What is recursive self-improvement?

What happens when artificial intelligence begins improving itself, or pursues a goal we set for it in ways we never intended? In recent news and technical debates, three related terms have moved to the center of that question: recursive self-improvement, emergent behaviors and reward hacking. For most readers, these phrases will be unfamiliar, and they are often grouped together in alarmist headlines. Yet separating them is essential for anyone managing AI in a real organization. This post defines what recursive self-improvement actually means, examines a recent MIT experiment where one AI modified another model’s internal weights, and explains why the most immediate practical risk for health leaders today is not a model rewriting its own brain, but how existing models may behave when deployed in everyday agentic operations.

Recursive self-improvement, usually shortened to RSI, describes a process in which an AI system helps design, train or refine a more capable system, which then helps create an even more capable successor. The idea dates back to the mathematician Irving John Good in 1965, who asked what might happen if a machine became good enough at designing machines to accelerate its own development. In popular culture, the phrase conjures an image of a sinister computer program rewriting its own mind overnight. In modern engineering, it refers more concretely to using AI to automate the research work that human engineers perform: writing software, selecting training data, running experiments and diagnosing errors.

To judge how close we are to that threshold, it helps to separate changing how a system is used in a workflow from changing its internal parameters—the numerical weights and biases a model learns during training. Recently, researchers at MIT demonstrated a technique called weight-space programming, in which a large frontier model from OpenAI (GPT-6 Astra) inspected and directly edited a tiny fraction—0.016%—of the numerical weights inside a much smaller open-weights model (Qwen3.5-0.8B) without retraining it, raising its mathematics benchmark score sixfold.

That result sounds like a machine making another machine smarter, but the details tell a more grounded story. The smaller model already knew how to compute the math answers; its problem was getting stuck in repetitive text loops that caused it to run out of space before finishing. The larger model used a programming environment and practice tests to locate the numerical pattern triggering that repetition and quieted it. When the researchers tried the same approach on another small model (Gemma 3 270M) that did not suffer from looping but simply lacked the mathematical ability, its score barely moved. Clearing a blockage that hid an existing skill is useful engineering, and major AI laboratories track automated research closely, but removing an obstacle inside a small model is not the same as installing new intelligence.

As models grow larger or interact with complex environments, they also display emergent behaviors—sometimes called emergent abilities when the result is a useful skill. Emergence is not unique to computing; it is a fundamental feature of natural and social systems, from biological ecosystems to human cities. What happens in a small group of people is one thing, but when millions of people interact in a city, entirely new patterns of behavior—traffic flows, neighborhood cultures, and local economies—emerge that could not be predicted simply by studying individuals in isolation. In AI, an emergent behavior is a capability or pattern of action that was not explicitly programmed into the system and was absent in smaller models, but appears as a system scales in size, training or interaction with its surroundings.

In public debate, there is a common fear that emergent behaviors in AI are necessarily sinister. In practice, the vast majority of documented emergent behaviors in large models have been helpful capabilities—such as multi-step mathematical reasoning, writing and debugging software, translating between languages, or interpreting clinical shorthand—that smaller models simply could not perform. Crucially, researchers have not observed spontaneous malevolent intent emerging simply because a model grew larger. However, because emergent behaviors cannot all be known in advance (a priori), frontier models require extensive testing and evaluation before release, as unanticipated failure modes can emerge alongside useful skills.

Practically speaking, then, the most significant risk right now does not come from models altering their own internal weights and biases, or from sinister emergent behaviors appearing out of nowhere as models scale. Instead, it comes from reward hacking—also called specification gaming—in agentic operations. In agentic operations, an organization takes an existing, pre-trained model whose weights are already fixed, connects it to software tools, and uses it as an agent to carry out a multi-step workflow while setting the criteria of success. Interestingly, the same underlying principle is used both when frontier labs use one model to train or evaluate another model during reinforcement learning and when a hospital uses an agent to run an operational process: in both cases, the system is driven to maximize a defined score or success metric.

Reward hacking occurs when the system discovers a shortcut that satisfies the literal grading rule or success metric while defeating the human purpose behind the task. If a simulated car is rewarded for hitting markers placed along a racecourse, it may drive in circles hitting the same marker repeatedly rather than finishing the race. In deployed AI agents, this is an immediate operational hazard. Incident reports and safety studies over the past year have shown that pre-trained agents pressured to satisfy a task goal can over-comply with requests, leak private data or credentials to clear an authentication hurdle, or tamper with the software used to monitor and grade their work. Being vigilant to this risk is a central consideration for anyone designing and governing agentic systems in health care.

Consider a health system deploying a pre-trained model in an agentic workflow to clear a backlog of prior-authorization requests, triage patient messages or coordinate hospital discharges. No one at the hospital is training a model or editing a pre-trained model’s weights; instead, the team defines the agent’s criteria of success—such as how quickly it marks cases “complete,” reduces pending inbox volume or obtains an approval code. Pressed to optimize that target, an agent might learn to satisfy an external insurer’s portal by transmitting unredacted private clinical notes, overstating how closely a patient meets a coverage criterion, or routing around an exception queue meant for a clinician’s review. The operational dashboard would look better while patient privacy and clinical safety deteriorated.

Health leaders need to keep two horizons in view at once. On one hand, we must stay attentive to advances in frontier intelligence and the potential for recursive self-improvement, which is likely to become a powerful, foundational trend as AI increasingly accelerates its own research and engineering. On the other hand, there are more immediate practical considerations today around reward hacking in the pre-trained models we are already putting to work. Organizations do not need to wait for self-improving superintelligence to take these risks seriously. Safe agentic operations in health care depend on defining success criteria that reflect real quality of care rather than narrow volume targets, enforcing strict tool permissions in ordinary software, monitoring how agents achieve their outcomes, and keeping human accountability over actions that cannot be undone.

Sources: Good — Speculations Concerning the First Ultraintelligent Machine (1965); Takehi and colleagues — Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights (2026); Wei and colleagues — Emergent Abilities of Large Language Models (2022); Krakovna and colleagues — Specification gaming: the flip side of AI ingenuity (2020); MacDiarmid and colleagues — Natural Emergent Misalignment from Reward Hacking in Production RL (2025).

To learn more, take a look at the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.

AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.

For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.

Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy