Weight-Space Programming: How LLMs Read and Edit Neural Network Weights

Weight-space programming is a method in which a general-purpose large language model (LLM) reads a neural network's raw weights and edits or writes them directly from a plain-language goal, with no gradient descent and no training loop. In a blog post published on November 2, 2026, four MIT CSAIL researchers report that GPT-6 Astra can already do this: it edited 0.016% of Qwen3.5-0.8B's weights and lifted the model's MATH-500 score from 7.9% to 48.1%.

The authors are Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab, first author of the original DSPy paper. The results are early and come from a single LLM, but the team published every prompt, response, and evaluation behind them.

In this article, I'll cover what weight-space programming is, what the experiments showed, where the method breaks down, and what it means for your work. If you need a refresher on the model doing the editing, see our guide to GPT-6 Astra.

What Is Weight-Space Programming?

Weight-space programming is the practice of asking an LLM to read, fix, or write a neural network's weights directly, around a natural-language spec, sometimes with no feedback from the environment at all. The LLM works out what each part of the network computes, or should compute, and sets the numbers accordingly.

That is a different route from how weights are normally produced. Today, weights come from gradient descent with optimizers like Adam or Muon, from networks trained to generate other networks' weights, or from compilers that turn formal programs into networks. In the MIT setup, GPT-6 Astra writes no training loop. It reasons about the weights the way an engineer reasons about code.

LLMs already optimize prompts, programs, and training code. The authors' argument is that this reasoning-driven optimization now reaches the weights themselves.

What the MIT Researchers Showed

The 8 experiments fall into three groups: reading weights, fixing them, and writing them from scratch. Most use small networks whose weights fit in Astra's context window as text; the larger ones add a Python sandbox and some a budget of forward passes.

Reading a network's weights

The team trained four policies with proximal policy optimization (PPO), the standard reinforcement learning algorithm for game agents, on grid versions of Breakout, Pong, Pac-Man, and Freeway, all with the same 17,253-weight architecture. Without being given the list of games, Astra named the right game for all 4 networks, reading clues like the Breakout agent's first-layer weights sitting along the bottom row, where the paddle is.

Two harder versions followed:

• Diffusion models: From the weights of 3 small DDPMs, with no forward pass allowed, Astra identified which one generates circles, squares, or triangles. It drew them as outlines although the models produce filled shapes, a hint that it read the weight geometry rather than simulating the network.
• Language models: Given only the 4 middle transformer blocks of 7.2-million-parameter character models, Astra named the training language (Python, romaji Japanese, or English) for all 3, in both attempts. It also found every copying head, also called an induction head, and named no other head as one. Our guide to multi-head attention explains what those heads do.

Repairing a broken policy

A PPO policy trained for 1 billion steps on a car-and-ramp level in the Kinetix environment drove straight at the ramp and stalled every time. Given the 171,664 weights, the scene, and one recorded episode, Astra explained the failure and wrote a fix in a single reply with no tools. The edit changed 3.5% of the weights and took the success rate from 0% to 100% across 1,024 episodes.

The fix is readable: a small extra path that backs the car up, then switches off once the car clears the strip.

Writing networks from scratch

For three MinAtar games, Astra got the rules, the source code, and an all-zero network, then wrote every weight as text in one reply, with no data, gradients, or tools. All three networks beat PPO agents trained for 100 million steps.

Breakout is the outlier because Astra wrote it as a program: the first layer computes where the ball will land, the second moves the paddle there. A 9x9 Go network with 1,480,274 weights, written in a Python sandbox, beat an equal-sized PPO network trained on 20 million games once Astra had 50 forward passes to test against.

Fixing reasoning loops in Qwen3.5-0.8B

The most practical result targets a known flaw. Qwen's own model card warns that Qwen3.5-0.8B is prone to thinking loops, and in the MIT tests 93.0% of its MATH-500 answers ran to the 32,768-token limit. Working in a Python kernel with 50 forward passes on practice problems from GSM8K and MATH, Astra traced the loops to one direction written by the MLPs of layers 15 and 19 and subtracted it from 64 output columns in each.

The share of answers hitting the token limit fell to 33.1%, and because loops are not specific to math, instruction following improved too. Budget mattered: with 10 or 25 forward passes the loops barely went away. Hugging Face's Ahsen Khaliq rebuilt the edited checkpoint from the logs and re-ran MATH-500 with the same settings, getting 51.0%.

Where Weight-Space Programming Falls Short

Weight-space programming falls short wherever the problem is not a single, localized fault in the weights. The authors report two clear failures, and both are informative.

• Sycophancy was invisible in the weights. Given an honest and a sycophantic fine-tune of Gemma 3 270M, Astra could not tell them apart from the weights alone in any of its four attempts. Once it could see their attention patterns, it picked out the sycophant in all three pairs.
• Gemma 3 270M didn't get better at math. Gemma rarely loops, so there was no single fault to remove, and Astra's best edit moved MATH-500 from 6.8% to 7.7%. As the authors put it, editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked.

The wider caveats: every experiment uses one LLM, the largest network has 0.8 billion weights, and this is a blog post with early results, not a peer-reviewed paper. The authors say they are still building a benchmark for the task.

What This Means for Data and AI Practitioners

For practitioners, weight-space programming is a research signal today, but not yet a tool. Two parts of it are worth tracking now.

The first is interpretability. Attribution methods like SHAP explain a model through its inputs, and mechanistic interpretability needs tools built for the job. Astra recovered induction heads with a known matrix-multiplication method the researchers never mentioned to it. If that holds up on larger models, "ask a frontier model what this layer does" becomes a first pass before the heavy tooling.

The second is targeted repair. When a small model has one narrow failure, like Qwen's loops, the usual fix is to fine-tune it on new data. A 131,072-weight edit leaves the rest of the model untouched, which the authors argue avoids the catastrophic forgetting fine-tuning can cause. I'd treat that as a hypothesis: the Gemma result shows the edit only works when there is a single culprit to find. The authors' wider wish list, from auditing models for backdoors without running them to unlearning capabilities without data, hasn't been shown yet.

How to Try Weight-Space Programming Yourself

The most direct way to test weight-space programming is to replay one of the experiments: the logs hold the exact prompts Astra received, and GPT-6 Astra is available as gpt-6-astra in the OpenAI API, in Amazon Bedrock, and in ChatGPT's paid plans. Experiment 1, naming the game from a policy's weights, is the easiest place to start:

1. Get the logs. Clone the lm_understand-logs repository. Each experiment folder has a README that maps every number in the post to the file it comes from.

2. Pick a tool-free experiment. Experiments 1, 2, 4, and 5 gave Astra no tools, so a single prompt reproduces each run. In Experiment 1, every run folder has a prompt.txt with the network description, the question, and all 17,253 weights as text.

3. Send it the way the authors did. They called gpt-6-astra through the Codex CLI (codex exec, prompt on stdin) with every built-in tool switched off, at high reasoning effort for Experiment 1. The rest of the configuration is in harness/codex_config.toml, but its defaults are gpt-6-luna at medium effort, so set the model and reasoning effort on each call, as the authors did.

4. Compare the answers. Each run folder also holds Astra's answer.md, and eval/ holds the scoring and the answer key.

A run used about 73,700 input tokens and under 2,000 output tokens, so at the API's $10 and $50 per million tokens, one replay costs under $1. The Python-sandbox experiments (3, 6, 7, and 8) need the authors' harness/sandbox.py and many more turns, so budget accordingly.

If you'd rather not run anything, the edited Qwen3.5-0.8B is on Hugging Face under Apache-2.0, with the hashes of the 2 changed tensors on its model card. The Hugging Face Space lets you type a problem and watch the base model loop while the edited one solves it.

Final Thoughts

Weight-space programming shows that a frontier LLM can read and edit neural network weights by reasoning, at least for small networks and single, localized faults. The Qwen fix is the result that will get attention, and it deserves it, partly because Hugging Face reproduced it within a day.

I'm more cautious about the bigger claims. One model, small networks, and no benchmark is a thin base for the authors' framing that "our neural network weights will soon be natively optimized by LLMs." Still, the failures are reported as carefully as the wins, and that makes me take the wins more seriously.

If you're keen to understand the transformer internals these edits target, I recommend our Developing Large Language Models track.