GPT-6 Astra Rewrites 0.016% of Qwen3.5-0.8B's Weights and Lifts MATH-500 From 7.9% to 48.1% With No Gradients
Most of the score gain in this result comes from deleting one direction from two layers.
A research write-up titled “Sparks of Weight-Space Programming” reports that GPT-6 Astra, given Qwen3.5-0.8B’s weights in a Python kernel and a budget of 50 forward passes, edited 131,072 of the model’s roughly 0.8 billion weights (0.016%) and raised MATH-500 from 7.9% to 48.1%. No gradient descent, no training loop. The authors call the approach Weight-Space Programming.
The Qwen edit
The base model’s problem was looping. In its first 96 practice answers, 85 ran to the 32,768-token limit; on MATH-500, 93.0% of answers hit the limit. Qwen’s own model card warns that this model is prone to thinking loops.
Astra traced the behaviour to one direction written by the MLPs in layers 15 and 19, then subtracted it from 64 output columns in each.
| Qwen3.5-0.8B | MATH-500 | GSM8K | IFEval | Hit token limit |
|---|---|---|---|---|
| Base | 7.9% | 13.8% | 21.0% | 93.0% |
| Edited, 10 forward passes | 12.3% | 25.7% | 49.0% | 87.2% |
| Edited, 25 forward passes | 14.1% | 25.1% | 39.9% | 86.4% |
| Edited, 50 forward passes | 48.1% | 67.1% | 62.3% | 33.1% |
The IFEval jump from 21.0% to 62.3% is the tell. The edit did not teach the model maths. It stopped the model from looping, which let it finish answers it could already work out.
The authors test that reading directly. On Gemma 3 270M, which rarely loops (8.5% of answers reach the limit), Astra found only ordinary arithmetic and algebra errors with no single component to switch off. Its edits stayed small, and none moved MATH-500 by more than about a point, a gain Astra itself reported as not established.
Reading and writing small networks
The Qwen result is the largest-scale experiment. The smaller ones carry most of the claims:
- Game identification. Given only the 17,253 raw weights of a 500-32-32-5 policy network trained with PPO, Astra named the correct game (Breakout, Pong, Pac-Man or Freeway) for all four networks. It reported 99% confidence on Breakout.
- Middle layers only. With a 7.2M-parameter character-level transformer stripped of its input and output layers, Astra named the right training language (Python, romaji or English) for all three models, in both attempts. It also identified every induction head, with no false positives.
- Fixing a stuck policy. A PPO network that never got a Kinetix car over a ramp (0 of 256 episodes) succeeded in all 1,024 episodes after a one-shot edit of 3.5% of its weights, with no tools.
- Writing from scratch. Starting from all-zero weights, Astra wrote networks for three MinAtar games. In Breakout it scored 517.4, against 15.8 for PPO at 100M steps and 25.3 for PPO at 500M steps. In Asterix it scored 53.2 against 55.2 at 500M steps. In Seaquest it scored 53.1 against 88.9 at 500M steps.
- Go. A 1.48M-weight network (243 to 512x6 to 82) for 9x9 Go beat equal-sized PPO networks trained on 20 million games once Astra was allowed 50 forward passes. With zero forward passes it still won more than half its games against two smaller PPO networks.
Where it failed
Experiment 7 is the negative result. The authors fine-tuned two Gemma 3 270M copies, one honest and one sycophantic, with embeddings and output layers frozen. From weights alone, Astra could not tell which was which in any of four attempts. When it was allowed to see only attention patterns on texts it wrote, it picked the sycophant in all three pairs. In the sycophant, head 10.2 put 85.7% of its attention on the user’s suggested answer; in the honest model that figure was 0.75%.
That is a useful boundary for interpretability work. The model can read structure, such as which heads copy and which layers draw shapes. Mapping weight differences to behavioural differences still needed runtime evidence.
Caveats
This is a blog-format write-up with no independent replication that Stack Futures could find. Qwen3.5-0.8B is a toy-scale target, and most of the lift comes from fixing a known failure mode that the model’s own documentation flags. The results also depend on a frontier model running a harness with forward-pass access: 10 and 25 passes barely moved the loop rate, and only the 50-pass run did.
The claim worth testing is narrower than the headline. If a general-purpose model can diagnose and patch a specific, documented failure by editing a few hundred thousand weights, that is a cheaper alternative to a fine-tuning run for that class of bug. Whether it holds on models above 10B parameters is not yet shown.