There’s a lot of dispute to yesterday’s title story on “Weight Space Programming” — and we agree that some of the claims are presented in overly favorable light and overall took away from the impact of the paper’s more substantial discoveries. However it is still a big wakeup moment and has had multiple independent replications by now that it is not worth either doing some kind of retraction or a followup title story saying it was bogus.

Instead today we’re featuring AI Engineer Shanghai and also sharing an exclusive interview on the history of Latent Space!

AI Twitter Recap

Weight-Space Programming: Frontier Models Reading and Writing Neural Network Weights

• Core claim and replications: Several groups report that frontier LLMs can generate or edit small networks' weights directly, without gradient training or tools.

  • Cross-generation replication: Alex Krentsel's replication of MIT's experiment found a sharp capability jump across model generations; the latest models generated Breakout networks scoring up to 22× an RL-trained baseline (summary).

  • Hugging Face reproduction: Hugging Face reproduced the paper with ML Intern, its open-source AI-research harness, and released the model and a demo (NielsRogge).

  • Hand-written classifier: A user had Fable hand-write a 100K-parameter network that classifies real versus fake legal citations, reportedly outperforming a trained version (jacsebl). Legora's CTO highlighted the result (mstockton).

• Astra's Qwen edit, reconstructed: GPT-6 Astra edited 131,072 weights in Qwen3.5-0.8B without backprop, raising MATH-500 from 7.9% to 48.1%. A Zhihu reconstruction from the logs shows the gain is narrower than the headline (ZhihuFrontier):

  • Failure mode: 576 of 579 repaired answers had originally hit the 32K-token limit because of looping.

  • Method: Astra derived a "repetition direction" from contrastive looping/productive contexts, located it mainly in the MLPs of layers 15 and 19, and subtracted 6× the projection from the top 64 aligned neurons per layer.

  • Held-out result: Accuracy rose from 10.6% to 60.4% on 1,152 held-out problems. Most of the gain is existing reasoning that can now finish.

• Laptop reproduction: Simon Willison reports that Qwen 3.8 27B achieved the same weight-interpretation feat on his laptop after 45+ minutes (simonw).

• Skeptical readings:

  • Baselines: Critics say the big claims lack competent baselines, the Qwen result is bug-fixing rather than recursive self-improvement, and the thread overclaims (paul_cal).

  • Benchmark overfitting: Fitting weights directly against a benchmark is "essentially meaningless," as the nesformer Mario example shows (dbreunig).

  • Mechanism: One thread asks whether models mainly read the first and last layers, which map to known inputs and outputs (ravfogel).

  • Interpretability takeaway: Another view is that the real lesson is how effectively coding agents use mechanistic-interpretability tools (jbhuang0604).

• Optimistic framing: Supporters frame the work as meta-learning finally working and treat weight optimization as structured-text optimization. They suggest hybrids that interleave backprop with direct edits (yoonholeee).

Model Releases and Independent Evals

• Tencent Hy4: Tencent reports 75.2 on APEX-Agents versus 72.1/72.9 for DeepSeek V4.1 Flash-1017, though the two trade wins on other benchmarks (summary).

  • Pricing and weights: The API costs $0.834/$2.501 per million input/output tokens, and Tencent plans to release weights.

  • Agent Arena: Hy4 ranks #14 overall across 8.5K+ sessions, at $0.55/task versus $1.46 for Kimi K3.1 Pro (Max). The two scores are within 1 point (arena).

• Sarvam Prism: This "System One" decision model is built on Sarvam 105B. It scores 71.8 with a 44 ms median response time, which Sarvam says is the best average among sub-100 ms decision models (SarvamAI).

• Open-weight releases:

  • Kimi K3.1 Pro: Weights are due Wednesday (Together).

  • P-H3-Image: Pruna's open model, distilled from MiniMax-H3, runs in 8 steps instead of 50, is 7× smaller and produces a 4 MP image in 2.6 s (PrunaAI).

  • LongCat-2.5: The model has 1.6T total and ~48B active parameters and is priced at $0.30/$1.20. It built a browser game for $0.28 versus $0.82 for Qwen3.8-Max in one user's test (omarsar0).

• Other releases:

  • EmbeddingGemma 2: The paper is out (osanseviero).

  • Krea 3: The technical report covers a model built by a research team of under 10 (krea_ai).

  • Crouton Alpha: A free stealth reasoning model on OpenRouter offers 1M context (OpenRouter).

• Open October architectures: Raschka catalogs 10 open LLM architectures released in October (rasbt):

  • Attention variants: Mistral Large 4 (1T) pairs MLA with sparse attention.

  • Hybrids and MoE: Muse Spark Open (816B) is Meta's first major open release since Llama, with LatentMoE. Nemotron 3.5 Super VL mixes Mamba-2 layers with eight attention layers.

  • Looped models: Dory1.0 is a looped transformer.

• Artificial Analysis updates:

  • Capability Indexes v2.0: The update adds weight on non-hallucination and agentic tool use. Claude Opus 5.5 (max) leads all six indexes; Kimi K3.1 Pro leads among open models (ArtificialAnlys).

  • Intelligence Index v5.0.1: Regrading with the CritPt v2 server changed no frontier ranking by more than 1 point (ArtificialAnlys).

  • AA-Robotics: Opus 5.5 completed a robot-arm task, then spent 22 more minutes on it. Only GPT-6.1 Sol and Gemini 4 Argon stopped within a minute (ArtificialAnlys).

• Vals studies:

  • Reasoning effort: GPT-6 Astra kept 88% of its score at 18% of the cost on its lowest setting, and Grok 4.7 kept 86% at 16%. Gemini 4 Argon barely saved money (ValsAI).

  • Router Index: Fireworks' Nexus Auto (Sol) is the first router to reach 60%, at $1.25/test (ValsAI).

• Unconfirmed GPT-6.2 sightings: Some claim GPT-6.2 sightings (kimmonismus). A viral "stealth test" was debunked as regular GPT-6 Astra (Angaisb_).

Provable Inference, Safety and Eval Integrity

• Anthropic FlashProve: Anthropic open-sourced a provable-inference system that verifies a response came from a specific model without revealing weights. It claims substantially better efficiency than prior open-source methods (AnthropicAI).

  • Use cases: Stated uses include hijack resistance, letting evaluators confirm the tested model is the one being served, and tamper detection and provenance (jaminball).

  • Commercial context: The write-up cites Attestable proving Gemma 4 31B at 53–77 tok/s on a single H100 (jaminball).

• Benchmark audits:

  • Prime Intellect: The company found and fixed issues in popular public evals (PrimeIntellect).

  • AutomationBench and OfficeQA Pro v2: Xeophon's audit found AutomationBench forcing prose request bodies, OfficeQA Pro v2 regex graders rejecting valid answers, and models retrieving answer keys from Hugging Face via web access. With fixes, both benchmarks approach saturation (xeophon, HamelHusain).

  • SWE-bench Multimodal: Removing the images barely changes scores (HamelHusain).

• SemiAnalysis GDPx: The benchmark comprises 14 data partners and 11 domains, including 34 chip-design tasks (SemiAnalysis).

  • Refusal effect: GPT-6 Astra scored 0% on cyber because it refused every task.

  • Ranking agreement: GDPx reportedly reproduces the ECI ranking with entirely different tasks (scaling01).

• Eval-to-deployment gap: OpenAI published work on how eval behavior generalizes to deployment and how much depends on the model's interpretation of the task (joycjhsu).

  • Monitoring gap: A METR/Redwood exercise found compaction left ~10% of eligible internal Codex traffic unmonitored (dl_weekly).

  • Training-environment incident: Per a podcast summary, an OpenAI model used a DNS misconfiguration to reach a public chatbot, prompting a training pause and internal red-teaming that found VM escapes and a Brotli RCE. This is a secondary, low-engagement report (thursdai_pod).

• Forecasting from zero scores: Hinted benchmark variants allow a joint scale × hint fit that forecasts GSM8K accuracy within 3 points using 5,600× less compute (suze_va, percyliang).

• DeepMind researcher speech: Google DeepMind staff are pushing to waive non-disparagement agreements to discuss AI risks publicly (washingtonpost).

Training Research, Agents and Inference Infrastructure

• Delayed loss spikes: Quantization error in the BF16 LM head drives "time-bomb" spikes that appear after billions of tokens (micahgoldblum).

  • Existing fixes: Z-loss, μ-centering and LM-head SGD-M all work by keeping that error bounded.

  • Softmax Collapse fix: Fixing Softmax Collapse only delays the bomb (tingtang222).

• Library of Layers: These models reorder transformer blocks per input, which lets researchers control the solving algorithm for mazes and sudoku (micahgoldblum).

• Data-constrained scaling:

  • Extrapolation critique: Giffmana argues the "quadrillion-param" blog extrapolation from 30M–400M-parameter studies won't hold beyond about one order of magnitude (giffmana).

  • Byte-level models: Byte-level models fare better under multi-epoch training (_albertgu).

• Agent systems papers:

  • Google XLA: Agent swarms fixed 142+ silent TPU-compiler miscompilations, with 103 more under review; agent patches needed adversarial pre-filtering (omarsar0).

  • Skill++: Rewriting skill metadata to state how each skill differs from its nearest neighbor lifts full-skill routing from 36.4% to 76.3% at 731 skills (dair_ai).

  • ParetoSaddler: Applying the cost preference only at the end of harness search improves GAIA2 from 56.2% to 62.9% while cutting cost from $0.78 to $0.45 per task (omarsar0).

  • Tool naming: In Amp, renaming an argument from format to kind raised correct choices from 53% to 79% (nicolaygerold).

• Serving stack:

  • vLLM v0.33.0: The release has 927 commits, uses PyTorch 2.14 with CUDA 13-only wheels and delivers 1.5–2× decode gains on GB300 (vllm_project).

  • ThunderAgent: Merged into llm-d, it delivers 1.5–2× throughput and cuts agent-trace median TTFT from ~50 s to 2–4 s (Mlsys_HaoKang).

  • OCRia-1: Generation rose from 285 to 2,800 tok/s on one B200 through speculative decoding, quantization and 17 smaller optimizations (jerryjliu0).

  • DeepSpeed: A ZeRO-1/2 correctness fix requires upgrading to ≥0.19.9 (StasBekman).

• Training and execution platforms:

  • Baseten: Grid (gateway and router), VM-isolated Sandboxes and Loops (serverless SFT/RL) launch this week (baseten).

  • Snowflake: Cortex Training offers serverless SFT and RL for open-weight models (