NVIDIA Nemotron 3.5 Bolt 74B Fits in 24GB at 2 Bits: Tops Its Class on GPQA Diamond, BrowseComp, Terminal-Bench
NVIDIA released Nemotron 3.5 Bolt 74B-A9B on October 15, a local-first agent model designed to run inside 24GB of memory while outperforming the 24GB-class field on agentic capability benchmarks.
The release is timed to RTX Spark laptops going on sale October 16, with the model available now on Hugging Face, Ollama, llama.cpp, LM Studio, Unsloth, and vLLM.
Architecture and Compression
Bolt 74B-A9B is a Mamba-2 and Transformer hybrid with Latent MoE layers and a C-RADIO v4-H vision encoder. It takes 74B total parameters down to 9B active per token using 22 routed experts in the parent model reduced to 9 per token. Inputs accepted: text, images, and video; output is text only.
The 24GB memory target is achieved through a two-stage compression pipeline:
- Star Elastic architecture search: compresses Nemotron 3.5 Super VL 120B-A12B down to a 74B skeleton, reducing per-layer expert counts and MoE intermediate widths while preserving hidden size and the Mamba hybrid structure
- Quantization-Aware Distillation (QAD): routed and shared expert weights quantized to IQ2_XXS at 2.0625 bits per element; Mamba linear layers in NVFP4; attention, embeddings, and LM head in Q8_0
At runtime on a 128K-token context, weights occupy 22.01GB, KV cache plus Mamba state plus activations 0.87GB, and the LiLiCorr speculative drafter 0.96GB — total 23.83GB excluding the vision encoder.
NVIDIA also ships a LiLiCorr speculative drafter (1.13B parameters) that achieves a mean acceptance length of 4.68 tokens with greedy decoding, reaching 84 TPS on an RTX Spark 128GB Surface Laptop Ultra.
Context length: up to 256K tokens. License: OpenMDW-1.1.
Benchmark Numbers
Comparison against 24GB-class open models, all vendor figures from the NVIDIA technical report:
| Benchmark | Nemotron 3.5 Bolt 74B-A9B 24GB | Qwen3.6-35B-A3B BF16 | Gemma4-26B-A4B BF16 | Nemotron 3.5 Lightning 30B-A3B BF16 |
|---|---|---|---|---|
| GPQA Diamond | 86.43% | 83.40% | 79.61% | 75.57% |
| BrowseComp | 66.75% | 48.74% | 26.30% | 36.81% |
| Terminal-Bench 2.1 | 50.98% | 44.38% | 37.22% | 23.46% |
| HLE (no tools) | 25.30% | 19.56% | 17.42% | 10.47% |
| WebArena (vision) | 50.86% | 36.60% | 38.20% | — |
| PinchBench | 87.82% | 88.07% | 74.70% | 83.43% |
| MCP-Atlas | 52.20% | 51.20%* | — | 46.40% |
| GDPVal-AA-V2 | 28.47 | 25.75 | 15.33 | 18.26 |
*NVIDIA measured Qwen3.6 at 51.2 on 500 public MCP-Atlas tasks; the published figure is 62.8.
The 2-bit quantization costs roughly 2-5 points versus the BF16 checkpoint on most tasks. GPQA Diamond drops from 88.83 (BF16) to 86.43 (IQ2), Terminal-Bench from 53.93 to 50.98.
Throughput comparison on RTX Spark 128GB (llama.cpp, coding workload):
| Model | TPS |
|---|---|
| Qwen3.6-35B-A3B UD-Q4_K_M | 96.63 |
| Gemma4-26B-A4B UD-Q4_K_M | 100.56 |
| Nemotron 3.5 Bolt 74B w. LiLiCorr | 83.99 |
| Qwen3.8-27B UD-Q4_K_M | 28.26 |
Bolt is slower than the two smaller MoE models at 4-bit, faster than the dense Qwen3.8. No independent benchmark results have been published yet.
What Else Shipped
NVIDIA released three models together:
- Nemotron 3.5 Super VL 120B-A12B — the large enterprise version, available now
- Nemotron 3.5 Bolt 74B-A9B 24GB — local agent model, available now
- Nemotron 3.5 Bolt E4B 4GB — edge model for GeForce RTX and Jetson devices, available October 23
The E4B targets voice assistants and embedded interfaces. All three are available through OpenRouter and as NVIDIA NIM endpoints, in addition to local deployment.
Context
Nemotron 3.5 Bolt is NVIDIA’s direct answer to Qwen3.6-35B-A3B as the default 24GB-class local model. The BrowseComp gap is the most striking: 66.75% vs 48.74%, an 18-point lead. Terminal-Bench and WebArena gaps are 6-14 points in Bolt’s favor, suggesting the agentic post-training on top of the compressed backbone held up.
The practical caveat: 24GB is the target, not the floor. NVIDIA’s own llama.cpp recipes need 28GB for a 65K context on a 64GB RTX Spark, and 32GB for the full 256K context on a 128GB machine. Ollama’s build sits at 25GB. Testing on a 24GB GPU is outside NVIDIA’s documented configurations.
Independent results will determine whether the 2-bit compression trades hold on real tasks, or whether the vendor benchmarks overstate the working accuracy.