NVIDIA Nemotron 3.5 Super VL Enters Vision Arena: 89.9% GPQA, 71.1% SWE-bench on One H200
NVIDIA added Nemotron 3.5 Super VL to three Arena leaderboards on October 19 — Vision Arena, Image-to-WebDev Arena, and Code Arena WebDev — five days after uploading the model to Hugging Face. The NVFP4-quantized variant is the deployable release: 120B total parameters, 12B active, runnable on a single H200 or two H100s.
The architecture is not a standard transformer. Nemotron 3.5 Super VL layers interleaved Mamba-2, Mixture-of-Experts, and attention blocks behind a C-RADIO vision encoder and connector. The language backbone traces back to Nemotron 3 Super Base. Multi-Token Prediction enables native speculative decoding at seven draft tokens per step using the LiLiCorr companion model. Context window: 1M tokens.
What the Benchmarks Show
The headline numbers are competitive with frontier reasoning and coding models:
| Benchmark | NVFP4 Score |
|---|---|
| GPQA Diamond | 89.90% |
| APEX Shortlist | 89.36% |
| LiveCodeBench v6 | 87.41% |
| GraphWalks | 90.34% |
| BrowseComp | 77.5% |
| SWE-bench Verified | 71.07% |
| SWE-bench Pro | 51.30% |
| Terminal-Bench 2.1 | 62.78% |
| OSWorld-Verified | 52.3% |
| CharXiv Reasoning | 79.88% |
| HLE (text-only) | 31.44% |
GPQA Diamond at 89.9% places it in range of frontier reasoning models. LiveCodeBench v6 at 87.4% is a strong coding score for a vision-capable model with 12B active parameters. GraphWalks at 90.3% is notably high — that benchmark probes structured reasoning over graph relations.
The NVFP4 quantized variant matches or slightly exceeds the BF16 parent on several benchmarks: GPQA Diamond (89.90 vs 89.52), GraphWalks (90.34 vs 89.71), IFBench (71.50 vs 70.86), OSWorld-Verified (52.3 vs 51.14). Quantization-aware distillation, applied after NVFP4 quantization using the supervised fine-tuning blend, preserved accuracy without the typical degradation — and in some cases improved it. Vision encoder, connector, Mamba convolutions, and embeddings remain in BF16; only the weight matrices are quantized.
Where It Underperforms
The tau2-bench numbers are the gap. tau2-bench measures agentic service performance across retail, airline, and telecom workflows — multi-turn tasks requiring tool use, policy adherence, and error recovery in realistic customer service scenarios.
| tau2-bench Category | NVFP4 Score |
|---|---|
| Retail | 69.30% |
| Airline | 75.75% |
| Telecom | 73.19% |
These scores fall below the 80% baseline that characterizes frontier agentic models. For context: Claude Opus 4.6 scores 91.9% retail and 99.3% telecom. Even Gemma-4-26B, a 4B active parameter model, scores 85.5% on retail tau2. Nemotron 3.5 Super VL’s tau2 numbers land below models it outperforms on GPQA by 15-20 points.
The pattern suggests the model was optimized for structured reasoning, coding, and vision tasks rather than conversational agentic workflows. SWE-bench Verified at 71.1% and OSWorld-Verified at 52.3% are further signals: both are below where current frontier coding agents sit on those benchmarks.
tau3-bench banking, a harder variant, scores 17.78% — consistent with the tau2 results.
Training Pipeline
NVIDIA used a seven-stage training pipeline. Stages 1 and 2 distilled from Nemotron 3 Ultra Base using 1T tokens of distillation, extended to 1M context, and ran 67B tokens of software-engineering mid-training at 256K context using repository data, issue-PR pairs, commit diffs, and agent trajectories. Stage 3 added the C-RADIO vision encoder. Stage 4 applied two-phase supervised fine-tuning on vision, reasoning, code, tools, chat, safety, and long-context data. Stage 5 combined reinforcement learning with verifiable rewards, RLHF, and multi-teacher on-policy distillation. Stage 6 boosted the MTP head against the frozen post-trained backbone. Stage 7 applied NVFP4 quantization with quantization-aware distillation.
The software-engineering emphasis in mid-training is reflected in LiveCodeBench (87.4%) and SWE-bench Verified (71.1%). The tau2 weakness is probably a consequence of what Stage 4 and 5 prioritized: reasoning and coding reward signals rather than service dialogue fidelity.
Deployment Footprint
Nemotron 3.5 Super VL NVFP4 runs on one H200, two H100s, or a DGX Spark (GB10). The reference vLLM configuration targets DGX Spark with one concurrent sequence and seven speculative tokens per draft, using the dflash speculation method. Reasoning mode is configurable: off, medium, or full. License is OpenMDW 1.1, which permits commercial use.
The BF16 version requires more hardware. For production deployments on standard NVIDIA data center configurations, NVFP4 is the practical choice — and the benchmark data confirms it performs as well as the full-precision model.
The Arena Story
Arena placement on Vision and Image-to-WebDev leaderboards will determine whether the benchmark paper numbers translate to human-rated quality on web and vision tasks. The Code WebDev Arena ELO is pending human evaluation runs. Given that the model’s coding scores (LiveCodeBench 87.4%, SWE-bench Verified 71.1%) are strong but not at the Claude Fable 5.x or GLM-5.3 tier, a mid-to-high Arena Code WebDev ELO is the expected range.
The Vision Arena is where the C-RADIO encoder gets tested outside structured benchmarks. MMLongBench-Doc at 62.82% and CharXiv Reasoning at 79.88% suggest competitive document understanding, though not at the top of the field.