← Back to feed

Google and Meta co-developed TorchTPU and released it as open source at PyTorch Conference 2026 on October 20. The project makes TPU a native PyTorch device — sitting alongside GPU and CPU in the standard backend hierarchy — and ships as torch_tpu v0.2.0 on PyPI.

Amazon announced TorchNeuron for Trainium on the same day, running an identical play: one configuration change routes PyTorch applications onto AWS’s own AI accelerator.

What Changed

Until now, getting PyTorch onto TPUs meant going through torch_xla. The library worked, but lazy tensor evaluation made execution timing opaque, small changes could retrigger full XLA graph recompilations, and the debugging surface differed substantially from standard PyTorch.

TorchTPU replaces that indirection. Tensors live on a tpu device like any other, eager mode runs immediately, and torch.compile(backend="tpu") handles graph-mode optimization for throughput workloads. The underlying XLA compiler still runs, but it’s hidden behind the standard PyTorch dispatch path rather than exposed as a separate API.

Hardware Support

TPU v7x (Ironwood) is the recommended target — it runs in CI on every pull request and is fully validated. TPU v6e, v5e, and v5p are marked experimental and are not covered by continuous integration.

Integrations Shipped at Launch

Six frameworks have production-ready TorchTPU integrations shipping at launch:

• Hugging Face Transformers v5: device_map="tpu" sends any Hub model to TPU without code changes. Diffusers follows the same pattern — pipe.to("tpu") for eager mode, one compile() call for production throughput.
• vLLM-TorchTPU (developed with Inferact): standard vLLM CLI and OpenAI-compatible server, with TPU-optimized Pallas kernels for attention, sparse MoE, and FP8/INT8/MXFP4 quantized matmul. Supports continuous batching, prefix caching, multi-LoRA, speculative decoding, and disaggregated prefill/decode.
• SGLang-TorchTPU (developed with RadixArk): same upstream SGLang worker and scheduler model, TPU execution underneath.
• Ray, TorchTitan, llm-d, LiteRT: all gain native TPU support through the same backend.

Hugging Face’s own benchmarks on a v6e-4 slice showed compiled mode on Qwen3-30B-A3B decoding 21.5x faster at batch size 64 compared to the default implementation, using a Tokamax ragged-dot kernel from the Kernels Hub.

vLLM-JAX and SGLang-JAX Remain

Google confirmed that the JAX-based vLLM and SGLang TPU backends will continue to be maintained. TorchTPU-backed serving is the recommended path for standard PyTorch model definitions; JAX backends remain available for workloads that require direct JAX model implementations.

Amazon’s Parallel Move

AWS announced TorchNeuron alongside TorchTPU at PyTorch Conference, targeting Trainium chips. The pitch is identical: roughly 85% of production AI workloads run on PyTorch, and a native device backend removes the porting cost that has historically kept both TPUs and Trainium on the sidelines of commercial deployment. Both moves are direct pressure on CUDA-based GPU infrastructure at the software layer.

Key Numbers

• Package: pip install torch-tpu==0.2.0
• Model examples included: Llama, Qwen, ResNet, DLRM, minGPT, UNet
• v6e-4 benchmark: 21.5x decode throughput gain at batch 64 with Hub kernel (Qwen3-30B-A3B)
• CI coverage: TPU v7x (Ironwood), 96.5% of Transformers library tests passing
• Repos: github.com/google-pytorch/torch_tpu, github.com/vllm-project/vllm-torchtpu, github.com/sgl-project/sglang-torchtpu