Skip to the yard
Waxsealai

TorchTPU Brings Native PyTorch Support to Google TPUs

Shipped
Shipper
David Davis
Readers
745

TorchTPU lets Hugging Face Transformers and Diffusers run on TPUs through PyTorch, with compiled execution, multi-chip sharding and specialized kernels.

TorchTPU Brings Native PyTorch Support to Google TPUs

TorchTPU changes the practical starting point for PyTorch teams considering Google’s AI accelerators: TPU support can now fit into familiar PyTorch workflows. Hugging Face says Transformers, Diffusers and Kernels can run natively on TPU, with models loaded onto the device through standard library interfaces. That could make it easier to evaluate TPU hardware without first adapting a project to a separate execution model.

A Tensor Processing Unit is Google’s accelerator for neural network workloads, particularly the matrix operations common in training and inference. TPUs are available through Google Cloud, from individual chips to pods containing thousands. That range makes them relevant to both experimentation and larger workloads, though the size of the available system alone doesn’t determine whether a model will run efficiently: software support, compilation and the workload’s shape all matter.

A PyTorch path to TPU hardware

PyTorch users have traditionally reached TPUs through PyTorch/XLA, the torch_xla package. That route supported TPU execution, but it introduced differences that could make ordinary development harder to follow. The source describes tensors as lazy, meaning operations might not execute at the point a developer expected. Small code changes could also cause graphs to be compiled again, while debugging those behaviors could be difficult to reason about.

TorchTPU adds TPU as a PyTorch backend, so developers can write regular PyTorch code and use the TPU as a device. It also includes Pallas kernels tuned for TPU hardware, distributed training support and integration with torch.compile for teams seeking additional performance. The reported installation process for a TPU VM takes one command, although the article doesn’t provide the command itself.

There’s a packaging detail to account for when installing: the extra package index selects a CPU build of torch. Without that index, pip may install the substantially larger CUDA build, which is not needed on a TPU. This is a reminder that a simpler device interface doesn’t remove environment setup; teams still need to match their PyTorch package to the accelerator they plan to use.

Hugging Face previously offered Optimum TPU to support inference and training for its models on TPU. With TorchTPU providing native support, Optimum TPU is no longer required to run Transformers and Diffusers models on the device. That reduces the number of TPU-specific layers in these workflows, while leaving performance dependent on each model, its configuration and the chosen execution mode.

Transformers and Diffusers workflows

Transformers v5 is presented as a shared source of model definitions that can run across different hardware. In the described TPU workflow, developers load a model with device_map="tpu". Eager execution is the default, which provides a direct route for trying a model. Developers can opt into torch.compile when they want compiled execution, or use a distributed setup when a workload calls for it.

Diffusers also treats TPU as a device. For prototyping or debugging, a pipeline can be moved with pipe.to("tpu"). Deployment can use a compiled mode with an additional compile call, while the pipeline and surrounding application code remain the same. The first call performs compilation; later calls reuse the graph when they use the same image size. That makes the initial run a different performance case from later runs, so teams measuring latency should account for compilation and compare repeated calls at matching sizes.

Models that exceed the capacity of one chip can be split across several. Passing a TensorParallelConfig to from_pretrained shards the transformer as it loads, avoiding the requirement that the full model fit on one chip. The article says many popular pipelines support this directly; other pipelines need a tp_plan. The Diffusers TPU guide provides the implementation details.

Specialized kernels and reported speedups

The Kernels Hub provides ready-to-use TPU kernels that can be loaded through the Kernels library. These kernels are written in Pallas, which gives developers lower-level control over TPU hardware. The article also points to kernels from Tokamax, including a ragged-dot kernel used with a Mixture-of-Experts model.

The reported gains are largest at batch size 64: decoding is 21.5× faster for Qwen3-30B-A3B and 12.8× faster for Gemma 4 26B-A4B. Those figures are for the Hub ragged-dot kernel compared with the default grouped_mm implementation, measured on a TPU v6e-8. The accompanying note specifies TP=4, meaning each model is split across four chips, with bf16 precision and batch size 1; the figure caption describes decode throughput. Because the reported speedups depend on model, batch and kernel choice, they should be treated as results for this setup rather than a general estimate for every TPU workload.

Decode throughput with the Hub ragged-dot kernel versus the default grouped_mm, for Qwen3-30B-A3B and Gemma 4 26B-A4B

Decode throughput on a TPU v6e-8, comparing the Hub kernel with the default implementation.

Compatibility is still being measured

Hugging Face says it is checking TPU support by running the Transformers test suite and is working toward nightly TPU testing. Results are intended to appear in the Transformers CI dashboard, alongside results for other accelerators. The test status is not complete: 96.5% of tests that run on TPU pass. The team reports filing 34 issues in the TorchTPU repository, with 19 already fixed.

Those test runs have also exposed bugs in Transformers, including problems in StaticCache and Qwen3 tensor-parallel plans, which Hugging Face says it fixed upstream. The dashboard and issue counts provide useful evidence of active compatibility work, but they don’t establish that every model or operation will run successfully. Developers bringing an existing project to TPU will still need to check their own model path and workload.

Transformers CI dashboard with TPU test results next to other accelerators

The Transformers CI dashboard displays TPU test results alongside other accelerators.

TorchTPU’s clearest near-term value is a more direct way to try PyTorch models on TPU, plus tools for compilation, multi-chip execution and specialized kernels. The reported benchmark gains show what tailored kernels can deliver for particular decoding workloads; the compatibility results show where the broader software path still needs work. For teams evaluating TPU, the practical next step is to test representative models and batch sizes, including the first compiled run, before drawing conclusions about cost or throughput.

Source: David Davis · huggingface.co

Same vesselRelated cargo