Skip to content
Neural Daily

ToolsNews

PyTorch 2.15 brings FP8 and faster MoE decoding to Macs

PyTorch 2.15 makes the MPS backend for Apple Silicon GPUs much faster. FP8 arrives, and mixture-of-experts models decode up to 6.5x faster than in 2.14. Most of the work came from the Hugging Face PyTorch team.

By: Mikola Máté Patrik · · 3 min read

PyTorch × Hugging Face cover: the PyTorch logo and the Hugging Face mascot above the title What you can expect on MPS in PyTorch 2.15
Image: Hugging Face · Press kit · Official blog thumbnail of the Hugging Face MPS in PyTorch 2.15 post · Source

What changes

MPS is PyTorch's backend for Apple Silicon GPUs. This year the Hugging Face PyTorch team ran real Hub models (language, image and video) to find where it falls short. They fixed the problems upstream in PyTorch, more and more often by replacing MPSGraph with dedicated Metal kernels (Hugging Face).

2.15 is still a release candidate. It needs an Apple Silicon Mac with macOS 15 or later and Python 3.11 or later. You can try it in a separate environment with: uv pip install torch==2.15.0 --index-url https://download.pytorch.org/whl/test/cpu

The key numbers

  • MoE decoding: on an M3 Ultra with torch.compile, Qwen3.6-35B-A3B goes from 9.4 to 61.3 tokens/s and LFM2.5-8B-A1B from 23.6 to 123.6 tokens/s. That is a 5x to 6.5x speedup over 2.14, thanks to a native grouped matmul with a dedicated decoding path, fewer GPU to CPU syncs and faster top-k routing.
  • Dense models: F.linear now uses matrix-vector kernels for single-row decoding inputs, a 1.23x geometric-mean speedup across 96 real BF16 layer shapes. Softmax over a 128,256-entry vocabulary drops from 57 µs to 7 µs, roughly 8x faster.
  • FP8: torch.float8_e4m3fn tensors now work on MPS, using half the memory of BF16. F.scaled_mm multiplies them directly on the GPU with tensorwise or rowwise scaling.
  • Video: Wan2.2's tiled VAE decode used to push driver memory to about 270 GiB; it now peaks at about 8 GiB after constant padding moved to Metal.
  • Image editing: on 2.14 the Qwen-Image-2.1 VAE encoder silently corrupted images on MPS (reported in ComfyUI). The same padding fix resolves it, raising PSNR against CPU from 10.06 dB to 29.95 dB.
Hugging Face bar chart: MoE decoding on an M3 Ultra with torch.compile gives about 1.3x more tokens per second with the new grouped matmul decode path
Hugging Face · Screenshot · Official benchmark chart from the Hugging Face MPS in PyTorch 2.15 post · Source

Source: Hugging Face. The new grouped matmul decode path alone gives about 1.3x more tokens per second on an M3 Ultra with torch.compile.

What else is in

More linear algebra runs on the GPU: complex64 LU factorization, LU solve and triangular solve, faster small-matrix inverses and faster QR. conv1d, padding, dropout and upsampling backward also got Metal kernels. MPS streams are now public from Python (torch.mps.Stream), and event timing is more reliable.

What is coming in 2.16

Scaled dot-product attention will run entirely on Metal. In HF's measurements on an M5 Pro, grouped-query attention decoding at 4K context drops from 465 µs to 65 µs. libtorch_cpu.dylib will shrink by about 25%. Some of this is still in review, so details may change.

If you run local LLMs or image generation on a Mac with Transformers, Diffusers or ComfyUI, this is a tangible upgrade: MoE models were notably slow on MPS until now. Since it is still an RC, benchmark your own model before switching.

    Produced with the help of AI, reviewed by an editor. How we make our articles

    Reviewed and edited by the Neural Daily desk. About

    Related articles