Blog

TokenSpeed-kernel is a standalone open-source subsystem for LLM inference kernels, using layered APIs and registry-based dispatch to separate runtime logic from hardware-specific backend implementations.

The TokenSpeed inference engine achieved a record-breaking 580 tps running the Qwen3.5-397B-A17B model on GPUs. This extreme performance for agentic workloads is driven by systematic elimination of memory copies, advanced kernel fusions, and fully overlapped CPU-GPU execution-keeping the GPU saturated at all times. On the functionality side, TokenSpeed also supports hybrid prefix caching and unified Prefill-Decode state transfers to handle complex agentic serving scenarios.

EAGLE 3.1 improves speculative decoding robustness with FC normalization and post-norm hidden-state feedback, with training support in TorchSpec and deployment support in vLLM.

Slides for the PyCon US 2026 talk on building high-performance LLM inference in pure Python with PyTorch custom ops.

TokenSpeed is a speed-of-light LLM inference engine designed from first principles for agentic workloads, with a compiler-backed modeling mechanism for parallelism, a high performance scheduler, a safe KV resource reuse restriction, a pluggable layered kernel system that supports heterogeneous accelerators, and SMG integration.

SMG moves CPU-bound serving work such as tokenization, detokenization, reasoning parsing, tool orchestration, multimodal preprocessing, and routing into a Rust gateway layer, keeping GPU inference engines focused on tensor execution while avoiding Python GIL bottlenecks.

We’re excited to announce TorchSpec, a torch-native framework for scalable speculative decoding training. TorchSpec streams hidden states directly from inference engines to training workers via Mooncake, eliminating the need to materialize massive tensors on disk or co-locate training with the target model. This design enables fully disaggregated pipelines where inference and training scale independently.

This post is jointly published with the PyTorch Foundation and is also available on the official PyTorch Blog.