Xiaomi's 1T MoE Hits 1000 tok/s — On Their Own Inference Stack

What happened

Xiaomi's MiMo team dropped MiMo-v2.5-Pro-UltraSpeed on its research blog with a single headline number: 1,000 output tokens per second on a 1-trillion-parameter mixture-of-experts model. The post — light on model weights, heavy on systems detail — frames the release as an inference-stack flex rather than a new pretrained model. The underlying weights are the same MiMo-v2.5-Pro that shipped earlier this quarter; what's new is TiLeRT, Xiaomi's serving runtime, and the engineering work to get a trillion-parameter MoE to spit out tokens at a rate normally reserved for 8B dense models on Groq.

The claimed throughput is measured at batch size 1, FP8 weights, FP8 KV cache, on a single 8×H800 node — the export-controlled SKU that Chinese labs have standardized on. Xiaomi reports a roughly 5× speedup over their own previous serving baseline and benchmarks against published numbers from Kimi K2 (around 200 tok/s) and DeepSeek-V3 (around 180 tok/s at similar parameter counts). The blog credits four specific techniques: a tree-structured speculative decoder trained on the model's own outputs, expert-parallel routing that pins hot experts to specific GPUs, an FP8 paged KV cache, and a custom CUDA kernel for the MoE all-to-all that avoids NCCL roundtrips.

No weights are released alongside the post. The model is accessible via Xiaomi's API and through the MiMo chat interface. There is no third-party reproduction yet, and the HN thread (182 points at time of writing) is — predictably — split between people excited about the systems work and people pointing out that a vendor-reported tok/s on their own infrastructure is the laxest possible benchmark.

Why it matters

The instinct, when a Chinese lab posts a big number, is to roll your eyes and wait for the asterisks. That instinct is correct for capability benchmarks — MMLU, HumanEval, the entire GPQA-flavored leaderboard treadmill — where contamination and prompt-tuning have made the public numbers nearly meaningless. But throughput is a different category of claim. tok/s on fixed hardware with fixed weights is reproducible in a way that "83.4 on MATH" is not. Either the kernels exist and the all-to-all is fast, or they don't. We just have to wait for somebody outside the building to check.

What's actually interesting here is the systems gap, not the model. For most of 2024, the conversation about Chinese frontier work was about pretraining — DeepSeek-V3's training run, the curriculum tricks, the cost-per-FLOP numbers. Through 2025, the conversation shifted to RL post-training and reasoning. Late 2025 and into 2026, the gap that's closing is serving: the difference between "we trained a competitive MoE" and "we can serve a competitive MoE at production economics." That's where Western labs still have a real lead, and it's where Xiaomi is now explicitly competing.

The specific techniques are worth reading carefully. Tree-structured speculative decoding isn't new — Medusa, EAGLE-2, and SpecInfer have all explored variants — but applying it to a 1T MoE where the draft model has to predict which experts will fire is genuinely hard. The blog claims 4-6 accepted tokens per draft call, which would be aggressive even for dense models. Expert-parallel routing with pinned hot experts is the same insight DeepSeek-V3's serving paper described: a small fraction of experts handle most tokens, and if you can predict which ones, you can avoid the cross-node all-to-all that murders MoE throughput. FP8 KV cache is the boring win — most labs are still on FP16 KV — and on H800s, where memory bandwidth is the actual bottleneck, halving the KV is a straight throughput multiplier.

Community reaction has been the expected mix. One top HN comment: "1000 tok/s on a 1T model with no third-party repro is in the same epistemic category as a benchmark screenshot from a private fork." Another, with more upvotes: "The kernel work is the story. Even if the headline number is 30% inflated, the all-to-all optimization is real and will show up in vLLM within six months." The second comment is probably right — the Chinese inference work consistently leaks into open serving stacks within a quarter, because the people writing the kernels publish papers and the people maintaining vLLM read them.

What this means for your stack

If you're building on hosted APIs, this changes nothing today. The model isn't open-weight, the API is in Chinese, and even if you could route to it, the latency from outside China would eat the throughput win. Watch the space, don't rewire.

If you're self-hosting MoE models — Mixtral, DeepSeek-V3, Kimi K2, Qwen3-MoE — the relevant question is when these techniques land in vLLM, SGLang, and TensorRT-LLM. Speculative decoding for MoE is the highest-impact change you can make to your serving stack right now, and it's underused outside the frontier labs. SGLang has experimental support; vLLM is behind. If you're at the scale where serving cost is a real line item, allocating one engineer-quarter to spec-decoding integration is probably the highest-leverage infra work available to you in 2026.

The FP8 KV cache win is more immediately actionable. If you're running H100s or H200s at FP16 KV today and your workload is memory-bound (long context, batch > 1), switching to FP8 KV via vLLM's existing support is a ~30% throughput gain for ~0.5 point of perplexity. The fact that Xiaomi is doing this on H800s — which have less memory bandwidth than H100s — means the gain on better hardware is larger, not smaller.

Looking ahead

The useful frame for releases like this isn't "did they beat OpenAI" — they didn't, and the comparison is silly. It's "how fast is the Chinese serving-infra stack closing the gap with the Western one, and which specific techniques will be commoditized into open serving runtimes by Q3." On current pace, tree-spec-decoding for MoE and FP8 paged KV will both be table-stakes in vLLM by mid-2026, and the people who built TiLeRT will have moved on to whatever the next bottleneck is — probably prefill throughput on million-token contexts. Bookmark the blog, ignore the headline number, and watch the kernel PRs.