Cursor open-sourced the MoE training kernel behind their Composer models. 2.37x faster than DeepSeek's DeepEP. Fuses all communication and computation into a single deterministic kernel. No CPU-GPU sync. Bitwise-identical results every run. Already training on tens of thousands of GPUs. Apache 2.0.
# The MoE Bottleneck
MoE layers are the bottleneck in large-scale training. Tokens bounce between GPUs and experts through separate kernel launches. Every boundary means a CPU-GPU sync, a scheduling delay, and wasted compute.
MoK collapses the entire dispatch-compute-combine chain into one kernel. No kernel launch boundaries. No CPU-GPU synchronization points. The result: bitwise-identical outputs every run, which makes debugging distributed training actually possible.
# The Numbers
| Solution | Forward Throughput | Deterministic | License |
|---|---|---|---|
| MoK (Cursor) | 2.37x baseline | Yes - bitwise | Apache 2.0 |
| DeepEP (DeepSeek) | 1.0x baseline | No | MIT |
| HybridEP + Megatron (NVIDIA) | ~1.0x baseline | No | Proprietary |
# How It Works
MoK fuses all-to-all dispatch, expert computation, and all-to-all combine into a single kernel with overlapped communication. Built specifically for GB300 NVL72 racks - 72 GPUs in one NVLink domain.
Fused Megakernel
All MoE communication and computation runs in one kernel launch. No scheduling boundaries. Uses Blackwell's Cluster Launch for scheduling across the NVLink domain.
Bitwise Determinism
Same inputs produce identical outputs every run. No floating-point non-determinism from async communication. Makes debugging at scale tractable.
Dual Precision
Supports both BF16 and MXFP8 precision modes. Router weight gradients computed via SonicMoE-style calculation fused into SwiGLU backward pass.
Hardware Requirements
NVIDIA Blackwell SM100/SM103 GPUs. Python 3.12+. PyTorch 2.10+. CUDA 13.0+. Designed for NVL72 rack topology.
Stuart H. Sul, Nash Brown, Henry Wildermuth, William Lin, and Federico Cassano from Cursor Research. The blog notes MoK was "designed so agents can adapt it to other platforms."
# Why Open-Source Matters
Cursor is an IDE company. Not a GPU cloud. Not a chip manufacturer. Yet they just published frontier training infrastructure that beats both DeepSeek and NVIDIA's own solutions.
DeepSeek open-sourced DeepEP and it accelerated MoE research across the industry. Now Cursor ships something 2.37x faster under Apache 2.0. The barrier to training MoE models keeps dropping. 487 stars and 50 forks in the first week - the community is already adapting it.
An IDE company just published state-of-the-art MoE training infrastructure that beats both DeepSeek and NVIDIA's own solutions. Mixture-of-Kittens is deterministic, open-source, and already proven at scale. The barrier to training MoE models just dropped significantly. If you have access to NVL72 racks, this is the kernel to use.
Next episode dropping soon
New episodes every Mon, Wed, Sat. Follow on LinkedIn to get notified.
Enjoyed this?
New episodes Mon, Wed, Sat.