cd ~/series/demystifying-ai
ORL: 2,788 Tokens Per Second From a Single RTX 5090
EP 57 arXiv: 2610.17894 Nov 14, 2026 5 min

ORL: 2,788 Tokens Per Second From a Single RTX 5090

Open Reasoning via Latent diffusion. Generate all tokens at once, refine iteratively - like an image diffusion model, but for text. 46x faster than autoregressive at the same quality. 3B and 8B checkpoints open-sourced.

Share:
// TL;DR

Researchers from Cambridge, EPFL, Anthropic, and Mistral introduce ORL (Open Reasoning via Latent diffusion), a model that generates text the way Stable Diffusion generates images: all tokens at once, then refined through iterative denoising. On a single RTX 5090, ORL hits 2,788 tokens per second - compared to roughly 60 tok/s for GPT-6 Comet's autoregressive decoding. It matches GPT-6 Luna quality on GPQA Diamond, MATH-500, and LiveCodeBench. The 3B and 8B checkpoints are fully open-sourced.

2,788
Tokens/Second
46x
Faster Than AR
3B / 8B
Open Checkpoints
~60
GPT-6 tok/s (AR)

# What Is ORL?

Every LLM you use today - GPT-6, Claude, Gemini, Llama - generates text one token at a time, left to right. Each token depends on the previous one. This is called autoregressive (AR) decoding, and it is inherently sequential. You cannot parallelize it.

ORL flips this entirely. Instead of predicting one token at a time, it generates all tokens simultaneously as a noisy initial draft in a continuous latent space, then refines the entire sequence through iterative denoising steps - exactly how Stable Diffusion generates images from noise.

AUTOREGRESSIVE
Step 1: The _ _ _ _ _ _ _
Step 2: The cat _ _ _ _ _ _
Step 3: The cat sat _ _ _ _ _
Step 4: The cat sat on _ _ _ _
  ...
Step 8: The cat sat on the warm mat .

Speed: ~60 tok/s (sequential)
ORL (DIFFUSION)
Step 1: ██ ███ ███ ██ ███ ████ ███ █
Step 2: Tha cet sot en tha werm met .
Step 3: The cat sat on the warm mat .
Step 4: The cat sat on the warm mat .



Speed: 2,788 tok/s (parallel)

The key insight: reasoning happens in a continuous latent space, not in discrete token space. The model thinks in vectors, not words. Only at the final step does it decode the latent representation back into text tokens. This means the entire "thinking" process is trivially parallelizable across GPUs.

# The Benchmarks

Speed is meaningless if quality drops. ORL does not drop quality. The 8B checkpoint matches GPT-6 Luna (the smaller, faster GPT-6 variant) across three of the hardest reasoning benchmarks.

GPQA DIAMOND
GPT-6 Comet71.2%
GPT-6 Luna64.8%
ORL-8B65.1%
MATH-500
GPT-6 Comet89.4%
GPT-6 Luna83.7%
ORL-8B84.2%
LIVECODEBENCH
GPT-6 Comet62.1%
GPT-6 Luna54.3%
ORL-8B53.9%
THROUGHPUT (1x RTX 5090)
GPT-6 Comet~60 tok/s
GPT-6 Luna~110 tok/s
ORL-8B2,788 tok/s

Read those throughput numbers again. 2,788 vs ~60 tokens per second. That is not a marginal improvement. That is a paradigm shift. ORL-8B is slightly below GPT-6 Luna on LiveCodeBench (53.9% vs 54.3%) but matches or exceeds on the other two benchmarks - while being 25x faster than Luna and 46x faster than Comet.

# How It Works

ORL has three core innovations that make diffusion-based text generation competitive with autoregressive models for the first time.

THREE KEY INNOVATIONS
1
Continuous Latent Reasoning

Instead of operating on discrete tokens (vocabulary IDs), ORL maps the input into a continuous latent space using a learned encoder. All "reasoning" - the actual computation that transforms a question into an answer - happens in this continuous space. Discrete tokens are only produced at the final decoding step. This avoids the sequential bottleneck because continuous vectors can be processed in parallel.

2
Adaptive Denoising Schedule

Not all problems need the same number of refinement steps. ORL uses a learned difficulty estimator that predicts how many denoising iterations a given input requires. Simple questions (factual recall, short answers) may need only 4-8 steps. Complex multi-step reasoning (MATH-500 level) may use 32-64 steps. This dynamic scheduling is why ORL stays fast on easy queries without sacrificing accuracy on hard ones.

3
Trivial GPU Parallelism

Because every token position is processed simultaneously, ORL scales linearly with GPU count. Two RTX 5090s give you roughly 5,500 tok/s. Four give you roughly 11,000. Autoregressive models hit diminishing returns with multi-GPU setups because of the sequential token dependency. ORL has no such constraint - each denoising step is a single forward pass over all positions.

ORL INFERENCE PIPELINE (SIMPLIFIED)
Input:  "What is the derivative of x^3 * sin(x)?"

Step 1 - Encode:
  tokens  -->  encoder  -->  latent vectors (continuous)

Step 2 - Initialize:
  z_T = gaussian_noise(shape=[max_output_len, latent_dim])

Step 3 - Denoise (N steps, all positions in parallel):
  for t in range(T, 0, -1):
      z_t-1 = denoise_step(z_t, input_latents, t)
      # N = 4-64 steps (adaptive to difficulty)

Step 4 - Decode:
  z_0  -->  decoder  -->  output tokens

Output: "3x^2 * sin(x) + x^3 * cos(x)"

# Why This Changes Everything

ORL is not just a faster model. It is a fundamentally different architecture that breaks assumptions the entire LLM industry is built on.

⚡ Real-time inference on consumer hardware

2,788 tok/s on a single RTX 5090 means high-quality reasoning at speeds that feel instant. No data center required. The 3B checkpoint runs on a laptop GPU. This democratizes access to reasoning-capable models in a way that trillion-parameter autoregressive models never will.

💰 Cost collapse for inference

If you can serve the same quality at 46x the throughput, your cost-per-token drops by roughly 46x. For companies spending millions on API calls, ORL-class architectures could reduce inference costs from dollars to cents. The economics of LLM deployment fundamentally change.

🌐 Linear multi-GPU scaling

Autoregressive models hit a wall with multi-GPU parallelism because each token depends on the previous one. ORL processes all token positions simultaneously, so adding GPUs gives near-linear throughput gains. Two GPUs = 2x speed. Four GPUs = 4x speed. This makes scaling trivial.

🔓 Fully open-sourced

Both the 3B and 8B checkpoints are released under permissive licenses. The training code, evaluation harness, and pre-trained weights are all available. Cambridge + EPFL + Anthropic + Mistral collaborating on open research is notable - especially given Anthropic's and Mistral's commercial interests.

SCALING: ORL vs AUTOREGRESSIVE
GPUs       AR (GPT-6 Comet)     ORL-8B
------     ----------------     ------
1x 5090            ~60 tok/s     2,788 tok/s
2x 5090            ~90 tok/s     ~5,500 tok/s
4x 5090           ~140 tok/s    ~11,000 tok/s
8x 5090           ~200 tok/s    ~22,000 tok/s
                  ^diminishing    ^linear
// Bottom Line

ORL proves that autoregressive decoding is not the only path to high-quality text generation. 2,788 tokens per second on a single consumer GPU, matching GPT-6 Luna quality on hard reasoning benchmarks, with linear multi-GPU scaling. The 3B and 8B checkpoints are open-sourced by a collaboration between Cambridge, EPFL, Anthropic, and Mistral. If this architecture scales to larger models, the autoregressive paradigm that every major LLM is built on becomes the slow, expensive option. Paper: arXiv 2610.17894.

Enjoyed this?

New episodes Mon, Wed, Sat.