cd ~/series/demystifying-ai
OpticalRAG: RAG in 16 Visual Tokens
EP 59 arXiv: 2611.13675 Nov 18, 2026 5 min

OpticalRAG: RAG in 16 Visual Tokens

Pixel-space compression for retrieval. Render the page, keep 16 query-conditioned tokens, beat text compressors on LongBench-v2.

Share:
// TL;DR

OpticalRAG (Liu et al., NeurIPS 2026 / arXiv:2611.13675) argues the RAG bottleneck is the compression modality, not raw information density. It renders docs as page images, encodes with a frozen ViT, retrieves via Encoder-Bridged Retrieval (no contrastive training), then Query-Driven Distillation packs the hit into 16 prefix tokens for frozen Qwen2.5-7B-Instruct. LongBench-v2: 35.79 vs closed-book 30.62 (+5.17). Best CAP on all three benches.

16
Prefix tokens
+5.17
LongBench-v2 lift
55.67
Avg CAP (best)
0
Contrastive pretrain

# The Bottleneck Is the Modality

Feeding long docs into the LLM is expensive. Aggressive text compression throws away layout, tables, and fine detail that a 1D token stream cannot keep. OpticalRAG says that is a modality choice, not a hard information limit.

TEXT COMPRESSION PATH
PDF / pages
↓ OCR / chunk
1D token stream
↓ prune / summarize
Hundreds of tokens · layout gone

Once structure is flattened to text, compressors cannot recover tables and figures cleanly.

PIXEL COMPRESSION PATH
PDF / pages
↓ render images
Frozen ViT (~100 visual toks / page)
↓ query distill
16 prefix tokens · layout kept

Modern visual encoders pack a rendered page with little semantic loss - a tradeoff text alone cannot match.

# Classic RAG vs OpticalRAG

Same job: retrieve evidence, answer with an LLM. Different thing stuffed into the prompt.

CLASSIC TEXT RAG
1 Embed query in text space
2 Retrieve text chunks
3 Paste chunks into context
4 LLM answers on a fat prompt
OPTICALRAG
1 Render docs → page images
2 Bridge query into visual space
3 Distill retrieved visuals → 16 toks
4 LLM answers on a tiny prefix

# System Architecture

Five stages. Two frozen giants. One small trainable middle.

END-TO-END FLOW
01 · RENDER
Docs → page images → frozen DeepEncoder
02 · BRIDGE (TRAIN)
Text query projected into visual feature space
03 · RETRIEVE
Top-k visual pages / tokens from the index
04 · DISTILL (TRAIN)
Cross-attn compressor: 256 → 16 prefix tokens
05 · GENERATE (FROZEN)
Qwen2.5-7B-Instruct reads the 16-token prefix
FROZEN (DO NOT TRAIN)
  • →DeepEncoder / ViT
  • →Qwen2.5-7B-Instruct
TRAINABLE (SMALL)
  • →Encoder Bridge projection
  • →Query-Driven Distillation

Plug-in stack: keep the backbone, train the bridge + compressor only. No expensive vision-language contrastive pretraining.

# Ingredient 1: Encoder-Bridged Retrieval

The hard part of pixel RAG: the user types text, the index holds visual features. OpticalRAG does not train a CLIP-style dual encoder from scratch.

TEXT QUERY → VISUAL SPACE (NO CONTRASTIVE)
Text query states
↓ shared projection (also used by compressor)
Projected into frozen ViT feature space
↓ nearest neighbors
Retrieved visual tokens / pages
USUAL APPROACH

Train contrastive text-image pairs so query and page live in one space. Expensive. Separate from the compressor.

OPTICALRAG BRIDGE

Project query states into the already frozen visual feature space with a projection shared with the compressor. Retrieve without a second contrastive stage.

# Ingredient 2: Query-Driven Distillation

Retrieved pages still cost too many visual tokens. A lightweight cross-attention compressor condenses them into a query-conditioned prefix the LLM can actually afford.

COMPRESSION FUNNEL · 256 → 16
RETRIEVED VISUAL FEATURES 256

Dense pack from retrieved page images (schematic width = budget before distill).

Query-Driven Distillation
cross-attn · query = Q · visuals = KV
PREFIX TOKENS 16
v1
v2
v3
v4
v5
v6
v7
v8
v9
v10
v11
v12
v13
v14
v15
v16

query-conditioned · injected as prefix

WHAT THE LLM ACTUALLY SEES
[v1 v2 ... v16]  <- 16 distilled visual prefixes
User: What does Table 3 report for X?
Assistant: ...

Not a generic summary of the page. The 16 tokens are shaped by the query, so different questions pull different evidence from the same pages.

# Why 16 Tokens Matter

Context windows grew. Cost and latency did not vanish. Compression is how RAG stays useful when every retrieved chunk eats budget.

CONTEXT BUDGET (ILLUSTRATIVE SCALE)
Raw multi-page RAG dump thousands
Text compressors (RECOMP / LLMLingua-style) hundreds
OpticalRAG prefix 16

Bar widths are relative for the article, not a paper-reported exact count for every baseline.

# How to Read CAP

Raw accuracy alone rewards fat prompts. CAP (compression-aware performance) rewards methods that beat closed-book while staying cheap.

INTUITION
S
method score
S₀
closed-book score
CAP ↑
when S > S₀ on a tiny budget

The paper still reports raw S even when S < S₀. The compression bonus only kicks in when retrieval actually helps.

HIGH RAW, LOW CAP

Paste a novel into context. Accuracy up a bit. Token bill huge. CAP stays modest.

OPTICALRAG STYLE

+5.17 on LongBench-v2 with 16 tokens. That is the CAP story - lift per budget.

# What the Paper Measured

Authors: Senhao Liu, Yuheng Zhang, Chunyu Wei, Yueguo Chen, Xinran Zhang (Renmin University of China). Accepted at NeurIPS 2026. Same backbone for comparisons: Qwen2.5-7B-Instruct. Eval is multiple-choice long-context style.

Code: github.com/brighterr/OpticalRAG.

LONGBENCH-V2 · SAME BACKBONE
Closed-book Qwen2.5-7B 30.62
OpticalRAG (16 toks) 35.79 · +5.17

Bar widths relative to OpticalRAG 35.79. CAP on this bench: 49.68 (best).

LONGBENCH-V2 · 35.79 / CAP 49.68

Closed-book 30.62 (+5.17). Best raw + CAP at 16 tokens.

BAMBOO · 61.00 / CAP 84.67

Win is CAP, not raw accuracy vs RECOMP / LLMLingua-v2.

LOOGLE-V2 · 23.54 / CAP 32.67

Best raw + CAP; xRAG CAP 32.54 is a near-tie on 1 token.

AVG CAP · THREE BENCHES
55.67
OpticalRAG
↓
Text comps.
↓
1-tok RAG

OpticalRAG avg CAP 55.67 is best among compared methods. Neighbor bars are schematic relative height.

# When This Matters for You

Not a universal "replace every RAG stack" paper. It is a sharp bet on layout-heavy docs + tiny context budgets.

LEAN PIXEL RAG

Layout-heavy PDFs, tiny token budgets, MC long-context evals, render pipeline available.

STAY ON TEXT

Clean text corpora, fat context OK, need exact quotes, text index already production.

Do credit

CAP wins on all three benches. LongBench-v2 raw +5.17 at 16 tokens. Frozen backbone + shared bridge design.

Do not overclaim

BAMBOO raw is not the win. Eval is multiple-choice. Not "beats all RAG in production."

// Bottom Line

RAG does not have to mean dumping text into the prompt. OpticalRAG shows page images + Encoder-Bridged Retrieval + Query-Driven Distillation can land a LongBench-v2 +5.17 in 16 tokens, with the best CAP across three benches - without contrastive vision-language pretraining. Treat it as a compression architecture win for layout-heavy retrieval. Then check open-ended tasks and production latency before ripping out the text index.

Enjoyed this?

New episodes Mon, Wed, Sat.