Researchers at ETH Zürich found that a lightweight linear probe trained on the final hidden states of an LLM can predict whether the model's answer is correct — before the answer is even generated. ECE (Expected Calibration Error) drops from 19.2% to 3.1% on TriviaQA. Works with Llama-3, Mistral, and Gemma. No fine-tuning. No extra inference cost. The model literally knows when it's about to hallucinate.
# The Confidence Problem
LLMs do not come with confidence scores. When a model says "Paris is the capital of France," it sounds just as confident as when it says "The Treaty of Westphalia was signed in 1645" (it was 1648). You get tokens, not uncertainty. In production, this is a critical gap. You cannot build reliable systems on outputs you cannot trust-score.
Current approaches to estimate confidence are expensive or unreliable. Self-consistency sampling generates N answers and checks agreement — costing N× inference. Verbalized confidence ("How sure are you?") is cheap but models systematically overestimate — they say "95% confident" when they're right only 60% of the time. Token probabilities from the softmax layer work for multiple-choice but fail for open-ended generation. All add latency, cost, or both.
# How Zero-Token Confidence Works
The core insight is almost embarrassingly simple: the model's hidden states already encode whether it knows the answer. Before generating a single output token, the last-layer hidden state at the final position contains a rich representation of the model's internal "certainty." A single linear probe can read this signal.
Standard forward pass. The model processes your question and prepares to generate. This happens anyway — no additional compute.
Before the model generates any tokens, grab the hidden-state vector at the last position. This is the model's internal representation of "what I'm about to say." Dimensions: 4096 for Llama-3 8B, 5120 for Llama-3 70B.
A single linear layer (hidden_dim → 1, sigmoid) takes the hidden state and outputs a confidence score between 0 and 1. The probe is ~16KB for a 4096-dim model. Training requires ~5,000 labeled examples and takes under 2 minutes on a single GPU.
Score > 0.8? Serve the answer. Score < 0.3? Route to retrieval, escalate to human, or abstain. In between? Add a disclaimer. The score is available before generation begins, so you can abort early.
import torch from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8B") tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3-8B") # The probe: one linear layer, ~16KB probe = torch.nn.Linear(4096, 1) # hidden_dim → 1 probe.load_state_dict(torch.load("confidence_probe.pt")) def get_confidence(prompt: str) -> float: inputs = tokenizer(prompt, return_tensors="pt") with torch.no_grad(): outputs = model(**inputs, output_hidden_states=True) # Grab last hidden state at final position hidden = outputs.hidden_states[-1][:, -1, :] # Run probe → confidence score confidence = torch.sigmoid(probe(hidden)).item() return confidence # 0.0 = "I'm guessing", 1.0 = "I know this" score = get_confidence("What year was the Treaty of Westphalia?") # score ≈ 0.34 → model is uncertain → trigger RAG or abstain
# Compared to Other Methods
| Method | ECE ↓ | Extra Cost | Latency | Needs Logprobs |
|---|---|---|---|---|
| Raw token logprobs | 19.2% | 0× | 0ms | Yes |
| Verbalized ("How sure?") | 14.8% | ~1.2× | +200ms | No |
| Self-consistency (N=5) | 8.1% | 5× | +1500ms | No |
| Zero-Token Confidence | 3.1% | ~0× | +0.4ms | No* |
* Requires access to hidden states, which means self-hosted models (vLLM, TGI, transformers) or API providers that expose hidden states. Does not work with OpenAI's API or Anthropic's API directly. Works with any model you host yourself.
# What You Can Build With This
Gate every LLM response with a confidence check. Low confidence → don't serve it. Zero additional latency to your users. In the paper's medical QA experiment, this filter eliminated 91% of incorrect answers while only blocking 12% of correct ones.
Only invoke retrieval when the model is uncertain. High confidence questions skip the vector DB entirely, saving 200-500ms latency and retrieval API costs. The paper shows this reduces RAG calls by 63% with no accuracy drop.
Small model unsure? Route to a larger model. Large model sure? Ship it. Build a cost-optimized cascade without manual threshold tuning. Use the probe on your 8B model to decide when to escalate to the 70B — saving 80% of inference cost.
Show users a confidence bar next to every answer. "I'm 92% sure" vs "I'm 34% sure — you might want to verify this." Honest AI by default. The probe's calibration means the displayed percentage is actually meaningful.
- ● Self-hosted only. Requires hidden state access. Won't work with OpenAI, Anthropic, or Google APIs (they don't expose hidden states).
- ● Domain-specific training. A probe trained on TriviaQA won't calibrate well on medical questions. You need ~5K labeled examples per domain.
- ● Pre-generation only. The probe fires before generation. If the model starts confidently but goes off the rails mid-answer, the probe won't catch it.
The model already knows when it's about to hallucinate. We just weren't listening. Zero-Token Confidence proves that the information is there, in the hidden states, before generation begins. A single linear probe — trainable in minutes on consumer hardware — unlocks it. No extra tokens. No extra cost. ECE from 19.2% to 3.1%. The limitation to self-hosted models is real, but if you're running Llama, Mistral, or Gemma on your own infra, there is no reason this shouldn't be in your inference pipeline by next quarter.
Transformers + GGUF: Local Inference Goes Native
HuggingFace Transformers now loads GGUF files natively. No llama.cpp wrapper. Direct torch integration.
Enjoyed this?
New episodes Mon, Wed, Sat.