Hallucination is the wrong word for what I keep running into. It implies the model is confused or malfunctioning. The more accurate description is confident incorrectness: the model produces plausible-sounding output, in well-formed prose, citing nothing, with no hedging, and the claim is simply false.
I have been building and operating an AI system that generates structured presentations from user-supplied text. That system runs roughly 1,000 inference calls per day across OpenAI and Anthropic APIs. Over time I have developed a working set of patterns for detecting, handling, and reducing confident incorrectness. This article documents what works in production and why.
The Core Problem: Why Models Sound Right When They Are Wrong
Large language models are trained on text that rewards confidence. Authoritative tone correlates with text that humans rate as high quality. The model learns, in effect, that hedging is a signal of lower-quality output. This creates a systematic bias toward sounding certain.
At the same time, the model has no access to ground truth at inference time. It cannot distinguish between a claim it has memorized correctly and a plausible interpolation it has generated on the fly. From the inside, both feel the same. From the output side, both look the same.
This matters most in three scenarios:
- Numerical claims: the model generates statistics, percentages, or dates from its training distribution rather than from your input.
- Proper nouns: names of people, companies, and products are reconstructed probabilistically, leading to subtly wrong spellings or merged identities.
- Structural constraints: when you ask the model to follow a JSON schema or a specific output format, it complies most of the time but drifts when the format conflicts with its training prior.
Pattern 1: Schema Enforcement Over Prompt Instruction
The least reliable way to get structured output from an LLM is to describe the format in prose. Return a JSON object with keys title, bullets, and summary works until it does not. The model may add extra keys, wrap the object in markdown code fences, or silently drop a key it has decided is redundant.
The more reliable pattern is to use structured output mode when the API supports it, or to validate against a schema immediately after inference and reject and retry if the output fails validation. In my system, every inference call for structured content goes through a Pydantic model. A failed validation triggers one automatic retry with the validation error appended to the prompt as context. This reduces formatting failures from roughly 8% to under 0.5%.
The key principle: do not describe what you want. Constrain the output space so the model cannot produce anything else.
Pattern 2: Grounding Claims in the Prompt, Not in the Model
If a fact matters, it has to be in the prompt. The failure mode here is subtle: the prompt might mention a topic, and the model fills in supporting details from training memory rather than from the prompt. The topic is correct; the details are invented.
The fix is aggressive grounding. For my use case, when a user provides source text, the system prompt explicitly instructs the model that all content in the output must be directly supported by the provided source material, that it should not add facts, statistics, or claims not present in the source, and that if the source does not support a claim, the model should omit it rather than invent it.
Then the source material is included in full, before the task instruction. The order matters. Material that appears earlier in the context window receives more weight in the attention mechanism, so placing the ground-truth source first and the task instruction second reduces confabulation measurably.
Pattern 3: Temperature and Sampling Strategy
Temperature does not control accuracy; it controls diversity. A low-temperature setting of 0.2 or below makes the model more deterministic, but it does not make it more factual. If the model’s most probable completion is wrong, a lower temperature just makes it wrong more consistently.
What temperature does usefully is reduce variance in format. For structured-output tasks, I run at temperature 0.2 to 0.4. For creative content, I run at 0.7 to 0.9. For factual extraction from provided source text, I run at 0.1. The rationale in that last case is not accuracy per se but consistency: if the source material contains the fact, I want the model to extract the same fact on every call.
Top-p sampling compounds with temperature. Running temperature 0.1 and top-p 0.95 effectively undoes most of the low-temperature benefit, because the nucleus is large enough to include many tokens. For high-consistency use cases, I set both low: temperature 0.1, top-p 0.1. This occasionally produces slightly stilted prose, but it is the right tradeoff when the output feeds into a structured artifact.
Pattern 4: Chain-of-Thought as a Reliability Signal
Chain-of-thought prompting is usually presented as a way to improve reasoning accuracy. That is true, but it has a second use: the reasoning trace is a reliability signal.
When I ask the model to reason through a task before producing the final output, I can inspect the trace for warning signs. A model that expresses uncertainty in its reasoning and then asserts the uncertain claim in its final output is a weaker output than one whose reasoning trace is consistent with its conclusion. I now run a lightweight secondary prompt to score the reasoning trace: did the model express uncertainty at any point in its reasoning, and if so, which claims should be flagged for human review?
This adds latency and cost, so I apply it only to high-stakes outputs. But for a production AI system where output quality directly affects user retention, the cost is justified.
Pattern 5: Retrieval-Augmented Generation as a Ground Truth Anchor
When user-provided text is long enough that it cannot fit in a single context window, the naive approach is to summarize or truncate. Both create reliability problems. Summarization introduces model judgment about what is important; truncation arbitrarily discards content.
RAG solves this by maintaining the original source in a retrieval index and pulling relevant chunks into the context window at inference time. The model is grounded in the retrieved text rather than in its own summarization of the full document.
In my system, chunks are stored with their source position embedded as metadata. When the model generates a claim that traces to a retrieved chunk, the claim can be verified back to source by position. This enables spot-checking without re-running inference.
What Does Not Work
Three patterns that are frequently recommended but unreliable in production:
Self-consistency voting: running the same prompt N times and taking the majority output. If the model has a systematic training-time bias toward a particular wrong answer, that answer wins the vote every time. Self-consistency catches random variance but not systematic bias.
Asking the model to rate its own confidence: the model assigns high confidence to wrong answers at roughly the same rate as correct answers. Self-assessment of confidence is not calibrated.
Negative prompting such as do not hallucinate or do not make up facts. This instruction has no measurable effect. The model does not have a separate hallucination mode it can turn off on request.
The Practical Baseline
For a production AI feature that requires reliable outputs, the minimum viable reliability stack is:
- Schema enforcement: structured output mode or immediate post-inference schema validation with one automated retry.
- Explicit grounding: source material in the prompt, with a prohibition on claims not supported by the source.
- Source-position metadata on chunks: so that any retrieved content is auditable.
- Temperature discipline: low temperature for structured or factual tasks, higher only where creative variation is actually desired.
- Human review hooks: route the subset of outputs that fail schema validation or trigger a low-confidence heuristic to a review queue rather than serving them directly.
None of these individually solves the problem. Together, they reduce confident incorrectness from a frequent occurrence to a manageable exception. LLM output reliability is not a binary property, and practitioners who treat it as one create systems that look good in demos and fail in production.
Conclusion
The models are improving. But the fundamental issue, that a model cannot distinguish between what it knows and what it is generating, is architectural, not a bug to be patched in the next release.
The practitioners who ship reliable AI features are the ones who treat the model as one component in a system, not as an oracle. They invest in the surrounding infrastructure: retrieval, validation, grounding, and review routing. The model does what it is good at; the system handles the reliability properties the model cannot provide for itself.
