Hallucination and limitations

Hallucination is a model producing something that sounds right and is wrong, with full confidence. Hallucination is the failure that matters most on a project, because the wrong number looks exactly like the right one.

Why models hallucinate​

A model predicts the most likely next text. Most of the time that is also the correct text. Sometimes it is not.

Type "the minimum concrete cover over the reinforcing steel is" and the model completes it with a number that appears in many documents. Whether that number is the one in your structural notes, for your exposure condition, in the edition of the standard your spec names, is a separate question. The model cannot tell the difference.

A model that lacks information rarely says so. Instead, it produces whatever fits the pattern:

• Invented references. A paragraph like Section 07 92 00, 2.4.C has the right shape and is not in the spec on file. Code sections do the same: a section number can look right and not exist in the adopted edition
• Plausible dimensions. A cover, slope, spacing, or clearance that is common somewhere, stated as if read off your sheet
• Mixed-up sources. A requirement from another project's spec, or one trade's scope attributed to another
• Confident counts. "14 type F2 fixtures on Level 2" after reading six of ten sheets

Models also have a knowledge cutoff. A new code cycle, a revised standard, an addendum issued this morning, or the firm's updated CAD manual is not in the model's memory unless you provide it.

Open figure

How Nomic limits hallucination​

A model always predicts, but you can limit what it predicts from.

The Nomic agent is grounded in your documents and cites them:

• Claims about the project cite the file and page, the BIM element, or the DWG entity
• Code and accessibility answers retrieve the governing text for the jurisdiction and cite the excerpt. The agent is designed never to state a clearance, slope, height, count, or section number from memory
• When geometry matters, the agent renders the page into a vision model rather than reading the text layer

Hallucination is still possible, and grounding makes it visible. Watch for a finding with no citation, a citation that opens on the wrong sheet, or an excerpt that does not say what the summary says.

Other limitations​

• Agreeableness. Push back and the model may agree when you were wrong. Agreement is not confirmation.
• Arithmetic and counting. Weaker than a spreadsheet. Ask for a table you can total, or use a tool that counts, rather than trusting a number in prose.
• Reading dimensions off pixels. A visual read is an estimate. Verify it in the source file.
• Incomplete inputs. With half the set missing, findings may come back as potential issues. Treat those as review prompts, and be wary of any finding that sounds certain.

Bias and hallucination​

Hallucination is fabrication: a reference, number, or fact with nothing behind it. Bias is a different failure. A model learns from the material it was trained on, so it leans toward the building types, climates, and conventions that show up most in that material. Ask for a typical wall assembly for a cold-climate hospital and you may get the assembly that is common in mild-climate housing.

Verification catches both, and you look for different things:

• Hallucination: a citation that opens on nothing, or on a paragraph that says something else
• Bias: an answer that is well sourced and fits a different project than yours, such as the wrong climate, building type, or regional practice

Giving the agent your own documents pulls answers toward your conditions and away from the typical ones. Your documents reduce bias but do not remove it, so read the answer against what you know about the site, the occupancy, and the region.

Who is accountable​

When an AI-assisted finding turns out to be wrong, the agent is not the one held accountable. The person who reviewed the finding and signed off is, the same as for any other work under their name. Citations make checking faster, but the responsibility to check stays with you. Your contracts and firm policy set the specifics (not covered in this course).

Verification​

Treat the output as a first draft, not the record.

Read the citation, not only the sentence, open the sheet, and check its revision. Your name or stamp goes on the result. Checking each citation also sharpens the review, because you read the source looking for one specific thing instead of skimming it.

Most of this course is about making verification more efficient: getting the model to show evidence, structuring work so errors surface early, and knowing what to delegate to the agent.

Next: how a model reads, and what that costs.

Next: Tokens, effort, and cost