Catch the Hallucinated Number Before It Hits the Deck

For a while I couldn’t square two things about AI and numbers that kept being true at the same time. Ask a tool “what was Q2 software spend” and it comes back right, again and again, straight out of the export you gave it. Then the same tool, same conversation, states an OpEx growth rate that a pocket calculator would refuse to get wrong, and states it beautifully. If it can read the numbers, why can’t it do the math?

The answer turned out to be the single most useful thing I know about working with these tools. Once you have it, you stop being surprised, and you know exactly which outputs to check before one of them lands, wrong, in front of your board with your name on the deck.

It’s not doing math at all

A language model predicts text. That’s the whole trick. When you ask for Q2 software spend, the most plausible next text is the figure sitting right there in your document, so retrieval mostly works; benchmark testing of top models finds them close to reliable on that kind of simple lookup. But when you ask for a growth rate, there is no growth rate sitting in the document to retrieve. The model has to produce a number that looks like the answer, and plausible-looking is the only standard it has. The same benchmark testing shows what that means in practice: on multivariate calculations, the kind where one figure is built from several others, accuracy collapses toward zero as the steps stack up.

That also explains the part that used to unsettle me most, which is the confidence. A calculator that can’t compute something says so. The model can’t say so, because it doesn’t know it isn’t computing. The wrong growth rate arrives in prose exactly as fluent as the right one. Nothing about the sentence changes when the number inside it breaks.

And for anyone in finance, notice where that lands. The stated figures, the ones it handles well, were never your job. The derived figures, the ratio, the variance, the trend, are your whole workbook. The tool is weakest precisely where your name is on the line.

The model doesn’t know it isn’t computing.

What the model tells you to check

Once the mental model is in place, the verification routine stops being a checklist you memorize and becomes obvious.

You recompute derived numbers yourself, from the raw source, because those are the ones the model was only ever guessing at. Not every one; on close week that’s fantasy, and a routine that needs heroics is gone by Wednesday. Just the load-bearing ones, the two or three figures a decision will actually rest on.

You ask for receipts, because citations are also just predicted text. A model can produce a plausible-sounding source the same way it produces a plausible-sounding number. The counter is to force contact with the actual document:

For the OpEx growth figure in your summary: name the exact source
document, then quote verbatim the lines and figures you used to
calculate it. Do not paraphrase.

A real citation survives that request and hands you back your own lines. A fabricated one produces a quote that isn’t in the file, and now you know before the room does. Say the tool wrote “OpEx rose 12%, driven mainly by headcount”: recompute the 12% from the raw lines, demand the quote, and if the headcount cost it cites isn’t in your ledger, you’ve caught an invented driver two minutes before it became a boardroom talking point.

And you sanity-check magnitudes, because a text predictor has no feel for scale. A revenue line that moved 40% while the business did nothing of the kind wouldn’t survive one glance against last period, which is exactly the glance the model can’t give it.

Where’s the edge of all this? I’m less sure than I’d like about exactly how many calculation steps it takes before the odds go bad; the benchmark’s answer is directional, more steps, worse accuracy. So I treat every derived number as unverified until checked, and I’d rather over-distrust than explain a wrong deck.

One thing I’ve stopped worrying about is what this does to the value of finance work. With 57% of CFOs expecting smaller teams because of AI, the fluent drafting is now the cheap part, and the judgment about whether a number deserves trust is the scarce part. That judgment is teachable, which is what this piece has been. If you want to map where else in your role it applies, audit your job into tasks; the pattern here is the verify-then-approve half of the four ways to work with AI, and the sibling question of what belongs in these tools at all is covered in what’s actually safe to put into AI at work.