Five ways to check an AI answer, from weakest to strongest

Most people check an AI's answer by reading it, and a clear, confident answer reads as if it has already been checked. Rereading is the weakest of the five checks I use in my own research, which I do with AI agents.

One question orders the checks: name one thing the check does not share with the original. If you reread your own answer, the check shares everything with it: the same person, the same assumptions, the same blind spots. A fresh AI chat shares less. A person with different training, working from a different source, shares the least. This is the old engineering idea of independent verification. The more a check shares with the original, the more likely it repeats the original's mistake. Sharing the same input data counts too.

Check the words as well as the numbers. When fresh AI agents redid a stretch of my work from scratch in July, every computed number came out the same. All the errors were in the sentences: a draft reply said some work was essentially finished when it was only a little over half done, a search reported that my number sequences were not yet published when every one of them was, and a real new result had gone unmentioned. "Finished", "not published yet" and "nothing new here" are claims, and they can be checked like a number.

The five checks, roughly from least to most independent:

1. Reread it yourself. This catches typos and obvious slips.
2. Check it against itself. Do the parts agree with each other and with the files they came from? This catches one part of the answer contradicting another, such as a total that does not match its own breakdown.
3. Redo it a different way, without looking at the first answer. In my research this means a second program written from scratch. Without code, rebuild the number yourself in a spreadsheet from the raw data. A fresh AI chat with none of the earlier conversation removes the first conversation’s wrong turns but keeps the model’s own habits and blind spots; a different AI model, or a different method, is more independent. If the AI found something specific, such as a solution, an example or a matching record, check that one thing directly; checking a found answer is usually much quicker than finding it.
4. Compare it with a trusted outside source that already has the answer. For me that is a published table or a public database. At work it might be the official system the numbers come from, a report that has already been filed, or the document the AI was summarizing. The source owes nothing to your work, which is why this ranks above level 3. Outside sources have errors too, so a mismatch means investigate both sides. I still use level 3 more often, because new work usually has no outside answer yet, and AI makes a second method cheap enough to do every time.
5. Hand it to someone whose only job is to break it. This level adds a motive to find faults, and it catches mistakes in the parts you were confident about. For my work on octal games, the mathematician who maintains the published reference tables for those games reviewed my program and reran my main computation on his own machines. If you work alone, a fresh AI chat told to find the flaw is a stand-in that ranks below level 4: it shares the model’s blind spots, and it will nearly always find something, so treat what it says as leads to check, not verdicts.

For teams: before the work starts, name the person who will try to break it and the date they will do it, so the job is theirs from the beginning and nobody has to volunteer to criticize a colleague afterwards. Record which level each important number reached, in the same place the number lives (the deck's notes, the ticket, the shared sheet).

To use it: take the AI-assisted answer you would least like to be wrong about, and check it one level higher than you did. If you only reread it, check it against itself. If you did that, redo it a different way, and see whether you get the same answer.

For specialists: the July recheck followed a model switch. I had chosen one Claude model, Fable 5; partway through a session I noticed that my sessions had been running on another, Opus 4.8, from July 13 to July 16. The session transcript records the model behind every message, which is how the switch was dated; it does not say why it happened. The trouble was that I had not chosen Opus 4.8 for this work; it is the same model I later picked on purpose to lead the parking-ticket project in other lessons. I asked for everything done under the other model to be redone by fresh agents, with new code and the original sources. The octal games are two-player games in which players take turns removing counters from heaps. The unmentioned result was a new rare value in the game 0.314, the kind of unusual value people who study these games watch for. The reviewer was Achim Flammenkamp, who maintains the reference tables. He reviewed my program and reran the main computation; my records do not say whether the rerun used his own code or mine, so I count it as a review plus a reproduction.