Five ways to check an AI answer, from weakest to strongest
Five ways to check an answer, from rereading it yourself to giving it to someone whose only job is to break it, and how to move up one level.
Guide
How to check numbers an AI gives you
Short lessons on checking a number an AI gives you before you rely on it. Each one is built on something that happened in my own research, and none needs a math background.
To see where each lesson fits in a project, from before you compute anything to writing up the result, read How the lessons fit together.
Start here
Read these four first, in this order.
Ask an AI to confirm each fact by name, and tell it where to check
Two AI helpers looked at the same data. The one told to confirm each column by name got it right. The other said two columns did not exist, and a direct check found both.
Write down the numbers you must match before you compute anything
Before an AI builds anything that produces numbers, write down the number it must reproduce, the result that would make you stop, and one case you can check by hand. On one project, making a program match a published paper's tables this way turned up errors in the paper.
A second program run to check the first gave the same wrong answer
Two versions of a counting program came up short by exactly the same amounts, and only a published table showed it. Two answers check each other only if they differ in the step most likely to be wrong.
Checking a number
Why a wrong AI answer looks right, and one question that exposes it
An AI's explanation for a counting error was specific, fit every number, and was wrong. The one question that exposed it, and how to use it on any explanation an AI gives you.
A program that worked at every size it was tested on broke at the next size up
A program an AI wrote matched every published count it was tested on, then crashed on the very next size. Describe a result by the cases you actually tested.
Recount the total from the list, and count each item once
A total I quote often read 34 in late August and 34 in early September, and the list underneath was wrong twice in between. Decide what one item is, count each one once, and recount the total from the list.
Keeping your records true
Keep a table of every claim and how well it is supported
One row per claim, a label for how well it is supported, the evidence, and who checked it. A spreadsheet works. Claims that turn out false stay in the table, marked false.
Read a change back before recording that it was made
Seven times in about a month, my AI agent's log of what it had done disagreed with the files themselves. Open the file and find the change before you write that it happened.
Check that your work was really backed up, whatever the tool said
Before you believe "saved" or "backed up", open the file from somewhere else. For a week my project's log said a folder of 96 data files was backed up, and none of them were.
A status page said "running" for six days after the work had stopped
Each update of the page copied the old wording forward. Check the real state before you repeat a status word.
An AI's instructions had the wrong numbers, and this time the AI caught it
One AI agent wrote another's instructions and gave it six wrong numbers to check its work against. The second agent looked them up in the project's files, used the right ones and reported the mistake. What in its instructions made that possible.
Before you publish
Before you publish, have a reader who knows nothing read it
Five AI readers with no background on my work read every public page of my two websites, and only 9 of the 43 pages were clear to them. Why my earlier editing missed it, and a prompt you can paste into a fresh chat to test any document.
One project, several lessons: AI agents and 25.8 million parking tickets
A team of AI agents tested whether 25.8 million Los Angeles parking tickets contained a finding worth publishing. The answer was "not yet". Each lesson below takes one part of that project.
Write up a "no" as carefully as a "yes": one AI research run from ten ideas to "not yet"
A checklist for writing up an idea that didn't work out, and the project it comes from: ten ideas, what happened to each, and the "not yet" verdict.
How to have separate AI helpers check each step of your work
At each major step, give the results to fresh AI sessions whose only job is to find what was missed, check what they say, and log what you accepted and why. How to do it with ordinary chat windows.
What quick AI checks caught before a project's expensive step, and when to stop adding them
Quick checks showed the two best ideas were partly done already, so the project stopped before its costliest step, and it stopped adding checkers once they only repeated each other.
Pick a comparison group that differs only in ways that cannot cause the pattern
The lead AI compared sweeping tickets with tickets that were bound to spread evenly over the hour, and a helper AI pointed out that such a comparison could not answer the question. How to pick one that can.
How an AI research run decided there was no paper yet
The same project step by step: what the lead agent and its eleven helpers did, the instructions behind each step, and how each claim was checked.
Two research projects, start to finish
The parking-ticket project next to a math project on twelve two-player games. Before extending the published tables for those games, the program had to match every number already in them, and the mathematician who keeps the tables later reran part of the new work on his own machines and got the same answers.
The research these lessons come from
My research follows three habits: tie every claim to an outside reference that could prove it wrong, get every computed number a second time by an independent route, and record every error the checks find. My research site, Cyborg Mathematics, has the full record: each result, how it was checked, and the evidence.