Guide · Checking a number
Why a wrong AI answer looks right, and one question that exposes it
A wrong answer from an AI is usually as confident and as specific as a right one, so reading it closely will not show the problem. Here is a case where an AI's explanation fit every number and was wrong, and the question that exposed it.
In August I was extending a published count of polyhexes, shapes made by joining hexagons edge to edge. A shape's size is the number of hexagons in it. On August 14 a comparison with the published counts in the OEIS, the public reference database for number sequences, showed the fast version of my counting program was short for shapes of 20, 21 and 22 hexagons: by 1, then 9, then 41 shapes. Every smaller size matched. A rerun of the slower version, about ten hours of computing, came out short by exactly the same amounts, so the two versions agreed with each other and were both wrong.
Some background on how both versions work. They find each shape by tracing its outline. To avoid counting the same shape twice, they give each shape a label and throw away any shape whose label was already seen. The label was the shape's set of corner points. If two different shapes share the same set of corners, one of them is thrown away, and the count comes up short.
That evening the AI agent running the project (Claude) wrote down a different cause. It said some shapes have an outline that comes back and touches itself at one corner, like a figure eight, and the tracing would miss those shapes. The explanation was specific, it fit the numbers, and it went into the project's notes as the cause. Nothing in it looked wrong, which is the usual case with a wrong AI answer: it is written as clearly and confidently as a right one, so how sure it sounds tells you little.
The question that exposes it: if this explanation is right, what else has to be true? Here, if tracing outlines really missed some shapes, then any program that traces outlines would come up short at sizes 20 to 22, whatever label it used.
That was tested on September 4. A helper agent counted the shapes again, still by tracing outlines, but labeled each shape by its set of edges instead of its set of corners. Its counts matched the published counts exactly at sizes 20, 21 and 22. Tracing outlines was not the problem; the corner label was. The helper's report also gave the reason the explanation could never have been right: on a grid of hexagons every corner is shared by exactly three edges, and an outline that touched itself at a corner would need four.
The corner-label problem had already been described in a project note written on July 10, which the AI session had not read in August.
You do not need to rewrite a program to use this. When an AI gives you a reason for something, write down one thing that would have to be true if the reason were right, and check that one thing. For example, if an AI says sales dropped last month because of a holiday, then the drop should sit in the holiday week and not in the other weeks. Look at the weeks before you repeat the reason in a report.
The full account of this error is in the post "An AI's explanation for my counting error was wrong, and how it was tested," in the Posts section of this site.