Guide · Already building with AI

A second program run to check the first gave the same wrong answer

Two versions of a counting program came up short by exactly the same amounts, because they shared the step that was wrong. A published table caught it. Two answers only check each other if they differ in the step most likely to be wrong, and that applies to chatbots, agents and colleagues.

I'm Tyler Orden, an independent researcher who does mathematics with AI agents. My research record is at Cyborg Mathematics.

Two answers that agree feel like a check. If they got there the same way, they are one answer counted twice. Two analysts working from the same spreadsheet will agree on its mistakes. So will two AI chatbots that both found the same web article: if two of them give you the same market size, ask each for the link it got the number from, and if both found the same article, you have one source.

In August, two versions of one of my programs gave the same wrong counts, and the second had been run specifically to check the first. The project counted polyhexes, shapes made by joining hexagons edge to edge, by size (the number of hexagons). Claude, the AI I work with, wrote and ran the programs under my direction. The main target was how many shapes of each size are their own mirror image, a count published only up to size 19. Every run also produced the total number of shapes of each size, and those totals were already published in the OEIS, the public reference database for number sequences, up to size 32. The rule, set before the run, was to compare every new total with the published one and throw out any size that did not match.

On August 14, the fast version of the program, which splits the work across many processors, came out short at sizes 20, 21 and 22: by 1 shape, then 9, then 41. Every smaller size matched. Those three sizes were thrown out, and a run at size 23 was stopped. The first suspicion was the splitting, so the slower version, which does everything in one sequence and was treated as the clean reference, was rerun to size 22, about ten hours of computing. Its totals were short by exactly the same 1, 9 and 41.

Both versions came from the same design, written by AI sessions, and both used the same shortcut for spotting duplicates: they traced each shape's outline and labeled the shape by the set of corner points on it. From size 19 on, the same set of points can sometimes be joined into two different outlines. At size 19 the two outlines always turned out to be the same shape turned around, so nothing was lost. From size 20 they can be different shapes, and one of them was thrown away as a "duplicate". Splitting the work was a real difference between the two programs, but not in the step that was wrong. Without the published totals, the slower run would have looked like confirmation.

What happened next: in September a helper agent counted the shapes again by a different method, labeling each shape by its edges instead of its corners, and matched the published totals exactly at sizes 20, 21 and 22. The mirror-image counts for sizes 20 to 32 were then computed by separately written methods that agreed with each other, and were published in the OEIS on September 12. How the cause was finally found, after a first explanation that was wrong, is in the lesson "Why a wrong AI answer looks right, and one question that exposes it."

Software engineers have measured this. Some safety-critical systems run several separately written versions of a program and take a vote, so one faulty version gets outvoted. In a 1986 experiment, Knight and Leveson found that separately written versions failed on the same inputs far more often than independent failures would. The same applies to AI: asking one model the same question several times, or asking a model to grade its own answer, gives agreement that shares the model's blind spots.

Before you count two answers as a check

The step most likely to be wrong here is ___. The two answers do that step differently because ___. If you can't fill the second blank, find a third answer that does that step another way, such as the original report or a number you can count yourself.

In the polyhex story, the first blank is "spotting duplicates". The fast and slow versions could not fill the second blank, and the published totals could.

If you build with code: two agents that read the same retrieved document, or call the same helper function, will repeat its errors and agree with each other. Check at least one result against something neither of them touched.

For teams: when two reviewers sign off on an analysis, the person who approves it asks each reviewer, before approval, what they checked and against what source. If both read the same summary or reran the same spreadsheet, record it in the review notes as one review, and get a second check that goes back to the raw data.

For specialists: the three counts are free (rotations and reflections identified), one-sided and mirror-symmetric polyhexes, OEIS A258206, A258204 and A258205 (oeis.org/A258205). Before extending anything, the programs reproduced all three exactly up to size 13. The duplicate label was a vertex-set canonical form of the self-avoiding boundary polygon; an edge-level canonical form reproduces A258206(20..22). Knight and Leveson, "An experimental evaluation of the assumption of independence in multiversion programming," IEEE Transactions on Software Engineering, 1986.