An AI's explanation for my counting error was wrong, and how it was tested

On August 14 a comparison with published counts showed an error in one of my counting projects. I had two versions of a program that counts polyhexes, shapes made by joining hexagons edge to edge: one built to run in parallel and one not. At sizes 20, 21 and 22 the two agreed with each other, and both came in short of the published counts in the OEIS, sequence A258206, by 1, then 9, then 41. Below size 20 everything matched, and my own published terms for sizes 14 to 19, in the companion sequences A258204 and A258205, were unaffected.

I had set up the comparison with the published counts because two programs that agree with each other can still share the same wrong assumption.

That evening Claude, the AI agent that runs the project under my direction, wrote down why it thought this had happened. The program finds each shape by tracing its outline, and the explanation was that some shapes have outlines that touch themselves at a corner, which the tracing would miss. It fit the numbers, so it went into the project's notes and planning files as the cause.

That explanation could not be true. On a honeycomb every corner joins exactly three edges, and an outline that touched itself at a corner would need four. It accounted for the gap, but nobody had tested it the way we test a number.

The test came on September 4. A helper agent extending the count past size 19 used a method that still traced each shape's outline, the step the explanation blamed, but recorded each shape by its edges instead of its corners. That method matched the published counts at 20, 21 and 22 exactly: 482,416, 1,467,094 and 4,436,536. If the outline explanation were right, no outline-tracing method could have matched them, so that result ruled it out. I could have run the same kind of test in August.

The real cause had been in my own files since July 10. A verification note in the project's folder, dated that day, described it: all three versions of the program recorded each shape by its set of corners, and from size 19 on, one set of corners can be joined into two different outlines. At 19 the two outlines were always the same shape turned around, so no error resulted, which the note called "Lucky, not safe." At 20 to 22, it said, "the luck runs out," and it listed the shortfalls, starting with 482,415 against a published 482,416. It also gave the fix: record each shape by its edges. The August explanation was written without reading that note.

Every file that carried the wrong explanation got a correction, and the paper idea it had suggested was dropped. One document never needed fixing: a proposal that had already gone out described the two programs as sharing "one reduction step," which was true under either explanation.

I now test the explanation for an error separately from the error itself. For a number, the test is whether an independent method gets the same value. For an explanation, the test is whether a method that still has the suspected cause, but differs somewhere else, gets the right answer; if it does, the suspected cause is ruled out. That test takes about an afternoon.

Now the project's own notes get read before anyone explains a failure. The July note was in a file next to the code, and its first lines said in capitals that the code was wrong past size 19. In August the session had looked only in the results folder, which is a different folder.

Drafted with Claude, like everything here, and checked against my research record before it went up.