Write down the numbers you must match before you compute anything

Before an AI builds you a forecast, a model or a count, write down three things: the known number it must reproduce, the result that would make you stop, and one small case you can check by hand. Do it before you see any output. An hour into the work you will want the result to be right, and a fair test is much harder to set then. Software testers call the first line a known-answer test; it is a small cousin of what researchers call preregistration.

No published number to match? Use last month's figure from the system of record, or a total you can count yourself. Keep the file for yourself rather than pasting it into the chat, so the AI is not tuning its work to your test.

Say an AI is helping you build next quarter's revenue forecast. For line 1, build the forecast using only data from before last year, and check that it predicts last year's actual revenue within a tolerance you choose now. Testing it on the data it was built from proves little, because it was fitted to that data. For line 2, write down how big a miss would make you rebuild instead of adjust. For line 3, take one month and work it out from the invoices.

I used this on a research project that extended a 2018 published count of benzenoids, molecule shapes made of hexagonal rings, sorted by number of hexagons. Claude, an AI, wrote and ran the counting program under my direction. The test file is dated in the project's version history to August 7, a week before the program existed. It required the program to reproduce the paper's counts exactly, and its printed percentages to the four decimal places shown, before any new row counted.

Some rows would not match, and each mismatch raised the same question: is the program wrong, or the paper? The clearest case was settled by the paper itself. One table listed 333 shapes in a group, and another table gave the number of pairs in that group as 55,270. Any 333 things form exactly 333 × 332 ÷ 2 = 55,278 pairs, and 55,270 is not a possible pair count for any number of things. A second, separate reading of the PDF confirmed the misprint was in the paper. In all, six errors in the paper turned up. Five were established while making the tables match, and the sixth turned up while reading the paper against its own tables during the write-up. Three percentages never matched because the paper printed the same row of three numbers in two tables; the program agreed with one table, so the row in the other had been copied there by mistake. That copy is one of the six errors. The paper's authors were sent the corrections and invited to comment.

Nobody set out to look for errors. They were found because the file said no new rows could be added until every mismatch with the source was explained.

For teams: these three lines are acceptance criteria. The person who owns the result writes them before the work starts, the team agrees who may change them, and they live in the same folder or ticket as the work, with every change added as a dated line.

For specialists: the source is Chapman, Foos, Hartung, Nelson and Williams, "Pairwise Disagreements of Kekulé, Clar, and Fries Numbers for Benzenoids: A Mathematical and Computational Investigation," MATCH Commun. Math. Comput. Chem. 80 (2018) 189-206. Tables 2 and 3 were reproduced exactly; 120 of the 123 disagreement percentages in Tables 5 to 7 matched to four decimals, and the other three are the Table 6 row at h = 12 duplicated from Table 7 (correct Kekulé-Clar values 3.0751, 3.1055, 2.9181). The pair-count errors are in Table 4 at (h, v) = (9, 34) and (9, 36); the second also prints 913,270 where C(1352, 2) = 913,276. Another error, the Figure 3 example with K = 13 at h = 7, does not occur in a full count, which rests on our program, cross-checked against a slower direct method on all 1,881 benzenoids with h ≤ 8. For the new rows, the totals at h = 14 and h = 15 matched the published counts in OEIS A018190; their disagreement percentages have no published reference to test against.