Guide · Already building with AI

An AI's instructions had the wrong numbers, and this time the AI caught it

An AI agent was told to check its work against six numbers that the agent writing its instructions had gotten wrong. It looked them up in the project's files, found no match, used the right numbers, and reported the mistake. What in its instructions made that possible, and a prompt you can reuse.

I'm Tyler Orden, an independent researcher who does mathematics with AI agents. My research record is at Cyborg Mathematics.

If you paste a target number into a request, an AI tool may well use it without checking it. A manager who types last quarter's figure into a brief from memory has the same problem. On August 28 one of my AI agents was handed wrong numbers, checked them against the project's files, and declined to use them. It happened once, and I haven't tested how often an agent does this.

The project was counting shapes made by gluing together copies of the Josehedron, a 3D shape whose copies fit together to fill space. How many different shapes you can make from 1 up to 6 copies was already published in the OEIS, the public reference database for number sequences, and the project kept copies of those published counts in its files.

The lead Claude session, which I was directing, handed a job to a helper agent, a separate AI session with its own written instructions: rebuild the project's core data on which copies touch which, then test that data by using it to reproduce the six published counts.

The instructions gave the counts as 1, 4, 16, 116, 903 and 8,551. The real ones are 1, 2, 15, 131, 1,360 and 15,133. The lead session hadn't copied them from the files. It had worked on two other shapes earlier in the same session and wrote the numbers from that earlier work, mixing the two up: the database entry numbers came from one shape and the six counts from the other. That is the part worth noticing: the agent writing the instructions supplied the wrong targets, and the agent reading them caught it.

Before using the six numbers, the helper searched the project's files for them. They appeared nowhere, and the files showed that the database entries named in its instructions belonged to a different shape. It set the six numbers aside and checked its work against the project's saved copies of the published counts. They matched exactly. Its report said: "a tasking discrepancy recorded, not followed." The lead session then reran all eight of the helper's tests itself, all eight passed, and it logged the mistake and the catch in the project log the same day.

The instructions never said "refuse numbers that look wrong." Three parts of them pointed the helper toward checking, and I can't tell which one made the difference:

They told the helper to read, before doing anything else, the project's file of tests, which was written before the work started and says what the results must match.
They told the helper where the project kept its data for this shape, described as holding the published values.
They told the helper that failure was an acceptable result: "A FAIL is a valid deliverable", followed by "record it precisely and stop; do not paper over."

If you add only one of these to your own instructions, the third costs one sentence.

Afterwards the project adopted a rule: any number that goes into an agent's instructions is copied from the project's files, never typed from memory. For anyone building with agents, that means passing a pointer to the data (this file, this row) instead of a value someone typed.

Paste with your request (attach the source report first)

The numbers in this message come from the attached file. Before you use any of them, find it in the file and quote the line or row you took it from. If a number in this message does not match the file, tell me which one and use the file's value. Telling me they don't match is a good result.

Asking for the quoted line matters because AI tools misread attachments, and a quote is quick to check. Check it against the file yourself, since a quote can be wrong too. This prompt is built from one case; I haven't tested it beyond that.

For teams: when a manager writes a brief with target numbers, attach the source instead of typing them, and add a line inviting the person doing the work to report any mismatch. When they find one, they note it in the brief's own thread or ticket, so the record of which number was wrong, and which source was right, sits next to the work.

For specialists: the Josehedron is a twelve-faced solid (four triangles, eight quadrilaterals) described by Mathias Bernhard (arXiv:2604.07160). Shapes are glued face to face and counted as the same if one can be rotated or reflected onto the other. The correct counts are the free polyforms of the Josehedron honeycomb, OEIS A398959 (oeis.org/A398959; fixed and one-sided counts are A398957 and A398958). The instructions cited A397708 and A397709, which belong to the sphenoid honeycomb, and the six values cited are the first terms of A385028.