Guide · New to this
Read a change back before recording that it was made
Seven times in about a month, my AI agent's log of what it had done disagreed with the files themselves. The fix is to look in the file, or have a script look, before anyone writes down that a change happened.
When ChatGPT, Copilot or an AI agent says it has changed a document, a spreadsheet or a slide, open the file and find the change before you tell anyone it is done. Search for a phrase from the new text. If it says it changed three things, look for all three.
For teams: the person or agent who made a change should not be the only one who confirms it. Decide who confirms (a second person, or a small script that searches the file for the expected text), do it before the work is marked done, and write "confirmed by" next to "done" wherever the team tracks work, such as the ticket or the change log.
Here is why. My research project is mathematics, and Claude, an AI agent, does most of the hands-on work: it edits papers and data files, runs checks, and after each step writes a line in a project log saying what it did. I read the log to know what has been done. The same agent did the work and wrote the log, and nothing compared the two.
Between August 23 and September 24, the log and the files disagreed seven times:
| Date | What the log said | What was true |
|---|---|---|
| August 23 | An edit to a paper was made. | The edit program searched for a passage, found no match, and changed nothing. |
| August 27 (two separate saves) | A new item had been added to the project status page. | The edit program crashed before writing anything, both times. |
| September 4 | A file was being kept out of git, the tool that keeps the project’s saved versions. | Git had been keeping the file all along. The note was written before the check that tested it had run. |
| September 12, 22 and 24 | A change was recorded and saved as done. | The check in front of it had failed, but the steps were joined so that a failure did not stop the save. |
In the first four, the sentence saying something happened was written before anyone looked. In the last three, the save did not wait for the check. The log gained 334 entries over those weeks and nobody went back over all of them, so there may have been more than seven. The seven that were found were caught by Claude itself, four of them within minutes, by reading the output of a later step or by a check that looked for the change.
Agents that report success before the work is finished are a known problem in AI agent work. The fix is old: confirm the result by looking at it, and do it mechanically where you can, since an agent rereading its own work can repeat its own mistake.
That prompt still relies on the AI's report of its own search, so for anything that matters, do the search yourself or have a script do it.
If you write scripts:
- Make edit tools fail loudly when they match nothing, instead of quietly changing nothing.
- Put the check in front of the step it protects as check || exit 1, or run it as its own step. Do not rely on set -e alone: it is switched off inside && and || chains and if conditions, and without set -o pipefail it ignores failures inside a pipeline.
- Better still, run the confirming search in a hook or a separate script that the agent does not control, so a failed check stops the save whatever the agent writes.
- If your project uses git, git show --stat lists the files the last commit actually touched (it shows nothing for merge commits by default); compare that with what its message says.