Guide · New to this
Keep a table of every claim and how well it is supported
One row per claim, a label for how well it is supported, the evidence, and who checked it. A spreadsheet works. Claims that turn out false stay in the table, marked false. An example from an AI research project on 25.8 million parking tickets.
When an AI hands you an analysis, some of its statements were checked and some were not, and both kinds read the same. If your write-up does not show which is which, your readers cannot tell either. A simple fix is a table with one row per claim; a spreadsheet works. Scientists have long kept something like it in a lab notebook, where entries are dated and never erased. With AI writing the analysis, the table is where you keep track of which sentences anyone actually checked.
Start with the claims your decision rests on, perhaps the top ten, rather than every sentence. Use these columns:
Three statuses are enough to start: CHECKED, NOT CHECKED, FALSE. The project below split NOT CHECKED three ways, because the reason something is unchecked tells you what to do next. Mark something CHECKED only when the evidence column says how to repeat the check, so someone else could redo it in a few minutes.
| Start with | The project used | Meaning |
|---|---|---|
| CHECKED | VERIFIED (green) | Checked against the source, by a method recorded in the evidence column that someone else can repeat. |
| NOT CHECKED | INFERRED (blue) | Fits what is known, but nobody has tested it. |
| NOT CHECKED | PARTIAL (yellow) | Some evidence is in, and the check is incomplete or points both ways. |
| NOT CHECKED | BLOCKED (orange) | Cannot be checked yet, because the information needed is somewhere you do not have. |
| FALSE | FALSE (red) | Tested and disproven, for the version that was tested. |
For teams: decide before the work starts who may mark a row CHECKED. A good rule: the person or agent who made a claim can mark it CHECKED only if the evidence column lets someone else repeat the check in a few minutes, and the "independent?" column is filled in honestly either way. Keep the table in one shared place, such as the project's shared spreadsheet, and have the reviewer look at the NOT CHECKED rows before any decision is made.
Here is how it looked on one project. A team of AI agents tested an idea in 25.8 million Los Angeles parking tickets, which the city publishes. An officer marks a parked car (the digital version of chalking a tire), comes back later and writes a ticket, and the time between the two might reveal how officers schedule their rounds, which might be new enough for a research paper. The AI doing the main work (the lead) checked the city's live database itself on September 12, and separate helper agents, running a different AI model, checked parts of the data and searched for earlier research. Some of the table's rows, with the checker added:
| Claim | Status | Evidence | Checked by / independent? |
|---|---|---|---|
| The city’s dataset has 25.8 million rows, and its license lets anyone reuse it. | VERIFIED | A count run on the city’s live database; the license read from the dataset’s own description. | The lead and, separately, a helper. Independent of each other; both read the city’s own records. |
| In a live sample, ticket times were recorded to the minute. | VERIFIED | Live sample of tickets. | Not recorded. Repeatable by anyone with the same query. |
| 729,430 tickets record the time an officer first marked the car. | VERIFIED | Live count. | Not recorded. Repeatable by anyone with the same query. |
| In 2022 data, the gaps between marking and ticketing form two clusters, a little over one hour and a little over two hours. | VERIFIED | A histogram of the gaps; rerun on the full year of 2022 in October with the same two clusters. | The project’s analysis, run twice. |
| The clusters come from one-hour and two-hour parking limits. | INFERRED | The cluster positions fit, but no ticket was matched to the limit posted on its street. | Not checked. |
| Nobody has published this before. | PARTIAL | A search found close work, including a patent, but no exact match; some papers could not be read in full. | Helpers that searched for earlier research. |
| Tickets come about 10 to 15 minutes after the posted time limit runs out. | BLOCKED | The posted limit is not in the ticket data. | Needs another source. |
| Officers come back to cars at fixed 15, 30 or 60 minute steps. | FALSE (for those steps) | About half a percent of gaps fell within two minutes of each of those steps, with no baseline to compare against. Other schedules were not tested. | The project’s analysis. |
Two rows show the habits that matter most. The observation (two clusters) and its explanation (parking limits) get separate rows, because only one of them was checked. And the false row stays in the table with its evidence and its limits, so readers can see what was tested and failed, along with what held up. That row also shows why the limits belong in it: the project's own table called the fixed schedule disproven, with no limits, and the wording above was narrowed afterward to what the test actually covered.
For bigger projects, keep the history too. Record which AI model did the work and what it was asked, and when a row changes, add a new dated line instead of editing the old one, so the table shows what was believed on each date.
AI models often label things as checked that they did not check, so their CHECKED-BY-AI rows are claims too. Redo the exact query or quote for the rows your decision rests on, and only then mark them CHECKED in your own table.