The guide · Case studies
Two research projects, start to finish
Two of my projects, told step by step from their working files: one ended with a written "not yet", the other extended published math tables. Click or tap a step to open it.
This site is about checking the numbers an AI gives you before you rely on them. Here are two of my own projects where AI did much of the work, checked in two different ways. In the first, AI agents checked each other: a lead agent tested its helpers' claims against the city's live data, and later sent fresh helpers to knock down the ideas that were left. It asked whether 25.8 million Los Angeles parking tickets held a new research paper, and decided "not yet". In the second, the check came from outside: an AI-written program had to reproduce published math tables exactly before extending them, and the person who keeps those tables then redid part of the work on his own machines.
What to take from them
- Check the one claim everything else rests on against the source yourself. In the parking run, a helper said two data fields did not exist; one query showed they did.
- Test an exciting explanation against the plain one. Two clusters in the parking data looked like officers' patrol habits and fit ordinary one- and two-hour parking limits just as well.
- Before trusting anything new, make the tool reproduce the old, known answers exactly. The games solver had to match every published number before it was allowed to extend them.
What I decided, and what the AI decided
Parking run: I chose the question and planned the process with ChatGPT, including the ground rules and the bar a paper had to clear. Claude Opus 4.8, the model I picked to lead, ran it and made the calls inside the run. I changed its direction twice, by pasting in a critique that narrowed it to one question and, that night, the rest of the plan with the rule that helpers check each stage. The lead decided on its own to skip most of the remaining audit stages and told me; the record does not show me approving or objecting.
Games project: I checked the published record first, required exact reproduction before any extension, ran the computations, and sent the results to the tables' keeper. An AI agent wrote the solver code.
1. Is there a new paper in 25.8 million parking tickets?
September 12 to 13, 2026. About an hour of agent work, split by a pause of about 22 hours.
TicketTracker is a map I built of Los Angeles parking tickets from the city's public data, which holds 25.8 million tickets. I wanted to know whether that data held a genuinely new research paper, and I handed the question to a team of AI agents.
I worked out the plan in a separate conversation with ChatGPT and pasted the first part of it in. Claude Opus 4.8, which I picked for this run, ran it as the lead agent: it wrote the research questions, sent out helper agents (Claude Sonnet), checked what they reported, and made the calls inside the run.
Two ground rules came first. "We did it in Los Angeles instead of somewhere else" does not count as new. And the work stays historical and aggregate, with nothing that could help someone track parking officers in real time: in September 2025, San Francisco had shut down a site called Find My Parking Cops within four hours of its launch.
Do not optimize for producing 10 ideas. Optimize for finding one genuinely good paper. If 9 ideas are already published and only 1 survives, that is an excellent outcome.
What came of itSuccess was defined as one good paper, and dropping ideas counted as progress.
The lead agent listed which fields the data actually contains, then wrote ten research questions, each tied to the fields it would need. Among them: do street-sweeping tickets pile up right after the posted start time; do certain kinds of vehicles get certain kinds of tickets; are tickets spread unequally across neighborhoods.
Before any helper ran, it wrote down which question it expected to fail: the neighborhood question, because it is already well studied. That question went into the first round anyway.
What came of itThe prediction held: the neighborhood question was the first to be dropped.
Eight helpers worked at once, each with one job. Four searched for existing research on the ten questions. One queried the data to confirm what it holds. One listed what could sink any paper on this data. One recommended methods, and one looked for journals. Meanwhile the lead ran its own queries against the city's live database.
Two questions were already answered. Three published studies already compare ticketing across Los Angeles neighborhoods. And "a few places get most of the tickets" is what anyone would expect: researchers showed long ago that crime and city activity bunch up in a few places.
The methods helper flagged a trap in the street-sweeping question: a sweeping ticket cannot be written before the sweeping window opens, so a jump in tickets at the start time is guaranteed by the rule and proves nothing.
One helper reported that two fields, a parking-meter ID and a patrol route, were not in the data. The lead's own query found both columns. The route column was filled in on about one ticket in six, and the meter column on a similar share, though its codes could not be used as meter numbers. The lead overruled that one claim and wrote down why.
Its queries also turned up a field no helper had mentioned: the time an officer first marked a parked car, the digital version of chalking a tire, before coming back to ticket it for staying too long.
What came of itTwo questions dropped and four set aside as weaker. Four went forward: which kinds of vehicles get which tickets, street-sweeping timing, the gap between the mark and the ticket, and, as a back-up, whether one ticket makes more nearby more likely.
A critique I pasted in from the ChatGPT conversation narrowed the run to one question: how long is the gap between the mark and the ticket, and do officers come back on a fixed schedule? The lead agreed. It ranked this question as the most likely to be new and street sweeping as the safest, and those two are the ones the rest of the run followed.
The other two ideas were not taken further. The lead's own query had found the vehicle signal real but modest, and the methods helper had rated the idea that one ticket makes more nearby more likely the riskiest, because patrol routes create the very clustering it would be looking for.
A helper told to kill the marked-car idea found a 2012 patent on working out patrol revisits from ticket times, and a commercial system built on that idea that is reported to be in use in Los Angeles.
The lead then measured the gaps on tickets from 2022. They form two humps, near 70 and 135 minutes, each about 10 to 15 minutes past one and two hours. That fits one-hour and two-hour parking limits with a little slack. The run found no spikes at multiples of 15, 30 or 60 minutes. That does not rule out every regular schedule: a patrol loop of about 65 to 70 minutes would also put humps near 70 and 135. Only matching each ticket to the time limit posted on its street can tell the two apart.
What came of itStill possible, leaning against: the remaining question needs each ticket's posted time limit, and that is not in the data.
After 1:45 am the run paused for most of the day. Late that evening, about 22 hours later, I pasted in the rest of the plan from the ChatGPT conversation. It asked for every step to be documented, and it added one rule: before moving past any major stage, the lead had to send two to four separate helpers to look for what it had missed, such as untested assumptions, research nobody had searched for, or problems with the method.
Until then the checking had gone one way: the lead tested its helpers' reports against the live data. The new rule added helpers checking the lead. The run applied it at once to the street-sweeping question and sent two.
One searched for existing work and found the burst of tickets after sweeping starts already shown in public: by CURB, a civic project in San Francisco, by a data-journalism project on this same Los Angeles data, and close to it in a 2026 paper and a Chicago study.
The other checked the method. It agreed with the first methods helper that the jump at the start time proves nothing, and it added two new problems. First, the rise in tickets after the start might come only from averaging streets whose sweeping starts at different times.
Second, a comparison the lead had run that evening did not show what the lead thought. The lead had found that red-zone and disabled-placard tickets spread evenly across the hour, and had counted that as support for the sweeping pattern. But those tickets have no start time to cluster around, so they would spread evenly anyway. The lead relabeled the result as a rough check that ticket types with no start time show no bunching at the top of the hour, and nothing more. (Finer rounding is a separate matter: the lead had already found that 24.5% of ticket times fall on a multiple of 5 minutes, against 20% if times were spread evenly.)
Fixing the sweeping analysis needs each ticket matched to its own street's start time. The schedule and the tickets use different route codes, so they could not be matched that way.
YOU MUST SPAWN A SMALL INDEPENDENT AUDIT TEAM BEFORE MOVING FORWARD. For EVERY major stage, spawn approximately 2-4 independent Sonnet-class subagents. Their job is NOT merely to perform the same task. Their explicit instruction is: "Audit this stage independently. Identify anything the principal investigator has missed, any assumptions that have not been tested, any additional data sources, methodological problems, literature we have not searched, alternative interpretations, statistical problems, reproducibility problems, or other work that should be completed before proceeding."
What came of itBoth remaining ideas were blocked by the same missing piece: a reliable match between each ticket and the posted rules on its street.
At the start, the plan had set a five-point scale for how much a paper would add, with 3 as the bar to start writing. The lead scored the best remaining idea 2. It also gave its own rough estimates, which are not calibrated probabilities: about 30 to 40 percent that the core idea is new, about 15 to 20 percent that the evidence is ready for peer review today.
It then asked whether a competent researcher could start writing a legitimate paper from this tomorrow, and recorded the answer no.
Eleven helpers ran in all, and the lead checked the city's live data itself six times. The plan had called for two to four helpers at each of its 25 stages and a five-reviewer panel at the end. The lead judged that these would repeat checks already done, skipped them, and flagged that to me. The record does not show me approving or objecting.
What came of itA written "not yet", with the three things that would have to change: a fuller search for earlier work, the street-by-street match, and a test of the sweeping pattern on single streets.
Read the full walkthrough, with the prompts behind each step and what each one teaches →
2. Extending the tables for twelve math games
July 3 to 17, 2026. Two weeks, on one laptop.
An octal game is a two-player game played with heaps of counters. On each turn you remove some counters, and the game's rules say how many you may take and whether you may split a heap. Each game is named by a short code that spells out those rules, such as 0.007.
Every heap size has a number attached to it, its value (formally its Sprague-Grundy value). A value of 0 means the player about to move loses with perfect play; the values also tell you how heaps combine. Mathematicians compute these values for larger and larger heaps, looking for the point where the list starts to repeat. For many of these games nobody knows whether it ever does, and the question has been open for decades.
For the games here, a fixed rule on each value's binary digits splits the values into two classes. One class turns up far less often than the other, and its members are called rare values. The reference page that tracks these games notes that if a game has only finitely many rare values, its values must eventually repeat. So when a table is extended, the thing people watch most closely is whether any new rare values turn up.
In July, these computations pushed twelve of the published tables further than they had gone.
Achim Flammenkamp keeps the reference tables for these games on his web page. The first step was to fetch his page and confirm it was still the record: it had last been updated in May 2021. A competing reimplementation on GitHub (Tomek Czajka's, last updated April 2025) did not go beyond his numbers, and I found no public results that did.
Next, with an AI agent (Claude) writing the code, I built a fast solver for one game, 0.007. It uses the same shortcut as Flammenkamp's tables, which relies on rare values being uncommon. Before it could extend anything, it had to reproduce every number Flammenkamp had published for that game, exactly. Those included where each of the values 1, 2, 4, 8 and so on up to 1024 first appears (eleven numbers), how many heaps have value 0 and where the last known one is (37, at heap 16,170), and how many rare values there are and where the last known one is (22,476, at heap 5,029,983).
A second, deliberately simple solver agreed with the fast one on every heap up to 60,000. The record shows no mismatch at any of these checks.
What came of itA solver that matched the published record exactly, ready for a long run.
Game 0.007 ran on one core of an Apple M3 Max laptop up to heap 2 to the 30th power, about 1.07 billion heaps. That is four times the published limit of 2 to the 28th. It took 7.06 hours. The run saved its progress as it went, and before launch I confirmed that a run stopped and resumed gives bit-identical output. It reproduced every published number at the old limit before any new value was trusted.
The same method then extended eleven more games. Four of them had new rare values past the old limits: 185,576 of them in game 0.114, 8,162 in 0.606, 2,932 in 0.006, and a single one in 0.314. In the other seven, none appeared, and none appeared in 0.007 either.
What came of itTwelve games extended past the published tables, each checked against the old numbers first.
The results went to Flammenkamp. He reran parts of the work on his own machines with his own programs: 0.006 up to 2 to the 27th and 0.606 up to 2 to the 29th both matched exactly, his frequency table for 0.114 matched, and he reproduced 0.007 over his old range. He also reviewed the solver's code, then took 0.007 one step further himself, to 2 to the 31st, with no new rare values.
What came of itThe work was confirmed independently, by the person whose tables it extended.
The 0.007 results went into the OEIS, the reference database for integer sequences, as entry A395248, published July 17. Flammenkamp's reference page credits the work in its support paragraph, where "sgv-sequence" is his term for a game's list of values: "In July 2026, Tyler Satchel Orden from Los Angelos (sic), USA, supports this database of octal-games by extending the sgv-sequence of lots of octal-games."
By early September, Flammenkamp had pushed all twelve games further on his own machines, so the limits now shown on his page are his own. The July work is credited there as a dated contribution.
What came of itThe results were published in the OEIS and credited on the reference page.
Where this comes from
Both projects are told from their original working files: the parking project's research folder, and the games project's computation records, emails and saved copies of the reference page. Dates and numbers come from those files. Analysis and drafting assisted by Claude (Anthropic). Each page says how its results were checked (recomputation, published values or expert review) and labels anything not yet checked.