Guide · Teaching a team
Write up a "no" as carefully as a "yes": one AI research run from ten ideas to "not yet"
A checklist for writing up an idea that didn't work out, and the AI research run it comes from: ten ideas about Los Angeles parking tickets, what happened to each, and the "not yet" verdict at the end.
A marketing test flops, a feature gets shelved, an AI-assisted analysis says an idea won't work. Usually the record is a line in a chat, and the next person with the same idea starts from zero. A "no" deserves the same write-up as a "yes". Writing down the branches you abandoned also lets readers see you didn't just report the one path that happened to work.
The checklist comes from an AI research run I directed in September. ChatGPT drafted the research brief from my description of the data, and Claude Opus 4.8, which I picked for this run, ran it as the lead agent, with eleven helper agents. The question was whether the 25.8 million parking tickets Los Angeles publishes contain enough for a new research paper. The brief said: "If 9 ideas are already published and only 1 survives, that is an excellent outcome." None of the ten survived to the paper stage, and the record says why for each:
| What happened | Ideas | Why |
|---|---|---|
| Dropped in the first round | Unequal ticketing across neighborhoods; a few places getting most tickets | Already well studied; expected result |
| Set aside as weaker | Fine changes as natural experiments; out-of-state cars; lag between registration expiring and a ticket; a four-way vehicle, violation, time and place pattern | Fines: no clean comparison, because of the 2020 pandemic pause. Out-of-state: earlier similar work and a fleet-vehicle confound. Registration lag: targeting could not be proven. Four-way pattern: the largest effects were true by definition |
| Carried forward, then not pursued | Vehicle-type "fingerprints" for violations; tickets as a chain where one ticket makes the next more likely | An outside critique advised dropping the first unless early data showed something large, and an early check found its new effects modest; the lead rated the second a follow-on |
| Carried to the end, verdict "not yet" | Street-sweeping ticket timing; the gap between an officer marking a car (the digital version of chalking a tire) and ticketing it | Earlier work existed for both, and both needed each ticket matched to the rules posted on its street, which the public data could not provide |
The two finalists went down for ordinary reasons. Helpers searching for earlier work found a 2012 patent and a commercial system reported to be in use in Los Angeles for the marking gap, and a public San Francisco project for street sweeping. A helper checking the method found that the jump in sweeping tickets at the start time is guaranteed, since no ticket can be written before the window opens. Matching tickets to street rules failed on the quick route: the sweeping schedule and the tickets use different route codes, and turning the schedule's 864 written route descriptions into a map was not attempted.
One finding was written down more strongly than its test allows. The marking gaps form two humps, near 70 and 135 minutes, about ten to fifteen minutes past the one-hour and two-hour marks, which fits posted one-hour and two-hour parking limits. The project's records list "officers return on a fixed schedule" as disproven. The test behind that only checked how many gaps fell within two minutes of exactly 15, 30 and 60 minutes, and only about one gap in ten is shorter than 70 minutes anyway, so that test looked at a small part of the data. A patrol loop of about 67 minutes, which would also put returns near 70 and 135, was never tested. A fixed schedule is not ruled out.
The verdict covered every item on the checklist: the answer to the brief's question (no, a competent researcher could not start a legitimate paper from this tomorrow), a score, the closest work by name, the confidence, three conditions that would all have to succeed to change the answer, with any one failure ending the idea, and the checks it skipped and why. I left it as the lead wrote it. What I asked for at the end was documentation complete enough for someone else to replicate the run.
For teams: make this half page the required close-out for any project that stops. The project owner writes it within a week of stopping, a second person reads it for item 5, and it goes in the shared folder where the next person with the same idea will search, under the idea's name rather than the date.
For specialists: the score used the brief's own scale, 0 observation, 1 interesting analysis, 2 incremental paper, 3 substantive contribution, 4 strong contribution, 5 unusually strong contribution, with manuscript work allowed only at 3 or above. The run scored 2. The brief also asked for percentages, and the lead agent gave about 30 to 40 percent that the core idea is new and about 15 to 20 percent that the evidence was ready for peer review. Those are the model's stated confidence, not calibrated probabilities. The three conditions were a fuller citation search, the street-by-street matching, and showing the sweeping rise holds on single street segments. The ticket total was a live count on September 12, 2026. The fixed-schedule check shares within two minutes of 15, 30 and 60 minutes were 0.43%, 0.63% and 0.42% on the full 2022 data (48,827 usable gaps); the record gives no baseline for those shares.