How an AI research run decided there was no paper yet

The lead listed which fields the data actually contains, then wrote ten candidate research questions, one sentence each, each tied to the fields it would need. A few of them: do street-sweeping tickets pile up right after the posted start time? Do certain kinds of vehicles get certain kinds of tickets? Is ticketing "self-exciting," so that one ticket makes more nearby more likely? Are tickets spread unequally across neighborhoods?

Before any helper ran, the lead wrote down which question it expected to fail: the neighborhood equity question, because it is already well studied. It went into the first round anyway, so the helpers could confirm or overturn the guess.

The lead sent eight helpers out at once, each with one job:

Do NOT overclaim novelty. ... Be skeptical and specific. Distinguish 'interesting' from 'publishably novel.'

Why it is written this way: a search helper tends to report "I found nothing" as "this is new." The brief makes it say what it did find.

The data-quality helper said two fields, a parking-meter ID and a patrol route, were not in the data. The data-feasibility helper, whose brief told it to check every column against the data itself, had reported both present. The lead queried the city's live database and found both columns: the route column filled in on 17.3% of tickets, about one in six, and the meter column on 15.3%, though its codes could not be used as meter numbers. It overruled the data-quality helper's claim, wrote the override down with its reason, and kept that helper's main point about rates, which was right. (Why the two helpers disagreed came down to how their briefs were written; the guide entry "Ask an AI to confirm each fact by name, and tell it where to check" tells that story.)

Result: two questions dropped and four set aside as weaker. Four went forward: which kinds of vehicles get which tickets, street-sweeping timing, the mark-to-ticket gap (from a field described in the next step), and, as a back-up, whether one ticket makes more nearby more likely.

While the helpers worked, the lead ran four quick queries of its own. Street-sweeping tickets do spike when the windows open: in a 2023 sample of 40,000, the count jumps from 242 in the 7 am hour to 11,644 in the 8 am hour. The vehicle question has real but modest signal.

And it found a field none of the helpers had mentioned: "marked time." When an officer marks a parked car (the digital version of chalking a tire) and comes back later to ticket it for staying past the limit, the ticket records when the car was first marked. About 2% of tickets from 2022 and 2023 carry it, and 729,430 tickets across all years. That opened a new question: how long is the gap between the mark and the ticket? In the lead's first sample of 4,000 tickets the typical (median) gap was 139 minutes. A later rerun on all 48,827 usable tickets from 2022 (after dropping gaps that were zero, negative or unreadable) gives 138.

At this point I pasted in a critique from the ChatGPT conversation. It said: focus on the mark-to-ticket gap alone, and sharpen it. The interesting quantity is the time past the posted limit, which would show how much slack officers give. And check whether the gaps bunch at regular intervals, which would mean officers circle back on a fixed schedule.

The lead agreed and re-ranked the four ideas still standing. The mark-to-ticket gap had the best chance of being new, and street sweeping was the safest; those two are the ones the rest of the run followed. The other two were not taken further. The lead's own query had found the vehicle signal real but modest, and the methods helper had rated the idea that one ticket makes more nearby more likely the riskiest, because patrol routes create the very clustering it would be looking for.

The lead sent one helper to try to kill the idea outright.

You are a HOSTILE novelty auditor. Your explicit goal is to KILL a research idea by finding prior work that already does it. ... Do not declare novelty just because an exact phrase is absent. ... Be adversarial. It is a SUCCESS to kill this idea if it deserves killing.

Why it is written this way: a helper asked whether an idea is new may say yes just because it found no exact match, so this brief tells it to find the closest existing work instead.

It came back with a 2012 patent (Dabell) that describes inferring patrol revisit times from ticket times, and a commercial system built on it, Conduent's CitySight, that is reported to be in use in Los Angeles. The helper rated the idea "wounded": weakened, but not ruled out.

The lead counted 729,430 tickets with a mark time across all years and measured the gaps on the 2022 tickets. They form two humps, near 70 minutes and near 135, each about 10 to 15 minutes past one and two hours. That fits one-hour and two-hour parking limits with a little slack. The lead also found no spikes at multiples of 15, 30 or 60 minutes: no sign of officers coming back every 15, 30 or 60 minutes.

It does not rule out every regular schedule. A patrol loop of about 65 to 70 minutes, with cars ticketed on the first or second pass, would also put humps near 70 and 135. The run's own claim register lists the parking-limit reading as an inference for this reason. The test that would separate the two is the same missing piece as before: each ticket's posted time limit.

Both that test and the original question, slack past the posted limit, need each ticket matched to the posted limit on its street, and that is not in the data. Verdict for this idea: possible, leaning against.

After 1:45 am the run paused for most of the day. Late that evening, about 22 hours after the last helper had reported, I pasted in the rest of the plan from the ChatGPT conversation. The full plan ran to 25 stages. It asked for a numbered research folder, a list of every helper agent, and a register of each claim with its status, and it added one new rule: before moving past any major stage, the lead had to send two to four separate helpers to audit it. Their job was to find what the lead had missed rather than redo the work: untested assumptions, research nobody had searched, other data, other explanations, and problems with the method, the statistics or reproducibility.

YOU MUST SPAWN A SMALL INDEPENDENT AUDIT TEAM BEFORE MOVING FORWARD. For EVERY major stage, spawn approximately 2-4 independent Sonnet-class subagents. Their job is NOT merely to perform the same task. Their explicit instruction is: "Audit this stage independently. Identify anything the principal investigator has missed, any assumptions that have not been tested, any additional data sources, methodological problems, literature we have not searched, alternative interpretations, statistical problems, reproducibility problems, or other work that should be completed before proceeding."

The run applied it straight away to the street-sweeping question, sending two helpers at the same time: another one told to kill the idea, and one told to check the method.

The first found that the burst of tickets after sweeping starts is already shown publicly, by a San Francisco civic project and by a data-journalism analysis of this same Los Angeles data, and that a Chicago study and a 2026 paper come close to the formal version. So the question could now lead, at most, to a better method for measuring a known effect.

The methods auditor agreed with the first methods helper (step 2) that the jump at the start time is guaranteed by the rule, so only the shape of the curve after it could be a finding. It then added two new points. First, even that shape might come only from averaging streets whose sweeping starts at different times. Second, a comparison the lead had run that evening measured something else.

In that comparison, red-zone and disabled-placard tickets spread evenly across the hour while sweeping tickets did not, which looked like support. But red-zone and placard tickets have no start time to cluster around, so they would spread evenly anyway, and that says nothing about sweeping. The lead relabeled the result as a rough check that ticket types with no start time show no bunching at the top of the hour, and nothing more. Finer rounding is a separate question: in its first data check the lead had found that 24.5% of ticket times fall on a multiple of 5 minutes, against 20% if times were spread evenly.

To study the curve properly, each ticket needs its own street's sweeping start time. The schedule and the tickets use two different route-code systems ("10P136 Th" versus "0350A"), so they cannot be matched by code. Matching by location would mean turning 864 written street descriptions into map shapes, which this run did not attempt.

The lead scored the best idea 2 out of 5, against the bar of 3 set at the start. It also gave its own rough estimates, which are not calibrated probabilities: about 30 to 40% that the core idea is new, and about 15 to 20% that the evidence is ready for peer review today. It then asked whether a competent researcher could start writing a legitimate paper from this tomorrow, and recorded the answer no.

It also wrote down the three things that would change the answer, any one of which could still rule the idea out. First, finish checking the roughly 379 papers that cite Hausman and Rapson (2018), a paper on comparing just before and just after a cutoff in time, and search a transportation research database directly. Second, build the street-level match between tickets and posted rules. Third, show the after-start pattern still appears when each street is looked at on its own.

The plan also called for an audit team at each of its 25 stages and a five-reviewer panel at the end. After eleven helpers, the lead judged that running the rest would only repeat checks already done, wrote that decision down with its reason, and told me. The record does not show me approving or objecting.