Pick a comparison group that differs only in ways that cannot cause the pattern

Say you change an email subject line and want to know whether more people opened it. Comparing it with last month's newsletter tells you little, because nearly everything else differs too: the list, the day, the content. The fair comparison splits the same list at random and sends both versions at the same time, which is what an email tool's A/B test does. A comparison where you expect no effect is called a placebo test or negative control. It only works if nothing else that differs between the two groups could produce the pattern by itself.

My AI lead agent (Claude Opus 4.8, which I picked for this run) missed this on a parking-ticket project, and a helper AI caught it. Using the tickets Los Angeles publishes, the lead asked whether street-sweeping tickets come early in the hour, since sweeping is posted to start on the hour. A sample of 20,000 sweeping tickets from 2023 put 34.8% of tickets in the first twenty minutes of the hour, against 33.3% for an even spread, 38.8% in the middle twenty minutes and 26.4% in the last twenty. So the busiest part was the middle of the hour, and the clearest feature was how few tickets came in the last twenty minutes.

For comparison it used tickets for rules with no start time: parking in a red zone, in a disabled spot without a placard, or at an expired meter. All three came out close to flat: red-zone and disabled-spot tickets within a point of a third in each twenty-minute part, meter tickets within two. Next to those flat groups, the sweeping tickets' lean toward the first forty minutes of the hour looked like support.

I had asked for a separate helper AI to check every major step, and the one sent to review this method was asked in its instructions whether tickets like the red-zone ones were valid comparisons. It said they were not. Those rules have no start time, so their tickets were going to spread evenly whatever was happening with sweeping. A flat comparison group that could not have come out any other way says nothing about why sweeping tickets are spread unevenly. The lead relabeled the result: it now counts only as a check that ticket times aren't rounded to the hour (if officers rounded, red-zone tickets would bunch at the hour too), and no longer as evidence about how officers work.

One group did share the on-the-hour start: no-stopping zones, whose no-stopping windows also begin on the hour. The lead left it out as a comparison for that reason, and its tickets split 38.4%, 34.2% and 27.4% across the three twenty-minute parts: more in the first twenty minutes than sweeping, and a similar shortfall in the last twenty. Reading it now, that group was the informative one. It suggests that this uneven spread is what any rule that starts on the hour produces, since no ticket can be written before the window opens, so the sweeping pattern by itself pointed to nothing special about sweeping enforcement.

The helper named comparisons that would speak to the question: the sweeping tickets compared with themselves at a fake start time, where no bunching should appear, and a check that the rise after the start has the same shape whether sweeping begins at 8:00, 10:00 or noon. Both need every ticket matched to its own street's schedule. The project never built that, so neither was run.

For teams: whoever presents a chart of results against a baseline fills in those two blanks in writing before the review, and the answers go in the review notes next to the chart. The reviewer's job is to look for anything else in the second blank.

For specialists: each sample was the first 20,000 tickets of that type that the city's data service returned for 2023, not a random draw, and minutes were pooled across all clock hours, so tickets in the middle of two-hour sweeping windows are mixed in. Sweeping split 34.8%, 38.8% and 26.4% across the three twenty-minute parts; red zone 33.4%, 33.6%, 33.0%; disabled placard 33.5%, 32.9%, 33.6%; meter 34.4%, 34.2%, 31.4%; no stopping 38.4%, 34.2%, 27.4%. If the 20,000 sweeping tickets were independent, 34.8% against 33.3% in the first twenty minutes would be about 4.5 standard errors, so that excess is detectable; tickets come in clusters (one officer, one route, one morning), so the real uncertainty is larger, and no cluster-aware test was run. In a separate sample of 40,000, 29% of sweeping tickets fell in the 8 o'clock hour, which mostly reflects when sweeping is scheduled. The helper's report called meter and red-zone tickets false placebos for the mechanism and recommended within-series controls (placebo cutoffs, comparisons across start hours).