Re-running our own numbers

Fast experiments produce results faster than anyone can trust them. After about two dozen small checks we stopped and had a separate pass recompute seven of them, some on runs the first pass had never seen.

Most numbers came back identical, which is the boring and good outcome. The useful part was the rest: a signal whose figures were right but which was empty in practice, a sign that had been swapped, and one claim that disappeared after correcting for the player.

Nothing was dropped from the data because of this. We keep everything we count. What changed is what we are willing to show: the empty signal stays out of the product, the corrected ones carry their corrected numbers, and the log says which is which. See R08 in research.