Pre-Registration Is What Separates a Finding From a Story

# Pre-Registration Is What Separates a Finding From a Story ## Claim Declaring what you will look for before you look is what makes a result evidence, because a question chosen after seeing the data cannot fail.  ## Evidence **Source**: [[3. Resources/Readwise/Articles/Full Systematic Literature Review PRISMA & PICO in 5 Steps]] — PRISMA/PICO systematic review protocol The systematic-review machinery is usually explained as thoroughness — the PRISMA flow diagram, the PICO question structure, two blind reviewers with a third for ties, eleven to sixteen months of work. But thoroughness is not what the protocol is defending against. A very thorough search that decides its inclusion criteria while reading the results produces a confident, well-cited, unfalsifiable conclusion. Pre-registration is the load-bearing element: the review's question, search strings, inclusion and exclusion rules, and analysis plan are filed before the corpus is read. What that buys is the ability to be *wrong*. If the criteria are fixed in advance, the evidence can come back against the hypothesis and there is no legitimate move available to reshape the question around it. The two-blind-reviewers rule serves the same end from a different direction — it prevents a single reader from resolving ambiguous inclusion calls in the direction they already believe. Without the registration step, everything downstream still runs. The flow diagram still fills in. The output looks identical. It just no longer distinguishes a finding from a story that fits. ## Scope & conditions - **Holds** wherever the analyst has discretion over what counts as relevant *and* an interest in the outcome — which includes most self-assessment, internal audits, and post-hoc incident analysis. - **Requires** that the registration be costly to revise silently. A plan you can quietly edit before publishing is not a pre-registration. - **Breaks** for genuinely exploratory work, where the point is to discover which question to ask. The error there is reporting exploration as if it had been confirmation. - **Weakens** where the outcome space is small and public — a binary that everyone can check does not need much protecting. - **Sharpens** as search becomes cheap. When you can retrieve support for nearly any proposition in seconds, the constraint that matters is having committed to the proposition first. ## Links - [[3. Resources/Permanent Notes/The Maker Should Not Grade Its Own Homework]] — because extends: pre-registration is the same defense applied along the time axis rather than across people. Fix the standard before the work, or bring in someone who did not do it. - [[3. Resources/Permanent Notes/A Nondeterministic Check Breaks the Loops Stop Condition]] — because extends: a stop condition chosen after seeing the output is not a stop condition. Both notes are about fixing the criterion in advance. - [[3. Resources/Permanent Notes/Encoding What Good Looks Like Is What Makes Agent Verification Trustworthy]] — because extends: encoding the standard up front is pre-registration for agent work, and it is what lets a run legitimately fail. - [[3. Resources/Permanent Notes/Sampling a Fraction of a Source Produces False Authority Over the Whole]] — because contrasts: that is the corpus-coverage failure, this is the criteria-timing failure. A review can be complete and still unfalsifiable, or rigorous and incomplete. - [[3. Resources/Permanent Notes/Retrospective Reinterpretation Looks Like Prediction After the Fact]] — because extends: the religious cognate of the same mechanism. A prophecy re-read after the event and a hypothesis re-written after the data are the same move. - [[3. Resources/Permanent Notes/Concentration Beats Coverage Under Retrieval]] — because contrasts on where review effort should go once retrieval is cheap. - [[3. Resources/Permanent Notes/A Metric Is Vanity Only Against Your Own Outcome Data]] — because applies-in: choosing which metric to report after seeing the numbers is this failure in dashboard form. - [[Personal Knowledge Management MOC]] — because applies-in. - [[3. Resources/MOCs/Evidence & Claims Literacy MOC]] — the entry condition for this map — what makes a result able to fail in the first place ## Implications - For this vault's own deep-read and audit loops, the transferable practice is small: write the specific question and the disqualifying condition into the run before opening the corpus, so a run can legitimately conclude "nothing here." - It is the strongest argument for the synthesis pass's own rule that a run creating zero notes is a valid run. If the only acceptable outcome is notes, the resonance filter has already been abandoned. - Applied to agent code-analysis: state in advance what would count as a real finding and what would disqualify one, or every plausible-looking artifact becomes a finding. - The eleven-to-sixteen-month figure is worth holding as a calibration. When a systematic review of a literature takes a year, a same-day synthesis is a different kind of object and should be labeled as one. ## Open Questions - What is the lightest pre-registration that still works at the scale of a single work session — one sentence, or does it need the disqualifying condition spelled out? - Does pre-registration help when the analyst is a model rather than a person, or does a model simply comply with whichever criteria it is given at the moment it is asked? - Where is the boundary between legitimate protocol amendment and question-reshaping, given real reviews do amend? ## Source - [[3. Resources/Readwise/Articles/Full Systematic Literature Review PRISMA & PICO in 5 Steps]] — the PRISMA flow, PICO framing, blind dual review, and the pre-registration step