# What a Recommendation Omits Identifies Its Real Selection Criterion ## Claim A recommendation list is diagnosed faster by what it leaves out than by what it includes, because the absence of the consensus option reveals the criterion actually in use. <!-- atomicity: 1 idea ✓ --> ## Evidence **Source**: [[3. Resources/Readwise/Articles/This Is NOT How You Evaluate Bible Translations]] — Dan McClellan McClellan disqualifies a "best Bible translations" video before examining a single verse. The tell is that it never mentions the NRSV or NRSVue — the translation he describes as widely considered the most faithful and critical English rendering of both testaments. > [!quote] > And the reason they omit it is because it doesn't prioritize dogma over data. The inference has a specific shape. If someone ranking by *accuracy* omits the option the field regards as most accurate, then accuracy is not the ranking criterion. Something else is, and the omission names it — here, agreement with the ranker's prior commitments. What makes this a general method rather than a domain argument is its cost. Checking every claim on a list requires the expertise the list purports to supply. Checking whether the obvious candidate is present requires only knowing what the obvious candidate is, which is far cheaper and often available to a non-expert. The test runs before the analysis and can end it. McClellan's later demonstration — that the creator's own objection to the NIV applies to three translations he recommended — confirms the diagnosis empirically. But the omission had already predicted it. ## Scope & conditions - **Holds** where a recognized consensus or default option exists, the audience would expect it, and the selection criteria are unstated. - **Requires** that the omitted option genuinely be well-known to the recommender. A list that omits something obscure is uninformed, not motivated, and the two failure modes call for different responses. - **Breaks** where the omission has a stated, sufficient reason — out of scope, wrong audience, unavailable in the relevant market, disclosed conflict. - **Weakens** where the field has no consensus pick. With genuine disagreement among experts, no single absence is diagnostic. - **Does not establish** that the included items are wrong. It establishes what the list was selected *for*, which is a claim about the selector rather than about the items. ## Links - [[3. Resources/Permanent Notes/Refuting a Claim on Its Own Premises Is Stronger Than Refuting Its Premises]] — because extends: both are moves that need no shared framework with the person being assessed. The omission test needs no expertise; the internal refutation needs no agreed premises. Same source, same author, two cheap methods. - [[3. Resources/Permanent Notes/Pre-Registration Is What Separates a Finding From a Story]] — because extends: stating criteria in advance is exactly what makes omissions non-diagnostic. A list with published criteria can defend an absence; one without cannot. - [[3. Resources/Permanent Notes/A Metric Is Vanity Only Against Your Own Outcome Data]] — because contrasts: that note assesses a metric's validity against outcomes. This assesses a selection's motive against expectations. One asks whether the number predicts, the other asks why this number was chosen. - [[3. Resources/Permanent Notes/Sampling a Fraction of a Source Produces False Authority Over the Whole]] — because extends: an omitted option and an unread portion of a corpus produce the same overconfidence, and neither is visible in the fluent output. - [[3. Resources/Permanent Notes/The Recommend Layer Is Where Vendor Markets Make Their Real Decisions]] — because applies-in: if recommendation is where vendor decisions are actually made, then auditing recommendation lists by their omissions is a procurement control. - [[3. Resources/Permanent Notes/A Trusted Intermediary Makes a False Claim Durable]] — because extends: an intermediary's selection is itself a claim, and its omissions are where the intermediary's interest shows. - [[Personal Knowledge Management MOC]] — because applies-in. ## Implications - Run the test first, because it is cheap and can terminate the evaluation. Before reading a comparison, list the two or three options you would expect any competent version to name, then check for them. - It transfers directly to software: benchmark suites that omit the obvious baseline, vendor comparisons that omit the incumbent, model evaluations that omit the strongest competitor. In each case the omission identifies the purpose. - The constructive corollary for anything you publish is to state selection criteria explicitly. Criteria convert an omission from evidence about your motives into a defensible scoping decision. - Distinguish motivated omission from ignorant omission before drawing conclusions. Both produce a bad list; only one tells you something about interest. ## Open Questions - Can the test be automated — a checker that compares a recommendation set against a known consensus set for a domain? - How reliably do people identify the "expected" option in fields where they are not expert? The test's cheapness assumes they can, which may be its weak point. - Does explicit criteria-stating actually reduce motivated omission, or does it just move the motivation into the criteria? ## Source - [[3. Resources/Readwise/Articles/This Is NOT How You Evaluate Bible Translations]] — the NRSV omission as the diagnostic, applied before any verse-level analysis