Better Answers With Jev for Reranking  🔗 **Source:** [Better Answers With Jev for Reranking](https://twitter.com/uglyrobot/status/2103195865280328031/?rw_tt_thread=True) > [!note]+ Document Notes > tags: [rag-evaluation, reranking, retrieval-cutoffs, agent-evaluation, cost-modelling] > author: Aaron Edwards (DocsBot) > source: Twitter/X long-form post > reference: https://github.com/onyx-dot-app/EnterpriseRAG-Bench > date created: 2026-09-25 > > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the author. > > Do not quote or attribute any other line to the named author. > <!-- deep-read:sections start --> ## 🏷️ Title **"Better answers with Jev for reranking"** — Aaron Edwards of DocsBot reports a two-stage experiment using Jev to score retrieval candidates and a relevance cutoff to decide how many chunks reach the answer model, in place of a fixed six. Roughly 860 words plus four charts. Answers improved; the bill rose 67.1 percent. **Pairs directly with the Vercel "Make Decisions with Jev" course page analysed in this same batch.** That document is the vendor's teaching material and contains no accuracy, latency or cost figures at all. This one is a customer's evaluation and contains little else. Together they are the marketing and the measurement of the same model, and the measurement is by far the more useful half. ## 🧭 Table of Contents 1. [References (in-page)](#-references-in-page) 2. [Summary](#-summary) 3. [Main Content](#-main-content) → [Notes](#️-notes) 4. [References](#-references) 5. [Paraphrase](#️-paraphrase) 6. [Newsletter](#-newsletter) 7. [Action Items](#-action-items) 8. [Hypotheses](#-hypotheses) 9. [Methodology](#-methodology) 10. [Questions](#-questions) 11. [Future Research](#-future-research) 12. [Implications](#-implications) 13. [Learning Styles Assessment](#-learning-styles-assessment) 14. [Accessibility Assessment](#-accessibility-assessment) 15. [DEIB Assessment](#-deib-assessment) 16. [Logical Validity](#-logical-validity) 17. [DARVO Analysis](#️-darvo-analysis) 18. [Conclusion](#-conclusion) 19. [Refactor Appendix](#-refactor-appendix) ## 🔗 References (in-page) The markup was enumerated before any claim about sourcing was made. **Outbound links — two, both to primary sources:** 1. [typesafe.ai](https://typesafe.ai/) — the vendor of the model under test, placed at first mention of Jev. 2. [github.com/onyx-dot-app/EnterpriseRAG-Bench](https://github.com/onyx-dot-app/EnterpriseRAG-Bench) — the public benchmark the experiment runs on, placed at the point the dataset is introduced. **This is the important one:** it makes the evaluation reproducible by a third party, which almost nothing in this genre is. **Images — four, all on Twitter's media CDN:** a lede card and three result charts, placed after the cutoff-sweep finding, after the cost finding, and at the head of the post. None carries alt text in the capture, and since three of them are the charts backing the numerical claims, their content is unavailable to anyone who cannot see them. **Sourcing assessment.** Two links, both outbound, both primary, both at the point of claim. The benchmark link in particular is the difference between a testimonial and an experiment. What is missing is not citation but *data*: no absolute counts appear anywhere, so a reader cannot recompute any of the percentages or their uncertainty. ## 📝 Summary **What happened.** DocsBot tested replacing a fixed six-chunk retrieval context with a dynamic one: score every candidate with Jev, keep those above a relevance cutoff, and let the context grow or shrink with the question. They ran two separate tests on the public synthetic EnterpriseRAG-Bench — a 500-question retrieval sweep holding the candidate pool constant, and a fresh 60-question end-to-end agent comparison scored on the 43 questions with sufficient gold evidence. At a 0.8 cutoff, gold-document discovery rose 7.4 percent while answer-context tokens fell 38.5 percent. In the agent run, strictly correct answers rose 23.5 percent, evidence was found on 21.7 percent more questions, the agent made 14.6 percent fewer searches, and uncached input to the answer model fell 50.1 percent. The answer model's share of the bill fell 35.9 percent. Total serving cost rose 67.1 percent. **Why it matters.** The headline result is the cost inversion, and the author explains its mechanism precisely: Jev scores the candidate pool *before* the cutoff prunes anything, so sending less to the answer model reduces that model's cost without touching the cost of ranking every candidate on every search. This is the general shape of every quality-improving filter placed upstream of an expensive step — the saving applies downstream of the filter, the charge applies upstream of it, and which dominates depends entirely on the ratio of pool size to surviving context. That is a transferable result independent of Jev, DocsBot, or RAG. **Stakeholders.** DocsBot and teams running retrieval-augmented agents; TypeSafe, whose model is being evaluated by a customer rather than by itself; anyone designing a two-stage retrieval pipeline; and, more broadly, anyone who has been asked to justify an AI quality improvement whose cost was measured only in the place the saving occurred. **Risks.** Three, all of which the author names himself. The answer-quality sample is small and *its paired uncertainty interval still includes no improvement* — which means the 23.5 percent headline is not statistically distinguishable from zero. The agent-run Jev variation also fetched candidates more deeply, so the answer gain cannot be attributed to Jev alone. And the cost figure excludes vector-store billing, so the true total is worse than 67.1 percent, not better. **Bottom line.** An unusually honest engineering write-up that reports a favourable quality result, declares it statistically inconclusive, identifies its own confound, and leads with the number that undercuts its own headline. The finding to keep is structural: measure the whole bill, not the part of it your change was designed to reduce. ## 📖 Main Content **The question being asked is a good one.** *"Why give every question exactly six chunks by default?"* A version lookup may need one strong match; a question spanning several policies may need many. Fixed-k retrieval is a convenience constant that nobody chose on evidence, and interrogating an arbitrary default is exactly the right instinct. The proposed replacement is a relevance cutoff: score candidates, keep what clears the bar, and if nothing clears it keep the single best — a sensible degenerate case that prevents the empty-context failure. **The experimental design is the strongest part of the document, and it is genuinely well constructed.** Two tests, deliberately separated because they answer different questions. The first is a 500-question direct retrieval sweep comparing each cutoff against the original first six chunks *drawn from the same candidate pool*. The author states the reason for that constraint explicitly: so that deeper fetching cannot take credit for a retrieval result. That is a real control, correctly identified and correctly applied, and it is the difference between measuring reranking and measuring "we also looked at more documents." The second is a fresh 60-question agent comparison that follows each question through search decisions, tool calls, the final answer, model usage and the Jev bill rather than stopping at the retrieval list. Correctness is judged on the 43 questions with sufficient designated gold evidence, with those gold chunks as the judge's authoritative source. And here the author volunteers the confound rather than hiding it: in the agent test, the Jev variation *also* fetched candidates more deeply — the very thing test one was built to exclude. So test two is not a clean attribution, and the document says so twice. **The cutoff sweep found a genuine interior optimum.** Permissive cutoffs still passed a lot of text through; strict ones saved context but began giving back evidence gains. At 0.8, gold-document discovery rose 7.4 percent while answer-context tokens fell 38.5 percent — better evidence on less context, which is the result one hopes for and does not usually get. The 0.7 cutoff found slightly more evidence at smaller savings; 0.9 cut far more context but narrowed the discovery gain enough to be unattractive. The author's framing of this is careful and worth noting: 0.8 is *"a good place to start a pilot. It is not a magic probability threshold."* Interior optima on one benchmark are not constants of nature, and he declines to sell it as one. **The cost result is the document's real contribution.** Line by line: uncached input to the answer model fell 50.1 percent; the answer model's portion of the bill fell 35.9 percent; total serving cost including Jev rose 67.1 percent. The two model-side numbers differ because output tokens and cached input were not reduced, only uncached input — so a halving of one input class produces roughly a third off the whole model bill. That internal consistency is a good sign for the rest of the figures. The mechanism is stated exactly: *"it scores the candidate pool before the cutoff removes anything, so sending fewer chunks to the answer model reduces its cost without erasing the cost of ranking those candidates on every search."* The filter's saving is downstream; the filter's charge is upstream and applies to everything. Note that this held even though the agent made 14.6 percent *fewer* searches — the per-search ranking cost still dominated. A figure the author does not state can be derived from the ones he does, and it is worth having. If the answer model dominated the original serving bill, then a 35.9 percent reduction in that component against a 67.1 percent increase in the total implies Jev's charge alone is roughly equal to the *entire original answer-model bill* — about 103 percent of it, or some 62 percent of the new total. If the original bill carried other material fixed costs, the implied Jev charge is larger still (around 110 to 137 percent for other-cost shares of 10 to 33 percent). This is an inference resting on an assumption the document does not confirm, and it is flagged as one — but any reading in that range makes reranking the single largest line item in the new architecture, not a marginal addition to it. **The qualitative review is two examples and it is honest in both directions.** On a synthetic question about a founder's spending target, the fixed-six agent picked the wrong figure and the Jev variation surfaced the designated evidence and answered from it. On another, the Jev answer *added a recovery step the evidence did not support*. The conclusion drawn is precisely the right one and is stated in six words: *"Better evidence helps. The answer model still has to use it carefully."* Improving retrieval does not constrain generation, and a pipeline that improves the evidence without constraining what is done with it has fixed one of two problems. **The proposed pilot is operationally sound.** Jev at a 0.8 cutoff *with a fallback to the original ranking if Jev fails* — an explicit degradation path, which is the detail most pilot designs omit. The stated watch list is correct and complete: multi-file questions, answer correctness, search turns, latency, and the complete serving bill. Latency is notable, because it is the one cost dimension the experiment did not measure at all. **The limitations paragraph is where this document earns its standing.** Small gold-scorable sample. *Paired uncertainty interval still includes no improvement.* The deeper-fetching confound, restated. Vector-store billing excluded from the cost calculation. Any one of those disclosures would be creditable; all four, in a post whose headline is "Better answers," is rare. **The closing line is the thesis and it generalises well beyond this experiment:** *"A retrieval chart can tell you the right document moved up. Only the full agent trace tells you what the bot said next, and what you paid for it."* ### 🗒️ Notes - **The confidence interval finding is the most important sentence in the document and it is in the penultimate paragraph.** An interval that includes no improvement means the 23.5 percent answer gain is not statistically distinguishable from zero at the stated sample size. Everything above it — the title, the section heading "The agent got better answers" — is written as though the improvement were established. Disclosed, but disclosed late. - **No absolute numbers appear anywhere.** Every figure is a relative percentage change, and the author flags this in bold. But that means 23.5 percent on 43 questions cannot be converted back into a count of questions by the reader. If the baseline were 17 correct, a 23.5 percent relative gain is four questions. Four questions out of 43 is exactly the magnitude one would expect the uncertainty interval to swallow — which is presumably why it does. - **The 500-question sweep and the 60-question agent run are not the same experiment.** The clean control is in the first; the headline number is from the second, which has the confound. That split is stated clearly and is easy to lose when the numbers are quoted onward. - **"No customer documents were sent to Jev"** — a synthetic public benchmark, stated plainly. Good practice, and it also means these results are about benchmark noise characteristics, not about DocsBot's real corpus. - **Hybrid search at alpha 0.5** is one configuration of one retrieval stack. The optimal cutoff almost certainly moves with alpha, with chunk size, and with the embedding model. - **Jev sees title, URL and content alongside the full question** — the stated reason being to distinguish a passage that *answers* the question from one that merely *shares its subject*. That distinction is the entire value proposition of reranking over similarity, and it is the right way to describe it. - **Fewer searches, higher cost.** The agent made 14.6 percent fewer document searches and the bill still rose 67.1 percent. Per-search ranking cost dominated the reduction in search count. - **Latency is never measured**, only listed as something to watch in the pilot. For an interactive documentation bot, a per-search ranking pass over the full candidate pool is a latency question at least as much as a cost one. - **The failure case is an over-addition, not an omission.** The Jev variation added an unsupported recovery step. Better evidence appears to have made the answer model more confident, not more careful — which is a hypothesis worth testing rather than an anecdote worth repeating. - **Capture quality is clean.** Full text, no truncation, no widget furniture. Only the four chart images are lost as information, and they carry the quantitative backing for three claims. ## 📚 References **Primary sources, both linked from the document:** - [EnterpriseRAG-Bench](https://github.com/onyx-dot-app/EnterpriseRAG-Bench) — the public synthetic benchmark. Described as containing *"scattered discussions, similar documents, and conflicting details"*, the noise characteristics that make enterprise knowledge hard to search. Publicly available, so the experiment is in principle reproducible — the single most valuable property of this write-up. - [TypeSafe](https://typesafe.ai/) — vendor of the model under test. **Companion document in this batch:** - "Make Decisions with Jev" (Vercel mini-course, analysed 2026-09-25) — the vendor-side framing of the same model. That analysis noted the absence of any accuracy, latency or independent evaluation figures; this document supplies the first and third for one task, and still not the second. **Not present and worth seeking:** - Absolute counts behind every percentage, and the width of the paired uncertainty interval. Without them the significance question cannot be assessed by a reader. - Latency measurements for the ranking pass. - Vector-store billing, explicitly excluded from the cost calculation. - A comparison against the established alternatives — a cross-encoder reranker, a commodity reranking API, or simply raising or lowering fixed k — none of which is tested. **Sourcing assessment.** Both links are outbound, primary, and at the point of claim, and one of them makes the work reproducible. This is well-sourced by the standards of any genre. The gap is in reported data, not in citation. ## 🗣️ Paraphrase *Restated in the analyst's words. None of the following is the author's phrasing.* **The premise.** The team questioned why their retrieval system always hands the answering model the same number of passages regardless of what was asked. Some questions are satisfied by a single precise match; others need evidence assembled from several places. So they tried scoring each retrieved candidate for relevance and keeping only those above a threshold, letting the amount of supplied context vary with the question. **The design.** Two separate experiments on a public, artificially constructed benchmark chosen for its difficulty — overlapping documents, scattered discussion, contradictory details. No real customer material was involved. The first experiment examined retrieval alone across five hundred questions, deliberately drawing both the new and old approaches from an identical candidate pool so that any improvement could not be explained by simply searching harder. The second followed sixty questions all the way through the agent's behaviour — its searches, its tool use, its final answer, and its costs — with correctness assessed on the forty-three questions that had adequate reference evidence. **The threshold finding.** Loose thresholds passed too much through; tight ones discarded useful material. One intermediate setting improved the rate at which the correct source document was found while substantially reducing the volume of text sent onward. The author is careful to present this as a reasonable starting point for a trial rather than as a discovered constant. **The cost finding.** In the full agent test, answers were more often strictly correct, reference evidence was located more often, the agent searched less, and the volume of fresh text sent to the answering model roughly halved. The answering model's share of the bill fell by about a third. The total bill nevertheless rose by two thirds. The reason is structural: the scoring step runs across every candidate before any pruning occurs, so the economy achieved downstream does not offset the charge incurred upstream on every single search. **The honest part.** The author states that the sample used to judge answer quality was small, that the range of statistical uncertainty around the improvement still includes the possibility of no improvement at all, that the improved configuration also searched more deeply so the gain cannot be credited to reranking alone, and that storage costs were left out of the cost comparison entirely. **The closing argument.** A chart of retrieval positions can show that the right document rose in the rankings. Only tracing the complete agent run reveals what the system actually said afterwards and what the whole thing cost. ## 📰 Newsletter **The reranking result that starts with the number that ruins it** Here is a thing that almost never happens. Someone runs an experiment, gets a favourable headline result, and then tells you in the second paragraph that the bill went up 67.1 percent, tells you near the end that the improvement's confidence interval still includes zero, tells you that the winning configuration also cheated slightly by searching harder, and tells you that the cost figure excludes storage so the real number is worse. Aaron Edwards at DocsBot tried replacing fixed six-chunk retrieval with dynamic retrieval: score every candidate with [Jev](https://typesafe.ai/), keep whatever clears a relevance cutoff, and let the context breathe with the question. Sensible premise — six is a number somebody typed once. The retrieval result is real and nicely controlled. At a 0.8 cutoff, gold-document discovery up 7.4 percent while answer-context tokens fell 38.5 percent, measured against the *same candidate pool* so that deeper fetching couldn't take the credit. Better evidence on less context. That is the good outcome. Then the agent test: strictly correct answers up 23.5 percent, evidence found on 21.7 percent more questions, 14.6 percent fewer searches, uncached input halved. Answer-model bill down 35.9 percent. Total serving cost **up 67.1 percent**. Why? Because the reranker scores the whole candidate pool *before* the cutoff throws anything away. You save on what you send downstream; you pay for everything upstream, on every search. Do the arithmetic he doesn't print and, if the answer model dominated the original bill, Jev's charge comes out roughly equal to the entire previous answer-model spend — the biggest line item in the new architecture, not a top-up on it. Every WordPress person who has ever been asked to justify a performance change should recognise this shape immediately. It is object caching with an expensive cache-warm. It is a query optimiser that runs a planner on every request. It is image optimisation that transcodes on upload for every size nobody requests. The saving lands where you were looking; the cost lands where you weren't. You do not know whether a change helped until you measure the whole bill, and "the whole bill" includes the thing your change added. The other line worth stealing, from the qualitative review: on one question the Jev-assisted agent *added a recovery step the evidence did not support*. Better evidence, worse discipline. Improving what the model sees does not constrain what the model does with it — and if you are building anything where a model's output drives a decision, that is the gap that will get you, not retrieval quality. And the closer, which deserves to be on a wall somewhere: > A retrieval chart can tell you the right document moved up. Only the full agent trace tells you what the bot said next, and what you paid for it. Pair this with the Vercel course page on the same model, which contains no accuracy figure, no latency figure and no cost figure whatsoever. One of these two documents is trying to sell you something and one of them is trying to find out. Only one of them is useful. ## ✅ Action Items **Core Performance** - The transferable finding is a measurement discipline, not a RAG result: any optimisation that adds a step must be measured on the *total* cost, including the added step, not on the component it was designed to reduce. That is directly applicable to caching, image pipelines, and query optimisation work. - Note the second-order version: fewer operations does not imply lower cost. The agent made 14.6 percent fewer searches and the bill still rose two thirds, because per-operation cost grew faster than operation count fell. **Core AI** - Adopt the two-test structure for any retrieval or ranking proposal: a controlled component test with the pool held constant, plus an end-to-end run that follows the behaviour through to output and cost. A component test alone cannot show what the system does next. - Record the failure mode explicitly: better evidence produced an unsupported additional step in one reviewed answer. Retrieval improvements do not constrain generation. **Documentation / Support** - If any documentation-search or support-bot work is contemplated, the pilot design here is directly copyable: a starting cutoff, an explicit fallback to the previous behaviour on failure, and a watch list covering correctness, search turns, latency and the complete bill. - Before adopting a reranker, test the boring baselines: raising or lowering fixed k, and a commodity cross-encoder. None is tested here. **DevRel** - This write-up is a good teaching artifact for how to report a result honestly: lead with the number that undercuts your headline, name your own confound, and state when your interval includes zero. Worth citing as a model. **OSPO** - Second data point in this batch on the same theme: a capability that depends on per-call vendor charges has an economic profile that only appears under end-to-end measurement. Relevant to any policy on AI tooling in project infrastructure. **Training** - The relative-versus-absolute distinction is a teachable statistics point with a live example: a bolded 23.5 percent that resolves to roughly four questions out of 43 and an interval including zero. **Meta / Hosting** - No action unless a wordpress.org search or support-routing project is live. If one is, the cost-measurement discipline above is the item to carry over. **Remaining subteams** (Core, Core Program Team, Mobile, Playground, Community, Marketing, Design, Photos, TV, Polyglots, Accessibility, CLI, Tide, Openverse, Themes, Plugins) and **FAIR / The WPOCC** - No action. A vendor-model retrieval experiment generates no work for these teams. ## 🧪 Hypotheses 1. **For any filter placed upstream of an expensive step, net cost is determined by the ratio of pool size to surviving context, not by the quality of the filter.** *Testable:* hold the reranker fixed and vary candidate pool depth; plot total serving cost against pool size. The crossover point where reranking becomes net-positive should be a function of that ratio alone. 2. **The 23.5 percent answer gain does not survive a larger sample.** The author states the paired interval includes no improvement. *Testable:* re-run the agent comparison on 400+ gold-scorable questions and report the interval. This is the single most valuable follow-up available. 3. **The deeper-fetch confound accounts for most of the agent-test answer gain.** Test one showed 7.4 percent discovery improvement with the pool held constant; test two showed 21.7 percent with deeper fetching allowed. *Testable:* re-run the agent comparison with the candidate pool held constant, as in test one. If the gain collapses toward 7.4 percent, the confound was doing the work. 4. **Better evidence increases answer-model overreach rather than reducing it.** One reviewed answer added an unsupported recovery step. *Testable:* code all 43 agent answers for unsupported additions and compare rates between arms. If the Jev arm overreaches more, improved retrieval is trading omission errors for commission errors. 5. **The optimal cutoff moves with retrieval configuration and is not transferable.** *Testable:* repeat the sweep at alpha 0.3 and 0.7, and with a different chunk size; if the optimum moves materially, 0.8 is a property of this stack, exactly as the author cautions. 6. **Reranking latency, unmeasured here, is the binding constraint for interactive use before cost is.** *Testable:* measure p95 end-to-end latency with and without the ranking pass at production pool sizes. ## 📊 Methodology **The source's method.** A two-arm comparison against a fixed-six baseline on a public synthetic benchmark, run at two levels of abstraction. Level one: a 500-question retrieval sweep across cutoff values, with both arms drawn from an identical candidate pool so that fetch depth is controlled — a real and correctly-motivated control. Gold-document discovery measured only on questions carrying a designated source document, which avoids scoring against absent ground truth. Level two: a fresh 60-question end-to-end agent run, scored for strict correctness on the 43 questions with sufficient gold evidence, judged against those gold chunks as authoritative, and instrumented through search decisions, tool calls, model usage and vendor charges. All results reported as relative change from baseline, stated in bold. Limitations declared: small answer sample, paired uncertainty interval including no improvement, an uncontrolled deeper-fetch confound in the agent arm, and vector-store costs excluded. The design's strength is the deliberate separation of the controlled component test from the uncontrolled end-to-end test, with the author telling the reader which is which. Its weakness is that the headline figures come from the uncontrolled one. **This analysis's method.** The captured body was read in full and its markup enumerated before any sourcing claim was made — two outbound links, both primary, both at the point of claim, plus four un-alt-texted chart images. The stated cost relationships were recomputed rather than accepted: a 35.9 percent reduction in the answer-model component against a 67.1 percent increase in the total implies, if the answer model dominated the original bill, a Jev charge of roughly 103 percent of that original component (about 62 percent of the new total), rising to roughly 110–137 percent under other-cost shares of 10–33 percent. That derivation rests on an assumption the document does not state and is flagged as an inference wherever it appears. The relationship between the 50.1 percent uncached-input reduction and the 35.9 percent bill reduction was checked for consistency and is consistent, given that output tokens and cached input were unaffected. The statistical claim — that an interval including no improvement makes the headline gain not distinguishable from zero — is a restatement of the author's own disclosure, not an independent analysis, since no interval width is published. **Limits.** The four charts were not visible, so three quantitative claims rest on the prose figures alone. The benchmark repository was not consulted. No absolute counts are available anywhere in the document, so no significance calculation could be performed and none is claimed here. ## ❓ Questions - What are the absolute numbers? How many of the 43 questions were strictly correct in each arm, and how wide is the paired interval? Without these the headline cannot be assessed by anyone. - What does the agent comparison show with the candidate pool held constant, as in test one? That single re-run would separate reranking from deeper fetching. - What is the ranking pass's latency at production pool size, and does it break the interactive experience before it breaks the budget? - What is the candidate pool depth? It determines the per-search ranking charge and therefore the entire cost result, and it is never stated. - How does this compare to a commodity cross-encoder reranker, or to simply tuning fixed k? No baseline other than fixed-six is tested. - Does the 0.8 optimum hold at other alpha values and chunk sizes, or is it a property of this configuration? - Is the unsupported-recovery-step failure a pattern or an anecdote? Two examples were reviewed; 43 were available. - What does the cost picture look like including vector-store billing, which is excluded? ## 🔮 Future Research - **Re-run the agent comparison at scale with the pool held constant.** This resolves hypotheses 2 and 3 simultaneously and is the highest-value follow-up in the document. - **Map the cost crossover.** Plot total serving cost against candidate pool depth to find where reranking becomes net-positive. That curve generalises to every upstream filter and is more useful than any single cost figure. - **Code the failure modes across all 43 answers.** Omission errors versus commission errors, by arm. The single reviewed overreach may be the more important finding than the correctness gain. - **Add latency to the measurement set.** It is on the pilot watch list and absent from the experiment. - **Test the boring baselines.** Fixed k at 3, 6, 10; a commodity cross-encoder. Without them, the comparison is against one arbitrary constant. - **Generalise the upstream-filter cost result.** State it independently of RAG and test it against caching, query planning, and preprocessing pipelines, where the same structure recurs. ## 💡 Implications **Practice.** Two directly usable lessons. First, measure the total bill, including the component your change adds — a saving reported only where you were looking is not a saving. Second, design retrieval pilots with an explicit fallback to prior behaviour on component failure, as this one does; that detail is what makes a pilot reversible. **Policy.** Any organisational standard for adopting AI components should require end-to-end cost measurement rather than component-level measurement, and should require that reported quality improvements state their uncertainty. This document meets both bars voluntarily; most do not, and the difference between a testimonial and an evaluation is exactly those two disclosures. **Security.** Largely not applicable to this source, with one genuine note: the author states plainly that *no customer documents were sent to Jev* and that the benchmark is public and synthetic. Sending corpus content to a third-party ranking service is a data-handling decision, and testing on synthetic data first is the correct sequence. Anyone adapting this pattern to a real corpus is making a materially different privacy decision than the one described here. **Education.** An excellent teaching text on two points that are hard to convey abstractly: why relative percentages without base rates mislead, and why a confidence interval that includes zero means the result is not yet a result. Both appear here with a real example attached, and the author is on the right side of both. **Funding.** The cost structure has a governance implication. A quality improvement that increases total serving cost by two thirds is an ongoing operational commitment, not a one-off engineering change, and it scales with query volume rather than with team size. For any volunteer or grant-funded project, a per-query vendor charge on *every search* is a fundamentally different budget object than a fixed subscription, and the distinction is easy to miss until the invoice arrives. ## 🧠 Learning Styles Assessment - **Visual.** Strong in intent, absent in capture. Four charts back the cutoff-sweep and cost findings; none has alt text and none is readable here. For a document whose argument is quantitative, that is a material loss — the charts presumably show the shape of the cutoff curve, which the prose can only summarise. - **Aural.** Not applicable to this source. No audio or video component. - **Verbal.** Very strong. Precise, unadorned technical prose. Every number is stated in context with its baseline, bolded for scanning, and accompanied by the mechanism that produced it. - **Kinesthetic.** Moderate, indirectly. The benchmark is public and linked, so a reader can in principle reproduce the experiment — the only hands-on affordance, and a real one. - **Logical.** The dominant mode and the document's strength. The whole piece is a chain of controlled comparisons with an explicit causal mechanism for the surprising result, plus a stated limitations set. A reader who reasons numerically can follow, check, and extend it. - **Social.** Weak. No discussion element; one author reporting one team's experiment. - **Solitary.** Strong. Self-contained and fully comprehensible alone in about eight minutes. - **Naturalistic.** Not applicable to this source. No environmental or natural-systems content. **Reading level.** Roughly US grade 13–14. Sentences are short and clear, but the vocabulary is unglossed and specialist throughout: reranking, chunks, hybrid search alpha, uncached input tokens, gold-document discovery, paired uncertainty interval, cutoff, candidate pool. A practitioner reads this easily; anyone outside retrieval engineering will follow the cost narrative and not the methodology. ## 📊 Accessibility Assessment Mapped to WCAG 2.2 AA, assessed against the captured artifact. 1. **Text alternatives (1.1.1 Non-text Content).** Fails, and materially. Four images, no alt text on any. Three are result charts carrying the quantitative backing for the cutoff sweep and the cost finding. A reader using assistive technology loses the evidence, not merely decoration. Mitigating factor: the prose states every key figure explicitly, so the *findings* survive even though the charts do not — which is itself good practice, probably unintentional. 2. **Time-based media (1.2.x).** Not applicable to this source. No audio or video. 3. **Adaptable / info and relationships (1.3.1).** Passes. Clear section headings, logical ordering, bolded key figures that carry emphasis semantically rather than by colour alone. 4. **Distinguishable (1.4.1 Use of Colour, 1.4.5 Images of Text).** Not determinable from the capture, and a live concern. Result charts commonly encode series by colour alone and commonly render their numbers as text within an image — both would fail. The charts cannot be inspected here to confirm either way. 5. **Navigable / link purpose (2.4.4).** Passes. Both links have meaningful anchor text — "Jev" at the vendor and "EnterpriseRAG-Bench" at the benchmark — each intelligible out of context. 6. **Readable (3.1.3 Unusual Words).** Fails for any non-specialist. A dozen or more domain terms are used without definition or expansion. Reasonable for the intended audience; a hard barrier outside it. 7. **Robust / parsing (4.1.x).** Passes as captured. Clean linear text, no widget furniture, no interface controls leaking into the reading order. **Overall.** The prose is well structured and the key numbers are all stated in text, which substantially limits the damage from the missing alt text. The single highest-value remediation is describing what each chart shows — particularly the shape of the cutoff curve, which is the one piece of information the prose genuinely cannot substitute for. ## 🌍 DEIB Assessment **Cultural breadth.** Narrow by genre and unmarked. English only, US-centric engineering idiom, and an implicit assumption of an English-language corpus throughout. The benchmark is described as containing scattered discussions and conflicting details, but not multiple languages. Relevance scoring across languages is a real and unexamined question, and for anyone considering this pattern for a multilingual documentation corpus it is the first thing that would need testing. **Inclusive language.** Clean. No gendered pronouns, no idiom, no metaphor drawn from sport, war or culture-specific reference. First-person plural throughout — "we wanted to test," "we ran," "where we go from here" — which frames the work as a team's shared inquiry rather than an individual's demonstration. The language of failure is notably blame-free: the unfavourable cost result is reported as a property of the architecture, not as anyone's mistake. **Neurodiversity support.** Good. Short paragraphs with one finding each, descriptive section headings that permit non-linear reading, key figures bolded for scanning, and an explicit bolded warning that all results are relative rather than absolute. Working-memory demand is moderate but well managed — each number is restated with its baseline rather than requiring the reader to carry it forward. The missing chart alt text is the one real cost, for readers who depend on the visual representation to hold a numerical relationship. **Representation.** One named author reporting a team's work; no individuals depicted or otherwise identified. Not applicable in the usual sense. Worth a neutral note that the work being examined — answering support and documentation questions — is work ordinarily performed by support staff, and the document frames it purely as a retrieval and cost problem. That is the appropriate frame for an engineering write-up, and it is also a frame in which the people whose work is being automated do not appear. **Psychological safety.** Genuinely exemplary, and the strongest DEIB dimension here. The author publishes a result that does not support his own headline, states that the uncertainty interval includes no improvement, names his own methodological confound unprompted, and reports a case where the improved system produced a worse answer. Publishing an inconclusive result in a field that rewards confident claims is a costly signal, and it models a norm — that being wrong in public is survivable and useful — that most engineering cultures assert and few practise. The one caution is structural rather than authorial: the honest disclosures sit in the last paragraph while the favourable numbers sit in the title and headings, so a fast reader gets the claim without the caveat. ## ✅ Logical Validity **Strong reasoning.** This is a well-constructed argument and most of its strengths are deliberate design choices rather than happy accidents. The candidate-pool control in test one is correctly motivated and correctly described — the author states *why* it exists, which is the mark of someone who understands what he is controlling for. The two tests are kept separate because they answer different questions, and the reader is told which is which. The surprising result comes with a stated causal mechanism rather than a shrug. The relative-not-absolute warning is bolded and placed before the results rather than after them. The interior optimum is presented as a pilot starting point and explicitly not as a constant. And the limitations paragraph volunteers four distinct weaknesses, including the one that most undermines the headline. **Fallacies.** *Relative risk presentation without base rates — present, and partially mitigated.* **Located** throughout the results: "strictly correct answers rose **23.5%**", "designated evidence was found on **21.7%** more questions", "gold-document discovery rose **7.4%**". **Why it misleads:** no absolute counts appear anywhere in the document, so a bolded 23.5 percent on a 43-question sample cannot be converted back into questions by the reader. If the baseline were 17 correct, the gain is four questions — a magnitude entirely consistent with the author's own disclosure that the interval includes no improvement. **Mitigation, which is real:** the author bolds an explicit warning that every figure is a relative change and not an absolute accuracy score, and separately states that the interval includes zero. He is not concealing the problem; he is reporting in a format that obscures it while telling you it does. **The fix:** publish the counts. "20 of 43 correct versus 17 of 43" is both more honest and more persuasive than a percentage, because it lets the reader do the arithmetic the percentage prevents. *Burying the qualifier — a structural fallacy of emphasis.* **Located** in the arrangement: the title is "Better answers with Jev for reranking" and a section heading reads "The agent got better answers." The disclosure that the paired uncertainty interval *still includes no improvement* appears in the penultimate paragraph. **Why it matters:** a result whose interval includes zero is not established, and everything above that sentence is written in the indicative as though it were. **Why this is a mild case rather than a serious one:** the author does disclose it, unprompted, and also leads paragraph two with the cost increase that undercuts his own headline — he is not building to a sale. **The fix:** state the inconclusiveness where the claim is made. "Strictly correct answers rose 23.5 percent, though at this sample size the improvement is not statistically distinguishable from zero" costs one clause and makes the headline honest. *Confounded attribution — identified and disclosed by the author himself.* **Located** in the agent test, where the Jev variation also fetched candidates more deeply — the precise variable test one was built to hold constant. **Why it matters:** the 21.7 percent evidence-discovery figure from the confounded test sits alongside the 7.4 percent figure from the controlled one, and the larger number is the memorable one. **Credit where due:** the author states the confound twice, including in his conclusion that *"we cannot credit Jev alone for the answer gain."* That is not a fallacy he committed; it is one he pre-empted. **The fix:** re-run the agent test with pool depth held constant. *Single-alternative framing — present by omission.* **Located** in the design: the only comparator is fixed-six retrieval. A commodity cross-encoder reranker, a different fixed k, or a cheaper scoring model would each be an obvious baseline, and none appears. **Why it matters:** the conclusion available from this experiment is "dynamic cutoffs with Jev beat one arbitrary constant," which is weaker than the impression left. **The fix:** one additional arm. **Tone of voice.** Plain, numerate, and conspicuously unsalesy. First-person plural, past tense, short declaratives carrying dense figures. No adjectival inflation anywhere — the word "promising" appears once and is immediately qualified. The two rhetorical flourishes are both structural rather than decorative and both do argumentative work: *"That last number is why we ran the full agent test"* justifies the methodology, and the closing *"A retrieval chart can tell you the right document moved up. Only the full agent trace tells you what the bot said next, and what you paid for it"* states the thesis. The register is a lab notebook written well, which is the correct register for what this is. ## 🛡️ DARVO Analysis **Deny.** No denial pattern, and the opposite is conspicuous. The document's central unfavourable finding — a 67.1 percent cost increase — is stated in the third paragraph, before any of the favourable results are developed. Four separate limitations are volunteered in the closing section, including the one that most undermines the headline. There is nothing being denied because nothing is being defended. **Attack.** No attack. No competitor, alternative approach, vendor or critic is named or disparaged. TypeSafe, whose model produced an unfavourable cost result, is referred to neutrally throughout and linked without comment. The benchmark's authors are credited by link. Even the failure case — where the improved system produced a worse answer — is reported without blame attached to the model, the vendor, or the approach. **Reverse victim and offender.** No reversal. There is no grievance, no injured party, no accusation, and no blame assigned in any direction. The unfavourable result is attributed to the architecture's structure rather than to anyone's decision. **Assessment.** No DARVO pattern present, and the positive finding is worth recording rather than only the absence. DARVO is defensive rhetoric under accusation; this document is the structural opposite — a voluntary disclosure of results that weaken its own headline, published by the party with the strongest incentive to suppress them. In a genre where vendor-adjacent evaluations routinely bury their inconvenient numbers, publishing "total serving cost was 67.1% higher" in the opening and "its paired uncertainty interval still includes no improvement" near the close is the behaviour the template exists to distinguish from its opposite. Section retained per the template. ## 🎯 Conclusion **Bottom line.** DocsBot replaced fixed six-chunk retrieval with a relevance-cutoff filter scored by Jev and measured the result twice — once on retrieval alone with the candidate pool held constant, once end-to-end through the agent's behaviour and bill. The controlled retrieval result is credible and modest: at a 0.8 cutoff, gold-document discovery up 7.4 percent while answer-context tokens fell 38.5 percent. The end-to-end result is larger, more attractive, and less trustworthy: answers up 23.5 percent, but on 43 questions, with a confound the author names, and with an uncertainty interval that by his own account still includes no improvement. And the bill rose 67.1 percent. The cost mechanism is the durable finding and it generalises well past this experiment. A filter placed upstream of an expensive step saves downstream of itself and charges upstream of itself, across the whole pool, on every invocation. Whether it is net-positive depends on the ratio of pool size to surviving context, not on how good the filter is. Here the charge implied by the published percentages is roughly the size of the entire original answer-model bill — an inference, but one that holds across any reasonable assumption about the original cost mix. What makes the document worth keeping is not the result but the reporting. It leads with the number that undercuts its own title, states its confound unprompted, declares its own interval inconclusive, and notes that the cost figure excludes storage and is therefore optimistic. Set beside the vendor's own course page for the same model — which contains no accuracy, latency or cost figure of any kind — it demonstrates the entire distance between marketing and measurement. **Next steps.** Treat the 7.4 percent controlled retrieval figure as the real result and the 23.5 percent as unestablished. If adopting the pattern anywhere, measure the total bill including the added component, hold pool depth constant when attributing gains, and measure latency, which nobody has measured yet. And publish absolute counts — the percentages here would be more convincing, not less, with the numbers behind them. ## 🔄 Refactor Appendix ### 📚 Literature Atoms #### [H1] Upstream filters charge on the pool, save on the survivors > [!quote] > The gap comes from when Jev runs: it scores the candidate pool *before* the cutoff removes anything, so sending fewer chunks to the answer model reduces its cost without erasing the cost of ranking those candidates on every search. **Context & use** - The mechanism behind a 50.1% reduction in uncached input, a 35.9% reduction in the answer-model bill, and a 67.1% increase in total serving cost. - Held even though the agent made 14.6% *fewer* searches — per-search ranking cost dominated the reduction in search count. - Generalises far beyond RAG: the same structure appears in cache warming, query planning, and preprocessing pipelines. Net effect depends on the ratio of pool size to surviving context, not on filter quality. - Derived figure, flagged as an inference: if the answer model dominated the original bill, the implied Jev charge is roughly 103% of that entire original component, or about 62% of the new total. **Tags:** #cost-modelling #reranking #systems-architecture <!-- atomicity: 1 idea ✓ --> #### [H2] The controlled test isolates reranking from deeper fetching > [!quote] > First, we ran a 500-question direct retrieval sweep, comparing each cutoff with the original first six chunks from the **same candidate pool** so that deeper fetching could not take credit for the retrieval result. **Context & use** - The single best methodological decision in the document, and the author states its purpose rather than leaving it implicit. - Produced the trustworthy number: 7.4% discovery gain at a 38.5% context reduction, cutoff 0.8. - The contrast matters: the agent test did *not* hold pool depth constant, and its larger 21.7% figure is therefore confounded — which the author says twice. - Use as the template for any component evaluation: control the thing that would otherwise take the credit. **Tags:** #experimental-design #rag-evaluation #confound-control <!-- atomicity: 1 idea ✓ --> #### [H3] Better evidence does not constrain what the model does with it > [!quote] > On another, the Jev answer added a recovery step the evidence did not support. Better evidence helps. The answer model still has to use it carefully. **Context & use** - Improving retrieval addresses omission errors; it does nothing about commission errors, and may increase them by raising the model's confidence. - One of only two reviewed examples, so this is an observation, not a rate — worth testing across all 43 answers. - Directly relevant to any design where a model's output drives a decision: the gap is between what the model sees and what it is permitted to do. **Tags:** #generation-errors #rag-limits #agent-behaviour <!-- atomicity: 1 idea ✓ --> #### [H4] The author declares his own result inconclusive > [!quote] > The result is promising, but this was a small gold-scorable answer sample. Its paired uncertainty interval still includes no improvement. **Context & use** - An interval including zero means the 23.5% headline is not statistically distinguishable from no effect at this sample size. - Published voluntarily, by the party with the strongest incentive to omit it, in a post titled "Better answers." - The placement is the only criticism available: this sits in the penultimate paragraph while the claim sits in the title. - Cite as a model of honest reporting, and as the reason to treat the controlled 7.4% figure as the real finding. **Tags:** #statistical-honesty #reporting-practice #uncertainty <!-- atomicity: 1 idea ✓ --> #### [H5] Component charts cannot show what the system did next > [!quote] > A retrieval chart can tell you the right document moved up. Only the full agent trace tells you what the bot said next, and what you paid for it. **Context & use** - The document's thesis, and the justification for having run the expensive second test at all. - Component-level metrics are necessary and insufficient: they cannot show downstream behaviour or total cost. - Transferable to any pipeline evaluation — a benchmark score on one stage is not a system result. **Tags:** #end-to-end-evaluation #measurement #systems-thinking <!-- atomicity: 1 idea ✓ --> ### 🧠 Permanent Candidates #### [P1] a-saving-measured-where-you-were-looking-is-not-a-saving **Claim.** An optimisation that adds a component must be measured on total system cost, including the added component, because the saving lands where the change was aimed and the charge lands where it was not. Component-level measurement will reliably report success for changes that are net-negative, and the sign of the error is always favourable to the change. **Evidence pointer.** [H1] — uncached input to the answer model fell 50.1 percent and that model's share of the bill fell 35.9 percent, while total serving cost rose 67.1 percent, because the ranking pass charges on the full candidate pool before any pruning occurs. Fewer searches (−14.6 percent) did not rescue it. **Scope & conditions.** Applies to any filter, cache, planner, or preprocessing step inserted upstream of an expensive operation. The net effect is governed by the ratio of pool size to surviving work, not by the quality of the inserted component — so a *better* filter does not fix a bad ratio. Does not apply where the added component's cost scales with the surviving set rather than the candidate set, which is the configuration worth designing toward. **Links.** - Connects to [P1] output-shape-determines-whether-a-system-can-be-checked from the Jev course-page analysis in this same batch. Both concern the gap between a property being true and a property being *measurable*: there, output shape decides whether an eval is affordable; here, measurement boundary decides whether a cost result is real. Together they make one argument — that what a system can demonstrate about itself is fixed at design time, by choices that look like implementation details. - Connects to the enforcement-over-instruction thread in the knowledge-system audit. "Measure the whole bill" is an instruction and will be forgotten; a cost check that fails when total spend rises is enforcement and will not. The same asymmetry the audit keeps finding. - Practical corollary for the deep-read pipeline itself: its own success criteria are all component-level (headings present, callout present, highlights created). Nothing measures whether the notes are read, used, or acted on — which is the total-bill question for a knowledge system. #### [P2] disclosed-inconclusiveness-is-the-strongest-signal-of-a-trustworthy-evaluation **Claim.** When a party with an incentive to report success instead publishes its own confound, its own inconclusive interval, and the number that undercuts its headline, those disclosures are better evidence of the work's reliability than any of its favourable results — because they are costly to make and cheap to omit. **Evidence pointer.** [H4] — the paired uncertainty interval including no improvement, stated voluntarily in a post titled "Better answers." Alongside it: the cost increase placed in the third paragraph, the deeper-fetch confound stated twice, and the exclusion of vector-store billing acknowledged. **Scope & conditions.** Applies to evaluations published by parties with a stake in the outcome — vendors, customers, and anyone reporting their own work. Does not apply to genuinely independent adversarial evaluation, where disclosure is the baseline expectation rather than a costly signal. The inverse also holds and is the more useful test in practice: an evaluation with *no* stated limitations should be read as an evaluation whose limitations were not looked for. **Links.** - Direct counterpoint to the vendor's own "Make Decisions with Jev" course page in this same batch, which contains no accuracy, latency or cost figure at all. Same model, two documents, and the useful one is the one that reports a worse outcome. - Connects to [H5] in the GoDaddy takeover analysis, also in this batch — the closing provenance note naming every source and admitting the FT original was read only through relays. Same underlying virtue: stating what you did not do is what makes what you did do credible. - The placement caveat belongs in the note: disclosure at the end of a document whose title asserts the opposite is honest but not yet fully effective. Trustworthiness is improved by disclosure; it is maximised by disclosure *at the point of claim*. ### ✏️ Fleeting Queue - Publish the counts. "20 of 43 versus 17 of 43" would be more persuasive than "23.5%", not less — and the fact that a percentage is more flattering than the count behind it is itself diagnostic. - The one re-run that resolves almost everything: the agent comparison with candidate pool depth held constant, as in test one. If the gain collapses toward 7.4 percent, the confound was doing the work. - Candidate pool depth is never stated anywhere, and it determines the entire cost result. Odd omission in an otherwise meticulous document. - Latency appears on the pilot watch list and nowhere in the experiment. For an interactive docs bot it may bind before cost does. - The over-addition failure — better evidence producing an unsupported recovery step — is potentially the more important finding than the correctness gain, and it rests on one example out of 43 available. - Two Jev documents arrived in this batch a day apart, one vendor and one customer. Worth remembering as a pairing when either is cited; neither is complete alone. - Applying [P1] to the deep-read pipeline itself is uncomfortable and probably correct: every success criterion it has is component-level. ### → Things 3 - [ ] Write the permanent note for "a saving measured where you were looking is not a saving" independently of this source — it is a systems-measurement principle, not a RAG finding, and it applies to the deep-read pipeline's own success criteria. - [ ] Pair this with the Vercel Jev course page when either is cited: vendor claims and customer measurement for the same model, and neither document is complete without the other. <!-- deep-read:sections end --> 📄 **Full text:** [[Full Document Contents/Tweets/Better Answers With Jev for Reranking.md|Better Answers With Jev for Reranking by Aaron Edwards]] ## Highlights - At DocsBot, we wanted to test a simple idea: why give every question exactly six chunks by default? ([View Highlight](https://read.readwise.io/read/01m3bckcj2w24v7h5n9j9zzjz4)) - Note: Good premise — fixed k is a convenience constant nobody chose on evidence. Fallacy (single-alternative framing, by omission): the only comparator tested is fixed-six. A commodity cross-encoder, a cheaper scoring model, or simply a different fixed k are each obvious baselines and none appears. The available conclusion is "dynamic cutoffs with Jev beat one arbitrary constant," which is weaker than the impression left. Fix: one additional arm. - Tags: atom/fleeting/f1, fallacy/single-alternative - so that deeper fetching could not take credit for the retrieval result. That isolates what reranking and truncation did to evidence discovery and context size. ([View Highlight](https://read.readwise.io/read/01m3bck0bzxhyga0rxgs2kdbfs)) - Note: H2 — The best methodological decision in the document, and the author states its purpose rather than leaving it implicit. Produced the trustworthy number: +7.4% discovery at -38.5% context, cutoff 0.8. The contrast matters — the agent test did NOT hold pool depth constant, so its larger +21.7% is confounded. - Tags: atom/literature/h2 - In this agent test, the Jev variation also fetched candidates more deeply, a distinction that matters when interpreting the answer gain. ([View Highlight](https://read.readwise.io/read/01m3bck9hg567dnfgz51ndyef9)) - Note: Fallacy (confounded attribution) — identified and disclosed by the author himself, twice. The agent test varies the precise thing test one was built to hold constant, so the +21.7% and +23.5% figures cannot be credited to reranking alone. Not a fallacy he committed; one he pre-empted. Fix: re-run the agent comparison with pool depth held constant — the single highest-value follow-up available. - Tags: fallacy/confounded-attribution - so sending fewer chunks to the answer model reduces its cost without erasing the cost of ranking those candidates on every search. ([View Highlight](https://read.readwise.io/read/01m3bcjxvd2zad4113f8epqwbn)) - Note: H1 / Z1 — The mechanism behind -50.1% uncached input, -35.9% answer-model bill, and +67.1% total cost. An upstream filter saves downstream of itself and charges upstream of itself across the whole pool, on every invocation. Held even though searches fell 14.6%. Derived (inference): if the answer model dominated the original bill, the implied Jev charge is ~103% of that entire original component. Generalises to cache warming, query planning, preprocessing. - Tags: atom/zettel/z1, atom/literature/h1 - On another, the Jev answer added a recovery step the evidence did not support. Better evidence helps. The answer model still has to use it carefully. ([View Highlight](https://read.readwise.io/read/01m3bck2zayj7kd9c8x484x0ey)) - Note: H3 — Improving retrieval addresses omission errors and does nothing about commission errors; it may increase them by raising the model's confidence. One of two reviewed examples out of 43 available, so an observation rather than a rate. The gap between what a model sees and what it is permitted to do is the one that matters wherever output drives a decision. - Tags: atom/literature/h3 - The result is promising, but this was a small gold-scorable answer sample. Its paired uncertainty interval still includes no improvement. ([View Highlight](https://read.readwise.io/read/01m3bck6rjaksm76d414m8edg2)) - Note: H4 — An interval including zero means the 23.5% headline is not statistically distinguishable from no effect. Published voluntarily by the party with the strongest incentive to omit it, in a post titled "Better answers." Two fallacies anchor here. (1) Relative risk without base rates: no absolute counts appear anywhere, so 23.5% on 43 questions cannot be converted back to a count — if baseline were 17 correct, the gain is four questions. Fix: publish "20 of 43 vs 17 of 43." (2) Buried qualifier: this sits in the penultimate paragraph while the claim sits in the title. Fix: state the inconclusiveness at the point of claim. - Tags: atom/zettel/z1, fallacy/buried-qualifier, fallacy/relative-risk-presentation - A retrieval chart can tell you the right document moved up. Only the full agent trace tells you what the bot said next, and what you paid for it. ([View Highlight](https://read.readwise.io/read/01m3bckf7qftm2f5d3zvd41hdg)) - Note: H5 — The thesis, and the justification for running the expensive second test. Component-level metrics are necessary and insufficient: they cannot show downstream behaviour or total cost. Transferable to any pipeline evaluation — including this deep-read pipeline, whose own success criteria are all component-level (headings present, callout present, highlights created) and none of which measures whether the notes are ever read or used. - Tags: action/h1