Better Answers With Jev for Reranking  🔗 **Source:** [Better Answers With Jev for Reranking](https://twitter.com/uglyrobot/status/2103195865280328031/?rw_tt_thread=True) > [!note]+ Document Notes > --- > tags: [rag-evaluation, reranking, retrieval-cutoffs, agent-evaluation, cost-modelling] > author: Aaron Edwards (DocsBot) > source: Twitter/X long-form post > reference: https://github.com/onyx-dot-app/EnterpriseRAG-Bench > date created: 2026-09-25 > --- > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the author. > > Do not quote or attribute any other line to the named author. > ## 🏷️ Title > **"Better answers with Jev for reranking"** — Aaron Edwards of DocsBot reports a two-stage experiment using Jev to score retrieval candidates and a relevance cutoff to decide how many chunks reach the answer model, in place of a fixed six. Roughly 860 words plus four charts. Answers improved; the bill rose 67.1 percent. > **Pairs directly with the Vercel "Make Decisions with Jev" course page analysed in this same batch.** That document is the vendor's teaching material and contains no accuracy, latency or cost figures at all. This one is a customer's evaluation and contains little else. Together they are the marketing and the measurement of the same model, and the measurement is by far the more useful half. > ## 🧭 Table of Contents > 1. [References (in-page)](#-references-in-page) > 2. [Summary](#-summary) > 3. [Main Content](#-main-content) → [Notes](#️-notes) > 4. [References](#-references) > 5. [Paraphrase](#️-paraphrase) > 6. [Newsletter](#-newsletter) > 7. [Action Items](#-action-items) > 8. [Hypotheses](#-hypotheses) > 9. [Methodology](#-methodology) > 10. [Questions](#-questions) > 11. [Future Research](#-future-research) > 12. [Implications](#-implications) > 13. [Learning Styles Assessment](#-learning-styles-assessment) > 14. [Accessibility Assessment](#-accessibility-assessment) > 15. [DEIB Assessment](#-deib-assessment) > 16. [Logical Validity](#-logical-validity) > 17. [DARVO Analysis](#️-darvo-analysis) > 18. [Conclusion](#-conclusion) > 19. [Refactor Appendix](#-refactor-appendix) > ## 🔗 References (in-page) > The markup was enumerated before any claim about sourcing was made. > **Outbound links — two, both to primary sources:** > 1. [typesafe.ai](https://typesafe.ai/) — the vendor of the model under test, placed at first mention of Jev. > 2. [github.com/onyx-dot-app/EnterpriseRAG-Bench](https://github.com/onyx-dot-app/EnterpriseRAG-Bench) — the public benchmark the experiment runs on, placed at the point the dataset is introduced. **This is the important one:** it makes the evaluation reproducible by a third party, which almost nothing in this genre is. > **Images — four, all on Twitter's media CDN:** a lede card and three result charts, placed after the cutoff-sweep finding, after the cost finding, and at the head of the post. None carries alt text in the capture, and since three of them are the charts backing the numerical claims, their content is unavailable to anyone who cannot see them. > **Sourcing assessment.** Two links, both outbound, both primary, both at the point of claim. The benchmark link in particular is the difference between a testimonial and an experiment. What is missing is not citation but *data*: no absolute counts appear anywhere, so a reader cannot recompute any of the percentages or their uncertainty. > ## 📝 Summary > **What happened.** DocsBot tested replacing a fixed six-chunk retrieval context with a dynamic one: score every candidate with Jev, keep those above a relevance cutoff, and let the context grow or shrink with the question. They ran two separate tests on the public synthetic EnterpriseRAG-Bench — a 500-question retrieval sweep holding the candidate pool constant, and a fresh 60-question end-to-end agent comparison scored on the 43 questions with sufficient gold evidence. At a 0.8 cutoff, gold-document discovery rose 7.4 percent while answer-context tokens fell 38.5 percent. In the agent run, strictly correct answers rose 23.5 percent, evidence was found on 21.7 percent more questions, the agent made 14.6 percent fewer searches, and uncached input to the answer model fell 50.1 percent. The answer model's share of the bill fell 35.9 percent. Total serving cost rose 67.1 percent. > **Why it matters.** The headline result is the cost inversion, and the author explains its mechanism precisely: Jev scores the candidate pool *before* the cutoff prunes anything, so sending less to the answer model reduces that model's cost without touching the cost of ranking every candidate on every search. This is the general shape of every quality-improving filter placed upstream of an expensive step — the saving applies downstream of the filter, the charge applies upstream of it, and which dominates depends entirely on the ratio of pool size to surviving context. That is a transferable result independent of Jev, DocsBot, or RAG. > **Stakeholders.** DocsBot and teams running retrieval-augmented agents; TypeSafe, whose model is being evaluated by a customer rather than by itself; anyone designing a two-stage retrieval pipeline; and, more broadly, anyone who has been asked to justify an AI quality improvement whose cost was measured only in the place the saving occurred. > **Risks.** Three, all of which the author names himself. The answer-quality sample is small and *its paired uncertainty interval still includes no improvement* — which means the 23.5 percent headline is not statistically distinguishable from zero. The agent-run Jev variation also fetched candidates more deeply, so the answer gain cannot be attributed to Jev alone. And the cost figure excludes vector-store billing, so the true total is worse than 67.1 percent, not better. > **Bottom line.** An unusually honest engineering write-up that reports a favourable quality result, declares it statistically inconclusive, identifies its own confound, and leads with the number that undercuts its own headline. The finding to keep is structural: measure the whole bill, not the part of it your change was designed to reduce. > ## 📖 Main Content > **The question being asked is a good one.** *"Why give every question exactly six chunks by default?"* A version lookup may need one strong match; a question spanning several policies may need many. Fixed-k retrieval is a convenience constant that nobody chose on evidence, and interrogating an arbitrary default is exactly the right instinct. The proposed replacement is a relevance cutoff: score candidates, keep what clears the bar, and if nothing clears it keep the single best — a sensible degenerate case that prevents the empty-context failure. > **The experimental design is the strongest part of the document, and it is genuinely well constructed.** Two tests, deliberately separated because they answer different questions. > The first is a 500-question direct retrieval sweep comparing each cutoff against the original first six chunks *drawn from the same candidate pool*. The author states the reason for that constraint explicitly: so that deeper fetching cannot take credit for a retrieval result. That is a real control, correctly identified and correctly applied, and it is the difference between measuring reranking and measuring "we also looked at more documents." > The second is a fresh 60-question agent comparison that follows each question through search decisions, tool calls, the final answer, model usage and the Jev bill rather than stopping at the retrieval list. Correctness is judged on the 43 questions with sufficient designated gold evidence, with those gold chunks as the judge's authoritative source. And here the author volunteers the confound rather than hiding it: in the agent test, the Jev variation *also* fetched candidates more deeply — the very thing test one was built to exclude. So test two is not a clean attribution, and the document says so twice. > **The cutoff sweep found a genuine interior optimum.** Permissive cutoffs still passed a lot of text through; strict ones saved context but began giving back evidence gains. At 0.8, gold-document discovery rose 7.4 percent while answer-context tokens fell 38.5 percent — better evidence on less context, which is the result one hopes for and does not usually get. The 0.7 cutoff found slightly more evidence  *What happened when we let Jev decide how much context our DocsBot RAG agents should read.* The answer model used **50.1% fewer uncached input tokens**. Strictly correct answers rose **23.5%** in our agent test. Then we added the reranker charges: total serving cost was **67.1% higher**. That last number is why we ran the full agent test: a retrieval chart can show leaner context and better evidence, but it cannot show what the agent does next or how much every search costs. At DocsBot, we wanted to test a simple idea: why give every question exactly six chunks by default? A version lookup might need one strong match. A question that pulls together several policies might need more. So we tried scoring search results with [Jev](https://typesafe.ai/), then letting a relevance cutoff decide how many chunks survived. #### Two tests, two different questions We used the public, synthetic [EnterpriseRAG-Bench](https://github.com/onyx-dot-app/EnterpriseRAG-Bench). It has the kind of noise that makes company knowledge hard to search: scattered discussions, similar documents, and conflicting details. No customer documents were sent to Jev. Hybrid search ran at alpha 0.5, and Jev saw the user's full question alongside each candidate's title, URL, and content so it could distinguish a passage that answered the question from one that merely shared its subject. We kept chunks above the selected cutoff; if none cleared it, we kept the highest-scored one. The final context could grow or shrink with the question. First, we ran a 500-question direct retrieval sweep, comparing each cutoff with the original first six chunks from the **same candidate pool** so that deeper fetching could not take credit for the retrieval result. That isolates what reranking and truncation did to evidence discovery and context size. Gold-document discovery was measured only on questions with a designated source document. Then we ran a fresh, 60-question agent comparison that followed each question through search decisions, tool calls, the final answer, model usage, and the Jev bill instead of stopping at the retrieval list. Correctness was judged on the 43 core questions with sufficient designated gold evidence; those gold chunks were the judge's authoritative source. In this agent test, the Jev variation also fetched candidates more deeply, a distinction that matters when interpreting the answer gain. **Every result below is a relative percentage change from fixed-six retrieval, not an absolute accuracy score.** #### The cutoff had a sweet spot At a permissive cutoff, Jev still sent a lot of text to the answer model. At a strict cutoff, it saved more context but began to give back evidence gains. The 0.8 relevance score cutoff landed between those outcomes: **gold-document discovery rose 7.4% while answer-context tokens fell 38.5%**.  The 0.7 cutoff found slightly more designated evidence, with smaller context savings; the 0.9 cutoff cut far more context, but its discovery improvement narrowed enough to make the extra pruning less attractive. In the separate controlled answer comparison, 0.8 also gave the strongest answer result among the cutoffs we tried. That makes 0.8 a good place to **start a pilot**. It is not a magic probability threshold. For questions that need several files, the more permissive 0.7 cutoff may be worth the extra context. #### The agent got better answers. The bill still grew. With Jev in the full agent run, strictly correct answers rose **23.5%** and designated evidence was found on **21.7% more questions**. The agent made **14.6% fewer document searches** in that run. Uncached input to the answer model fell **50.1%**. The answer-model portion of the bill fell **35.9%**. But still total serving cost, including Jev, rose **67.1%**.  The gap comes from when Jev runs: it scores the candidate pool *before* the cutoff removes anything, so sending fewer chunks to the answer model reduces its cost without erasing the cost of ranking those candidates on every search. The answer review showed both sides of the tradeoff. On one synthetic question about a founder's spending target, the fixed-six agent picked the wrong figure; the Jev variation surfaced the designated evidence and answered from it, the sort of correction we hoped a better evidence filter could make. On another, the Jev answer added a recovery step the evidence did not support. Better evidence helps. The answer model still has to use it carefully. #### Where we go from here at DocsBot The best pilot would be including Jev in our retrieval process with a 0.8 relevance cutoff with a fallback to the original ranking if Jev fails. We would watch questions that need evidence from multiple files, along with answer correctness, search turns, latency, and the complete serving bill. A 0.7 cutoff deserves its own look on multi-file questions. The result is promising, but this was a small gold-scorable answer sample. Its paired uncertainty interval still includes no improvement. The full agent variation also fetched candidates more deeply, so we cannot credit Jev alone for the answer gain. Our cost calculation excludes vector-store billing too. A retrieval chart can tell you the right document moved up. Only the full agent trace tells you what the bot said next, and what you paid for it., Better Answers With Jev for Reranking, Aaron Edwards, 01m3b0d8m0zk4yj977khjqj1xs.