You're Doing RAG Wrong  🔗 **Source:** [You're Doing RAG Wrong](https://twitter.com/akshay_pachaar/status/2052743644411765230/?rw_tt_thread=True) > [!note]+ Document Notes > --- > tags: [rag, retrieval-augmented-generation, vector-search, embeddings, knowledge-engineering, llm-infrastructure] > author: Akshay (@akshay_pachaar) > source: X (Twitter) > reference: https://twitter.com/akshay_pachaar/status/2052743644411765230 > date created: 2026-06-21 15:00 > --- > # 🏷️ Title: You're doing RAG wrong > ## 🧭 Table of Contents > - [📝 Summary](#-summary) > - [📖 Main Content](#-main-content) > - [🗒️ Notes](#️-notes) > - [📚 References](#-references) > - [🗣️ Paraphrase](#️-paraphrase) > - [📰 Newsletter](#-newsletter) > - [✅ Action Items](#-action-items) > - [🧪 Hypotheses](#-hypotheses) > - [📊 Methodology](#-methodology) > - [❓ Questions](#-questions) > - [🔮 Future Research](#-future-research) > - [💡 Implications](#-implications) > - [🧠 Learning Styles Assessment](#-learning-styles-assessment) > - [📊 Accessibility Assessment](#-accessibility-assessment) > - [🌍 DEIB Assessment](#-deib-assessment) > - [✅ Logical Validity](#-logical-validity) > - [🛡️ DARVO Analysis](#-darvo-analysis) > - [🎯 Conclusion](#-conclusion) > ### 🔗 References > - [Blockify (Iternal Technologies)](https://github.com/) — the preprocessing layer the thread promotes (open-source; thread links a "try it yourself" URL). > - [LangChain](https://www.langchain.com), [LlamaIndex](https://www.llamaindex.ai), [Haystack](https://haystack.deepset.ai) — orchestration frameworks the thread positions as downstream of the real problem. > ## 📝 Summary > - **What happened.** A technical Twitter thread by Akshay (@akshay_pachaar) argues that nearly every RAG pipeline rests on an unexamined assumption — that a *chunk of text* is the right unit of knowledge to embed — and that this upstream choice causes most retrieval failures people try to patch downstream with rerankers, hybrid search, and threshold tuning. > - **Why it matters.** If the unit of embedding is wrong, no amount of retrieval-algorithm tuning fixes it. The thread reframes RAG quality as a *data-modeling* problem, not a search-tuning problem, and proposes embedding a "question-answer packet" (one validated claim plus typed governance fields) instead of a prose window. > - **Key stakeholders.** Enterprise RAG teams (especially those with versioned, access-controlled corpora across SharePoint/Confluence/Git), vector-DB and embedding vendors, and Iternal Technologies, whose Blockify product implements the proposed "IdeaBlock" unit. > - **Risks and tradeoffs.** The headline metrics (40x corpus reduction, 3x fewer tokens, 2.3x relevance, 13.55% accuracy lift) all come from one vendor's internal 17-document benchmark — unsourced, unreproduced, and marketing-shaped. The pipeline adds a preprocessing/distillation layer (LLM cost, human SME validation, dedup tuning) before the vector store. > - **Bottom line.** Fix the knowledge unit, not the retrieval algorithm: embed atomic question-answer claims with governance metadata, deduplicate to canonical blocks, and the retrieval surface sharpens. The claim is plausible and well-argued; the numbers need independent verification. > ## 📖 Main Content > - **The core thesis.** "A chunk of text is the wrong unit of knowledge to embed." Chunkers split on token count, so a chunk is a *structurally neutral container* with no idea boundary, no version identity, and no access metadata. The model retrieving it can't know what's missing. > - **Three failure modes traced to the chunk.** (1) *Idea fragmentation* — you retrieve half a table or a conclusion with no argument. (2) *Version collision* — enterprise corpora hold the same doc in a dozen near-identical versions; top-K returns five copies, current and deprecated mixed, and the LLM blends them into confident wrongness. (3) *No governance anchor* — access control gets bolted onto the orchestrator instead of living in the data. > - **The proposed fix: the IdeaBlock.** Instead of a prose window, embed a *claim*: one question, its validated 2–3 sentence answer, and typed governance fields (clearance level, version state, source) on one object. Because user queries are already questions, matching a question to a stored answer becomes *structural*, not just semantic. > - **Vector-geometry argument.** In Blockify's internal benchmark (17 docs, 298 pages), mean cosine distance from query to best block was 0.1585 for IdeaBlocks vs 0.3624 for naive chunks — a 2.29x reduction. > - **Counterintuitive distillation result.** Less data, more accuracy: 2,042 raw IdeaBlocks deduplicated (80–85% similarity, 3–5 rounds) to 1,200 canonical blocks; word count dropped 88,877 → 44,537; the distilled set beat the undistilled by 13.55% on vector accuracy. Rationale: fifteen near-duplicates create fifteen competing vectors that split probability mass and depress the canonical match. > - **The seven-stage pipeline.** Scoping (define Org > BU > Product > Persona hierarchy) → Ingestion (DOCX/PDF/PPT/images/MD/HTML → fine-tuned LLaMA 3 / QWEN 3.5 / Gemma → draft IdeaBlocks, 1,000–4,000 char input) → Chunking/extraction (context-aware, outputs Q-A pairs) → Semantic deduplication (cluster + merge via a second tuned LLM; GPU or Intel Xeon CPU) → Auto-tagging (clearance, version state, product line, export-control, privacy) → Human validation (5–10 SMEs, 1–2 hrs/quarter each) → Export (JSON-L or API to Azure AI Search / Pinecone / Milvus / Vertex; embeddings via OpenAI/Bedrock/Mistral/Jina/open-source). > - **Application-layer payoff.** Query construction simplifies (question→answer match); governance moves into the data layer (a sales engineer and a legal reviewer query the same index but the *blocks* carry the access boundary); updates propagate from one record instead of hunting dozens of near-duplicates. > - **The framing analogy.** "RAG stacks are beginning to grow a distillation layer between parsing and vectorization, the way web stacks grew a CDN layer between origin and browser." Build it yourself (clustering + LLM summarization + schema enforcement) or use Blockify. > - **Contested/uncertain.** Every quantitative claim is vendor-internal and single-source. "Almost no one examines" the chunk assumption is rhetorical, not measured. The thread is also a product promotion for Blockify/Iternal Technologies, which is disclosed but shapes the framing. > ### 🗒️ Notes > - The unit-of-knowledge critique is the load-bearing idea: a chunk "knows nothing about where its ideas begin or end, which version of a document it came from, who is allowed to see it." > - Splitter behavior is the mechanism: "the splitter cuts wherever the token count runs out" → fragmented retrieval. > - Version problem is framed as enterprise-specific: same doc in a dozen near-identical versions across SharePoint, Confluence, Git → top-K returns deprecated + current mixed. > - IdeaBlock = question + validated answer + typed governance fields on one object; "one fact per unit, nothing more." > - Key reframe: "When your index stores answers to questions, the match becomes structural, not just semantic." > - Geometry metric: 0.1585 vs 0.3624 cosine distance = 2.29x reduction (17 docs / 298 pages, Blockify internal). > - Distillation: 2,042 → 1,200 canonical blocks; 88,877 → 44,537 words; +13.55% vector accuracy distilled vs undistilled. > - Redundancy theory: near-duplicates "distribute probability mass" and pull down the canonical match score — "Your vector index isn't a hard drive you want to fill. It's a retrieval surface." > - Pipeline is seven stages, each with defined I/O so "failures are localizable and reproducible." > - Models named: fine-tuned LLaMA 3 / QWEN 3.5 / Gemma; dedup uses a second specially tuned LLM; GPU-optimized with Intel Xeon CPU fallback. > - Governance becomes data, not orchestrator logic: clearance (PUBLIC/INTERNAL/CONFIDENTIAL/SECRET), version state (Current/Deprecated/Draft/Approved). > - Single-record update propagation is the maintainability win vs "finding all of them, which at enterprise scale is not a tractable problem." > - Underlying principle: "The chunk is a parsing convenience that became a retrieval assumption." The fix is "a better unit," not "a better retrieval algorithm." > - CDN analogy positions the distillation layer as an emerging architectural tier. > - Promotional close: "It's open-source! Try it yourself here→" plus a reshare CTA and the author's handle —  There's a new approach that: * cuts corpus size by 40x. * reduces tokens per query by 3x. * improves vector search relevance by 2.3x. And it doesn't touch your retrieval algorithm, your reranker, or your embedding model. It fixes something upstream that almost no one examines. Every RAG pipeline starts with the same assumption: a chunk of text is the right unit of knowledge to embed. That assumption is almost never examined. And it's the source of most of the retrieval failures people try to fix downstream. ### Why the Chunk Is a Bad Unit A chunk of text is a structurally neutral container. It knows nothing about: * where its ideas begin or end * which version of a document it came from * who is allowed to see it Since a chunk has no idea boundary, the splitter cuts wherever the token count runs out.  You end up retrieving half a table, or a conclusion with no argument, or a claim stripped of the context that makes it true. The model has no way to know what’s missing. The version problem is just as bad. Most enterprise corpora have the same document in a dozen near-identical versions across SharePoint, Confluence, and Git. Top-K retrieval returns five copies of the same paragraph, current and deprecated mixed together. The LLM blends them into an answer that’s confidently wrong. Because the chunk carries no metadata either, there’s nowhere to attach access control in the data itself. Role filters, version state, clearance level: all of that ends up as logic bolted onto the orchestrator, disconnected from the content it’s supposed to govern. LangChain, LlamaIndex, and Haystack sit above all of this. They orchestrate retrieval from whatever you put in the vector store. Most stacks have nothing between the document parser and the vector store. That gap is where all three problems compound. ### A Better Unit: The Question-Answer Packet The chunk fails because it’s structurally agnostic. The fix is to make the unit of knowledge structurally explicit. Instead of embedding a window of prose, you embed a claim: one question, its validated answer, and governance fields as typed schema. One fact per unit, nothing more.  Your queries are already questions. When your index stores answers to questions, the match becomes structural, not just semantic. You’re not hoping the right paragraph floats to the top. You’re matching a question to its answer directly. **[Blockify](https://github.com/iternal-technologies-partners/blockify-agentic-data-optimization)**, a preprocessing layer from Iternal Technologies, implements this as a structure it calls an IdeaBlock: a question, its validated answer, and typed governance fields like clearance level, version state, and source, all on the same object. It mirrors the shape of how users actually query a RAG system, as questions. The key insight: when you embed a question-answer packet instead of a text window, your embedding represents a single atomic claim, not a chunk of narrative that happens to contain it. This has a measurable consequence on vector geometry. In Blockify’s internal benchmark across 17 documents and 298 pages, the average cosine distance from query to best-matching block was 0.1585 for IdeaBlocks versus 0.3624 for naive chunks. That’s a 2.29x reduction in retrieval distance. ### **The counterintuitive finding: less data, more accuracy** Most people expect that shrinking the corpus hurts retrieval. That’s not what happens with semantic distillation. In Blockify’s internal benchmark, the pipeline produced 2,042 raw IdeaBlocks from the source documents.  After iterative deduplication at 80-85% similarity across 3-5 rounds: * 2,042 blocks collapsed to 1,200 canonical IdeaBlocks * Word count dropped from 88,877 to 44,537 * Distilled dataset outperformed undistilled by 13.55% on vector accuracy The reason redundant copies hurt rather than help: fifteen near-duplicates of the same paragraph create fifteen competing vectors in the same region of embedding space. Retrieval distributes probability mass across all of them, pulling the match score down for the canonical version. Collapse them into one canonical block and the signal sharpens. Your vector index isn’t a hard drive you want to fill. It’s a retrieval surface, and redundancy degrades it. ### **The pipeline: Documents to IdeaBlocks** The fix is a preprocessing pipeline that runs before anything touches the vector store. Blockify’s processing runs in seven stages. Each stage has a defined input and output, so failures are localizable and reproducible.  #### **Stage 1: Scoping** Before any document is parsed, you define the index hierarchy: Organization > Business Unit > Product > Persona. This determines which blocks get tagged to which access tier and shapes how deduplication runs later. #### **Stage 2: Ingestion** Documents enter as DOCX, PDF, PPT, PNG/JPG, Markdown, or HTML. The parser hands off to the LLM layer, running fine-tuned LLaMA 3/QWEN 3.5/Gemma4 (and other custom foundation model variants), which converts raw chunks into draft IdeaBlocks: one critical question, one validated answer of two to three sentences, and typed governance fields. Input size is bounded at 1,000 to 4,000 characters, with 2,000 as the practical midpoint. #### **Stage 3: Chunking and extraction** Context-aware splitting means the LLM converts chunks to draft IdeaBlocks rather than cutting on token count alone. The output of each chunk is a question-answer pair, not a window of prose. #### **Stage 4: Semantic deduplication** This is where the retrieval surface gets cleaned. Blocks are clustered by cosine similarity at an 80-85% threshold across three to five iterative rounds. Near-duplicates merge into a single canonical block via a second specially tuned LLM. The pipeline is optimized for GPU, but you can also run it on extra CPU capacity via the Intel Xeon optimized version. The result is a dataset where every vector in the index represents a distinct claim, not one of fifteen near-identical copies competing for the same retrieval slot. #### **Stage 5: Auto-tagging** Each block receives typed metadata: clearance level (PUBLIC, INTERNAL, CONFIDENTIAL, SECRET), version state (Current, Deprecated, Draft, Approved), product line, export control flags, and data privacy labels. Applied by the pipeline, not the document author. #### **Stage 6: Human validation** A product corpus of 2,000 to 3,000 IdeaBlocks splits across five to ten SMEs, each spending one to two hours per quarter on their slice. Structured claims with source citations attached, not raw documents. #### **Stage 7: Export** Validated blocks push to the vector database via API or export as JSON-L. Supported vector stores: Azure AI Search, Pinecone, Milvus, Vertex Matching Engine. Supported embedding models: OpenAI, Bedrock, Mistral, Jina, open-source. The pipeline sits between document parser and vector store regardless of which combination is in use. ### **What Changes at the Application Layer** The unit you embed determines what the application layer can do.  * **Query construction gets simpler:** your queries are already questions. When the index stores answers to questions, the match is structural rather than probabilistic. You stop tuning similarity thresholds to compensate for semantic mismatch between query shape and document shape. * **Governance moves into the data layer:** role-based access, version state, clearance level are typed fields on each block, not logic bolted onto the orchestrator. A sales engineer querying the same index as a legal reviewer gets a different dataset not because the retrieval layer filters it, but because the blocks themselves carry the access boundary. * **Updates propagate from a single record:** when a spec changes, you update one IdeaBlock. Every application querying that block gets the corrected answer on the next request. With naive chunking, the same fact lives in dozens of near-duplicate passages across multiple documents. Updating it means finding all of them, which at enterprise scale is not a tractable problem. The architecture doesn’t change how you query. It changes what you can trust about the answer. ### **The Underlying Principle** The chunk is a parsing convenience that became a retrieval assumption. It has no idea boundary, version context or access state. Retrieval stacks have spent years patching that mismatch: rerankers, hybrid search, threshold tuning, prompt engineering. All of it downstream of the real problem. The fix is not a better retrieval algorithm. It’s a better unit. RAG stacks are beginning to grow a distillation layer between parsing and vectorization, the way web stacks grew a CDN layer between origin and browser. You can build it yourself with clustering, LLM-based summarization, and schema enforcement. Or you can use something purpose-built for it like Blockify. Either way, the chunk-as-unit assumption is the bug, and fixing it at the data layer pays off more than tuning retrieval ever will. It's open-source! #### **[Try it yourself here→](https://github.com/iternal-technologies-partners/blockify-agentic-data-optimization)** Thanks for reading! If you found it insightful, reshare with your network. Find me →[@akshay\_pachaar](https://x.com/@akshay_pachaar) ✔️ For more insights and tutorials on LLMs, AI Agents, and Machine Learning!, You're Doing RAG Wrong, Akshay 🚀, 01kvkwg17p1f71br21sfrncc9c.