# How To Evolve Your LLM Self-Hosting Platform > [!info] Ported from an existing deep-read (Refactor Appendix present). CNCF/KubeCon talk from LY Corporation's production LLM platform (~650B tokens/month, 10 models, 30 internal tenants), with real benchmarks (not vendor marketing) for four inference-optimization techniques. > [!question] Lingering Questions > - The evaluation framework is explicitly disclosed as non-standard (the presenters' own practical lens, not an industry benchmark) — how would the same four techniques score under a different, published evaluation framework? > - PD (prefill/decode) disaggregation caused outright benchmark timeouts under insufficient prefill capacity — is there a safe minimum provisioning ratio below which the technique should not be attempted at all, or is "assessment phase" the right posture indefinitely absent a stronger signal? > - Does the human-in-the-loop vs. autonomous latency distinction hold up outside LLM serving — is it a general property of any agentic system's resource allocation, or specific to GPU-bound inference? > [!tip] Article Ideas > - "Agentic doesn't mean latency-sensitive" — the talk's sharpest correction to a common oversimplification, worth a short standalone piece since it applies to agent design decisions well beyond infrastructure (e.g., which of the vault's own AI-agent workflows actually need low-latency response vs. which just need eventual completion). > [!link] Cross Reference > - [[3. Resources/MOCs/AI Cost & Tokenomics MOC]] — token-based rate limiting and the throughput/latency tradeoff are direct tokenomics-infrastructure content > - [[3. Resources/MOCs/AI Agent Safety MOC]] — the human-in-the-loop vs. autonomous distinction bears on agent design and oversight posture generally > [!quote] Vocabulary > - **TTFT / ITL** — time-to-first-token / inter-token latency, the two latency metrics used throughout the evaluation > - **PD (prefill/decode) disaggregation** — separating the compute-bound prompt-ingestion phase from the memory-bound token-generation phase onto dedicated GPU pools ## Bibliographic capsule CNCF/KubeCon conference talk, YouTube. Two LY Corporation engineers describe their production on-premises, shared-GPU LLM serving platform and walk through real benchmarks for four inference-optimization techniques (quantization, speculative decoding, KV-cache-aware routing, PD disaggregation), each scored on system performance, architectural complexity, and use-case suitability. Category: video, filed under Podcasts. ## Author's thesis (as synthesized) Inference-optimization techniques should be adopted in order of favorable benefit-to-architectural-complexity ratio against your actual workload mix, not by hype or theoretical benefit — and "agentic" is not a synonym for "latency-sensitive": whether a human is waiting synchronously on each turn is the real determinant, independent of whether the workload is technically multi-turn. ## Key arguments (≤3) 1. **Agentic workloads split by human-in-the-loop vs. autonomous, not just by being agentic.** Human-in-the-loop agentic tasks need low latency; autonomous long-running agent tasks care more about throughput than latency — a non-obvious refinement of the common "agentic = latency-sensitive" assumption. 2. **KV-cache-aware routing produced the largest measured effect in the talk.** Routing requests to the pod already holding the relevant conversation's cached prefix dropped time-to-first-token from over 1,200ms to ~200ms on simulated multi-turn agentic traffic. 3. **PD disaggregation gains are conditional, and misconfiguration actively hurts.** Insufficient prefill capacity under prefill-heavy load caused outright benchmark timeouts, not just under-delivery — the most complex technique evaluated, held at "assessment phase" in the presenters' own roadmap pending clearer workload justification. ## Claim atoms ### Agentic doesn't mean latency-sensitive > [!quote] ID: rw_01kze55sqnyfhmd0s2nqmhv8ty > "for human in the loop task where human interact often with agent latency is very important but for autonomous agent that runs a long task latency may be less important and throughput may become more important." Where/why: corrects a common oversimplification. The top two production use cases at LY Corporation are agentic (multi-turn), but the actual latency requirement is determined by whether a human is synchronously waiting on each turn — not by the agentic/non-agentic label itself. ### Speculative decoding preserves exact output quality Where/why: because the base model verifies every drafted token from the smaller draft model, output is mathematically identical to base-model-only generation — a genuine quality-free speedup, unlike quantization. The magnitude of benefit depends on draft-model predictability (51% acceptance on natural conversational data vs. 15% on random input, though even the worst case showed only marginal baseline advantage). ### PD disaggregation can make things worse, not just underdeliver > [!quote] ID: rw_01kze55sqnyfhmd0s2nqmhv8ty > "It is important to note that this is not simple optimization. It requires careful configuration as our benchmark showed. An imbalance between PD resource allocation and the actual workload can lead performance degradations very easily." Where/why: the highest-complexity technique evaluated; benchmark timeouts under insufficient prefill capacity, latency regressions under insufficient decode capacity. Kept in "assessment phase" on their own roadmap rather than adopted, an explicitly disciplined, non-hype-driven decision. ### Adopt techniques in order of benefit-to-complexity ratio, not hype Where/why: the presenters' explicit roadmap — quantization, speculative decoding, and KV-cache-aware routing adopted soon (favorable ratio); PD disaggregation held pending clearer justification. A generalizable infrastructure-adoption discipline beyond LLM serving specifically. ## Permanent candidates - **Agentic doesn't mean latency-sensitive; human-in-the-loop does** — genuinely uncaptured elsewhere in the vault and cross-domain applicable to agent design generally. Promoted below. - The benefit-to-complexity adoption discipline and the PD-disaggregation misconfiguration risk are solid but more infrastructure-specific — connected via Cross Reference rather than promoted this pass. ## Processing checklist - [x] Ported claim atoms from the existing deep-read - [x] Checked existing Permanents before creating new ones - [x] Connected into [[3. Resources/MOCs/AI Cost & Tokenomics MOC]] ## Links - [[3. Resources/MOCs/AI Cost & Tokenomics MOC]] — GPU-serving cost/throughput tradeoffs and token-based infrastructure design - [[3. Resources/Literature Notes/How To Evolve Your LLM Self-Hosting Platform]] — source transcript/notes