Stop Trusting AI; Start Verifying It With Ari Joury

Stop Trusting AI; Start Verifying It With Ari Joury ![rw-book-cover](https://i.ytimg.com/vi/AQCrFnNxnyM/sddefault.jpg) 🔗 **Source:** [Stop Trusting AI; Start Verifying It With Ari Joury](https://www.youtube.com/watch?v=AQCrFnNxnyM) > [!note]+ Document Notes > tags: [ai-verification, ai-trust, agentic-ai, code-quality, financial-services-ai] > author: Ari Joury > source: O'Reilly > reference: https://www.youtube.com/watch?v=AQCrFnNxnyM > date created: 2026-07-16 08:53 > > # 🏷️ Title: Stop Trusting AI; Start Verifying It with Ari Joury >  ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ## 📝 Summary - Ari Joury (founder/CEO of Wangari Global, misheard as "Wonga" in the auto-captions) recounts how his team first treated AI coding tools as a "magic code completion tool," which quietly produced spaghetti code and architectural decay — within six weeks nobody could tell which module was human-written or why it existed. - The stakes became concrete on a pilot with the Zurich Insurance Group (a tier-one insurer): an AI system generating regulatory commentary output a plausible-sounding but meaningless sentence ("Executives should prioritize dynamic asset liability and keep stats"). Actuaries called it complete nonsense — embarrassing because it was fluent, not because it was obviously wrong. - Root cause named explicitly: LLMs are optimized for plausibility, not accuracy — a fatal mismatch in regulated domains like financial services. - The fix Joury describes: stop treating AI as a black box and start treating it as an engineering component. Use tools like Manus and Claude Code as design/spec collaborators (non-deterministic, good for ideation), and deterministic orchestrators like LangGraph/LangChain for production agents (same prompt, same result, reproducible). - Concrete verification architecture: hardcode deterministic calculations rather than trusting the model to do math; use a separate agent layer to sanity-check plausibility; generate commentary with one agentic layer and critique it with another before anything reaches a client; trace every output number back to its source data; and feed client approve/disapprove decisions back into the system as a learning signal. - Risks/tradeoffs: this verification scaffolding is expensive to build and clearly was not there from day one — it was built reactively after a regulatory near-miss. The clip does not address cost, latency, or team size implications of running multiple critique agents per output. - Bottom line: "trust but verify" isn't a slogan here — it's an architecture where every AI output is checked by hardcoded logic, a second agent, and eventually a human client before it ships. ## 📖 Main Content Joury frames the talk around a failure mode he calls "architectural decay": when AI-generated code accumulates without engineering discipline, the codebase becomes illegible even to the team that built it, in a matter of weeks rather than years. This is presented as more insidious than a one-time bug because it's gradual and because the code often *looks* fine — the same trap the video's title targets ("stop trusting," not "stop using"). The core case study is a regulated-industry failure: a descriptive-commentary feature built for the Zurich Insurance Group, a tier-one insurer, generated a sentence that sounded like professional financial writing but was semantically empty. Joury's read on why this happened is a specific and useful frame: LLMs are trained/tuned toward plausibility (sounding right) rather than accuracy (being right), and that gap is invisible until a domain expert (the actuaries) checks the output. His prescribed fix has two parts. First, a tooling split: use non-deterministic, high-capability models (Manus, Claude Code) for design and specification work where variability is fine or even useful, and deterministic orchestration frameworks (LangGraph, LangChain) for the actual production agent pipeline, where the same input must always produce the same output. Second, a verification architecture: hardcode anything that can be hardcoded (numeric calculations), use agents only to check plausibility of those hardcoded results, generate narrative commentary with one agent and critique it with a separate agentic layer, and require human client sign-off as a final gate — with disapprovals feeding back into the system. Uncertainties: the clip doesn't specify what "the AI algorithm learns" means mechanically (fine-tuning? prompt/context updates? a retrieval store of rejected outputs?). It also doesn't quantify the cost or latency overhead of running multiple critique passes per output, which matters a lot for anyone trying to replicate this for their own regulated-industry AI feature. The company name is inaudible/mistranscribed as "Wonga" in the auto-captions — external sourcing (Wangari Global's own newsletter) confirms Joury is founder/CEO of Wangari Global, and the auto-caption is a mishearing. ### 🗒️ Notes #### ⏱️ [00:00:00](https://www.youtube.com/watch?v=AQCrFnNxnyM&t=0) AI as a magic box causes architectural decay - Opens cold: "AI degrades." Team treated AI as a "magic code completion tool" — vague prompt in, impressive-looking code out. - Within under 6 weeks, nobody can tell whether a module was written by a human or AI, or why the code exists — this is the "architectural decay" claim. - Framed as worse than an isolated bug because it's silent and cumulative. #### ⏱️ [00:01:04](https://www.youtube.com/watch?v=AQCrFnNxnyM&t=64) The Zurich Insurance Group failure - Pilot with Zurich Insurance Group (tier-one insurer): a feature auto-generates descriptive commentary on data, a regulatory requirement — "you can't mess it up." - The system output: "Executives should prioritize dynamic asset liability and keep stats" — fluent, meaningless. - Actuaries called it complete nonsense; embarrassing precisely because it "sounded sophisticated." - Root-cause line: LLMs "were optimized for plausibility and not accuracy" — and in financial services "that's a no-go." #### ⏱️ [00:01:59](https://www.youtube.com/watch?v=AQCrFnNxnyM&t=119) Treating AI as an engineering component - Stop treating AI as a magic box; treat it as a real engineering component. - Use Manus and Claude Code as collaborative partners for design and specification. - Use LangGraph (and LangChain) to orchestrate the actual production agents — called out as "die-hard deterministic": same prompt, same result, every time. Contrasted directly with Manus/Claude Code, which are non-deterministic but good for scaffolding and refactoring. #### ⏱️ [00:02:40](https://www.youtube.com/watch?v=AQCrFnNxnyM&t=160) AI as engineering peers — design over coding - Reframes the working relationship: treat these tools "as kind of engineering peers" — highly capable, but capable of really messing up. - Practical shift: interrogate what you actually want, shift more effort to design than to coding, and design tests and edge cases upfront specifically to prevent a repeat of the Zurich-style embarrassing output. #### ⏱️ [00:03:10](https://www.youtube.com/watch?v=AQCrFnNxnyM&t=190) Grounded, deterministic, and agent-critiqued outputs - Current system: numbers are hardcoded/calculated deterministically, not trusted to the model — then a separate agent layer checks plausibility (in case "the wrong model gets chosen"). - Commentary generation is its own agentic system; output (numbers + comments) is critiqued by another agentic layer — described as "throwing darts at this thing until it's really bulletproof." #### ⏱️ [00:03:56](https://www.youtube.com/watch?v=AQCrFnNxnyM&t=236) Client approval loop and the trust-but-verify bottom line - Only after the internal critique loop does the output reach the client, who can approve or disapprove. - Every disapproval is a learning signal for "the AI algorithm," not just the team. - Closing framing, the video's thesis in one line: "You trust the AI to do its job, but you're also verifying it at every single step." - Business framing: this moves output quality "from nonsense at times to utterly high quality as possible," and every data point is traceable back to its original source. ## 📚 References - [The Hidden Architecture of AI Failure — Ari Joury, Wangari](https://newsletter.wangari.global/p/the-hidden-architecture-of-ai-failure) — companion piece on the five types of structural debt (technical, operational, evaluation, integration, governance) that stall enterprise AI pilots. - [From Physicist to AI Founder — Ari Joury, Wangari](https://newsletter.wangari.global/p/from-physicist-to-ai-founder) — background on Joury's path to founding Wangari Global. - [Trust but Verify — O'Reilly Radar](https://www.oreilly.com/radar/trust-but-verify/) — adjacent O'Reilly piece (different author, Andrew Stellman) on the same "trust but verify" framing applied to AI-generated code review. - [AI Is Writing Our Code Faster Than We Can Verify It — O'Reilly Radar](https://www.oreilly.com/radar/ai-is-writing-our-code-faster-than-we-can-verify-it/) — related O'Reilly coverage of the verification gap. ## 🗣️ Paraphrase - Joury's team initially let AI coding tools run on vague prompts because the output looked impressive, and that habit quietly wrecked the codebase's legibility within weeks — a decay pattern he says is common wherever teams treat AI as a magic box rather than an engineering input. - The wake-up call came from a real client: an insurer's regulatory-commentary feature produced a grammatically fluent but factually empty sentence, and the fix that followed wasn't "use a better model" — it was building a whole verification architecture: deterministic math instead of AI-guessed math, one agent generating and another agent critiquing, and a human client as the final gate, with every rejection feeding back into the system. ## 📰 Newsletter Short, sharp, and worth four minutes: Ari Joury of Wangari Global walks through what happened when his team's AI coding habits met a regulated financial-services client. The short version — an AI-generated sentence that sounded like expert commentary but meant nothing, caught by actuaries at a tier-one insurer before it went further. The useful version is what he built in response: a split between non-deterministic tools (Manus, Claude Code) for design work and deterministic orchestration (LangGraph/LangChain) for production, plus a critique-agent layer that has to approve every output before a human client ever sees it. If you're shipping AI features into anything regulated — or just tired of AI-generated code turning to spaghetti in a few sprints — this is a tight four minutes on what "trust but verify" looks like as an actual system rather than a slogan. ## ✅ Action Items #### DevRel - Developer advocacy: none surfaced — this is a single practitioner's war story, not a DevRel program. - Education: could be repurposed as a short case-study segment in a "verifying AI output" workshop, citing the Zurich Insurance Group example as a concrete failure mode. - Developer experience: none surfaced. - Developer marketing: none surfaced. - Community management: none surfaced. - Internal/external/cross-project: none surfaced. #### OSPO - Not applicable to this source — no open-source governance, licensing, or policy content in the clip. #### Community - Not applicable to this source — no maintainer, contributor, or sponsor content. #### WordPress - Not applicable to this source — unrelated to WordPress core, ecosystem, or community programs. #### The WPCC - Not applicable to this source. ### FAIR - Not applicable to this source — no funding, provenance, or governance-artifact content. ## 🧪 Hypotheses - H1: Non-deterministic AI tools (e.g., Claude Code, Manus) produce measurably higher architectural-decay rates (module churn without clear authorship/rationale) than deterministic orchestration frameworks over the same time period. - H2: LLM-generated financial commentary has a materially higher plausibility-without-accuracy rate than LLM-generated code, because natural-language fluency masks factual emptiness more effectively than syntax masks logic errors. - H3: Adding a second "critique agent" layer measurably reduces the rate of client disapprovals compared to single-pass generation, holding the underlying generation model constant. - H4: Teams that shift effort from "coding" to "design and specification" when working with AI produce fewer downstream regulatory or accuracy incidents than teams that stay prompt-and-accept. - H5: Hardcoding deterministic calculations (rather than letting an LLM compute them) reduces numeric-error incident rates to near zero, while narrative-commentary error rates remain nonzero even with a critique layer. ### 📊 Methodology - H1: Instrument commits by tool (Claude Code/Manus vs. LangGraph-orchestrated agents) over a fixed window; measure code churn, "unknown authorship" flags, and time-to-comprehension for a fresh reviewer. - H2: A/B test the same LLM generating (a) code and (b) financial narrative from equivalent-quality prompts; have domain experts (engineers vs. actuaries) rate plausibility and accuracy separately. - H3: Run production outputs through single-agent vs. two-agent (generate + critique) pipelines; compare client disapproval rates on matched output samples. - H4: Survey or interview teams on design-time-to-coding-time ratio; correlate with logged incident counts (regulatory flags, client rejections). - H5: Compare error rates on a benchmark set of financial calculations done by (a) LLM directly, (b) hardcoded logic with LLM plausibility-check, across a few hundred trials. ### ❓ Questions - What exactly does "the AI algorithm learns" from a client disapproval mean mechanically — fine-tuning, RAG memory, prompt augmentation, or a human-curated rule update? - What's the latency and cost overhead of a generate-then-critique two-agent pipeline versus single-pass generation, at the volume a tier-one insurer would need? - How does Wangari Global decide what's safe to hardcode versus what still needs an LLM (e.g., commentary is generated by an agent — why not template-based generation with LLM-assisted phrasing only)? - Is "architectural decay" within six weeks specific to this team's workflow, or a broader pattern Joury has observed across clients? - How is "plausibility" operationalized as a checkable property by the critique agent — is there a rubric, or is it another LLM judgment call (same failure mode one level up)? ### 🔮 Future Research - Empirical study of architectural-decay rate as a function of AI-tool determinism (non-deterministic assistants vs. deterministic orchestrators) across real codebases. - Comparative study of single-pass vs. multi-agent critique pipelines specifically for regulated-industry natural-language generation (financial, medical, legal commentary). - Investigation into whether "critique agents" themselves suffer from the same plausibility-over-accuracy bias as generation agents, and what breaks that recursion. ### 💡 Implications - Practice: teams shipping AI-generated content into regulated domains likely need a deterministic-core-plus-agentic-check architecture, not a single capable model, before they can trust outputs enough to remove humans from the approval loop. - Policy: regulators overseeing insurance/financial reporting may need explicit standards for what "traceable back to original data" means for AI-assisted commentary, given this is presented here as a design choice rather than a requirement. - Security: hardcoding critical calculations reduces one attack/error surface (LLM math hallucination) but concentrates risk in the critique-agent layer, which is not itself claimed to be deterministic. - Education: the "design over coding" shift Joury describes is a concrete, teachable reframe for engineers new to agentic AI development. - Funding models: none discussed in this clip. ### 🧠 Learning Styles Assessment - Reading level estimate: Flesch-Kincaid grade approx. 9–10 for the spoken transcript (short sentences, some technical jargon like "LangGraph," "agentic layer," "deterministic," offset by a conversational, first-person narrative style). #### 👀 Visual learners - Present: none — this is a talking-head interview clip with no on-screen diagrams, charts, or code. - Improve: a simple pipeline diagram (generate agent → critique agent → hardcoded calc check → client approval → feedback loop) would make the architecture Joury describes far easier to retain. #### 🔊 Aural learners - Present: the source is already audio/narration — well suited to this learning style as-is. - Improve: a slightly slower pace or a brief pause after the Zurich Insurance example would help retention of the punchline root cause (plausibility vs. accuracy). #### 🗣️ Verbal learners - Present: concrete analogy ("magic box" vs. "engineering component," "engineering peers," "throwing darts at this thing"). - Improve: a one-line glossary distinguishing "deterministic" vs. "non-deterministic" tooling would help viewers less familiar with the terms. #### 🧩 Kinesthetic learners - Present: none — no hands-on exercise, sandbox, or reproducible steps in this short clip. - Improve: a companion checklist ("hardcode what's calculable, critique what's generated, gate on human approval") would give something actionable to apply immediately. #### 🧠 Logical learners - Present: a clear causal chain (vague prompting → architectural decay → regulatory failure → engineering-discipline fix → verification architecture). - Improve: a decision table (what to hardcode vs. what to delegate to which type of agent) would sharpen this for logical learners. #### 🧑‍🤝‍🧑 Social learners - Present: none — solo interview format, no discussion prompts. - Improve: a discussion question like "where in your own stack are you trusting plausibility over verified accuracy?" would invite social/team discussion. #### 🧘‍♀️ Solitary learners - Present: the narrative format supports individual reflective viewing well. - Improve: a short self-audit prompt ("list the AI outputs in your pipeline that are currently unverified") would extend this for self-paced learners. #### 🌿 Naturalistic learners - Present: none — no ecosystem, sustainability, or environmental framing in this clip. - Improve: not a natural fit for this content; skip rather than force an analogy. ### 📊 Accessibility Assessment #### Screen reader semantics - Present: standard YouTube video page semantics (title, channel name) are screen-reader accessible; no additional in-video structure to assess since this is a plain interview format. - Improve: n/a beyond what YouTube's own player provides. #### Keyboard navigation - Present: standard YouTube player keyboard controls apply. - Improve: n/a — no custom interactive elements in the video itself. #### Contrast and readability - Present: talking-head video with normal lighting; no on-screen text overlays to assess for contrast. - Improve: adding key-term lower-third captions (e.g., "LangGraph," "Zurich Insurance Group") at high contrast would aid readability during fast, jargon-dense speech. #### Text resizing and reflow - Not applicable to this source — no responsive text layout in a video format. #### Media and captions - Present: auto-generated captions exist (used to produce this transcript) but contain errors (e.g., "Wonga" instead of "Wangari," "on 1,000 side" is garbled). - Improve: publish corrected, human-reviewed captions/transcript, since the auto-captions currently misrepresent the company name and at least one phrase. #### Forms and interactions - Not applicable to this source — no forms or interactive elements in the video. #### Standards reference - Gaps map to WCAG 2.2 AA 1.2.2 (Captions - Prerecorded): auto-captions exist but are not fully accurate, which is a partial gap against the accuracy expectation of that criterion. ### 🌍 DEIB Assessment #### Cultural breadth - Not applicable to this source — single speaker, one company example (a Swiss/European insurer), no multi-region framing to assess. #### Inclusive language - Present: no gendered or exclusionary language observed; jargon (LangGraph, agentic layer) is defined implicitly through context. - Improve: a one-line definition of "deterministic" for non-technical viewers would broaden accessibility of the content, though this is a minor gap for a short technical clip. #### Neurodiversity support - Present: content is chunked into distinct narrative beats (setup, failure, fix) even without formal headings, which aids followability. - Improve: on-screen chapter markers or captions naming each beat would help viewers who benefit from explicit structure. #### Representation - Not applicable to this source — solo speaker interview; no cast of characters to assess for representation. #### Psychological safety - Present: Joury is candid about his own team's mistake ("we fell into that same trap") rather than framing failure as someone else's fault — a respectful, low-blame framing. - Improve: none needed; the tone is already appropriately self-critical without being alarmist. ### ✅ Logical Validity #### Strong reasoning - Causal narrative from specific incident to general principle: the Zurich Insurance Group example (~00:01:04) is used as concrete evidence for the broader claim that "LLMs were optimized for plausibility and not accuracy" — this is a reasonable inductive move from one well-described case to a general design principle, and it's strengthened by naming a real client and a real professional check (actuaries flagging the sentence as nonsense). - The proposed fix maps directly onto the diagnosed failure mode: since the problem was "plausible but wrong," the fix (hardcode what's checkable, add a critique layer, gate on human approval) directly targets the plausibility/accuracy gap rather than proposing an unrelated solution. This gives the argument internal coherence. #### Fallacies - Possible false dichotomy at ~00:01:59–00:02:25: framing tools as either "magic box" (bad) or "engineering component" (good) somewhat collapses a spectrum of intermediate practices (e.g., prompt engineering discipline without full deterministic orchestration). Fix: acknowledge partial/intermediate approaches exist between "vague prompting" and "full deterministic pipeline." - Minor overgeneralization risk: the single Zurich Insurance Group anecdote is used to support a general claim about LLMs being optimized for plausibility over accuracy — true as a widely-documented property of LLMs, but presented here with n=1 direct evidence within the clip itself (the general claim is independently well-supported elsewhere, just not proven by this one story alone). Fix: cite the broader literature on LLM calibration/plausibility bias rather than relying solely on the anecdote. #### Tone of voice - Candid, self-implicating, and practitioner-pragmatic — e.g., "we fell into that same trap, actually," "which was rather embarrassing for us." Not defensive or sales-oriented; reads as an honest postmortem rather than a pitch, despite being an interview likely tied to O'Reilly promotional content. ### 🛡️ DARVO Analysis - Deny: not applicable — Joury does not deny the failure occurred; he states it plainly ("they told us complete nonsense"). - Attack: not applicable — no attack on the accusers (the actuaries); their judgment is treated as correct and authoritative. - Reverse victim/offender: not applicable — Joury positions his own team as the party that made the mistake, not as a victim of unreasonable client expectations. - Overall: no DARVO pattern present. This is a self-critical postmortem narrative, structurally the opposite of DARVO (owns the error, credits the people who caught it, describes the fix rather than deflecting). ## 🎯 Conclusion The core claim survives scrutiny: AI systems optimized for fluency will produce confident, wrong output in regulated domains unless something outside the model itself checks accuracy — and Joury's team learned this the hard way with a real insurer. The practical takeaway for anyone building AI features that touch numbers, compliance, or client-facing commentary: don't let one model generate and self-certify its own output. Hardcode what's calculable, use a second agent (or human) to critique what's generated, and keep a traceable path from every output back to its source data. Next step for practitioners: audit your own AI pipeline for any step where a single model's output ships without a second, differently-sourced check. ## 🔄 Refactor Appendix — Stop Trusting AI; Start Verifying It with Ari Joury ### 📚 Literature Atoms (H3) #### [H1] Vague AI prompting causes silent architectural decay > [!quote] ID:rw_AQCrFnNxnyM_h1 > "Under the hood, what happens is that your code base just becomes spaghetti code in ways that a human would never even dare to do." **Context & use**: - ~[00:00:23](https://www.youtube.com/watch?v=AQCrFnNxnyM&t=23) - Names the specific mechanism of AI-driven codebase decay: not a single bug, but accumulated illegibility from unreviewed AI output. **Tags**: #xref **Highlight**: [view highlight](readwise://)  #### [H2] LLMs optimize for plausibility, not accuracy > [!quote] ID:rw_AQCrFnNxnyM_h2 > "These LLMs were optimized for plausibility and not accuracy, despite my apparently say. And in financial services, that's a no-go." **Context & use**: - ~[00:01:37](https://www.youtube.com/watch?v=AQCrFnNxnyM&t=97) - Root-cause diagnosis for the Zurich Insurance Group incident; the caption has a garbled clause ("despite my apparently say") but the core claim is clean and citable. **Tags**: #xref #aphorisms **Highlight**: [view highlight](readwise://)  #### [H3] Trust the AI to act, but verify at every step > [!quote] ID:rw_AQCrFnNxnyM_h3 > "You trust the AI to do its job, but you're also verifying it at every single step." **Context & use**: - ~[00:04:08](https://www.youtube.com/watch?v=AQCrFnNxnyM&t=248) - Closing thesis line of the video; directly restates the title's "stop trusting, start verifying" framing as a working principle rather than a slogan. **Tags**: #xref #aphorisms **Highlight**: [view highlight](readwise://)  ### 🧠 Permanent Candidates (H4) #### [P1] Deterministic-core-plus-agentic-critique architecture **Claim** (one sentence, Courtney's voice): When an AI pipeline feeds regulated or client-facing output, I should hardcode anything calculable and route only the generative, non-deterministic parts through a second agent that critiques the first before a human ever sees it. **Evidence pointer**: [[#H2]] **Scope & conditions**: Holds when output touches numbers, compliance, or client trust; less critical for low-stakes internal tooling where a wrong plausible answer costs little. **Links**: - [[AI verification]] — because extends - [[Trust but verify code review]] — because contrasts on domain (code vs. financial narrative)  ### ✏️ Fleeting Queue - [F1] Look into what "the AI algorithm learns" from a client disapproval means mechanically (fine-tune vs. RAG vs. rule update) — Wangari Global doesn't specify in this clip #xref - [F2] Compare this critique-agent pattern against Andrew Stellman's "Trust but Verify" O'Reilly piece on AI-generated code review — same principle, different domain #xref ### → Things 3 (max 2 actions) - [ ] Read the companion piece "The Hidden Architecture of AI Failure" by Ari Joury for the five-types-of-debt framework — reading #things  📄 **Full text:** [[Full Document Contents/Articles/Stop Trusting AI; Start Verifying It With Ari Joury.md|Stop Trusting AI; Start Verifying It With Ari Joury by Ari Joury]] ## Highlights - AI degrades. When we started building our core product H uh our last year, we fell into that same trap, actually. So, we treated AI as this magic code completion tool. We built this thing and we and we were very proud of it, right? Because you can just use a vague prompt and then the AI produces something that looks really great and you feel really good about yourself. But, under the hood, what happens is that your code base just becomes spaghetti code in ways that a human would never ([View Highlight](https://read.readwise.io/read/01kxnfvvz9w1sebykxtpp2cxpq)) - Note: Mechanism of AI-driven architectural decay: vague prompting → impressive-looking but illegible codebase. - Tags: atom/literature/h1 - We showed it to the actuaries. They told us complete nonsense. It just sounded sophisticated, which was rather embarrassing for us because these LLMs were optimized for plausibility and not accuracy, despite my apparently say. And in financial services, that's a no-go. So, we had to build a whole entire system around that um to make sure that this never happens again. So, what we do is we stop treating AI as ([View Highlight](https://read.readwise.io/read/01kxnfvzm8k116jg31hdnwtwj1)) - Note: Root cause: LLMs optimized for plausibility, not accuracy — a no-go in regulated financial services. - Tags: atom/literature/h2, aphorisms - approve or disapprove that. And every time that they disapprove, not only we, but also the AI algorithm learns. So, that's basically the idea. You trust the AI to do its job, but you're also verifying it at every single step. What this means in practice from a business point of view is that you get from nonsense at times to utterly high quality as possible. Grounded in the data, we built a system in such a way that you ([View Highlight](https://read.readwise.io/read/01kxnfw31mz6hsmx0wdcbd0bfr)) - Note: Closing thesis: trust the AI to act, verify at every step. - Tags: aphorisms, atom/literature/h3