Shipping Secure, Reliable and High Performance AI Agents

Shipping Secure, Reliable and High Performance AI Agents ![rw-book-cover](https://img.youtube.com/vi/EG53QTV8D-E/maxresdefault.jpg) 🔗 **Source:** [Shipping Secure, Reliable and High Performance AI Agents](https://www.youtube.com/watch?v=EG53QTV8D-E) > [!note]+ Document Notes > tags: [ai-agents, agent-security, durable-execution, llm-failover, inference-latency, financial-services] > author: Dana and David (Gradian Labs, as spelled in the transcript) > source: LeadDev (YouTube) > reference: https://www.youtube.com/watch?v=EG53QTV8D-E > date created: 2026-09-29 10:30 > > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the author. > > Do not quote or attribute any other line to the named author. > > # 🏷️ Title: Shipping secure, reliable and high performance AI agents >  ## 🧭 Table of contents - [📝 Summary](#-summary) - [📖 Main content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning styles assessment](#-learning-styles-assessment) - [📊 Accessibility assessment](#-accessibility-assessment) - [🌍 DEIB assessment](#-deib-assessment) - [✅ Logical validity](#-logical-validity) - [🛡️ DARVO analysis](#️-darvo-analysis) - [🎯 Conclusion](#-conclusion) - [🔄 Refactor appendix](#-refactor-appendix--shipping-secure-reliable-and-high-performance-ai-agents) ## 📝 Summary - **What happened.** LeadDev published a 26:13 conference talk, uploaded 2026-09-22, by two engineers from a company the auto-captions spell "Gradian Labs" (most likely Gradient Labs; verify before citing). The video description is empty, so there are no speaker bios or links. The captions call the speakers "Dana" (also rendered "Dai" and "deny") and "David." They describe two years of running customer-support agents for banks and fintechs. - **The claim.** An "agent" is a runtime harness around a model, and reliability, security and performance come from the harness, not the model. The talk backs that with three concrete designs: scoped procedures with per-message tokens (security), Temporal durable execution with provider failover (reliability), and compiled procedure graphs plus dynamic reasoning plus pre-filtered guardrails (performance). - **Why it matters.** The failure mode in regulated support is a compliance breach, not a odd chatbot answer. The talk's example is a customer who agrees to pay 200 pounds on Friday when the payment is unaffordable, which resolves the conversation and still breaks consumer-protection rules. - **Key stakeholders.** The speakers' company and its bank and fintech customers, end customers (including a car-insurance caller after a crash), Temporal (workflow vendor), OpenAI, Azure, Anthropic and Google (model providers), and Intercom and Zendesk (named as pre-authenticated channels). - **Risks and tradeoffs.** No numbers are given for latency savings, cost savings, failover frequency, guardrail count reduction or filter recall. The talk describes practice, not measurement. Each speed technique trades determinism or coverage for latency, and the talk names only one mitigation: the guardrail pre-filter fails open. - **Bottom line.** The architecture is specific and internally consistent, and the strongest parts (per-message scoped tokens, a yes/no identity tool that keeps ground truth out of context, and a mean-latency failover trigger) are directly reusable. Treat the performance claims as unquantified. ## 📖 Main content **Harness as the unit of design.** The opening argument (00:00:48 to 00:02:15) rejects two definitions of "agent": a model with tools, and a large prompt with everything in it. The speakers define an agent as an execution harness that controls what the model may see, which tools it may call, which data it may read, and how it should behave. In financial services the harness also sets the objective function. A generic support bot wins by closing the conversation. A bank agent wins only if the close is compliant. **Control plane versus runtime.** The platform splits into a stable control plane (procedures, resources, tools, tokens, integrations, shared model infrastructure) and a per-conversation, per-turn runtime (which skills, which retrieved data, which hidden data, which guard rules and preconditions). The reason given at 00:02:58: exposing the whole control plane at runtime costs money on model API bills, loses context, and degrades performance. **Procedures as the bridge.** A "procedure" converts business policy into runtime policy. The example is disputes and chargebacks, where the Mastercard rules booklet runs to hundreds of pages. Scoping a procedure forces decisions about which resources, tools, identity requirements and escalation paths the agent gets inside that boundary. **Security, part one: identity.** Chat and email arrive pre-authenticated in most cases (in-app messaging, Intercom, Zendesk). Voice is harder because phone numbers spoof easily. The agent asks for information the customer should know, and the choice of question trades security against convenience: a policy number is hard to guess but hard to find after a car crash. Verification runs through a black-box tool that returns pass, fail or a follow-up question. The ground truth never enters the model's context, so prompt tricks can't extract it. Inside the tool, inputs get normalized (filler words stripped, spoken numbers converted, identifiers split into components) and compared phonetically, so "John's Smith" matches "John Smith." **Security, part two: least privilege.** Each procedure lists the read endpoints ("resources") and write endpoints ("tools") it needs, and the agent can call only those while executing it. Long procedures branch into subprocedures, so the address-update write endpoint becomes reachable only inside the update-address subprocedure. Token design went through three iterations: one static token for everything (too broad), one token per conversation scoped to the user (expiry unbounded, since conversations last minutes to months), and one token per outgoing message scoped to the user with a fixed short expiry (the version they ship). **Reliability: durable execution.** Agent requests run long, cost money and need interruption. Retrying a whole request after a mid-flight error, say 10 seconds in, spikes latency and doubles LLM spend. Building this by hand means queues, checkpoint tables, retry logic and cancellation signals. The speakers use Temporal: a conversation workflow receives a signal per customer message and spawns a child turn workflow, which calls activities (database, network, LLM completion). Temporal checkpoints every activity result, so a restart doesn't re-buy the completion. Waiting for the customer sleeps the workflow without holding CPU on a worker. **Reliability: model failover.** OpenAI models are reachable through both OpenAI and Azure, so the system calls one and fails over on HTTP 500s (not 400s), rate limiting or excess latency. Latency is subtle: during one OpenAI incident P90 stayed flat while mean latency spiked, so a timeout set at P99 never fired even though every call slowed. For critical paths they configure a fallback prompt and fallback model across vendors, and say a global Google outage the previous year was absorbed by failing over to OpenAI or Anthropic. **Performance: four budgets.** Quality, latency, cost and safety are treated as budgets to allocate together. The original path (search knowledge base, find procedure, strongest model, maximum reasoning, every guardrail) felt safe and cost latency and money. Three fixes: compile procedures offline into a directed control-flow graph so runtime estimates the likely next nodes instead of attending over thousands of tokens (voice needs sub-second turns); toggle reasoning up only when the customer message is out of distribution, and play holding phrases to avoid dead air; and pre-filter guardrails with tiny decision-tree models trained on embeddings of agent messages, tuned for high recall, failing open. **Uncertain or contested in this source.** No measured results appear for any of the three performance techniques. The claim that pre-filters run in "tens of milliseconds" and the count of "hundreds of guardrails" are stated without evidence. The Google outage anecdote gives no date. The "Gradian Labs" name and the product names in the captions are unverified. ## 🗒️ Notes #### ⏱️ [00:00:04](https://www.youtube.com/watch?v=EG53QTV8D-E&t=4) Who they are and why financial support is different - Two speakers introduce their company as AI-native customer support for financial institutions, covering AML, fraud, collections, lending, underwriting and disputed chargebacks. - The failure they care about is reliability, security and compliance, not a strange chatbot answer. - The talk promises lessons from a couple of years in production at institutions and fintechs. > [!quote] [00:00:04] > essentially what we're doing is we're connecting frontline customer support conversations with the back office workflows. #### ⏱️ [00:00:48](https://www.youtube.com/watch?v=EG53QTV8D-E&t=48) Defining an agent as a harness - "AI agents" is called overloaded. They exclude a model with tools and a giant stuffed prompt. - The harness controls visibility, tools, data access and behavior, and reliability, security and performance are built into it. - Generic support optimizes for resolving the conversation, which is the wrong objective in finance. #### ⏱️ [00:02:15](https://www.youtube.com/watch?v=EG53QTV8D-E&t=135) The objective function and the control-plane split - Worked example: a customer promises 200 pounds on Friday. That closes the conversation and may still be an unaffordable commitment that triggers consumer-regulation trouble. - The harness defines what success means. The platform splits into a stable control plane (procedures, resources, tools, tokens, integrations, shared model infrastructure) and a per-turn runtime. - Runtime decides skills, retrievable data, hidden data, guard rules and preconditions. #### ⏱️ [00:03:41](https://www.youtube.com/watch?v=EG53QTV8D-E&t=221) Procedures turn business policy into runtime policy - Giving the agent the whole control plane costs money and causes context loss, so a bridging abstraction was needed. - Example: the Mastercard dispute booklet runs to hundreds of pages. - Scoping questions: which resources, tools, identity requirements and escalation paths belong inside a given procedure boundary. #### ⏱️ [00:04:23](https://www.youtube.com/watch?v=EG53QTV8D-E&t=263) Two security challenges: right person, minimum access - David takes over. The agent converses with customers and must read and write their data. - Challenge one is confirming identity. Challenge two is least privilege on read and write endpoints. - Chat and email are usually pre-authenticated through in-app messaging, Intercom or Zendesk. Email is harder to spoof. #### ⏱️ [00:05:50](https://www.youtube.com/watch?v=EG53QTV8D-E&t=350) Voice identity verification and the convenience tradeoff - Voice is the hard case because phone numbers spoof easily. The agent asks for something the customer should know: email, policy number, balance. - Policy numbers resist guessing but are a bad ask from a customer who just crashed a car. - Transcribed speech is messy: filler words and sound-alike mis-transcriptions. #### ⏱️ [00:07:18](https://www.youtube.com/watch?v=EG53QTV8D-E&t=438) The black-box verification tool - The agent gets a tool that answers yes, no or a follow-up question. Ground truth stays out of the agent's context so nobody can trick the agent into revealing it. - The tool normalizes both inputs (filler removal, number words to digits, identifier splitting), compares components phonetically, and returns accept, reject or a request to spell part of the answer. > [!quote] [00:07:18] > we don't want to place the ground truth in the agent's context um for security reasons #### ⏱️ [00:08:44](https://www.youtube.com/watch?v=EG53QTV8D-E&t=524) Least privilege through procedures - A procedure is a sequence of steps. The example is ordering a new card: confirm the address, then call the order-card tool. - On the slide, a yellow box marks a readable data point and a green box marks a callable tool. - Runtime access is limited to the read endpoints (resources) and write endpoints (tools) that the running procedure needs. #### ⏱️ [00:10:10](https://www.youtube.com/watch?v=EG53QTV8D-E&t=610) Subprocedures keep scopes narrow - A procedure with many tools defeats scoping, so procedures branch into subprocedures. - Example: if the customer moved, an update-address subprocedure runs first, and the address write endpoint is reachable only inside it. #### ⏱️ [00:10:56](https://www.youtube.com/watch?v=EG53QTV8D-E&t=656) Token scoping, three iterations - Every tool call becomes an HTTP request with a token in the header. Open questions: who issues it, its scope, its expiry. - Version one, a static token for all conversations and users, is too broad. Ordering a card for customer A must not permit customer B. - Version two, one token per conversation scoped to the user, fails on expiry because a conversation can last minutes, days or months. > [!quote] [00:11:40] > it's one token per message that's scoped to the user and for that we can set a fixed expiry #### ⏱️ [00:12:23](https://www.youtube.com/watch?v=EG53QTV8D-E&t=743) Why agent requests break the retry-everything habit - A classic HTTP request returns in about 100 ms and gets retried whole on error. - An agent request fans out into several LLM API calls. It is long, expensive and interruptible. - Retrying at 10 seconds in spikes latency and doubles LLM cost. #### ⏱️ [00:13:50](https://www.youtube.com/watch?v=EG53QTV8D-E&t=830) Rebuilding durable execution from backend primitives - Long-running work goes in a queue. Each LLM response is saved to a database as intermediate state, in a loop, with a queue step per iteration. - Interruptibility adds a cancellation signal that invalidates state. - The point: this is boilerplate (queues, schemas, checkpointing, retries, cancellation) just to call an LLM a few times. #### ⏱️ [00:15:17](https://www.youtube.com/watch?v=EG53QTV8D-E&t=917) Temporal workflows and activities - The problem is called durable execution. They use Temporal and say other options exist without comparing them. - A conversation workflow receives a signal per customer message and spawns a child turn workflow. - Activities hold non-deterministic work (database, network, LLM completion). Each activity result is checkpointed, so a failure mid-turn restarts without re-buying the completion. #### ⏱️ [00:16:44](https://www.youtube.com/watch?v=EG53QTV8D-E&t=1004) Sleeping without holding CPU - After replying, the workflow pauses until the customer answers or a timeout fires, and no worker CPU stays occupied. - The speakers contrast this with a naive sleep call. #### ⏱️ [00:17:26](https://www.youtube.com/watch?v=EG53QTV8D-E&t=1046) Provider failover for one model - OpenAI models sit behind both OpenAI and Azure APIs. The system calls one and fails over on 500s (not 400s), rate limits or slow responses, spending the other's rate-limit headroom. - Latency failover is subtle. Setting the timeout at P99 usually works. #### ⏱️ [00:18:10](https://www.youtube.com/watch?v=EG53QTV8D-E&t=1090) Deceptive latency incidents and cross-model failover - In an OpenAI incident, P90 stayed flat while mean latency spiked. Everything slowed without crossing the timeout. - The same multi-provider pattern applies to Anthropic and Google. Critical paths carry a fallback prompt and fallback model. - They report surviving a global Google outage the previous year by failing over to OpenAI or Anthropic. No date or duration is given. #### ⏱️ [00:19:37](https://www.youtube.com/watch?v=EG53QTV8D-E&t=1177) Performance as four budgets - Dana returns. The four verticals are quality, latency, cost and safety. Teams usually optimize a subset, typically quality and safety at the price of cost and latency. - Their goal: change the inference path so all four budgets improve together, rather than picking a faster model or skipping guardrails. #### ⏱️ [00:21:03](https://www.youtube.com/watch?v=EG53QTV8D-E&t=1263) The old path and the three fixes - Three years earlier: search the knowledge base, find the procedure, use the strongest model, maximize reasoning, run every guard. It felt safe and was slow and expensive. - Top three fixes: distill procedures into graphs, toggle reasoning dynamically, pre-filter guardrails. #### ⏱️ [00:21:48](https://www.youtube.com/watch?v=EG53QTV8D-E&t=1308) Distilling procedures into graphs - Human-written procedures are word salad for a model. Voice turns need sub-second latency, and thousands of tokens don't fit. - Offline, on every procedure save, each conditional branch, exception, partial rule and subprocedure becomes a node, and transitions become edges. - At runtime the system estimates the most likely next node or top-k nodes given history and the latest customer message, and passes only those to the LLM. #### ⏱️ [00:23:57](https://www.youtube.com/watch?v=EG53QTV8D-E&t=1437) Dynamic reasoning - Most turns follow a simple procedure with trivial customer input, so maximum reasoning every turn is unnecessary. - Reasoning turns on for out-of-distribution input: curveballs, pushback, unavailable information. - Knowing a turn will run slow lets the agent send a holding response instead of dead air. #### ⏱️ [00:24:39](https://www.youtube.com/watch?v=EG53QTV8D-E&t=1479) Guardrail pre-filtering and close - They run hundreds of guardrails. A closing line like "let me know if that helped" doesn't need a financial-advice check. - Tiny decision-tree models, trained on embeddings of agent messages against the available guards, run in tens of milliseconds and are tuned for high recall. - If the filter times out or errors, the guardrail runs. The talk ends with an offer to chat and a hiring pitch. > [!quote] [00:25:23] > we would fail open. If it times out or if it fails, we would still run the guardrail. ## 📚 References The transcript itself contains no links, and the video description is empty. The following are primary sources for entities the speakers name. - [Shipping secure, reliable and high performance AI agents, LeadDev](https://www.youtube.com/watch?v=EG53QTV8D-E): the source video, 26:13, uploaded 2026-09-22. - [LeadDev](https://leaddev.com): the conference-talk publisher. - [Gradient Labs](https://gradient-labs.ai): the speakers' company. The captions spell it "Gradian Labs." Confirm the spelling and URL before citing. - [Temporal](https://temporal.io): the durable-execution platform the speakers use. Its docs define workflows, activities and signals. - [OpenAI API](https://platform.openai.com/docs) and [Azure OpenAI Service](https://learn.microsoft.com/azure/ai-services/openai/): the two routes to the same OpenAI models used for failover. - [Anthropic API](https://docs.anthropic.com) and [Google Gemini API](https://ai.google.dev): the cross-model fallbacks. - [Intercom](https://www.intercom.com) and [Zendesk](https://www.zendesk.com): the support channels named as pre-authenticated. - Mastercard dispute rules: the "booklet" the speakers reference. The likely official title is the Mastercard Chargeback Guide; verify against Mastercard's documentation. - Principle of least privilege: the security concept the token and procedure scoping implements. NIST SP 800-53 control AC-6 is the standard citation. ## 🗣️ Paraphrase The speakers say an AI agent isn't a model plus tools. It's the harness that decides what the model sees, what it can call and what counts as success. In finance the success definition is the hard part, because an answer that closes a conversation can still be a regulatory problem. So they keep a stable control plane of policy and integrations, and at runtime they hand the model only the slice a given procedure needs. A procedure is business policy rewritten as runtime policy, and it lists the exact read and write endpoints the agent may touch. For security they do two things. Identity checks run through a yes/no tool that never shows the model the correct answer, and it matches speech-to-text output phonetically. Write access comes from tokens that last one message and cover one customer, because a per-conversation token would live too long. For reliability, they argue that hand-rolling queues, checkpoints and cancellation for LLM calls is wasted effort, so they run turns as Temporal workflows that checkpoint each completion. They also fail over between providers, including across vendors, and watch mean latency because a flat P90 hid a slowdown once. For speed, they turned free-text procedures into precompiled graphs, raise reasoning effort only when a customer says something unexpected, and use small decision trees to skip guardrails that can't apply to the message at hand. When the filter is unsure or broken, the guardrail runs. ## 📰 Newsletter **Scope the agent, not just the prompt.** LeadDev's 26-minute talk from two engineers at a financial-support AI company is a working checklist for anyone shipping agents that can write to real systems. Three ideas travel well to WordPress work. First, mint credentials per message: a token scoped to one user with a one-minute expiry limits what a hijacked or confused agent can do, and it's the pattern to copy for any WordPress Abilities API or REST integration that lets a model act on a site. Second, keep secrets out of model context by giving the agent a tool that returns pass or fail instead of the answer. Third, measure mean latency during provider failover, not just tail percentiles, because the speakers saw a flat P90 while everything slowed. There's no benchmark data in the talk, so read the speed claims as a design description. Still, the compile-the-procedure-into-a-graph idea maps neatly onto documentation-driven assistants: turn a long handbook page into a small decision graph offline, then let the model see only the next step. Links: [the talk](https://www.youtube.com/watch?v=EG53QTV8D-E), [Temporal](https://temporal.io), [LeadDev](https://leaddev.com). ## ✅ Action items #### DevRel - Developer advocacy: turn the per-message scoped token pattern into a demo, with a short-lived credential minted for each agent action, and publish the expiry and scope numbers. - Education: run a workshop on "what does the agent see" that has attendees list, for one procedure, the exact resources and tools it needs and nothing else. - Developer experience: add a least-privilege section to any agent quickstart showing scoped credentials and a verify-without-revealing tool. - Developer marketing: don't quote latency or cost savings from this talk in Pantheon or community material. It publishes none. - Community management: add a "harness, not model" framing note to agent discussion channels so security threads start from scope, not prompt wording. - Internal/external/cross-project: ask the speakers whether they'll share failover thresholds and pre-filter recall figures for a follow-up write-up. #### OSPO - Policy and governance: write down which agent actions require human approval and which escalation paths a procedure must declare. - License compliance: if Temporal or a similar engine enters any workflow, record its license and hosted-service terms before adoption. - Security management: add "ground truth outside model context" and "per-message token expiry" to the agent threat-model checklist. - Software inventory: list workflow engines and multi-vendor model routes as network dependencies with availability SLOs. - Developer support: publish a note on failover triggers that includes mean latency, not only timeouts at a tail percentile. - Community engagement: no direct action from this source. - Strategic alignment: set a KPI for the share of agent turns that run at full reasoning, so dynamic reasoning is measurable. #### Community - Maintainers: pilot a guardrail pre-filter idea on a moderation or triage bot and log skipped-versus-run counts and misses for 30 days. - Contributors: document which agent tools are read-only and which write, so contributors can review scope changes in a diff. - Sponsors: no direct action from this source; nearest relevance: request cost-per-completed-task reporting from any sponsored agent pilot. - Ecosystem: no direct action from this source; nearest relevance: a shared failover-incident write-up format across projects. #### WordPress - Core: no direct action from this source; nearest relevance: token scoping for REST and application passwords. - Core Performance: no direct action from this source; nearest relevance: measure mean and tail latency separately for any model-backed feature. - Core AI: add least-privilege and per-action credentials to responsible-AI guardrails for the Abilities API, and require that secrets stay out of model context. - **Core Program Team:** no direct action from this source; nearest relevance: cross-team review of which abilities an agent can call. - Mobile: no direct action from this source. - Playground: build a browser demo of a two-endpoint procedure showing that the write ability only appears inside its subprocedure. - Community: no direct action from this source. - Training: draft a lesson on harness design (visibility, tools, data access, behavior) using the card-ordering procedure as the example. - Marketing: no direct action from this source. - Support: evaluate a verify-identity black-box tool for any support workflow that touches account data. - Design: no direct action from this source; nearest relevance: holding-phrase patterns to avoid dead air in voice UI. - Documentation: add a "procedures as scoped runtime policy" glossary entry if agent docs reference the pattern. - Photos: no direct action from this source. - TV: no direct action from this source. - Polyglots: no direct action from this source; nearest relevance: phonetic matching would need per-language handling. - Accessibility: no direct action from this source; nearest relevance: voice agents need transcription-error tolerance for speech-disabled users. - Meta: no direct action from this source; nearest relevance: credentials for .org tooling agents should be per-action and short-lived. - CLI: no direct action from this source; nearest relevance: scope WP-CLI-backed agent tools to named commands. - Hosting: add "provider failover triggers on mean latency" to cross-host reliability checklists. - Tide: no direct action from this source; nearest relevance: prefilter-then-check could reduce scan cost. - Openverse: no direct action from this source; nearest relevance: a cheap pre-filter before expensive moderation checks, tuned for recall. - Themes: no direct action from this source. - Plugins: no direct action from this source; nearest relevance: review of agent-callable plugin abilities for scope creep. #### The WPOCC - Programs: no direct action from this source. - Projects: no direct action from this source; nearest relevance: a documentation sprint on agent-scope patterns. - Funding experiments: no direct action from this source; nearest relevance: ask any funded agent tooling to report cost per completed task. ### FAIR - Technical independence: multi-provider failover reduces single-vendor dependency, so document it as a design goal for any FAIR-adjacent AI tooling. - Policies and covenants: require agent decisions and the tokens' scope to be logged so actions are auditable after the fact. - Productized governance artifacts: add a "model and workflow-engine dependency" row to operator scorecards. ## 🧪 Hypotheses 1. Issuing one token per agent message, scoped to one user with an expiry of 60 seconds or less, cuts the blast radius of a compromised turn to a single user and a single reply window, compared with a per-conversation token. 2. Keeping ground truth out of model context and exposing only a pass, fail or follow-up tool lowers successful extraction of the true value by prompt attack to near zero, compared with placing the value in context. 3. Durable execution with per-activity checkpointing lowers LLM spend on failed turns by at least 30% compared with whole-request retry, on a workload where 5% or more of turns hit a mid-flight error. 4. A failover trigger based on mean latency detects a provider slowdown earlier than a P99 timeout when P90 stays flat. 5. Compiling procedures into a node graph and passing only the top-k next steps reduces prompt tokens per turn by more than 50% with no drop in procedure-adherence accuracy. 6. A tiny decision-tree guardrail pre-filter tuned for high recall skips more than half of guardrail runs while missing fewer than 1% of messages that a full guardrail would have flagged. ## 📊 Methodology - **Design for H1 and H2.** Build a test agent with two credential modes and two secret-handling modes. Run 200 scripted injection attempts (extraction, cross-user access, replay after conversation end) against each configuration. Record successful cross-user calls and successful secret extractions. Count how long a stolen token stays usable. - **Design for H3.** Replay 10,000 recorded turns with injected failures at random points, once with whole-request retry and once with checkpointed activities. Compare total completions billed and wall-clock latency at p50 and p95, taken from provider invoices. - **Design for H4.** Replay historical latency traces, including the incident shape described (flat P90, rising mean), and compare detection time for a P99 timeout, a P90 threshold and a mean-latency threshold. Report detection delay and false-trigger rate. - **Design for H5.** Take 50 real procedures. Compare adherence on 1,000 labeled turns between full-text prompting and graph-plus-top-k prompting. Measure prompt tokens and latency. - **Design for H6.** Train the pre-filter on labeled guardrail outcomes with a time-based split. Report recall on flagged messages, skip rate and added latency, and pre-register the recall floor. - **Controls throughout.** Fix the model, prompt and seed per arm, run three repeats, and log every skipped guardrail for later audit. ## ❓ Questions - What is the token lifetime in seconds, and how is it enforced when a reply takes longer than expected? - How does the identity tool cap guessing attempts, and is there a lockout after repeated fails? - What phonetic algorithm handles names and emails in other languages and accents? - What recall does the guardrail pre-filter reach, and how is a missed flag detected in production? - How much does graph compilation cost when a procedure changes often, and how do stale graphs get caught? - What does the graph-based next-step estimator use: embeddings, a classifier, or a model, and how does it handle a customer who jumps between branches? - How does the system decide a message is out of distribution, and what is the false-negative cost? - What did the Google outage failover cost in output quality when the fallback prompt ran on a different model? ## 🔮 Future research - **Measured tradeoffs for scoped procedures.** The talk names the design but gives no data on task success versus scope width. A public benchmark of narrow-scope versus full-scope agents would settle the claim. - **Calibration of out-of-distribution triggers.** Toggling reasoning on surprise depends on detecting surprise. Error rates for that detector are unpublished. - **Failover semantics across vendors.** Prompt and model fallbacks change behavior. Research on drift between primary and fallback outputs in regulated settings is thin. - **Prompt injection against the pre-filter.** A small model trained on embeddings can be evaded by crafted agent output. This is unexplored for guardrail routing. - **Audit trails for skipped guardrails.** A skip is a decision. Standards for logging and reviewing skips don't exist in the talk. ## 💡 Implications - **Practice.** Scope credentials per message and tools per procedure, and keep secrets behind a verify tool. These patterns apply outside finance. - **Policy.** Regulated deployments need a documented rule for what the agent can do when a customer's stated intent (pay 200 pounds) conflicts with affordability. - **Security.** Failing open on the guardrail pre-filter favors safety over latency. Teams copying the design should confirm that choice matches their risk posture. - **Education.** The harness-versus-model framing is a compact way to teach why agent safety isn't solved by choosing a better model. - **Funding.** Reliability work (durable execution, multi-vendor routing) is real engineering cost. Budgets that count only model spend miss it. ## 🧠 Learning styles assessment - **Reading level estimate.** Flesch-Kincaid Grade Level of about 8 to 9 on the transcript, based on short spoken sentences with heavy filler. Flesch Reading Ease lands near 65. Domain terms (durable execution, P90, activity, guardrail) raise effective difficulty to an intermediate-developer audience. #### 👀 Visual learners - Present: the transcript references slides: a procedure with yellow (data) and green (tool) boxes, a latency chart showing flat P90 and rising mean, and a procedure-to-graph diagram. - Improve: describe each slide aloud. The latency chart and graph diagram carry information the narration only partly states. #### 🔊 Aural learners - Present: continuous narration by two speakers with a clear handoff at each section. - Improve: a 2-minute audio recap of the three techniques would help listeners who can't see slides. #### 🗣️ Verbal learners - Present: concrete stories (200 pounds on Friday, the car-crash caller, the Mastercard booklet) explain each idea. - Improve: define P90, mean versus tail latency, and "out of distribution" in one clause each. #### 🧩 Kinesthetic learners - Present: worked examples such as the card-ordering procedure and the Temporal workflow structure. - Improve: no repository, snippet or lab. A minimal Temporal turn-workflow sample would let viewers build it. #### 🧠 Logical learners - Present: the token iterations run as a clear progression, each version fixing the last version's flaw. - Improve: the performance section states conclusions without figures. Add the token, latency and cost numbers per technique. #### 🧑‍🤝‍🧑 Social learners - Present: two speakers alternate, and the close invites questions afterward. - Improve: add discussion prompts, or a linked channel for follow-up. #### 🧘‍♀️ Solitary learners - Present: the talk is self-contained and short enough to rewatch by section. - Improve: end with a checklist viewers can score against their own agent (scope, token lifetime, checkpointing, failover trigger). #### 🌿 Naturalistic learners - Present: no explicit natural-world framing. - Improve: skipping unneeded guardrail and reasoning calls saves compute. State the energy or resource effect if it can be measured. ## 📊 Accessibility assessment #### Screen reader semantics - Present: YouTube supplies player and page landmarks, and the auto-transcript is machine-readable. - Improve: slide content (procedure diagram, latency chart, graph) is not spoken in full. WCAG 2.2 AA 1.1.1 (Non-text Content) and 1.2.5 (Audio Description, Prerecorded) are unmet for those visuals. #### Keyboard navigation - Present: the YouTube player is keyboard operable with visible focus, which passes 2.1.1 and 2.4.7 through the platform. - Improve: the video has no chapters (metadata lists none), so keyboard users can't jump between security, reliability and performance. Add chapters for 2.4.5 (Multiple Ways) as a practical aid. #### Contrast and readability - Present: unverifiable from a transcript. - Improve: conference slides often use small text. Confirm 1.4.3 (Contrast Minimum) on the latency chart and diagrams, and enlarge labels for mobile viewing. #### Text resizing and reflow - Present: the player scales, and captions resize independently. - Improve: text burned into slides can't resize, so 1.4.4 (Resize Text) and 1.4.10 (Reflow) fail for that content. #### Media and captions - Present: an auto-generated transcript exists and is the basis for this analysis. - Improve: captions garble proper nouns and terms: "Gradian Labs," "Dana" as "Dai" and "deny," "aras" for what is probably ARAs or a similar acronym, "entropic of an AI" for Anthropic, "Zenesk" for Zendesk, "fintex" for fintechs, and "Intercom" alone as a transcription of "intercom." This is a 1.2.2 (Captions, Prerecorded) failure in substance. The description is also empty, which drops the links and speaker credits that 2.4.4 (Link Purpose) style context would supply. #### Forms and interactions - Present: none in the video. - Improve: the hiring call at the end has no link or address in the transcript. Put it in the description as selectable text. #### Standards reference - Primary gaps against WCAG 2.2 AA: **1.2.2** (inaccurate captions on names and vendors), **1.2.5** (no audio description for slides), **1.1.1** (chart data in visuals only), **1.4.4 and 1.4.10** (burned-in slide text). Captions are the highest-impact fix. ## 🌍 DEIB assessment #### Cultural breadth - Present: the examples span a UK-flavored payment scenario (pounds), a car-insurance caller, and global providers (OpenAI, Azure, Google, Anthropic). - Improve: the identity design assumes English speech and email addresses. Cover names and identifiers in other languages and how phonetic matching behaves for them. #### Inclusive language - Present: direct, plain language. Jargon is mostly explained by example. - Improve: "curveball" and "word salad" are idioms that translate badly. "Detractable" in the captions is probably a mis-transcription of a different word. #### Neurodiversity support - Present: the structure is predictable: security, then reliability, then performance, with a named handoff. - Improve: no chapters and no on-screen agenda were captured. Add chapter markers so viewers can navigate by topic. #### Representation - Present: two speakers with a co-presenting format. - Improve: no customer, regulator or third-party voice appears. One outside comment on the tradeoffs would broaden the view. #### Psychological safety - Present: the tone is candid about past mistakes ("this used to be us about three years ago"), which models admitting an imperfect first design. - Improve: a short statement that the numbers are internal and unpublished would help viewers judge their own results fairly. ## ✅ Logical validity #### Strong reasoning - **Progressive elimination, 00:10:56 to 00:12:23.** Each token design fixes the previous flaw: static token is too broad, per-conversation token expires too late, per-message token gets a fixed expiry. The conclusion follows from the stated constraints. To clarify further, state the actual expiry value. - **Causal mechanism, 00:13:06.** If a request runs long and each step costs money, a whole-request retry at 10 seconds repeats paid work. The mechanism is stated and checkable. Adding a measured retry-cost figure would finish it. - **Counter-intuitive evidence, 00:18:10.** A flat P90 with a spiking mean shows a timeout at a tail percentile can miss a real slowdown. This is a concrete case against a common default and is easy to reproduce. - **Design argument, 00:07:18.** Keeping ground truth out of context prevents extraction by prompt trickery. The claim is structural, and it holds without a benchmark. #### Fallacies - **Unsupported quantification, 00:20:19 and 00:24:39.** "significant cost of uh well cost and latency" and "tens of milliseconds" carry no baseline or measurement. Fix: publish before and after latency and cost per turn. - **Unquantified generalization, 00:19:37.** "which we managed to weather" (the Google outage) gives one anecdote as evidence the failover works. Fix: report failover rate, quality delta, and how many incidents were covered. - **Survivorship framing, 00:21:03.** "this feels great because feels safe" describes an old design as safe, but the talk provides no failure data to show it was. Fix: state what the maximal path actually prevented. - **Appeal to experience, throughout.** "everything that we've learned from the last couple of years" is the warrant for the whole talk. The lessons are plausible, but the evidence is the speakers' report. Fix: include incident counts and outcomes. - **Ambiguity from captions.** "aras" and "detractable" can't be treated as stated facts. Fix: corrected captions. #### Tone of voice - Conversational, practical and engineer-to-engineer, with contractions and second person ("you don't want to have to like spin up all these queues"). Humor is light ("congratulations. You're very lucky" about the Mastercard booklet). Confidence is highest on security and reliability, where mechanisms are named, and lowest on performance, where the section is rushed at 00:23:13 ("I'm just going to time uh so I might skip through this one a bit faster") and figures are absent. ## 🛡️ DARVO analysis Not applicable to this source. DARVO describes a response pattern to accusation or criticism, and this talk is a technical presentation with no accuser or dispute. For completeness across the three prongs: **Deny**: nothing is denied, and the speakers describe their earlier expensive design as a real cost. **Attack**: no critics or competitors are attacked, and Temporal is recommended with other options acknowledged. **Reverse victim/offender**: no such move appears. ## 🎯 Conclusion The talk's durable claim is that agent safety, uptime and speed come from the harness. The strongest concrete lessons are three. Scope credentials to one user and one message with a fixed expiry. Keep secrets outside model context behind a pass, fail or follow-up tool. Fail over on mean latency, not only on tail timeouts, because the speakers saw a flat P90 during a real incident. The performance section is the weakest as evidence. Graph compilation, dynamic reasoning and guardrail pre-filtering all sound sensible, but no token counts, latencies, recall figures or cost numbers appear, and the section runs short on time. The auto-captions also mangle the company name, a speaker name, a vendor and at least one acronym. **Next steps for readers:** audit your own agent for the four security questions (who issues each token, its scope, its expiry, what the model can see), then instrument one workflow to measure retry cost and mean-versus-tail latency before adopting any of the performance techniques. Check the company name and product details against the speakers' own site, since the captions can't be trusted for them. ## 🔄 Refactor Appendix — Shipping secure, reliable and high performance AI agents > Density: 2–5 Literature atoms (H3 pad), 0–3 Permanent candidates (H4 pad). > Emitted: 5 Literature, 2 Permanent. Resonance filter applied: the Google-outage and > hiring remarks were dropped as unevidenced or off-topic. > > Konik claim-header rules: present tense, declarative, ≤10 words, searchable, unique. > > Atomicity: one declarative sentence per atom. "and"/"also" → split. > > Tag pool: #bmf · #addendum · #xref · #articleseed · #storystem · #share · #aphorisms ### 📚 Literature Split Pad #### [H1] An agent is a runtime harness > [!quote] [00:00:48] > a more useful mental model is essentially a runtime harness around a model. **Context & use**: - 00:00:48, right after the speakers reject "models with tools" and "a massive prompt" as definitions. - Anchors the whole talk: reliability, security and performance are properties you build into the harness, not properties you inherit from the model. **Tags**: #xref #articleseed  #### [H2] Ground truth stays outside model context > [!quote] [00:07:18] > we don't want to place the ground truth in the agent's context um for security reasons **Context & use**: - 00:07:18, explaining the black-box identity tool that returns pass, fail or a follow-up question. - The mechanism is structural: what the model never sees, a prompt attack can't extract. **Tags**: #xref #share  #### [H3] One token per message limits blast radius > [!quote] [00:11:40] > one token per message that's scoped to the user and for that we can set a fixed expiry **Context & use**: - 00:11:40, the final of three token designs after static and per-conversation tokens failed on scope and expiry. - Reusable for any agent that writes to real systems. The reply window is short and known, so the expiry can be short and fixed. **Tags**: #xref #share  #### [H4] Whole-request retry doubles agent cost > [!quote] [00:13:06] > if you retry the entire thing, like 10 seconds in, you'll suddenly like have a spike in user latency. You'll double your LLM costs. **Context & use**: - 00:13:06, the motivating problem for durable execution, contrasted with a 100 ms HTTP retry. - Explains why checkpointing each LLM completion pays for itself on long, interruptible turns. **Tags**: #xref  #### [H5] Tiny models decide which guardrails run > [!quote] [00:24:39] > we run tiny models essentially trained on the embeddings of the agents messages and the guards that we have available. **Context & use**: - 00:24:39, the pre-filter that skips irrelevant guardrails; it is tuned for high recall and fails open (00:25:23). - No recall figure or skip rate is given, so treat the effect size as unknown. **Tags**: #xref #articleseed  ### 🧠 Atomic Split Pad #### [P1] Scope agent access per action **Claim** (one sentence, Courtney's voice): An agent's credential should cover one user and one action window, so a confused or hijacked turn can't reach anything else. **Evidence pointer**: [[#H3]] · [[#H2]] **Scope & conditions**: Holds when the reply window is short and bounded. Breaks when a single action can run for hours, or when the credential service becomes a latency bottleneck. Combine it with procedure-level endpoint scoping so the token isn't the only control. **Links**: - [[Principle of least privilege]] — because extends - [[Static API tokens]] — because contrasts on scope and lifetime - [[WordPress Abilities API]] — because applies-in agent-callable site actions  #### [P2] Failover triggers should watch mean latency **Claim** (one sentence, Courtney's voice): A timeout set at a tail percentile can miss a provider slowdown that shifts the whole distribution, so failover logic should also watch the mean. **Evidence pointer**: [[#H4]] **Scope & conditions**: Applies to model-provider failover where P90 or P99 looks flat during an incident. The speakers show one OpenAI incident as evidence, so the claim needs a second case before it generalizes. Mean latency is sensitive to outliers, so pair it with a change-detection threshold. **Links**: - [[Multi-provider model failover]] — because extends - [[Durable execution]] — because evidence-for reliable retries after a failover - [[Latency percentiles]] — because contrasts on what each statistic detects  ### ✏️ Fleeting Queue - [F1] Confirm the company name, since captions say "Gradian Labs" and the likely form is Gradient Labs: https://www.youtube.com/watch?v=EG53QTV8D-E #fleeting #inbox - [F2] Identify the acronym the captions render "aras" at 00:00:04, likely a fraud or compliance term named in the customer-support list #fleeting #inbox - [F3] Open question: what is the pre-filter's recall at 00:25:23, and how do they detect a missed guardrail in production #fleeting #inbox - [F4] Open question: does per-message token minting add latency at the credential service, and how is it cached #fleeting #inbox - [F5] Article seed: "Retry the step, not the request" as a rule for agent backends, built on H4 and Temporal's activity checkpointing #fleeting #inbox #articleseed ### → Things 3 (max 2 actions) - [ ] Verify company name, speaker names and the ambiguous acronym against the company's own site before citing this talk — research, 15 min #things - [ ] Audit one agent workflow for token scope, token expiry and whether ground truth appears in model context — security review #things  📄 **Full text:** [[Full Document Contents/Articles/Shipping Secure, Reliable and High Performance AI Agents.md|Shipping Secure, Reliable and High Performance AI Agents by Danai Antoniou and David (Gradient Labs)]] ## Highlights - prompt where you just shove things in and hope for the best. Um, a more useful mental model is essentially a runtime harness around a model. And the reason that we think about it that way is because you we're going to talk about ([View Highlight](https://read.readwise.io/read/01m34hzszvmmwq3kp3hxkd89hk)) - Note: Agent = runtime harness around a model; reliability, security, and performance come from the harness, not the model. - Tags: atom/literature/h1 - this verification is we give the agent access to a tool. It's like a black box where it can get effectively a yes or no answer. And this is intentional because we don't want to place the ground truth in the agent's context um for security ([View Highlight](https://read.readwise.io/read/01m34hztyv69mbdrxdvzeebh7z)) - Note: Identity verification as a yes/no tool: the secret never enters the model's context, so prompt injection can't extract it. - Tags: atom/zettel/z1, atom/literature/h2 - that we landed on is uh one where which is issued every time we need to reply to the customer. So it's one token per message that's scoped to the user and for that we can set a fixed expiry ([View Highlight](https://read.readwise.io/read/01m34hzvsgfyr6jdwpfjqwj5yr)) - Note: Token scope ladder: static → per-conversation (expiry unbounded) → per-message, user-scoped, fixed short expiry. - Tags: atom/literature/h3 ## New Highlights added 2026-09-29 at 11:23 AM - [00:00:04] Hey folks, thanks for coming. I'm Dana and this is David and we're from Gradian Labs. We're building AI native customer support for financial institutions. So despite all the SEO buzzwords here, um, essentially what we're doing is we're connecting frontline customer support conversations with the back office workflows. So think less FAQ chatbot and more high stakes higher risk workflows like AML fraud uh aras collections lending and underwriting disputed chargebacks all the good stuff uh the financial institutions have to deal with. The interesting problem here is that it changes the setup less in terms of failures less about you know oh the chatbot just gave a weird answer and more about uh a reliability a security [00:00:48] and a compliant compliance uh failure. So the talk is going to be today about everything that we've learned from the last couple of years shipping those uh agents in production in financial institutions and fintex that you know and use every day. So a bit of background um I find the term AI agents uh significantly overloaded. So just to kind of set the scene here we're not talking models with uh tools and we're not talking a massive prompt where you just shove things in and hope for the best. Um, a more useful mental model is essentially a runtime harness around a model. And the reason that we think about it that way is because you we're going to talk about reliability, security, and performance. And you don't inherit those system ([View Highlight](https://read.readwise.io/read/01m3psgwebqne1t97wwbscdf5t)) - Note: [H1] An agent is a runtime harness - Tags: atom/literature/h1 - [00:07:18] this verification is we give the agent access to a tool. It's like a black box where it can get effectively a yes or no answer. And this is intentional because we don't want to place the ground truth in the agent's context um for security reasons because otherwise you could effectively ask the agent or trick it into giving you the the real answer. Um, so what the agent will do is it will take the answer from the user and the ground truth and in input it into this tool and get back either a pass, a fail or a follow-up question that it could ask. Um, so what I'll do is I I'll dive a little bit deeper into how this this verify identity uh box works which I built. Um, so at a high level what it does is it will take the two uh inputs um uh and do a bunch of things to [00:08:01] normalize them. So it might strip out filler words. Uh it might strip out filler words. Uh you know convert uh verb uh word forms of a numbers to numeric form. Uh you know split identifiers up into components and so on. Um and then what it's going to do is compare those two uh the components from both sides phonetically. So we don't do exact string matching here. So in this example, John's Smith would match John Smith because it sounds the same and it's effectively the correct answer. And then finally, yeah. So what will happen is the um this uh this the model that does this will output either accept, direct, reject, or perhaps a follow-up question where we might ask the user to ([View Highlight](https://read.readwise.io/read/01m3psgzf9c5tsj0j6ft00ww0j)) - Note: [H2] Ground truth stays outside model context - Tags: atom/literature/h2 - [00:11:40] the agent to be able to suddenly order a card for customer B by accident. Um, so you ideally want the scope of this token to be limited to the user that you're helping. So then you might improve this and say, okay, let's uh issue a token at the start of the conversation that's scoped only to the user you're helping. Um but the issue here is uh the expiry will be too long because we don't know when the conversation will end. A conversation might last a minute or an hour, a day and in some cases months. So you don't want to give issue a token that has an expiry of say 3 months. So the uh most secure version of this that we landed on is uh one where which is issued every time we need to reply to the customer. So it's one token per message that's scoped to the user and for that we can set a fixed expiry [00:12:23] because we know exactly how long we will take to respond. And this is you know no longer than a minute for example. Cool. Okay. So that was a couple of examples of some of the challenges we've faced in security. Uh now I'm going to touch a bit more about uh reliability. Um so before I I talk about durable execution I'll try and uh give an example of or motivating example. So, um, as I'm sure you're all familiar with, um, you know, if you're dealing with a classic HTTP server, you can send it a request. You might get back a response within, say, 100 milliseconds or so. Um, and if you encounter an error, you'll probably just retry the entire request, and that's completely normal. ([View Highlight](https://read.readwise.io/read/01m3psh24dt0y6q9rngxcgjzqs)) - Note: [H3] One token per message limits blast radius - Tags: atom/literature/h3 - [00:13:06] However, with Agentic systems, it's much more common to send a request and then behind the scenes, there will be several uh LLM API calls um and you'll eventually get back a response. Now, the key difference here is that uh these are long running, they're expensive, and they're often you want them to be interruptible. So, if the user changes their mind midway through, you want to be able to interrupt the sequence. Uh, and what this means is that if you get an error midway through, um, you know, dealing with a request, you don't want to retry the entire thing because if you retry the entire thing, like 10 seconds in, you'll suddenly like have a spike in user latency. You'll double your LLM costs. Uh, and it's just an all round bad experience. [00:13:50] So, how do we solve for this? I'll try and reason about this using kind of like more classic backend primitives and then motivate uh the the solution we'll come to. So if your system is longunning and expensive then you okay let's say you start out with a backend service and because it's long running you don't want to respond synchronously so you put the request in a queue and uh because you kind of want to be able to checkpoint your progress along the way what you might do is you say okay well every time we call an LLM API we'll save the response in a database and kind of update some intermediate state and we do this in a loop now you don't want a single request to hog this backend service so might what you might then introduce is some kind of like cue. So ([View Highlight](https://read.readwise.io/read/01m3psh4wq9q0r1yfswtd0j1d4)) - Note: [H4] Whole-request retry doubles agent cost - Tags: atom/literature/h3 - [00:24:39] dead air. Um last but not least uh pre-filtering guardrails. Um most of the time you don't need to run every guardrail. We have I think hundreds of guardrails uh that we run. But you know if our agent is saying let me know if that helped or if you need anything else from me. You don't need to run a financial advice guardrail. Um and the way we decide whether when to run a guardrail or not is u we've trained our own tiny uh decision tree models. You know back when it used to be called ML and not AI. Um, we run tiny models essentially trained on the embeddings of the agents messages and the guards that we have available. Um, and during runtime those models are extremely fast. You're talking at tens of milliseconds. And during runtime uh [00:25:23] what we do is we have an agent message that has been generated. We run it through the pre-filtering gates uh which essentially are optimized for significantly higher recall. Um what you're optimizing for is just deciding whether you should run a guardrail or not. And if the pre-f filter says yes, then we run the guardrail. Otherwise, we skip it completely. And obviously, we would fail open. If it times out or if it fails, we would still run the guardrail. So, these are just a couple of examples of uh how we've been optimizing performance, but also keeping security and uh reliability in mind. Um would love to chat afterwards if anyone's any questions. And also, um we're hiring ([View Highlight](https://read.readwise.io/read/01m3psh84f0h2ejn26h19x0jbr)) - Note: [H5] Tiny models decide which guardrails run - Tags: atom/literature/h3