Protecting Releases, Fixing Bugs With Sentry & LaunchDarkly  🔗 **Source:** [Protecting Releases, Fixing Bugs With Sentry & LaunchDarkly](https://www.youtube.com/watch?v=uyesTWSDllI) > [!note]+ Document Notes > tags: [feature-flags, guarded-releases, error-monitoring, ai-code-remediation, release-automation, progressive-delivery] > author: Peter McCarron and Tom Totenberg (Sentry & LaunchDarkly) > source: Sentry (YouTube) > reference: https://www.youtube.com/watch?v=uyesTWSDllI > date created: 2026-09-25 08:51 > > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the author. > > Do not quote or attribute any other line to the named author. > > # 🏷️ Title: Protecting releases, fixing bugs with Sentry & LaunchDarkly > <!-- deep-read:sections start --> ## 🧭 Table of contents - [📝 Summary](#-summary) - [📖 Main content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning styles assessment](#-learning-styles-assessment) - [📊 Accessibility assessment](#-accessibility-assessment) - [🌍 DEIB assessment](#-deib-assessment) - [✅ Logical validity](#-logical-validity) - [🛡️ DARVO analysis](#️-darvo-analysis) - [🎯 Conclusion](#-conclusion) - [🔄 Refactor appendix](#-refactor-appendix--protecting-releases-fixing-bugs-with-sentry--launchdarkly) ## 📝 Summary - **What happened.** Sentry published a 55:10 livestream recording on 2026-09-24. Peter McCarron (staff technical product marketing manager, Sentry, Denver) and Tom Totenberg (head of release automation, LaunchDarkly, Minneapolis) walk through a hack project that chains three things: a LaunchDarkly "factory" of agents that wraps a pull request in a feature flag and adds Sentry instrumentation, a LaunchDarkly guarded release that reads Sentry error metrics and auto-reverts the flag, and Sentry's Seer autofix that opens a pull request against the Sentry issue the revert exposed. The two hosts used to work together at LaunchDarkly (00:04:24). The auto-transcript garbles both vendor names throughout: "Launch Sharkly," "Launch Darkley," "launch directly," and "century" for Sentry; Seer appears as "Seir," "SER," "Sierra," and "Sears." Quotes below keep the transcript spelling. - **The claim.** Agents now generate code faster than humans can review it, so the review step can't be the safety net. The hosts' answer is to move safety into runtime: every risky change ships dark behind a flag, exposure ramps by cohort, deterministic statistics compare the exposed cohort against the held-out cohort on four error metrics, and the flag flips off on its own when the exposed cohort degrades. Agents do the setup chores (classify, flag, instrument, clean up) and the aftermath (root cause, plan, fix), not the go/no-go decision. - **Why it matters.** For anyone running release testing, the mechanism separates cleanly from the products. Deploy and release are decoupled; rollback becomes a config change with no redeploy; regression detection is cohort-based rather than before/after; and the human review moves from "read every diff" to "review the proposed fix on Monday." Each of those is a practice you can adopt with or without these vendors. - **Key stakeholders.** Sentry (error monitoring, Seer autofix, Sentry MCP server), LaunchDarkly (flags, guarded releases, the agent factory in early access), engineering teams absorbing AI-generated code volume, on-call engineers, and the platform teams at hosts who decide whether flags and error telemetry are first-class in their release pipelines. - **Risks and tradeoffs.** This is two vendors demoing their own products on a self-built hack project. The demo bug is a deliberately named flag ("enable broken signin," 00:28:33) hit by a scripted load generator, not organic traffic. The 27-user, 89%-error revert (00:34:58) shows the mechanism works on a bug every user hits; it says nothing about subtle regressions. The factory is described as a prototype with a public repo and an early-access link, with the full release "in a month or two" (00:20:02). The repo, EAP, and privacy links were dropped in the livestream chat, not the video description, so they're unavailable from this source. One factual inconsistency: Seer is described as "not its own model" at 00:41:23 and as running on "custom models that we have built and trained" at 00:47:09. - **Bottom line.** The durable ideas are the four-metric guarded release, exposed-versus-held-out comparison, and "deterministic core, agents at the edges" (00:40:41). The product specifics are a prototype. Take the pattern, verify the products separately. ## 📖 Main content **The framing: review doesn't scale.** Tom opens with the "software factory" vocabulary (dark software factories, AI DLC or ADLC, 00:04:24) and a claim that the ratio of AI-generated to human-written code is exploding. He shows a slide but reads no number from it, so the figure is unsourced in this source. His conclusion is that "we cannot review our way out of this explosion of code" (00:05:49). He grounds the human cost in two secondhand references: an Ethan Mollick line (transcript "Mollik") about automating the wrong half of the job when agents make every interesting decision and leave people the approvals, exceptions, and failures (00:05:49), and Charity Majors' "it's my loop, I'm not in the loop" (00:06:31). A third reference, a Facebook engineer's social post about being "never more stressed or more bored," is an unsourced anecdote. **What LaunchDarkly does, in three minutes.** A GitHub Pages site carries a parent flag ("new UI") gating a rebrand, with child flags for a footer, a hero, and an AI chatbot named Finn (00:12:54). Tom does a deliberate anti-pattern: a big-bang release to 100% with no reviewers, no schedule, no metrics, no comment (00:12:54). Finn is broken. Because Finn has its own child flag, he turns it off without losing the rest of the rebrand (00:13:38). The lesson he draws is decoupling deploy from release, trunk-based development, and the line that the broken 10% shouldn't sacrifice the working 90% (00:14:21). A screen-share mistake at 00:14:21 (the presenters shared a media window, so viewers saw a tiny frame) forces a replay; Peter later reframes that as "live feedback" and an argument for using data instead (00:22:51). **Guarded releases: the deterministic middle.** The bottleneck after a bad release is how fast a human can connect the dots across five simultaneous changes and dashboards (00:15:45). The proposed fix is that LaunchDarkly and Sentry both speak OpenTelemetry and have an integration, so Sentry telemetry gets enriched with LaunchDarkly audience data. Four metrics are named: 500 response rate, Core Web Vitals, HTTP error rate, and non-HTTP exception rate (00:16:29). If any degrades for the exposed cohort, the flag flips back. Tom is explicit that this layer is "deterministic stuff, this is not throwing agents at it" (00:15:45). At 00:34:14 he adds the detail that makes the mechanism work: the comparison is people who got variation 1 versus people who got control, so a generic Sentry error metric still isolates the culprit because exposure, not the endpoint, is the discriminating variable. **The agent factory: chores, not decisions.** LaunchDarkly's prototype chains agents into a graph (00:17:54). Tom's stated design goal is to lift agent configuration out of YAML, Markdown, and skill files into a "configuration plane" that sits beside the features, so you can swap models and prompts under the same runtime control as the features themselves. The stages named at 00:08:40 and 00:18:37: classify whether a change needs a flag and give it a risk score; implement the flag in code and in LaunchDarkly; judge the flagging agent's output; verify the flag key matches in both places; author telemetry; review; and, later, open PRs that remove stale flags after a healthy release (00:19:20). It runs as a GitHub Action in the demo but "could run basically anywhere" (00:18:37). A public repo and an early-access link were shared in chat. **The end-to-end demo.** Peter's Coffee Lab app (grind calculator, coffee directory, equipment profile, Postgres backend, hosted on Railway) gets a sign-in button in the navbar. He pushes only the React component (00:27:08). The GitHub Action runs the factory, which adds a flag key, a `release-flags` JSON schema (flag key, target variation, notes) authored by Claude, a Sentry span, and logging inside the component (00:27:51). He merges. In LaunchDarkly he uses the "ship it" flow: select the two auto-authored Sentry metrics, enable auto rollback, roll out to 50%, leave a comment (00:29:16 to 00:29:58). The framing is a Friday 5 p.m. release before a happy hour. Typing a username into the new sign-in crashes the page (00:30:40). A script hammers the endpoint to simulate users. A second hack service, Beacon, also on Railway, listens for guarded-release events and logs the release state (00:32:08). At about 27 unique users with 89% hitting the error, LaunchDarkly reverts the flag; Beacon sees the revert, finds the matching Sentry issue, and starts a Seer autofix (00:34:58 to 00:35:42). **Seer: root cause to PR, with configurable stopping points.** Sentry's new consolidated inbox shows whether an issue is diagnosed, assigned, and how far the fix has progressed (00:35:42). Seer runs root cause analysis, plans, writes the fix, and opens a PR; you can stop it at root cause, at plan, or let it go "full yolo" (00:37:07). Peter demonstrates PR iteration: he comments asking for NextAuth (transcript "next off") and Seer updates the PR's auth route (00:37:50 to 00:38:33). Seer also re-runs against failed CI checks until the PR is mergeable (00:39:16). In the Q&A Peter says Seer ingests spans, traces, error messages, session replays, breadcrumbs, and code files (00:47:09), that the RCA can be handed to Cursor, Claude Code, or GitHub Copilot as a smaller payload that keeps their context window small (00:42:06 to 00:42:48), and that it's reachable through the Sentry MCP server (00:46:24). **The thesis, stated late.** Tom's closing argument at 00:39:58 to 00:41:23 is the most useful two minutes: he says he doesn't trust agents, that regression detection uses "deterministic telemetry, deterministic stats models," and that agents surround that core on both sides (setup before, remediation after). The factory idea is blending deterministic verification, which is fast, cheap, scalable, and repeatable, with agents, which are flexible but probabilistic. **Contested and unverifiable in this source.** The AI-to-code ratio slide has no number read aloud. The demo bug is scripted and total; no partial regression is shown. Guarded-release thresholds are described as "statistical significance" (00:48:34) with no test, sample size, or significance level given. The Seer model question gets two answers. LaunchDarkly's agent runtime is a Claude model on LaunchDarkly's own Bedrock instance (00:50:02, transcript "clawed"); this is a vendor statement, not verified here. Compliance claims (BAAs for HIPAA, FedRAMP Moderate, data sovereignty instances, 00:52:12) are stated, not shown. ## 🗒️ Notes #### ⏱️ [00:00:04](https://www.youtube.com/watch?v=uyesTWSDllI&t=4) Pre-show, poll, housekeeping - Two minutes of chat warm-up. A live poll shows most attendees know Sentry and a couple are new to both products (00:00:48). - Housekeeping: recorded, Q&A open throughout, survey at the end with a random t-shirt draw (00:01:32, 00:54:21). - Agenda: introductions, software factories, orchestrating releases, what happens when things go wrong (00:02:14 to 00:02:58). #### ⏱️ [00:02:58](https://www.youtube.com/watch?v=uyesTWSDllI&t=178) Introductions - Tom Totenberg heads release automation at LaunchDarkly, from Minneapolis; years on standardizing how features get exposed to users, the last two on agents taking over parts of that. - Peter McCarron is a staff technical product marketing manager at Sentry, from Denver. Both previously worked together at LaunchDarkly (00:04:24). #### ⏱️ [00:04:24](https://www.youtube.com/watch?v=uyesTWSDllI&t=264) Software factories and the review bottleneck - Terms in circulation: software factories, dark software factories (machines running, lights off), AI DLC or ADLC. The industry hasn't settled on a name (00:05:06). - Claim: an explosion in AI-generated code relative to human code, shown on a slide; no figure read out. "We cannot review our way out" (00:05:49). - Two attributed lines and one anecdote: Ethan Mollick on automating the wrong half of the job; Charity Majors' "it's my loop"; a Facebook engineer's post about being stressed and bored reviewing "slop grenades" (00:06:31 to 00:07:15). > [!quote] [00:05:49] > leave us people with the approvals, the exceptions and the failures, then we have automated the wrong half of the job. #### ⏱️ [00:07:57](https://www.youtube.com/watch?v=uyesTWSDllI&t=477) Where the two vendors sit, and the agent stages - Both products live in "safe releases, issue remediation": signals and control points so the non-functional chores are handled (00:07:57). - The prototype rests on deterministic features: flag hygiene, Sentry tracing, logging, and error handling best practices (00:08:40). - Stages: classify whether a change needs a flag; generate the flag in LaunchDarkly and in code; verify both match; author metrics and telemetry; stage the release with an exposure plan; clean up afterward (00:08:40 to 00:09:23). - Tom notes Seer is already a shipped product and the factory agents are ones LaunchDarkly is publishing with design partners (00:09:23). #### ⏱️ [00:10:49](https://www.youtube.com/watch?v=uyesTWSDllI&t=649) LaunchDarkly basics: the big-bang release done wrong - Peter's framing: nobody is reinventing the process; the goal is to hand the boring checklist steps to automation (00:10:49). - Demo site on GitHub Pages with a QR code so attendees can watch flags flip on their phones (00:12:12). - Parent flag "new UI" gates a rebrand; child flags gate footer, hero, and the Finn chatbot. Tom releases to everyone with no reviewers, no schedule, no metrics, no comment (00:12:54). - Finn is broken. Its child flag lets him turn Finn off while keeping the rebrand (00:13:38). > [!quote] [00:14:21] > so that we can move independently and not let the 10% that's broken sacrifice the 90% that's still working. #### ⏱️ [00:15:03](https://www.youtube.com/watch?v=uyesTWSDllI&t=903) Recovery replay, and what guarded releases measure - Screen-share glitch: the media window was shared, viewers saw a tiny frame; Tom replays the flip (00:14:21 to 00:15:03). - The limiting factor after a bad release is human decision speed across five simultaneous changes (00:15:45). - Both products speak OpenTelemetry and share an integration. Metrics named: 500 response rate, Core Web Vitals, HTTP error rate, non-HTTP exception rate. Any degradation for the exposed cohort flips the flag off (00:16:29). - Works across 50 concurrent releases because Sentry telemetry is enriched with audience data, so the platform can attribute degradation to the exposed feature (00:16:29 to 00:17:11). > [!quote] [00:15:45] > But still, the limiting factor at this point is how quickly humans can make that decision, right? #### ⏱️ [00:17:54](https://www.youtube.com/watch?v=uyesTWSDllI&t=1074) The agent graph and cleanup - Agents are chained into a graph; the same runtime control that governs features governs agents, so models and prompts become swappable configuration rather than YAML and skill files (00:17:54). - Per-PR tasks: classify, risk score, implement the flag, judge the flagging agent, verify the flag key in code and in LaunchDarkly, create telemetry, review (00:18:37). - Cleanup: PRs that remove stale flags after a healthy release (00:19:20). - A public repo with the skills and graph-walking code, plus an early-access link; full release expected "in a month or two" (00:19:20 to 00:20:02). #### ⏱️ [00:20:44](https://www.youtube.com/watch?v=uyesTWSDllI&t=1244) Peter's three steps - The factory work is pre-release; production is where "the rubber meets the road" (00:21:27). - Step 1: protect releases with flags so broken code rolls back fast (00:22:10). Step 2: use data (Sentry) rather than waiting for a Reddit thread or live chat complaints (00:22:51 to 00:23:34). Step 3: let bots fix, not build: automate debugging (00:23:34 to 00:24:16). - Stack: Sentry with Seer autofix, LaunchDarkly guarded releases, Railway hosting. Cursor cloud agents were planned, swapped for Seer (00:24:59). Repeated caveat: a hack project by an engineer and a marketer. #### ⏱️ [00:24:59](https://www.youtube.com/watch?v=uyesTWSDllI&t=1499) Coffee Lab demo: from bare PR to instrumented code - Coffee Lab: grind calculator, coffee directory, equipment profile. Goal: a navbar sign-in (00:25:41 to 00:26:24). - Peter pushes only the new React component, no flag, no instrumentation (00:27:08). - A GitHub Action runs the factory: adds a flag key, Sentry metric definitions, and a `release-flags` JSON file (flag key, target variation, notes) written by Claude; inserts a Sentry span and logging into the component (00:27:51 to 00:28:33). #### ⏱️ [00:28:33](https://www.youtube.com/watch?v=uyesTWSDllI&t=1713) Ship it: configuring the guarded release - Two flags appear: "notify for beacon deployment" and "enable broken signin" (00:28:33). - Scenario: Friday 5 p.m. release, then happy hour, then Monday (00:29:16). - "Ship it" flow: pick the two auto-authored Sentry metrics, enable auto rollback (the "guarded" part), roll out to 50%, review and save with a comment (00:29:16 to 00:30:40). - Peter lands in the exposed cohort; typing his username crashes the page (00:30:40). #### ⏱️ [00:31:23](https://www.youtube.com/watch?v=uyesTWSDllI&t=1883) Simulated traffic, Beacon, auto-revert - A script hammers the endpoint to generate unique users (00:31:23). - Beacon, a hack service on Railway beside the app and its Postgres table, listens for guarded-release events and logs "running" state (00:32:08 to 00:32:50). - Tom clarifies the measurement: variation 1 versus control, health of exposed people rather than health of services (00:33:32 to 00:34:14). - At about 27 users with 89% hitting the error, LaunchDarkly reverts. Beacon sees the revert, finds the Sentry issue, starts an autofix (00:34:58 to 00:35:42). > [!quote] [00:34:14] > We're looking at the health of individual people um which is kind of cool. #### ⏱️ [00:35:42](https://www.youtube.com/watch?v=uyesTWSDllI&t=2142) Seer inbox and PR iteration - New consolidated inbox: diagnosed, assigned, fix status per issue (00:35:42 to 00:36:24). - Stopping points: root cause only, plan only, or full autofix to PR (00:36:24 to 00:37:07). - PR iteration: a comment asking for NextAuth makes Seer revise the PR's auth route (00:37:50 to 00:38:33). Seer re-runs on failed CI checks until mergeable (00:39:16). - Peter's summary: end to end with no manual intervention; "all I did was turn the flag on" (00:39:16). #### ⏱️ [00:39:58](https://www.youtube.com/watch?v=uyesTWSDllI&t=2398) Deterministic core, agents at the edges - Tom: agents are impressive and he doesn't trust them. Regression detection is deterministic telemetry and stats models; agents wrap setup and remediation (00:39:58 to 00:40:41). > [!quote] [00:40:41] > blend our good old deterministic verification, which is fast and cheap and scalable and repeatable with these agents which are flexible but probabilistic #### ⏱️ [00:41:23](https://www.youtube.com/watch?v=uyesTWSDllI&t=2483) Q&A: what Seer is, why not one error - Seer described as "not its own model," a feature inside the platform that runs RCA, plans, and proposes a fix; can hand off to Cursor, Claude Code, or Copilot with a smaller payload (00:41:23 to 00:42:48). - Revert wasn't triggered by one error: "30 something users." Exposure units can be users, requests, tenants, or organizations; measure exposed versus held-out (00:42:48 to 00:43:30). - PR iteration "just released": ingests comments and failed CI (00:44:56). > [!quote] [00:43:30] > and then measure the effect on that audience versus the ones who are held out. #### ⏱️ [00:44:56](https://www.youtube.com/watch?v=uyesTWSDllI&t=2696) Recap - Instrument releases (LaunchDarkly factory), release with guardrails (guarded releases fed by Sentry), let robots clean up (Seer autofix) (00:44:56 to 00:45:39). - Seer RCA is reachable from the Sentry MCP server for bespoke workflows (00:46:24). #### ⏱️ [00:46:24](https://www.youtube.com/watch?v=uyesTWSDllI&t=2784) Q&A: Seer's model, ELI5 responsibilities - Second answer on Seer: an in-house solution on custom models trained on Sentry data; ingests spans, traces, error messages, session replays, breadcrumbs, code files (00:47:09). - Tom's ELI5: Sentry is measurement, LaunchDarkly is control; together they form feedback loops; a kill switch fires globally in a fraction of a second once statistical significance is reached, no redeploy (00:47:51 to 00:49:19). - Peter's addition: Sentry is the "context engine," LaunchDarkly the "action point" (00:49:19). #### ⏱️ [00:50:02](https://www.youtube.com/watch?v=uyesTWSDllI&t=3002) LaunchDarkly's three layers, privacy answers - Layer 1: opinionated agent runtime, a Claude model on LaunchDarkly's Bedrock instance. Layer 2: bring your own key on that infrastructure. Layer 3: APIs, CLIs, MCP for fully custom agents (00:50:02 to 00:50:45). - Sentry: doesn't train on customer data by default, documents subprocessors, lets you disable generative AI features (00:50:45 to 00:51:28). - LaunchDarkly: BAAs for HIPAA, FedRAMP Moderate, data sovereignty instances (00:52:12). #### ⏱️ [00:52:57](https://www.youtube.com/watch?v=uyesTWSDllI&t=3177) Close - Restated: Seer RCA can feed your own agent via the MCP server (00:52:57). - Off-topic chat (a D&D weekend), survey reminder, t-shirt draw, recording reminder, thanks (00:53:39 to 00:55:04). ## 📚 References The transcript itself contains no links; the following come from the video description or are primary sources for named entities. The factory repo, early-access form, and privacy documentation links were posted in the livestream chat and don't appear in the description, so they're not recoverable from this source. - [Protecting releases, fixing bugs with Sentry & LaunchDarkly](https://www.youtube.com/watch?v=uyesTWSDllI): the source video, 55:10, uploaded 2026-09-24 by Sentry. - [Sentry](https://sentry.io): error monitoring vendor; publisher. - [Sentry Seer](https://docs.sentry.io/product/ai-in-sentry/seer/): the AI debugger demonstrated (root cause, plan, autofix, PR iteration). The transcript renders it "Seir," "SER," "Sierra," and "Sears." - [Sentry MCP server](https://docs.sentry.io/product/sentry-mcp/): named at 00:46:24 and 00:47:09 as the route to Seer RCA from external agents. - [LaunchDarkly](https://launchdarkly.com): feature flag and runtime control vendor. - [LaunchDarkly guarded releases](https://docs.launchdarkly.com/home/releases/guarded-releases): the auto-rollback mechanism at 00:29:58. - [LaunchDarkly Sentry integration](https://docs.launchdarkly.com/integrations/sentry): the OpenTelemetry-based link named at 00:15:45. - [OpenTelemetry](https://opentelemetry.io): the shared telemetry standard both vendors cite. - [Railway](https://railway.com): hosting for the Coffee Lab app and the Beacon service. - [NextAuth.js](https://next-auth.js.org): the auth library Peter asks Seer to add at 00:37:50 (transcript "next off"). - [Ethan Mollick](https://www.oneusefulthing.org): Wharton professor quoted at 00:05:49 (transcript "Mollik"). - [Charity Majors](https://charity.wtf): Honeycomb cofounder, "it's my loop" line at 00:06:31. - [FedRAMP](https://www.fedramp.gov): the authorization level LaunchDarkly claims at 00:52:12. ## 🗣️ Paraphrase Tom's opening argument is that AI made the code-review step the new bottleneck. If agents write most of the code and humans approve it, the humans get the boring half of the job and the interesting decisions still happen somewhere they can't see. His proposal is to stop treating the diff review as the safety net. Ship every risky change dark behind a flag, expose it to a slice of users, and let a statistics engine compare the people who got the change against the people who didn't. When the exposed group's error rate rises, the flag turns itself off in under a second, no redeploy. Agents earn their keep doing the setup nobody enjoys: deciding whether a change needs a flag, creating it in both code and the flag service, adding the spans and logs, and later deleting the flag once the release is healthy. Peter's half picks up at the moment the flag flips off. Sentry has the error, the trace, the breadcrumbs, and the code. Seer reads all of it, writes a root cause, plans a fix, opens a PR, and keeps revising the PR against comments and failed CI until it's mergeable. You choose how far it goes: stop at root cause, stop at plan, or let it run. The pitch is a Friday 5 p.m. release you don't think about until Monday, when a PR is waiting. Both hosts repeat that the detection layer is deterministic and the agents sit around it, and that this is a hack project with a public repo, not a shipped integration. ## 📰 Newsletter **Flags in, fires out.** Sentry and LaunchDarkly spent 55 minutes this week showing a hack project that's worth watching for the pattern even if you never buy either product. A PR ships with no flag and no instrumentation; a GitHub Action adds both; a guarded release rolls it to 50%; when 27 users hit a crash (89% of the exposed cohort), the flag reverts itself, and Sentry's Seer opens a fix PR that keeps revising against comments and CI. The line to remember comes from LaunchDarkly's Tom Totenberg at 00:40:41: deterministic verification is fast, cheap, and repeatable; agents are flexible but probabilistic; put the agents around the deterministic core, not inside it. For WordPress folks, the transferable ideas are cohort comparison and decoupled rollback. Core has no runtime flag system, but hosts do, and the four metrics named (500 rate, Core Web Vitals, HTTP error rate, exception rate) are exactly what a release-day sweep already watches by hand. Caveats: the demo bug is scripted and total, the factory is in early access, and the repo link lived in the chat rather than the description. Links: [the video](https://www.youtube.com/watch?v=uyesTWSDllI), [guarded releases docs](https://docs.launchdarkly.com/home/releases/guarded-releases), [Seer docs](https://docs.sentry.io/product/ai-in-sentry/seer/). ## ✅ Action items #### DevRel - Developer advocacy: build a talk-sized demo that shows exposed-versus-held-out comparison on a WordPress plugin update, using a flag and one error metric, and publish the cohort sizes and the threshold that fired. - Education: run one office-hours session on "deploy is not release," using the 00:13:38 Finn example as the warm-up and a plugin activation as the WordPress equivalent. - Developer experience: add a "ship dark, ramp by cohort, revert by flag" section to any release quickstart, with the four metrics from 00:16:29 as the default watch list. - Developer marketing: don't repeat the AI-to-code ratio claim (00:05:06) in any material; the slide had no number read aloud. Cite the mechanism instead. - Community management: when the factory repo link surfaces, pin it with the "prototype, early access" caveat the hosts gave at 00:24:16. - Internal/external: ask Sentry which answer on Seer's model is correct (00:41:23 versus 00:47:09) before describing it in any doc. #### OSPO - Policy and governance: decide whether agent-authored flags and instrumentation in a PR count as human-reviewed code, and write the rule before a factory-style action lands in a repo. - License compliance: the factory writes files into your repo (a `release-flags` JSON, Sentry spans). Check the prototype repo's license before adopting any of its skills or graph code. - Security management: an agent with write access to both the codebase and the flag service is a new privilege set. Threat-model the GitHub Action token and the LaunchDarkly API key it holds. - Software inventory: flags created by agents need owners and expiry, or the cleanup agent at 00:19:20 becomes the only thing preventing flag debt. Track them in the inventory. - Developer support: publish guidance on which Seer stopping point (root cause, plan, full PR) is the default for each repo class. - Community engagement: both vendors' MCP servers are the extension points named. Track whether upstream projects standardize on MCP for this. - Strategic alignment: set a KPI for mean time from bad-release detection to exposure-zero; the demo implies sub-second once the threshold fires, and your baseline is probably minutes. #### Community - Maintainers: pilot cohort-based error comparison on one plugin release using an existing error tracker, and record whether the exposed cohort is separable in your telemetry today. - Contributors: document the human checkpoint. If an autofix PR arrives, who reviews it and against what criteria. - Sponsors: if a vendor sponsors a flag or monitoring pilot, publish the cost, the number of reverts, and the false-revert count. - Ecosystem: run one cross-project drill where a scripted bad release triggers an auto-revert, so the runbook is tested before a real one. #### WordPress - Core: no direct action. Core has no runtime flag service; the nearest relevance is the Beta and RC ramp, which is a manual cohort ramp. - Core Performance: Core Web Vitals is one of the four guarded-release metrics (00:16:29). Compare that watch list with the Performance team's release-day dashboards. - Core AI: the "deterministic core, agents at the edges" rule at 00:40:41 is a guardrail worth writing into any responsible-AI doc: agents propose, deterministic checks decide. - **Core Program Team:** note the vocabulary drift (software factory, ADLC, dark factory) so cross-team notes don't adopt vendor terms without definitions. - Mobile: no direct action from this source; nearest relevance is staged rollout in app stores, which is a cohort ramp without a kill switch. - Playground: a blueprint that toggles a plugin feature by a query flag and shows an error count per cohort would teach the mechanism in one page. - Community: no direct action from this source. - Training: a short lesson on deploy-versus-release using the Finn example at 00:13:38. - Marketing: no direct action from this source. - Support: the 00:22:51 point about learning of breakage from a Reddit thread is the support team's daily reality. Evaluate whether error-tracker alerts reach support before users do. - Design: no direct action from this source. - Documentation: if hosting docs describe feature flags, add the exposed-versus-held-out framing from 00:34:14; before/after comparison is the common misconception. - Photos: no direct action from this source. - TV: no direct action from this source; nearest relevance is caption quality, which this video's auto-captions fail on every product name. - Polyglots: no direct action from this source. - Accessibility: no direct action from this source. - Meta: WordPress.org deploys are already flag-gated in places; document which metrics gate them. - CLI: a `wp` command that reports error rate for a cohort would be the WordPress half of this demo. Scope it. - Hosting: the strongest fit. The four metrics, cohort ramp, and flag revert map onto how a host ships platform changes. Compare current practice against 00:16:29 to 00:17:11 and 00:29:16 to 00:29:58. - Tide: no direct action from this source. - Openverse: no direct action from this source. - Themes: no direct action from this source. - Plugins: plugin auto-updates are a 100% big-bang release with no cohort and no metric. That's the anti-pattern at 00:12:54. Worth a written comparison. #### The WPOCC - Programs: no direct action from this source; nearest relevance is that The WP Open Community Collective could fund a maintainer stipend for release-safety tooling. - Projects: a release-team clinic on flag-gated rollouts would reuse the demo's structure. - Funding experiments: if a grant funds a monitoring pilot, require the revert count and false-revert count as reported outcomes. ### FAIR - Technical independence: both halves of the demo are hosted SaaS. A FAIR-aligned version needs a self-hostable flag store and error tracker; note the dependency before adopting the pattern. - Policies and covenants: auto-revert decisions should be logged with the metric, the cohort sizes, and the threshold, so a revert is auditable after the fact. - Productized governance artifacts: add "rollback mechanism" and "exposure control" rows to operator scorecards. ## 🧪 Hypotheses 1. On a WordPress site behind a host-level flag, an auto-revert triggered by a cohort error-rate comparison reduces time-to-exposure-zero from the current manual median (likely 10 to 60 minutes) to under 60 seconds after the threshold fires. 2. Cohort-based comparison (exposed versus held-out) isolates the culprit release with at least 90% precision when five releases ramp concurrently, where before/after comparison drops below 50% precision. 3. The auto-revert fires on total failures like the demo's crash within 30 exposed users, but for a regression affecting under 5% of the exposed cohort it needs more than 1,000 exposed users to reach the platform's significance threshold. 4. Agent-authored flag wrapping matches human-authored flag placement (same component boundary, same default variation) in at least 80% of PRs on a mid-size React codebase, and below 60% on a PHP codebase with template-level rendering. 5. Seer-style autofix PRs are merged without human edits in fewer than 40% of cases for crashes with a single stack trace, and fewer than 10% for regressions that surface only as a metric shift. 6. Flag debt grows unless cleanup is automated: repos using an agent factory without the stale-flag cleanup step accumulate more than 20% dead flags within 90 days. ## 📊 Methodology - **H1.** Instrument one host-level flag on a staging fleet of 50 sites. Inject a scripted failure at 50% exposure, record the timestamp the error metric crosses threshold and the timestamp the flag reads off for all sites. Compare against 10 recent manual rollbacks from incident logs. - **H2.** Run five synthetic releases concurrently on a test app, one carrying a fault. Repeat 100 times with randomized fault assignment. Score whether cohort comparison and before/after comparison each name the right release. - **H3.** Vary fault rate within the exposed cohort (100%, 20%, 5%, 1%) and record exposed-user count at revert. Report the platform's stated significance rule alongside; if it isn't documented, that's the finding. - **H4.** Take 50 merged PRs with human-placed flags from a React repo and 50 from a PHP repo. Strip the flags, run the factory, and have two reviewers score placement agreement blind. - **H5.** Over 60 days, log every autofix PR by issue type (crash with stack trace versus metric-only regression), record whether it merged unedited, edited, or closed. - **H6.** Compare flag counts and last-evaluated dates across repos with and without the cleanup agent at day 0, 45, and 90. - **Controls.** Same codebase, same load generator, same thresholds across arms. Pre-register the precision and merge-rate cutoffs before looking. ## ❓ Questions - What statistical test and significance level does the guarded release use, and what's the minimum exposed cohort before it can fire (00:48:34 says "statistical significance" with no detail)? - Which is true about Seer: "not its own model" (00:41:23) or "custom models that we have built and trained" (00:47:09)? - How does the factory decide a change needs a flag, and what's its false-negative rate on changes that later cause incidents? - Does the flag-key verification step (00:18:37) catch a key that exists in both places but gates the wrong code path? - What does "enrich Sentry telemetry with audience data" (00:16:29) look like in the SDK: a tag per flag evaluation, a context object, or something else? - When the auto-revert fires, does the held-out cohort's error rate get re-checked to rule out a site-wide incident that isn't the release's fault? - What happens to a Seer PR when the underlying flag is turned back on before the fix merges? - Is the "configuration plane" for agents (00:17:54) versioned and auditable the way flag changes are? ## 🔮 Future research - **False-revert rate.** No one in the session names the cost of a revert that fires on noise. A study of guarded releases in production, counting reverts later judged unnecessary, would size the tradeoff the demo skips. - **PHP and template-rendered apps.** Every demo artifact is React or a static page. Whether agent flag placement works when the feature boundary is a PHP template partial or a WordPress hook is untested here. - **Cohort telemetry in WordPress.** WordPress error tracking usually lacks a stable user or session identity for logged-out traffic. How to define an exposure unit (request, session, site) for cohort comparison on a CMS is an open design problem. - **Agent-written instrumentation quality.** The factory adds spans and logs. Whether those spans carry the right attributes to isolate a regression, versus adding noise, deserves measurement. - **Flag debt with and without cleanup agents.** H6 above, but longitudinal and across organizations. - **Statistical method disclosure.** Vendor guarded-release products describe "significance" without publishing the test. Independent documentation of what each platform does would help buyers compare. ## 💡 Implications - **Practice.** Release testing shifts from "did the RC pass the checklist" to "can the RC be withdrawn from a cohort in seconds." A team that can't answer the second question hasn't finished the release plan, regardless of how good the checklist is. - **Policy.** Auto-revert is a production change made by software. Change-control policies that require a human approver for every production change need a carve-out with logged evidence (metric, cohort, threshold), or the mechanism can't run. - **Security.** The factory holds write access to code and to the flag service; Seer holds read access to code, traces, and session replays and write access to PRs. That's a wide privilege surface for an automated actor. Least privilege and token scoping matter more here than in a read-only monitoring integration. - **Education.** The deploy/release split and the exposed/held-out comparison are teachable in one example each (00:13:38 and 00:34:14). Both belong in any release-engineering curriculum before tool-specific content. - **Funding models.** Flags and error tracking are recurring SaaS costs priced per seat or per event. Open source projects and small hosts need self-hostable equivalents or the pattern stays enterprise-only. - **Vendor literacy.** A demo where the flag is named "enable broken signin" and the traffic is scripted proves the pipeline is wired up. It doesn't prove detection sensitivity. Read vendor demos for what the fault injection can and can't show. ## 🧠 Learning styles assessment - **Reading level estimate.** Flesch-Kincaid Grade Level on the transcript body: about grade 7 to 8, Flesch Reading Ease near 70. The speech is conversational with short clauses; the comprehension load comes from undefined terms (guarded release, OpenTelemetry, RCA, MCP, ADLC), which push the effective audience to working engineers. #### 👀 Visual learners - Present: live UI for both products, the flag toggles, the Sentry inbox, PR diffs, and a Beacon log stream; slides for the factory stages and the three-step recap. - Improve: no single diagram shows the full loop (PR, factory, flag, cohort, metric, revert, autofix). The screen-share error at 00:14:21 cost a segment. Zoom into the flag configuration and the metric selection at 00:29:16; they're the two screens that carry the mechanism. #### 🔊 Aural learners - Present: two-voice conversation with Q&A, which breaks up monologue. - Improve: Peter says he talks fast and suggests 0.75x playback (00:54:21). A 3-minute audio recap of the three steps would serve commuters. Auto-captions garble every product name, so audio-only listeners have an edge over caption readers. #### 🗣️ Verbal learners - Present: the ELI5 at 00:47:51 (Sentry is measurement, LaunchDarkly is control) and the "context engine" versus "action point" framing at 00:49:19 are clean verbal handles. - Improve: guarded release, OpenTelemetry, RCA, MCP, BYOK, and FedRAMP go undefined. One clause each. #### 🧩 Kinesthetic learners - Present: a QR code lets attendees watch flags flip on their phones (00:12:12), and a public repo is shared. - Improve: the repo link is in the chat, not the description, so replay viewers can't follow. Put it in the description with a five-step quickstart. #### 🧠 Logical learners - Present: the deterministic-core argument at 00:39:58 to 00:41:23 is a clear structure: fast, cheap, repeatable checks in the middle; flexible, probabilistic agents around them. - Improve: no numbers for the significance test, the minimum cohort, or the AI-to-code ratio slide. Show one worked threshold calculation. #### 🧑‍🤝‍🧑 Social learners - Present: live poll, chat prompts, Q&A answered by name (James, Sean, Peter), survey with a prize. - Improve: no post-event community space is named. A discussion thread on the repo would keep the conversation going. #### 🧘‍♀️ Solitary learners - Present: the recording and the recap slide let a solo viewer skim. - Improve: no chapters, so self-paced navigation means scrubbing a 55-minute video. A reflection prompt ("what's your current time from bad release to exposure zero?") would turn the recap into an exercise. #### 🌿 Naturalistic learners - Present: nothing explicit; the "factory" metaphor is industrial, not ecological. - Improve: flag debt and stale instrumentation are lifecycle problems. Framing the cleanup agent at 00:19:20 as lifecycle management would land with this group. ## 📊 Accessibility assessment #### Screen reader semantics - Present: YouTube supplies landmarks and player controls; the auto-transcript is machine-readable. - Improve: the factory-stage slide (00:08:40), the metrics list, the flag UI, and the Beacon logs carry content the speakers only partly voice. WCAG 2.2 AA 1.1.1 (Non-text Content) and 1.2.5 (Audio Description, Prerecorded) aren't met for those screens. #### Keyboard navigation - Present: the player is keyboard operable with visible focus. - Improve: nothing in the video adds barriers. 2.1.1 and 2.4.7 pass via the platform. Without chapters, keyboard users can't jump by topic; adding chapters helps 2.4.5 (Multiple Ways) in spirit. #### Contrast and readability - Present: unverifiable from the transcript; dashboards and code diffs on screen. - Improve: the 00:14:21 incident shows viewers got a tiny frame for a full segment. Code diffs at default resolution risk 1.4.3 (Contrast Minimum) failures on mobile. Zoom during diffs and flag configuration. #### Text resizing and reflow - Present: player and captions scale. - Improve: burned-in slide text and the QR code don't reflow. 1.4.4 (Resize Text) and 1.4.10 (Reflow) fail for in-frame text. Speak the URL the QR code points to. #### Media and captions - Present: auto-captions exist and produced this transcript. - Improve: captions render LaunchDarkly six different ways ("Launch Sharkly," "Launch Darkley," "launchly," "launch directly"), Sentry as "century," Seer as "Seir," "Sierra," and "Sears," MCP as "MTP," HIPAA as "HIPPA," and NextAuth as "next off." That's a 1.2.2 (Captions, Prerecorded) failure in substance: captions exist but are wrong on every proper noun. Human-corrected captions are the fix. #### Forms and interactions - Present: the poll, the chat Q&A, and the survey happened in the livestream platform, not in the video. - Improve: links shared in chat should also be in the description as selectable text; 3.3.2 (Labels or Instructions) applies to the survey and EAP forms those links point to. #### Standards reference - Primary gaps against WCAG 2.2 AA: **1.2.2** (inaccurate captions on all product names), **1.2.5** (no audio description for slides, UI, and logs), **1.1.1** (slide content conveyed visually only), **1.4.4 / 1.4.10** (burned-in text and QR code). No chapters in a 55-minute recording is the biggest navigation gap and the cheapest to fix. ## 🌍 DEIB assessment #### Cultural breadth - Present: two US-based presenters (Denver, Minneapolis); a nod to international attendees "past their coffee time" (00:00:48). Compliance references are US-centric (HIPAA, FedRAMP) with a mention of data sovereignty instances (00:52:12). - Improve: name one non-US regulatory frame (GDPR is the obvious one) alongside FedRAMP, and pick a demo scenario that isn't a Friday happy hour. #### Inclusive language - Present: no exclusionary terms; second person throughout; "folks" as the collective noun. - Improve: "full yolo" (00:37:07), "slop grenades" (00:06:31), and "robots" for agents are casual and fine, but "the robots work for us" framing could be swapped for a literal description in written follow-ups. #### Neurodiversity support - Present: a stated agenda at 00:02:14, a three-step structure at 00:22:10, and a recap at 00:44:56 give predictable scaffolding. - Improve: no chapters, a screen-share failure mid-demo, and rapid speech (self-acknowledged at 00:54:21) raise processing load. Chapters and corrected captions are the two fixes. #### Representation - Present: two men from two vendors; questioners named James, Sean, and Peter. - Improve: no customer, no independent practitioner, no on-call engineer voice. One outside voice describing a real incident would improve both representation and evidence. #### Psychological safety - Present: the hosts joke about their own mistakes (the screen share, the unreviewed flag flip) rather than anyone else's. Tom's "I don't trust them" about agents (00:39:58) gives skeptical viewers permission to be skeptical. - Improve: the "don't tell my boss" line at 00:12:54 is a joke, but the anti-pattern it describes (unreviewed production changes) is a blameless-culture topic worth a serious sentence. ## ✅ Logical validity #### Strong reasoning - **Deductive, 00:13:38 to 00:14:21.** If each component is gated by its own flag, then turning off one component's flag leaves the others untouched. Valid, and demonstrated live. Clarify by naming the precondition: the child flags must gate independent code paths. - **Causal, 00:15:45 to 00:17:11.** Human decision speed bounds recovery time; automating the decision removes that bound. The mechanism (metric threshold triggers flag off) is stated and demonstrated at 00:34:58. Strengthen by stating the false-positive cost. - **Design inference, 00:33:32 to 00:34:14.** A generic error metric can't name the failing endpoint, but exposure cohort can name the failing release. That's a correct observation about which variable discriminates. Clarify by explaining how the cohort label reaches Sentry. - **Explicit epistemic split, 00:39:58 to 00:41:23.** Deterministic checks for the decision, probabilistic agents for setup and remediation. The argument names the property that justifies each assignment (repeatability versus flexibility). This is the session's strongest reasoning. - **Concession against interest, 00:24:16 and 00:42:48.** Repeated "hack project" caveats, and the admission that the revert was scripted across 30-plus users rather than organic. These raise the credibility of the rest. #### Fallacies - **Unsupported quantification, 00:05:06.** "a genuine explosion of the eyes to code ratio" with "good data to back this up," but no number, source, or date is spoken. *Fix:* read the figure and its source aloud, or drop the slide. - **Anecdotal evidence, 00:06:31.** A remembered social post from an unnamed Facebook engineer stands in for evidence that review load is harming engineers. *Fix:* cite a survey or drop to "one example I saw." - **Demonstration by rigged case, 00:28:33 to 00:34:58.** A flag named "enable broken signin," a crash on the first keystroke, and a load script produce an 89% error rate. This proves the wiring, not detection sensitivity, yet the recap at 00:45:39 treats it as proof the approach works. *Fix:* state what the demo does and doesn't show, or add a partial-failure run. - **Inconsistent claim, 00:41:23 versus 00:47:09.** Seer is "not its own model on its own... a feature inside the platform," then "runs on some custom models that we have built and trained." Both can be partly true (a feature built on custom models), but as stated they conflict. *Fix:* one sentence that says which components are in-house models and which are orchestration. - **Appeal to convenience, 00:29:16 and 00:35:42.** The Friday 5 p.m. release and the Monday-morning PR are motivating stories, not arguments; they assume the auto-revert never misfires and the fix is right. *Fix:* pair the story with the revert precision and PR merge rate. - **Undisclosed interest, throughout.** Two vendors demo their own products, and the hosts are former colleagues (00:04:24). The session is honest about being a Sentry event, but never says the integration and factory are things they sell. *Fix:* a one-line disclosure. #### Tone of voice - Conversational, self-deprecating, and fast, with running jokes about each other's demo mistakes. Peter is the emcee and marketer voice ("let's go break some stuff," 00:24:16); Tom is the engineering voice, and his register turns serious exactly where the argument gets strongest (00:39:58 to 00:41:23, "I don't trust them"). Superlatives are rare; caveats are frequent ("hack project" appears at 00:08:40, 00:24:16, 00:32:08). The Q&A answers on privacy (00:50:45 to 00:52:12) are the most scripted-sounding segment. Reading rule for this source: trust the mechanism explanations, verify every product-status claim. ## 🛡️ DARVO analysis Not applicable to this source. DARVO (Deny, Attack, Reverse Victim and Offender) describes a response to accusation, and this is a vendor demo with no accuser. The nearest challenge is an audience question at 00:42:48 asking whether the revert fired on a single error, which Peter answers directly by stating the cohort size and the scripted simulation. For completeness: **Deny**: nothing is denied; the hosts volunteer their own mistakes (screen share at 00:14:21, unreviewed flag flip at 00:12:54) and prototype status. **Attack**: no competitor or critic is disparaged; the only critical framing is aimed at unreviewed AI code volume in general. **Reverse victim/offender**: no such move appears. Emotional cues are humor and enthusiasm, not defensiveness. ## 🎯 Conclusion The mechanism this session teaches stands on its own: ship dark behind a flag, ramp exposure by cohort, compare the exposed cohort against the held-out cohort on a small fixed set of error metrics, and let a deterministic rule revert the flag when the exposed cohort degrades. Agents belong on both sides of that core (setup before, remediation after), not inside it. Tom states that rule at 00:40:41 and it's the line worth keeping. The product specifics are a prototype. The factory is in early access with a full release promised in a month or two, the demo bug is scripted and total, the significance test is unnamed, and Seer's architecture gets two different descriptions. The repo and privacy links live in a chat log the recording doesn't carry. **Next steps for readers:** measure your current time from bad-release detection to exposure zero, because that number is what this pattern attacks. Check whether your error telemetry can carry an exposure cohort label today; if it can't, cohort comparison is impossible regardless of tooling. If you run WordPress at a host, map the four metrics at 00:16:29 onto your release-day watch list and note which ones you already have. Then verify the vendor claims (statistical method, Seer's model, factory availability) against the vendors' own docs before repeating any of them. ## 🔄 Refactor Appendix — Protecting releases, fixing bugs with Sentry & LaunchDarkly > Density: 2–5 Literature atoms (H3 pad), 0–3 Permanent candidates (H4 pad). > Emitted: 5 Literature, 2 Permanent. Resonance filter applied. The AI-to-code ratio slide, > the Facebook engineer anecdote, and the compliance claims were dropped as unevidenced or promotional. > > Konik claim-header rules: present tense, declarative, ≤10 words, searchable, unique. > > Atomicity: one declarative sentence per atom. "and"/"also" → split. > > Tag pool: #bmf · #addendum · #xref · #articleseed · #storystem · #share · #aphorisms ### 📚 Literature Split Pad #### [H1] Per-component flags let 10% break without losing 90% > [!quote] [00:14:21] > not let the 10% that's broken sacrifice the 90% that's still working. **Context & use**: - 00:14:21, right after turning off the Finn chatbot's child flag while the rest of the rebrand stays live. - The precondition matters: child flags must gate independent code paths. A single parent flag gives you all-or-nothing rollback, which is the plugin auto-update situation in WordPress. **Tags**: #xref <!-- atomicity: 1 idea ✓ --> #### [H2] Human decision speed bounds rollback time > [!quote] [00:15:45] > the limiting factor at this point is how quickly humans can make that decision **Context & use**: - 00:15:45, transitioning from the manual flag flip to guarded releases. - This is the problem statement the whole session answers. Recovery time equals detection time plus decision time plus action time; flags make action near-zero, so decision becomes the bottleneck to automate. **Tags**: #xref <!-- atomicity: 1 idea ✓ --> #### [H3] Cohort health replaces service health as the release signal > [!quote] [00:34:14] > the health of individual services. We're looking at the health of individual people **Context & use**: - 00:34:14, Tom's aside while the load script runs, explaining why a generic error metric still isolates the culprit. - The discriminating variable is exposure, not endpoint. For WordPress, this raises the design question of what the exposure unit is for logged-out traffic (request, session, or site). **Tags**: #xref #articleseed <!-- atomicity: 1 idea ✓ --> #### [H4] Deterministic checks decide; probabilistic agents surround them > [!quote] [00:40:41] > blend our good old deterministic verification, which is fast and cheap and scalable and repeatable with these agents which are flexible but probabilistic **Context & use**: - 00:40:41, Tom's closing argument after saying he doesn't trust agents. - The property pairing is the reusable part: assign the go/no-go to whatever is repeatable, assign the chores and the investigation to whatever is flexible. This generalizes beyond releases to any automation design. **Tags**: #xref #aphorisms #share <!-- atomicity: 1 idea ✓ --> #### [H5] Regression detection compares exposed against held-out > [!quote] [00:43:30] > and then measure the effect on that audience versus the ones who are held out. **Context & use**: - 00:43:30, answering whether the revert fired on one error; Tom generalizes the exposure unit to users, requests, tenants, or organizations. - Before/after comparison confounds the release with everything else that changed in the window. Held-out comparison controls for that, which is why the "one bad connection" false positive doesn't fire. **Tags**: #xref <!-- atomicity: 1 idea ✓ --> ### 🧠 Atomic Split Pad #### [P1] Flag-gated exposure turns rollback into a config change **Claim** (one sentence, Courtney's voice): A change that ships dark behind its own flag can be withdrawn without a redeploy, so recovery time is bounded by detection, not by build and deploy. **Evidence pointer**: [[#H1]] · [[#H2]] **Scope & conditions**: Holds when the flag gates an independent code path and the flag service evaluates at runtime. Breaks for schema migrations, data backfills, and anything the flag can't un-run. Also breaks when one parent flag gates many components, which reintroduces all-or-nothing rollback. **Links**: - [[Deploy is not release]] — because extends - [[WordPress plugin auto-updates]] — because contrasts on exposure control, auto-updates are a 100% big-bang release with no kill switch - [[Release-day sweep]] — because applies-in RC testing at a host <!-- atomicity: 1 idea ✓ --> #### [P2] Release regressions need a held-out cohort, not a before/after window **Claim** (one sentence, Courtney's voice): An error metric attributes a regression to a release only when the telemetry carries who was exposed, so the comparison is exposed versus held-out rather than this hour versus last hour. **Evidence pointer**: [[#H3]] · [[#H5]] **Scope & conditions**: Requires an exposure unit that's stable enough to label (user, session, request, tenant) and enough exposed traffic to reach whatever significance rule the platform uses. Breaks for logged-out CMS traffic with no session identity, and for changes exposed to 100% at once. **Links**: - [[Error monitoring for WordPress]] — because applies-in defining the exposure unit - [[A/B testing statistics]] — because extends - [[Before/after incident analysis]] — because contrasts on confounding <!-- atomicity: 1 idea ✓ --> ### ✏️ Fleeting Queue - [F1] Find the LaunchDarkly factory prototype repo and EAP link; both were shared in the livestream chat, not the description: https://www.youtube.com/watch?v=uyesTWSDllI #fleeting #inbox - [F2] Resolve the Seer contradiction: "not its own model" at 00:41:23 versus "custom models we built and trained" at 00:47:09, against Sentry's Seer docs #fleeting #inbox - [F3] Open question: what is the exposure unit for cohort comparison on logged-out WordPress traffic, request or session or site? The 00:34:14 and 00:43:30 segments are the concrete framing #fleeting #inbox #articleseed - [F4] Map the four guarded-release metrics (500 rate, Core Web Vitals, HTTP error rate, non-HTTP exception rate, 00:16:29) onto the current release-party watch list and note gaps #fleeting #inbox - [F5] Article seed: "Deterministic core, agents at the edges" as a design rule for any automation, with this session's release pipeline as the worked example #fleeting #inbox #articleseed - [F6] Check whether the guarded-release docs publish the statistical test and minimum cohort; 00:48:34 says only "statistical significance" #fleeting #inbox ### → Things 3 (max 2 actions) - [ ] Measure current time from bad-release detection to exposure zero on one recent incident and record the number — release testing baseline, 20 min #things - [ ] Check whether Sentry events from the WordPress fleet can carry an exposure-cohort tag today, and note what's missing if not — error monitoring, 30 min #things <!-- deep-read:sections end --> 📄 **Full text:** [[Full Document Contents/Articles/Protecting Releases, Fixing Bugs With Sentry & LaunchDarkly.md|Protecting Releases, Fixing Bugs With Sentry & LaunchDarkly by Sentry]] ## Highlights - [00:14:21] of um uh types of of good practices that allow us to do uh things like trunkbased development and decoupling of processes between teams so that we can move independently and not let the 10% that's broken sacrifice the 90% that's still working. Now this is the wrong way to do it. Uh what we really should do is to start actually automating some of this. Um, and so in order to do that, part of what I'll show you is the end result of what we will be building toward, and that is to make sure that number one, we are doing some slow rollouts of um Oh, can't see the screen. Oh, no. Should I go back? Is that because we were still sharing the uh the the media? >> Sorry, we were still sharing the resource. That's on me. Uh, so we switch this back. So, maybe just back up just [00:15:03] like a little bit. Maybe flip your flag back over to uh the original the original state. >> You got it. Here, tell you what. So, I'm going to go in reverse. I'm not going to try the backwards talking thing that we always see in the movie in the movies, but uh instead, we'll just kind of, you know, go through uh and you can take my word for it. If you do want to scan the QR code as I as I go back through this and super speed time, we can go right ahead. Um but there you go. Uh it's basically as fast as that, right? To be able to uh to to say, "Hey, I'm doing my release. I'm recovering from my release. I'm making sure that each individual component that is wrapped in flags uh is actually behaving the the way that we want to." that for that fast uh uh fast feedback loops, right? This fast recovery. So, once again, I've done the the first release. I'm going to turn off ([View Highlight](https://read.readwise.io/read/01m3can51f8pfh51f2rfdnzzwq)) - Note: [H1] Per-component flags let 10% break without losing 90% - Tags: atom/literature/h1 - [00:15:45] that broken chatbot. If you're really quick on the trigger and able to get that QR code in, you'll still see that uh that that chatbot go away. There you go. It's gone basically as soon as I can make that decision. But still, the limiting factor at this point is how quickly humans can make that decision, right? which is uh rellyant upon being able to to go connect the dots in Sentry for example and say okay I've got all my monitoring dashboards up we release like five different things which one of those caused the problem and so the the beautiful part here is that by bringing the two uh technologies together here which launch sharkly speaks a lot of open telemetry centry speaks open telemetry we also have a good integration we can actually automatically measure this now to be clear this is still uh deterministic [00:16:29] stuff this is not throwing agents at it. We'll get to the agents in just a moment. Um, the idea here is that we should be able to measure all sorts of things and have and as we gradually expose this new functionality to a greater and greater audience. We should be able to measure how that's going for those people who are exposed. In this case, I've got stuff like 500 response rate, core web vitals, HTTP error rate, and nonHTTP exception rate. And if any of those start degrading, we'll flip that back off. And that doesn't matter if that's uh the only rollout you've got going on or if this is one of 50 releases you've got going on today. We'll still be able to figure out uh because we are enriching all that that good century telemetry with the audience data and then analyzing that and figuring out based on which audiences are exposed to which features uh which ([View Highlight](https://read.readwise.io/read/01m3can928048btn885pcmvymf)) - Note: [H2] Human decision speed bounds rollback time - Tags: atom/literature/h2 - [00:34:14] failing. Um but the specificity comes down to who was exposed because again we want to make sure that this is working for the humans. Um and so being able to uh to see hey did the people who were exposed to this versus the people who weren't um you know did they get a degraded experience in some way like that's the problem that we're solving. So it's an interesting like layer on top of some of the traditional uh observability too right which is like covering a lot of ground getting deep into like the health of individual services. We're looking at the health of individual people um which is kind of cool. Um I I like that approach. >> Yeah. And so the whole thing here is that again we're counting these users and ah perfect timing. We just like synced that up perfectly. We've hit about 27 users and the nice thing is [00:34:58] that Sentry is like firing off and telling us that this is totally bad like like something really wrong is happening right now and Launch Darkly's platform is smart enough to know okay this is well exceeded our threshold. We should like every like the majority of users 89% of our users are hitting this like horrible flag. uh we know that this feature is completely aborted and we should get rid of it. So if I go back over to beacon over here, one of the things you're going to notice is that we have this nice little beacon release and it said that hey this got reverted. So there was a problem here. And so part of the logic of what we built in here is that we said okay well if that's the case let's go find the corresponding sentry issue and let's kick off an autofix. So not only did we prevent the broken code from going out to our users, ([View Highlight](https://read.readwise.io/read/01m3candetts9m8fym1932nbvp)) - Note: [H3] Cohort health replaces service health as the release signal - Tags: atom/literature/h3 - [00:39:58] cool, right? It's very cool. Um, and so, so something that I that I really want to drill into here because like this is fantastic. I I it's been so cool to jam on this with you, but I I feel like it's really uh it is simultaneously amazing what agents can do right now and I don't trust them, right? Like w with all sorts of stuff and I want to be able to verify all that. So what both Sentry and Launch Darkly are doing here are essentially um making sure that you can use all of the good underlying foundational stuff. Uh like this is not throwing agents at a body of data and asking them to tell us whether something got better or worse. This is deterministic. telemetry deterministic stats models to detect regressions. Um, and then we're like [00:40:41] surrounding that with agents both to help with the setup of the of the the monitoring and and all the setup of the flags as well as then helping us like investigate and fix the underlying issue because the the deterministic stuff in between there has already contained or or even completely, you know, put out that fire so that it's not exposed to end users anymore, right? So, um, I I part of what excites me about the overall idea of a software factory is that as an industry, I think we're really starting to figure out good ways to be able to blend our good old deterministic verification, which is fast and cheap and scalable and repeatable with these agents which are flexible but probabilistic and and we got to know when and where to use both ([View Highlight](https://read.readwise.io/read/01m3cankjmdhsnntwbnbp4hnfc)) - Note: [H4] Deterministic checks decide; probabilistic agents surround them - Tags: atom/literature/h3 - [00:42:48] and letting it burn through a bunch of tokens like trying to figure out what exactly went wrong, uh, Seir can handle that information for you and then we just have like a much smaller single payload that goes out into your context window that keeps your context window small and then helps you resolve it. And as far as the question of like the only one error, yeah, we didn't we didn't just do that based off one trigger. Uh, I don't know if you saw we were on the launch darkly screen. We had like 30 something users that got impacted by our terrible bug. It was clearly very broken. It was all the same error, but we were simulating and uh I think that's an important thing for guarded releases and Tom can keep me honest here. The whole idea is that you do want to spread it across multiple individual users because you don't want to run into a circumstance where you like have it shut off because one person just happened to [00:43:30] have a bad internet connection >> and and and at risk of overcomplicating it, it's one uh uh multiple units, right? So like we tend to think of rolling out user by user, but it could be like an invisible underlying API upgrade that's theoretically invisible, I should say. And so we're wanting to measure request by request and randomize and analyze that way or tenant by tenant or organization by organization, something like that. But yeah, let's let's uh roll it out to a progressively increasing audience or whatever that audience is and then measure the effect on that audience versus the ones who are held out. So that's how it works. Pretty cool stuff. So, I will say uh James, you know, happy like one of the things that is nice about Sierra is that we can automate the PR process. Uh we're always trying to make it better. So, appreciate any feedback. Feel free to follow up ([View Highlight](https://read.readwise.io/read/01m3canst9nvyq010p4s028fhh)) - Note: [H5] Regression detection compares exposed against held-out - Tags: atom/literature/h3