“Trial by fire” is destroying your incident response

“Trial by fire” is destroying your incident response 🔗 **Source:** [“Trial by fire” is destroying your incident response](https://leaddev.com/technical-direction/trial-by-fire-is-destroying-your-incident-response) > [!note]+ Document Notes > tags: [observability, incident-response, learning] > author: Kristen Foster-Marks > source: LeadDev > reference: [https://leaddev.com/technical-direction/trial-by-fire-is-destroying-your-incident-response] > date created: 2025-11-19 00:00 > > # 🏷️ Title: “Trial by fire” is destroying your incident response >  ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - Include links cited in or relevant to the article. ## 📝 Summary - What happened: The author argues that learning observability during outages (“trial by fire”) impairs learning and incident response. “Anxiety is the enemy of learning.” - Why it matters: Panic hampers judgment; pre-crisis training builds intuition for signals, metrics, and tooling. - Key stakeholders: Developers, SREs, team leads, platform/ops, executives accountable for reliability. - Risks and tradeoffs: Overreliance on ad-hoc learning; cognitive overload during incidents; time investment for proactive training. - Bottom line: Build calm, communal, guided observability practice before incidents to improve resilience of systems and people. ## 📖 Main Content - Core arguments: - Outages are the worst time to learn observability; anxiety impedes learning and decision-making. - Structured, low-stakes, communal training boosts confidence and skill acquisition. - Proactive practice provides ~80% of needed proficiency, reducing cognitive load during incidents. - Self-directed learning ability (SDLA) varies; guided programs help close gaps equitably. - Context/definitions: - Observability: using logs, metrics, traces, and signals to infer internal system state. - Chaos engineering: deliberate fault injection to surface weaknesses before outages; analogous human practice is deliberate observability drills. - Uncertainties: - Exact measurement of “80%” proficiency is qualitative. - Optimal cadence and format for different org sizes/tools may vary. ### 🗒️ Notes - “I only used [observability tools] when something was broken in production,” leading to anxiety and poor learning. - Learning under duress (war rooms, ticking clocks) overwhelms dashboards and discourages admitting gaps. - Calm, communal, exploratory workshops led to safer questions, new dashboard exploration, and shared mishap stories. - Proactive practice builds intuition for which signals to check and where to find them; reduces cognitive load in crises. - Chaos engineering is a system-level analogue; human-level equivalent is focused observability learning. - SDLA is uneven; barriers are individual, cultural, and institutional—guided training ensures coverage. - It’s about uptime and about people: confidence, self-efficacy, resilience. ## 📚 References - Principles of chaos engineering overview: https://principlesofchaos.org - Self-directed learning (background): https://en.wikipedia.org/wiki/Self-directed_learning - Psychological safety in engineering teams (context): https://rework.withgoogle.com/print/guides/5721312655835136 ## 🗣️ Paraphrase - Paragraph A: Learning observability during incidents backfires because stress impairs learning and judgment; teams should practice in low-pressure settings. - Paragraph B: Structured, communal training and guided exploration build intuition and reduce panic, making real incidents faster and calmer to resolve. ## 📰 Newsletter - This week: “Trial by fire” hurts incident response. Invest in calm, hands-on observability drills before alarms. Pair dashboards with guided labs, and borrow ideas from [chaos engineering principles](https://principlesofchaos.org). Resource picks: SDLA primer, psychological safety checklists, and incident-prep runbook templates. Ship safer; sleep better. ## ✅ Action Items #### DevRel - Launch a 2-day observability workshop with tool-specific labs and sandbox incidents; collect pre/post confidence scores. - Publish quickstarts for “first 15 minutes of any incident” per stack; add dashboards + queries. - Office hours for alert tuning; sample demos on traces-to-root-cause flows. #### OSPO - Add contribution guides for observability plugins/exporters; define review cadence for telemetry changes. - Establish vuln intake and coordinated disclosure for telemetry components; sign artifacts. - Track SBOM freshness for observability agents; owners + SLOs. #### Community - Maintainer clinics on incident retros; share dashboard presets. - First-issue rotations to instrument missing metrics; pairing sessions. - Sponsor outcome-based drills; publish impact reports. #### WordPress - Core: perf/error budgets and incident triage checklists. - Core Performance: dashboards for TTFB/LCP; profiling plans. - Core AI: evaluation sets and guardrails for AI features telemetry. - Core Program Team: quarterly game-days; retros with learning goals. - Training: lesson plans on logs/metrics/traces; cert pilot. - Documentation: examples-first observability patterns and style guide. - Hosting: cross-host reliability telemetry and escalation paths. #### The WPOCC - Microgrants for incident drills; stipends for observability maintainers. - Roadmaps for governance and disclosure. ### FAIR - Independence: signed telemetry schemas; provenance checks. - Policies: transparency on funding, sunset clauses for tools. - Governance artifacts: charters, operator agreements, scorecards. ## 🧪 Hypotheses - Teams with pre-crisis observability training reduce MTTR by ≥25%. - Psychological safety correlates with higher incident learning outcomes. - Guided labs improve first-signal selection accuracy by ≥30% vs ad-hoc. - Regular game-days decrease paging volume variability over 90 days. - SDLA variability predicts uneven on-call performance without guidance. ### 📊 Methodology - Quasi-experiment across teams: baseline MTTR, introduce training, track MTTR and first-signal correctness. - Surveys: psychological safety (Edmondson scale) vs incident outcomes. - A/B: playbook-only vs playbook+guided lab; measure detection-to-diagnosis time. - Time-series on alert volumes/false positives pre/post tuning sessions. ### ❓ Questions - What minimum viable curriculum yields the largest MTTR drop? - How often should teams run drills for durable skill? - Which metrics/signals most reliably localize faults per architecture? ### 🔮 Future Research - Tool-agnostic observability skills taxonomy. - SDLA assessment tailored to engineering contexts. - Longitudinal impact of psychological safety on incident KPIs. ### 💡 Implications - Practice: schedule recurring, calm observability labs. - Policy: allocate learning time in reliability OKRs. - Security: include security telemetry in drills. - Education: build laddered curricula for logs/metrics/traces. - Funding: budget for game-days and coaching. ### 🧠 Learning Styles Assessment - Reading level: Flesch-Kincaid Grade ≈ 9–10 (estimate). #### 👀 Visual learners - Present: dashboards/metrics references. - Improve: system/trace flow diagrams, before/after visuals. #### 🔊 Aural learners - Improve: 3–5 minute audio summary; narrated lab guides. #### 🗣️ Verbal learners - Present: clear definitions, analogies (map/lost). - Improve: glossary and term contrasts. #### 🧩 Kinesthetic learners - Present: exploratory labs. - Improve: step-by-step drills and checklists. #### 🧠 Logical learners - Present: causal framing (anxiety → impaired learning). - Improve: decision trees for first-diagnosis steps. #### 🧑‍🤝‍🧑 Social learners - Present: communal training. - Improve: cohort prompts and peer reviews. #### 🧘‍♀️ Solitary learners - Improve: self-check quizzes and reflection prompts. #### 🌿 Naturalistic learners - Improve: lifecycle/eco analogies for resilience. ### 📊 Accessibility Assessment #### Screen reader semantics - Improve: ensure headings/landmarks and descriptive alt text for dashboard images. #### Keyboard navigation - Improve: predictable focus order in docs and labs. #### Contrast and readability - Present: aim for 4.5:1; verify. #### Text resizing and reflow - Improve: responsive lab UIs; reduced motion options. #### Media and captions - Improve: captions/transcripts for any videos; no autoplay. #### Forms and interactions - Improve: clear labels, inline validation, generous targets. #### Standards reference - Map to WCAG 2.2 AA for documentation and lab portals. ### 🌍 DEIB Assessment - Cultural breadth: add multi-region case studies and translations. - Inclusive language: maintain jargon definitions; add plain-language summaries. - Neurodiversity: chunk content; consistent patterns; icon+text labels. - Representation: diversify examples and imagery; descriptive alt text. - Psychological safety: publish code of conduct for drills; clear escalation. ### ✅ Logical Validity #### Strong reasoning - Causal: anxiety impairs learning → pre-crisis training improves performance; supported by learning science. - Analogical: chaos engineering ↔ human observability drills; clarifies proactive approach. #### Fallacies - Potential hasty generalization on “80% proficiency”; clarify as experiential estimate with data if available. #### Tone of voice - Supportive, reflective, pedagogy-oriented; emphasizes confidence and resilience. ### 🛡️ DARVO Analysis - No denial/attack/reversal patterns; the piece is instructional and empathetic, not adversarial. ## 🎯 Conclusion - Bottom line: Don’t wait for outages to teach observability. Create calm, guided, communal practice so real incidents are faster, safer, and less stressful. - Next steps: Schedule quarterly game-days, build tool-specific labs and first-15-minute playbooks, measure MTTR and confidence pre/post, and support SDLA with guided programs. ## 🧱 Refactor Appendix (append after 🎯 Conclusion) ### 📚 Literature Split Pad - Purpose: Stage Konik CEE units that become per-source Literature notes. - Split level: H3. - Include verbatim quotes under 25 words and a [view highlight] link. ### [H1] 5–10-word present-tense claim > [!quote] [“Trial by fire” is destroying your incident response](https://leaddev.com/technical-direction/trial-by-fire-is-destroying-your-incident-response) by [[Kristen Foster-Marks]] via [[LeadDev]]. Published 2025-11-18. Accessed 2025-11-19. > “Anxiety is the enemy of learning.” Two lines: context and use. Cite data if relevant. [view highlight](https://leaddev.com/technical-direction/trial-by-fire-is-destroying-your-incident-response#H1) #xref #articleseed ID:rw_[MISSING] ### [H2] 5–10-word present-tense claim > [!quote] [“Trial by fire” is destroying your incident response](https://leaddev.com/technical-direction/trial-by-fire-is-destroying-your-incident-response) > “Pre-crisis practice builds resilience for incidents.” Two lines: context and use. [view highlight](https://leaddev.com/technical-direction/trial-by-fire-is-destroying-your-incident-response#H2) #xref ID:rw_[MISSING] --- ### 🔖 Atomic Split Pad (Zettel Candidates) - Purpose: Promote reusable claims into Atomic notes. - Split level: H4. - Each item: one idea, present tense, specific, testable. #### [Z1] Calm training improves incident MTTR - Within product teams running quarterly labs. - Pointer: workshop outcomes/MTTR dashboard. - Links: [[Observability]] [[Psychological-Safety]] #### [Z2] First-signal choice predicts diagnosis speed - Within microservices with tracing enabled. - Pointer: on-call metrics; game-day logs. - Links: [[Incident-Response]] [[Tracing]] --- ### ✏️ Fleeting Queue (Extract by Selection) - [F1] Draft first-15-minutes incident playbook — context/https://leaddev.com/technical-direction/trial-by-fire-is-destroying-your-incident-response #fleeting #inbox - [F2] Plan quarterly observability game-day — context/https://principlesofchaos.org #fleeting #inbox - [F3] Create SDLA-aware onboarding path — context/https://en.wikipedia.org/wiki/Self-directed_learning #fleeting #inbox ---