We Halved Our Continuous Integration Pipeline  🔗 **Source:** [We Halved Our Continuous Integration Pipeline](https://leaddev.com/technical-direction/we-halved-our-continuous-integration-pipeline) > [!note]+ Document Notes > tags: [continuous-integration, developer-experience, platform-engineering, test-flakiness, engineering-metrics] > author: "[MISSING]" > source: LeadDev > reference: https://leaddev.com/technical-direction/we-halved-our-continuous-integration-pipeline > date created: 2026-09-07 > > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the author. > > Do not quote or attribute any other line to the named author. > > # 🏷️ Title: We halved our continuous integration pipeline > <!-- deep-read:sections start --> ## 🧭 Table of Contents - [[#📝 Summary]] - [[#📖 Main Content]] - [[#🗒️ Notes]] - [[#📚 References]] - [[#🗣️ Paraphrase]] - [[#📰 Newsletter]] - [[#✅ Action Items]] - [[#🧪 Hypotheses]] - [[#📊 Methodology]] - [[#❓ Questions]] - [[#🔮 Future Research]] - [[#💡 Implications]] - [[#🧠 Learning Styles Assessment]] - [[#📊 Accessibility Assessment]] - [[#🌍 DEIB Assessment]] - [[#✅ Logical Validity]] - [[#🛡️ DARVO Analysis]] - [[#🎯 Conclusion]] - [[#🔄 Refactor Appendix — We halved our continuous integration pipeline]] ## 🔗 References - Source: LeadDev, "We halved our continuous integration pipeline" — https://leaddev.com/technical-direction/we-halved-our-continuous-integration-pipeline - Byline not present in the captured content; the piece is written in first person by an unnamed engineer (references "our three-person engineering platform team"), so author is [MISSING] rather than invented. - 9 outbound links checked via `paras2.py --links` (confirmed present, not asserting "uncited"): a self-link to the article, plus links to LeadDev's own pieces on staff engineering, CI pipeline troubleshooting, security-as-everyone's-responsibility, LLM operator views, redesigning a broken delivery flow, AI-assisted-code prompting, standardizing codebases, and engineering budgets during a downturn. All are LeadDev's own related-reading links, not primary sources for the numbers in this piece. ## 📝 Summary - **What happened:** An engineering team that cut CI pipeline time from 60 to 22 minutes in a 2025 cost-justified program watched it degrade again within six months as AI-assisted development quadrupled pipeline volume against 7% more compute. By May, only ~25% of merge-train runs finished green, ~4,000 runs were canceled in one month, and the internal "build and test" developer-experience survey score hit 39/100. A three-person platform team then spent six weeks moving most end-to-end tests off the merge train, rebalancing parallel test groups by historical timing data, and tightening fail-fast, bringing runtime to 15–16 minutes with a 95% green rate. - **Why it matters:** The piece argues CI speed is not a one-time project but a recurring cost that scales with usage, and that AI-assisted coding is a volume multiplier that breaks pipelines sized for pre-AI merge rates. - **Key stakeholders:** the author's ~400-person company (200+ developers), the three-person platform team named as the second fix's owner, and any org treating a past CI win as permanently solved. - **Risks and tradeoffs:** moving end-to-end tests off the merge train and onto a two-hour schedule means a failing offloaded test now blocks deployment until the owning team signs off, rather than blocking the merge itself — a tradeoff the piece names directly. - **Bottom line:** the author's closing frame is to treat "pipeline time as a product metric" with an owner and a quarterly target (75th percentile: 15 minutes in Q3, 10 minutes by year end), rather than a project that gets closed out. ## 📖 Main Content The piece is a first-person postmortem structured as two build cycles around the same pipeline. **2025: the arithmetic-driven fix.** The team converted "CI is slow" into a budget request by computing that at 45 merges and 340 pipeline runs per day, each minute of merge-train time cost roughly six engineering-hours of blocked time daily. That framing funded three changes: choosing a more consistent (if not fastest) AWS instance family, splitting the pipeline so fast checks (lint, unit tests, security scans) didn't wait on the full build, and dropping browsers from test workers that didn't need them. A quarantine mechanism automated flaky-test handling — three unrelated MR failures in half a day triggered a 24-hour quarantine, with a $100/month LLM comparing failure logs to the triggering code change. Runtime dropped from 60 to 22 minutes (64%) for a 10% infrastructure cost increase, against an estimated $65,000/month in recovered engineering time at a conservative $50/hour rate. > [!quote] ID:rw_01m1y15g2cpx0tnsa4zsq8y1hv > "It was in May when one of our staff engineers tossed out a number in our Slack channel that put an end to an internal debate I'd been having with myself. Roughly only 25% of the runs on our merge train were completing successfully." **The regression.** Over the following six months the backend test suite grew from 2,338 to 4,100 tests, the frontend suite tripled, and merges per month doubled — organic growth the 2025 fixes were sized for. Then AI-assisted development decoupled pipeline volume from headcount: volume on the main repository grew more than 4x at peak while compute capacity grew only 7%. None of the 2025 mechanisms had failed; they were simply outrun. Out-of-memory and exit-137 errors appeared on node workers, cascading cancellations took out unrelated builds queued behind a failing one, and by May roughly a quarter of runs finished green, a quarter failed on unstable infrastructure, and the rest were canceled behind failures. **2026: rework, not tuning.** The platform team's fix wasn't another round of the same optimizations — it restructured what ran on the merge train at all. End-to-end tests (the slowest, most product-realistic tests) were split into a mandatory "critical" 15% and a "deprecated" majority, the latter moved to a two-hour schedule on main rather than blocking every merge, with dashboards to confirm critical tests still caught pre-production bugs. Runtime fell from 31 to 18 minutes (42%) immediately, and the successful-to-failed/canceled ratio went from 1:3 to 25:1 on day one. Parallel test groups were then rebalanced using actual production timing data rather than an even split, cutting the longest group's time by roughly a third. Fail-fast tightened from killing a run after up to 70 failed jobs to killing it one minute after the first failure. ## 🗒️ Notes **Numbers worth keeping as reference points** - 2025 program: 60 → 22 min (64% cut); infra cost +10% (~$1,680/month); ~1,300 engineering hours/month recovered, ~$65,000 valued at $50/hr and a 30–50% "blocked time is lost" discount. - Instance selection: C7a (AMD) ~90 seconds faster on average than C7i (Intel) but with 10-minute runtime variance vs. 6 minutes for Intel — the team chose consistency over raw average speed, plus more workers, for ~12 minutes saved at 2.5% more cost. - Regression scale: backend tests 2,338 → 4,100; frontend suite tripled; merges/month doubled organically, then pipeline volume grew >4x at peak under AI-assisted development against 7% more compute. - Developer-experience survey: "build and test" scored 39/100, the lowest of any category that quarter. - May crisis: merge train >30 min; ~4,000 runs canceled in the month; roughly 25% green / 25% failed on infra / 50% canceled behind a failure. - 2026 fix, delivered by a 3-person platform team in 6 weeks: 31 → 18 min (42% cut) immediately from moving e2e tests off the train; parallel-group rebalancing cut the longest group by ~1/3 to 15–17 min; fail-fast changed from "up to 70 jobs" to "1 minute after first failure." - Current state (as written): 15–16 min runtime, ~95% green, 130 merges/day or 22/hour capacity, <20 min median merge-to-main, 75th percentile run time 33 → 20 min, CI runner costs down 40%, merge-train-pool cost down 70%. ~20 flaky events/day still detected and auto-skipped; described as "weather," not a bug that gets fixed once. - Forward targets: 75th percentile 15 minutes by Q3, 10 minutes by end of year, owned by a platform team against a stated budget. **Structural pattern across both fixes** - 2025 fix optimized existing infrastructure and process (instances, phasing, quarantine, caching). - 2026 fix changed what belongs on the critical path at all (which tests run synchronously vs. on a schedule) — a reclassification, not a re-tuning. ## 📚 References - [We halved our continuous integration pipeline](https://leaddev.com/technical-direction/we-halved-our-continuous-integration-pipeline) — the source article itself. - [The reality of being a staff engineer](https://leaddev.com/career-development/the-reality-of-being-a-staff-engineer) — related reading, not cited as evidence for any claim in this piece. - [How to identify and fix CI pipeline problems](https://leaddev.com/technical-direction/how-identify-and-fix-ci-pipeline-problems) — related reading. - [Why security is everyone's responsibility](https://leaddev.com/culture/why-security-everyones-responsibility) — related reading. - [LLMs: an operator's view](https://leaddev.com/leadership/llms-an-operators-view) — related reading, relevant to the $100/month LLM flake-triage mention but not linked from that specific claim. - [How to redesign a broken delivery flow](https://leaddev.com/technical-direction/how-to-redesign-a-broken-delivery-flow) — related reading. - [7 prompting strategies to sharpen your AI-assisted code](https://leaddev.com/software-quality/7-prompting-strategies-to-sharpen-your-ai-assisted-code) — related reading, relevant to the "AI-assisted development" volume claim but not a source for it. - [How to standardize codebases across teams](https://leaddev.com/software-quality/how-standardize-codebases-across-teams) — related reading. - [Rethinking your engineering budget during a downturn](https://leaddev.com/career-development/rethinking-your-engineering-budget-during-downturn) — related reading. - No third-party or independently verifiable source backs any of the article's own numbers (percentages, dollar figures, test counts); they are the author's internal reporting. ## 🗣️ Paraphrase A team fixed their CI pipeline once in 2025, cutting it from an hour to 22 minutes by picking steadier infrastructure, splitting fast checks from slow ones, and automating flaky-test quarantine — paid for by turning minutes of blocked engineering time into a dollar figure. They declared victory. Over the next six months the test suite nearly doubled and merges doubled too, which the fixes could absorb — until AI-assisted development quadrupled pipeline volume against 7% more compute, and the same mechanisms that worked at the old scale buckled: workers ran out of memory, one failure canceled everyone queued behind it, and by May a quarter of runs succeeded, a quarter failed on infrastructure, and half were canceled. The second fix wasn't more tuning — it moved most end-to-end tests off the merge train entirely, kept only the 15% that must not break, rebalanced parallel test groups using real timing data instead of an even split, and killed a failing run within a minute instead of letting up to 70 jobs run first. The team is now down to 15–16 minutes with a 95% green rate, and the author's stated lesson is to treat pipeline time as an owned, budgeted product metric rather than a project you finish. ## 📰 Newsletter **Your CI fix has a shelf life — especially now.** A team cut their pipeline from 60 to 22 minutes in 2025 with the standard playbook: better infrastructure, split fast/slow checks, automated flaky-test quarantine. Six months later it was worse than before they started — not because anything broke, but because AI-assisted development quadrupled their pipeline volume while compute grew 7%. The fix that actually stuck wasn't another round of tuning; it was moving most end-to-end tests off the merge train's critical path entirely, keeping only the ~15% that can't be allowed to break. For plugin teams running PHPUnit and Playwright in GitHub Actions: the volume-multiplier problem applies directly if AI-assisted commits are increasing your merge frequency, and the "critical vs. deprecated e2e" split is a concrete, portable pattern for keeping Playwright runs off the blocking path without dropping coverage. ## ✅ Action Items #### DevRel Not applicable to this source. No developer-advocacy, community, or external-facing education content. #### OSPO Not applicable to this source. No licensing, governance, or SBOM content. #### Community Not applicable to this source. Internal engineering-org case study, no external contributor or maintainer dimension. #### WordPress - **Core / plugin CI (concrete, source-supported):** the article's "critical vs. deprecated end-to-end tests" split maps directly onto a Playwright suite in a WordPress plugin repo — keep a small, curated set of Playwright specs (the ones covering activation, core admin flows, and anything that would break a site) as required checks on PRs, and move broader/exploratory Playwright coverage to a scheduled workflow (e.g. every 2 hours or nightly on `main`) with a dashboard or Slack notification if it catches something a merged PR missed. - **Fail-fast for PHPUnit matrices:** the source's change from "kill after up to 70 failed jobs" to "kill 1 minute after the first failure" maps to setting `fail-fast: true` (or a tight `max-parallel`/cancel-on-first-failure equivalent) on a GitHub Actions PHP/WP version matrix, so a single incompatible combination doesn't burn the full matrix's runner minutes before failing. - **Flaky-test triage:** the source's LLM-assisted quarantine (three unrelated failures in half a day → 24-hour quarantine, log-vs-diff comparison) is a pattern worth a lightweight equivalent for a plugin's PHPUnit/Playwright suite — even a manual "3 strikes → skip and file an issue" rule captures the core idea without the $100/month tooling. - **Test-timing-based sharding:** the source's move from evenly split parallel groups to production-timing-based groups maps to using recorded per-job duration to size PHPUnit/Playwright matrix shards instead of splitting test files evenly, which is directly implementable in a GitHub Actions matrix once job timings are logged. - Core Performance / Core AI / Mobile / Playground / Community / Training / Marketing / Support / Design / Documentation / Photos / TV / Polyglots / Accessibility / Meta / CLI / Hosting / Tide / Openverse / Themes / Plugins: not applicable — the source has no content addressing any of these teams' domains. #### The WPOCC Not applicable to this source. No governance, fellowship, or funding content. ### FAIR Not applicable to this source. ## 🧪 Hypotheses 1. Pipeline volume driven by AI-assisted commits grows faster than headcount-based capacity planning accounts for, so teams using AI coding assistants will see CI capacity problems before teams that aren't, at equivalent headcount. 2. Moving non-critical end-to-end tests off the blocking merge path reduces merge-train runtime more than any single infrastructure or caching optimization, for suites where e2e tests are a large share of total runtime. 3. A quarantine mechanism that separates "flaky" from "broken" reduces wasted retries by a measurable amount (the source claims ~150/month) without materially reducing real-defect catch rate. 4. Sizing parallel test groups by historical per-test timing data rather than an even split reduces the longest-group runtime by roughly the imbalance-to-balance ratio of the original split. 5. Tightening fail-fast thresholds (fewer failed jobs tolerated before cancellation) reduces average wasted CI-minutes per failed run roughly in proportion to the threshold reduction. 6. Treating pipeline time as an owned metric with a quarterly target sustains improvements longer than a one-off optimization project, measured by whether the pipeline degrades again within a comparable window (the source's own 2025 fix regressed within ~6 months without an owner). ## 📊 Methodology The piece reports internal operational metrics (queue depth, percentages, dollar costs, test counts, survey scores) from the author's own organization, gathered informally — the triggering data point was "a staff engineer tossed out a number in Slack." No methodology section, no description of how percentages or dollar valuations were computed beyond the stated $50/hour assumption and 30–50% "blocked time is lost" discount, and no comparison group or control. All figures are self-reported and internally sourced; nothing in the piece is externally auditable. ## ❓ Questions - What specific mechanism inside "AI-assisted development" drove the 4x pipeline-volume increase — more commits per engineer, larger diffs per commit, more speculative/exploratory branches, or something else? The piece names the effect but not the cause. - How was the $65,000/month figure computed given the stated 30–50% "blocked time is lost" range — which end of that range, or an average, and does it net against the $1,680/month infrastructure increase? - Did the 24-hour quarantine + LLM triage ever mask a real (non-flaky) regression, and if so, how was that discovered? - What is the actual current headcount and merge rate, and how does the new 15-minute/10-minute target scale if AI-assisted volume keeps growing faster than compute? - Why did the "critical" e2e tests land at exactly 15% of the total — was that threshold chosen by risk assessment, by a runtime budget, or by some other criterion? ## 🔮 Future Research - A published methodology for translating "blocked engineering time" into dollar figures that other teams could reuse and audit, rather than each team inventing its own discount rate. - Longitudinal tracking of whether the 2026 fix (test reclassification) degrades on the same ~6-month timescale as the 2025 fix (infrastructure tuning) did, which would test whether structural fixes are more durable than tuning fixes. - A general framework for capacity planning that treats AI-assisted-coding volume as a first-class input alongside headcount, since the piece shows headcount-based planning failed here. - Comparable case studies from teams that never let pipeline ownership lapse, as a counterfactual to this piece's single before/after story. ## 💡 Implications - **Practice:** CI health needs a named owner and a recurring target, not a project that gets closed. The piece's own regression happened specifically because nobody owned it after the 2025 win. - **Policy/budgeting:** the arithmetic-to-budget conversion (minutes of pipeline time × engineering hours blocked × hourly rate) is a reusable technique for justifying platform-team investment, independent of this specific pipeline. - **Security:** not addressed in the source; automated LLM analysis of failure logs and code diffs implies sending potentially sensitive code/log content to a third-party model, which the piece never discusses as a risk. - **Education/onboarding:** the "critical vs. deprecated e2e" split is a transferable pattern for any team drowning in slow end-to-end suites, independent of the specific AI-volume trigger described here. - **Funding models:** the shift of the merge-train pool from "biggest CI line item" to a 70%-cheaper one shows that infrastructure spend and process redesign are not substitutes — the 2025 fix increased spend 10% for a runtime win, while the 2026 fix cut both runtime and cost simultaneously by changing scope rather than infrastructure. ## 🧠 Learning Styles Assessment - **Reading level estimate:** approximately Flesch-Kincaid grade 11–12 — long, clause-heavy sentences ("Due to the out-of-memory errors and exit-137 errors on the node worker and the infrastructure not being able to handle the load properly...") and dense technical/financial vocabulary (arithmetic-as-budget-justification, percentile runtime, quarantine periods). #### 👀 Visual learners - Present: none. No diagrams, charts, or screenshots in the captured content despite heavy numeric data that would suit a chart. - Improve: a before/after timeline chart (2025 fix → regression → 2026 fix) and a runtime-vs-volume line chart would make the central "volume outran capacity" argument immediately legible. #### 🔊 Aural learners - Present: none. - Improve: a 2–3 minute narrated summary of the two fix cycles and their numbers. #### 🗣️ Verbal learners - Present: well served — the piece is prose-heavy with defined terms (merge train, quarantine period, fail-fast) used consistently. - Improve: a short glossary block for CI-specific terms (exit-137, merge train, 75th percentile run time) would help readers unfamiliar with the vocabulary. #### 🧩 Kinesthetic learners - Present: the concrete before/after numbers double as a checklist a reader could apply to their own pipeline. - Improve: an explicit step-by-step "audit your own pipeline" checklist mirroring the two fix cycles. #### 🧠 Logical learners - Present: well served — the piece is essentially a causal chain (volume up, capacity flat → cascading failures → targeted restructuring → measured recovery) with numbers at each step. - Improve: a table mapping each intervention to its measured effect would make the causal claims easier to audit at a glance (see Logical Validity below for why this matters). #### 🧑‍🤝‍🧑 Social learners - Present: none — a solo first-person account with a rhetorical closing question to the reader. - Improve: comparable accounts from other teams, or a discussion prompt inviting readers' own numbers. #### 🧘‍♀️ Solitary learners - Present: well served — written for a reader auditing their own pipeline against this one. - Improve: none needed. #### 🌿 Naturalistic learners - Not applicable to this source. ## 📊 Accessibility Assessment #### Screen reader semantics - Present: as captured, plain paragraph text with no headings inside the body (the source's own subheadings, if any, were stripped by the `<p>` extraction that produced this file). - Improve: semantic `<h2>`/`<h3>` subheadings for each phase (2025 fix, regression, 2026 fix) would let screen-reader users navigate the two-cycle structure without reading linearly. #### Keyboard navigation - Not assessable from the captured plain-text content; depends on the live page's markup. #### Contrast and readability - Not assessable from the captured plain-text content. #### Text resizing and reflow - Not assessable from the captured plain-text content. #### Media and captions - Present: no media in the captured content. - Improve: not applicable — text-only piece. #### Forms and interactions - Not applicable to this source. #### Standards reference - Cannot map to WCAG 2.2 AA criteria beyond structural heading use (2.4.6) without access to the live page's DOM. ## 🌍 DEIB Assessment #### Cultural breadth - Narrow by design — a single company's internal CI numbers, not generalized across contexts, regions, or team sizes. #### Inclusive language - Neutral throughout. The author's gender/pronouns are not stated in the captured content; no pronouns are used to refer to them here. #### Neurodiversity support - Mixed: dense, clause-heavy sentences and heavy numeric load work against low-effort scanning, though the two-cycle before/after structure gives predictable shape once identified. - Improve: shorter sentences and a numbers-only summary table. #### Representation - The only people named are unattributed roles ("one of our staff engineers," "our three-person engineering platform team"); no individuals are credited by name. #### Psychological safety - The tone is self-critical rather than blaming — the author frames the regression as their own mistake ("I declared victory and moved on. That was the mistake.") rather than attributing it to any team or individual. ## ✅ Logical Validity #### Strong reasoning - The core causal chain — pipeline volume grew 4x while compute grew 7%, so the same mechanisms that worked at old scale failed — is a clear, quantified, falsifiable claim, not an assertion. Location: paragraph beginning "Then came AI-assisted development." - The 2026 fix's before/after pairing (31→18 min immediately upon moving e2e tests off the train, 1:3→25:1 success ratio "on the first day") ties a specific intervention to a specific, time-bound measurement, which is stronger evidence than the vaguer 2025 numbers. #### Fallacies - **Post hoc / multiple simultaneous causes.** The 2025 section changes instance family, splits the pipeline, drops browsers, adds quarantine, and fixes deprecated-metric calls all within one described program, then attributes the full 60→22 minute result to "no single change" — an honest hedge, but the piece still assigns per-change savings (12 min from instances, 3.5 min from quick wins) that don't obviously sum to the total, and no breakdown is given for the split-pipeline or quarantine contributions specifically. Fix: a full itemized breakdown, not partial. `#fallacy/composition` - **Survivorship framing.** The piece is a single success story told after the fact; it doesn't say whether other interventions were tried and failed, so the two fixes described may look more inevitable in hindsight than they were in practice. Fix: name what was tried and discarded. - **Unstated denominator.** "150 wasted retries per month evaporated" and "20 flaky events a day" are given without the total retry or event volume they're a fraction of, making it hard to judge how large the improvement actually was. Fix: state the base rate alongside the reduction. #### Tone of voice Direct, self-critical, practitioner-to-practitioner, with a closing rhetorical question aimed at the reader ("Do you have an owner and a number for next quarter or just another happy story from last year?") that functions as both a call to action and a mild challenge. ## 🛡️ DARVO Analysis Not applicable to this source. There is no accusation, denial, or victim/offender framing — the piece is a technical postmortem in which the author explicitly owns their own mistake ("I declared victory and moved on. That was the mistake.") rather than deflecting it onto any other party. ## 🎯 Conclusion The durable claim is structural, not numeric: a CI pipeline optimized for a given commit volume will degrade again when that volume changes, and AI-assisted development is now a volume multiplier that decouples pipeline load from headcount-based capacity planning. The fix that held wasn't re-tuning the 2025 mechanisms harder — it was reclassifying what belongs on the blocking path at all (15% critical e2e vs. the rest on a schedule) and sizing parallel work by real timing data instead of an even split. Practical next step for a reader auditing their own pipeline: before adding infrastructure, ask which currently-blocking checks could move to a scheduled path without losing pre-production bug coverage, and name an owner with a runtime target before declaring the fix finished. ## 🔄 Refactor Appendix — We halved our continuous integration pipeline ### 📚 Literature Atoms (H3) #### [H1] AI-assisted development quadrupled pipeline volume against 7% more compute > [!quote] ID:rw_01m1y15g2cpx0tnsa4zsq8y1hv > "the pipeline volume on our main repository increased over fourfold at peak, compute capacity grew 7%" Names the specific mechanism that broke a pipeline the team believed was already fixed: capacity planning tied to headcount didn't account for AI-assisted-coding volume growth. No breakdown of what inside "AI-assisted development" drove the 4x figure. Tags: `#xref` **Highlight**: [view highlight](readwise://01m1y15g2cpx0tnsa4zsq8y1hv) <!-- atomicity: 1 idea ✓ --> #### [H2] A three-strikes rule triggers automated quarantine and LLM-assisted flake triage > [!quote] ID:rw_01m1y15g2cpx0tnsa4zsq8y1hv > "any test that had three unrelated merge requests fail within half a day would enter a 24-hour quarantine period, and a large language model (LLM) costing $100 per month would analyze the error logs" A concrete, cheap ($100/month), portable mechanism: threshold-triggered quarantine plus automated log-vs-diff comparison to flag likely flakes before the third failure rather than after. Tags: `#xref` **Highlight**: [view highlight](readwise://01m1y15g2cpx0tnsa4zsq8y1hv) <!-- atomicity: 1 idea ✓ --> #### [H3] Reclassifying end-to-end tests cut merge-train time 42% in one day > [!quote] ID:rw_01m1y15g2cpx0tnsa4zsq8y1hv > "the average runtime of the merge train dropping from about 31 minutes to 18 minutes, which is a 42% decrease" The fix that actually held wasn't more infrastructure tuning — it was moving the slowest, least-critical tests off the blocking path entirely. Immediate, large, measured effect from a scope change rather than a performance optimization. Tags: `#xref` <!-- atomicity: 1 idea ✓ --> ### 🧠 Permanent Candidates (H4) #### [P1] A capacity fix sized to one volume driver silently expires when a different driver takes over **Claim** (one sentence, Courtney's voice): A pipeline, budget, or process tuned to absorb growth from one source (headcount) can fail without warning once a different, faster-growing source (AI-assisted commit volume) takes over, because nothing in the original fix was monitoring for a change in *what* was driving growth. **Evidence pointer**: [[#H1]] **Scope & conditions**: Applies whenever a system is capacity-planned against a proxy metric (headcount, user count, request rate) rather than the thing that actually consumes capacity; holds until someone notices the proxy has decoupled from the real driver, as happened here via a developer-experience survey score rather than a capacity alarm. **Links**: - [[3. Resources/MOCs/AI Agent Safety MOC]] — because applies-in capacity-planning-under-AI-assisted-workflows <!-- atomicity: 1 idea ✓ --> #### [P2] Reclassifying what runs on the critical path beats re-tuning what already runs there **Claim** (one sentence, Courtney's voice): When a process is failing under load, changing which steps are mandatory-and-blocking versus scheduled-and-advisory produces a larger, faster win than optimizing the mandatory steps further, because it removes work from the bottleneck instead of making the bottleneck's existing work faster. **Evidence pointer**: [[#H3]] **Scope & conditions**: Requires that some blocking work is not actually load-bearing for the outcome being protected (here, pre-production bug catch rate was preserved by dashboards checking the offloaded tests still worked) — without that verification step, offloading risks silently losing coverage. **Links**: - [[3. Resources/MOCs/AI Agent Safety MOC]] — because applies-in CI/CD pipeline design for WordPress plugin repos using GitHub Actions <!-- atomicity: 1 idea ✓ --> ### ✏️ Fleeting Queue - The piece never states what specifically inside "AI-assisted development" caused the 4x volume increase — more commits, larger diffs, more speculative branches — worth flagging next time a similar claim surfaces. `#xref` - Automated LLM analysis of failure logs against code diffs implies sending code/log content to a third-party model; the piece never raises this as a security or data-handling question. `#xref` ### → Things 3 (max 2 actions) - [ ] Check whether any WordPress plugin repo's Playwright suite could split into a small mandatory "critical" set vs. a scheduled "broader coverage" set, following the source's 15%-critical pattern. `#things` - [ ] Check whether current GitHub Actions matrices for PHPUnit use `fail-fast: true` or an equivalent early-kill setting, per the source's fail-fast tightening. `#things` <!-- deep-read:sections end --> 📄 **Full text:** [[Full Document Contents/Articles/We Halved Our Continuous Integration Pipeline.md|We Halved Our Continuous Integration Pipeline by [MISSING]]] ## Highlights - It was in May when one of our staff engineers tossed out a number in our Slack channel that put an end to an internal debate I’d been having with myself. Roughly only 25% of the runs on our merge train were completing successfully. A majority were failing due to flaky tests or got canceled since another run that was ahead of them failed. ([View Highlight](https://read.readwise.io/read/01m1y1h4j8s8xb0rr7bkrx7wqc)) - Note: H1 — the trigger number that surfaced a pipeline no one had been watching. #xref - Tags: xref - Retrying became a practice that nobody even doubted; as such, we automated the disbelief: any test that had three unrelated merge requests fail within half a day would enter a 24-hour quarantine period, and a large language model (LLM) costing $100 per month would analyze the error logs. ([View Highlight](https://read.readwise.io/read/01m1y1h72cyqdcv935fy4xyp8p)) - Note: H2 — cheap, threshold-triggered quarantine plus LLM log-vs-diff triage; portable pattern for any CI. #xref - Tags: xref - That is what I will do anywhere else because I will treat pipeline time as a product metric and allocate a platform team and/or a product manager who is in charge of meeting a target and a budget. That is what we currently do. This team is responsible for a 75th percentile 15 minutes target in Q3, and ten minutes target by the end of the year. ([View Highlight](https://read.readwise.io/read/01m1y1h9n3nat71zaq81f2jnk3)) - Note: H3 — closing lesson: pipeline health needs an owner and a recurring target, not a closed project. #xref - Tags: xref