We Halved Our Continuous Integration Pipeline  🔗 **Source:** [We Halved Our Continuous Integration Pipeline](https://leaddev.com/technical-direction/we-halved-our-continuous-integration-pipeline) > [!note]+ Document Notes > --- > tags: [continuous-integration, developer-experience, platform-engineering, test-flakiness, engineering-metrics] > author: "[MISSING]" > source: LeadDev > reference: https://leaddev.com/technical-direction/we-halved-our-continuous-integration-pipeline > date created: 2026-09-07 > --- > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the author. > > Do not quote or attribute any other line to the named author. > # 🏷️ Title: We halved our continuous integration pipeline > ## 🧭 Table of Contents > - [[#📝 Summary]] > - [[#📖 Main Content]] > - [[#🗒️ Notes]] > - [[#📚 References]] > - [[#🗣️ Paraphrase]] > - [[#📰 Newsletter]] > - [[#✅ Action Items]] > - [[#🧪 Hypotheses]] > - [[#📊 Methodology]] > - [[#❓ Questions]] > - [[#🔮 Future Research]] > - [[#💡 Implications]] > - [[#🧠 Learning Styles Assessment]] > - [[#📊 Accessibility Assessment]] > - [[#🌍 DEIB Assessment]] > - [[#✅ Logical Validity]] > - [[#🛡️ DARVO Analysis]] > - [[#🎯 Conclusion]] > - [[#🔄 Refactor Appendix — We halved our continuous integration pipeline]] > ## 🔗 References > - Source: LeadDev, "We halved our continuous integration pipeline" — https://leaddev.com/technical-direction/we-halved-our-continuous-integration-pipeline > - Byline not present in the captured content; the piece is written in first person by an unnamed engineer (references "our three-person engineering platform team"), so author is [MISSING] rather than invented. > - 9 outbound links checked via `paras2.py --links` (confirmed present, not asserting "uncited"): a self-link to the article, plus links to LeadDev's own pieces on staff engineering, CI pipeline troubleshooting, security-as-everyone's-responsibility, LLM operator views, redesigning a broken delivery flow, AI-assisted-code prompting, standardizing codebases, and engineering budgets during a downturn. All are LeadDev's own related-reading links, not primary sources for the numbers in this piece. > ## 📝 Summary > - **What happened:** An engineering team that cut CI pipeline time from 60 to 22 minutes in a 2025 cost-justified program watched it degrade again within six months as AI-assisted development quadrupled pipeline volume against 7% more compute. By May, only ~25% of merge-train runs finished green, ~4,000 runs were canceled in one month, and the internal "build and test" developer-experience survey score hit 39/100. A three-person platform team then spent six weeks moving most end-to-end tests off the merge train, rebalancing parallel test groups by historical timing data, and tightening fail-fast, bringing runtime to 15–16 minutes with a 95% green rate. > - **Why it matters:** The piece argues CI speed is not a one-time project but a recurring cost that scales with usage, and that AI-assisted coding is a volume multiplier that breaks pipelines sized for pre-AI merge rates. > - **Key stakeholders:** the author's ~400-person company (200+ developers), the three-person platform team named as the second fix's owner, and any org treating a past CI win as permanently solved. > - **Risks and tradeoffs:** moving end-to-end tests off the merge train and onto a two-hour schedule means a failing offloaded test now blocks deployment until the owning team signs off, rather than blocking the merge itself — a tradeoff the piece names directly. > - **Bottom line:** the author's closing frame is to treat "pipeline time as a product metric" with an owner and a quarterly target (75th percentile: 15 minutes in Q3, 10 minutes by year end), rather than a project that gets closed out. > ## 📖 Main Content > The piece is a first-person postmortem structured as two build cycles around the same pipeline. > **2025: the arithmetic-driven fix.** The team converted "CI is slow" into a budget request by computing that at 45 merges and 340 pipeline runs per day, each minute of merge-train time cost roughly six engineering-hours of blocked time daily. That framing funded three changes: choosing a more consistent (if not fastest) AWS instance family, splitting the pipeline so fast checks (lint, unit tests, security scans) didn't wait on the full build, and dropping browsers from test workers that didn't need them. A quarantine mechanism automated flaky-test handling — three unrelated MR failures in half a day triggered a 24-hour quarantine, with a $100/month LLM comparing failure logs to the triggering code change. Runtime dropped from 60 to 22 minutes (64%) for a 10% infrastructure cost increase, against an estimated $65,000/month in recovered engineering time at a conservative $50/hour rate. > > [!quote] ID:rw_01m1y15g2cpx0tnsa4zsq8y1hv > > "It was in May when one of our staff engineers tossed out a number in our Slack channel that put an end to an internal debate I'd been having with myself. Roughly only 25% of the runs on our merge train were completing successfully." > **The regression.** Over the following six months the backend test suite grew from 2,338 to 4,100 tests, the frontend suite tripled, and merges per month doubled — organic growth the 2025 fixes were sized for. Then AI-assisted development decoupled pipeline volume from headcount: volume on the main repository grew more than 4x at peak while compute capacity grew only 7%. None of the 2025 mechanisms had failed; they were simply outrun. Out-of-memory and exit-137 errors appeared on node workers, cascading cancellations took out unrelated builds queued behind a failing one, and by May roughly a quarter of runs finished green, a quarter failed on unstable infrastructure, and the rest were canceled behind failures. > **2026: rework, not tuning.** The platform team's fix wasn't another round of the same optimizations — it restructured what ran on the merge train at all. End-to-end tests (the slowest, most product-realistic tests) were split into a mandatory "critical" 15% and a "deprecated" majority, the latter moved to a two-hour schedule on main rather than blocking every merge, with dashboards to confirm critical tests still caught pre-production bugs. Runtime fell from 31 to 18 minutes (42%) immediately, and the successful-to-failed/canceled ratio went from 1:3 to 25:1 on day one. Parallel test groups were then rebalanced using actual production timing data rather than an even split, cutting the longest group's time by roughly a third. Fail-fast tightened from killing a run after up to 70 failed jobs to killing it one minute after the first failure. > ## 🗒️ Notes > **Numbers worth keeping as reference points** > - 2025 program: 60 → 22 min (64% cut); infra cost +10% (~$1,680/month); ~1,300 engineering hours/month recovered, ~$65,000 valued at $50/hr and a 30–50% "blocked time is lost" discount. > - Instance selection: C7a (AMD) ~90 seconds faster on average than C7i (Intel) but with 10-minute runtime variance vs. 6 minutes for Intel — the team chose consistency over raw average speed, plus more workers, for ~12 minutes saved at 2.5% more cost. > - Regression scale: backend tests 2,338 → 4,100; frontend suite tripled; merges/month doubled organically, then pipeline volume grew >4x at peak under AI-assisted development against 7% more compute. > - Developer-experience survey: "build and test" scored 39/100, the lowest of any category that quarter. > - May crisis: merge train >30 min; ~4,000 runs canceled in the month; roughly 25% green / 25% failed on infra / 50% canceled behind a failure. > - 2026 fix, delivered by a 3-person platform team in 6 weeks: 31 → 18 min (42% cut) immediately from moving e2e tests off the train; parallel-group rebalancing cut the longest group by ~1/3 to 15–17 min; fail-fast changed from "up to 70 jobs" to "1 minute after first failure." > - Current state (as written): 15–16 min runtime, ~95% green, 130 merges/day or 22/hour capacity, <20 min median merge-to-main, 75th percentile run time 33 → 20 min, CI runner costs down 40%, merge-train-pool cost down 70%. ~20 flaky events/day still detected and auto-skipped; described as "weather," not a bug that gets fixed once. > - Forward targets: 75th percentile 15 minutes by Q3, 10 minutes by end of year, owned by a platform team against a stated budget. > **Structural pattern across both fixes** > - 2025 fix optimized [](https://leaddev.com/technical-direction/we-halved-our-continuous-integration-pipeline)[motivating engineers-07](https://leaddev.com/technical-direction/we-halved-our-continuous-integration-pipeline) **Key takeaways:** * **A 64% CI speedup didn’t hold**. Growth in tests, merges, and AI-assisted development outpaced the fix within months. * Offloading the right tests beats chasing raw speed. Splitting critical from deprecated end-to-end tests cut runtime and flipped the pass/fail ratio almost overnight. * CI speed needs an **owner and a target**, not a **one-time fix**. Treating it as a recurring product metric is what kept the gains from disappearing again. It was in May when one of our [staff engineers](https://leaddev.com/career-development/the-reality-of-being-a-staff-engineer) tossed out a number in our Slack channel that put an end to an internal debate I’d been having with myself. Roughly only 25% of the runs on our merge train were completing successfully. A majority were failing due to flaky tests or got canceled since another run that was ahead of them failed. The queue had 50+ runs at its peak level, and some other developer was in it for close to two hours and still was 14 behind in the queue. In our software company, with a total of 400 people, more than 200 of whom are developers, shipping a change had become a gamble. The problem here is that the same pipeline was actually our success story for the year in 2025, where we’d brought down the time of the pipeline from an hour to 22 minutes. Here is what happened after that, and how I realize now that [continuous integration (CI)](https://leaddev.com/technical-direction/how-identify-and-fix-ci-pipeline-problems) speed and stability is the bill you keep paying. #### The decisions we made that led to the first 64% cut Arithmetic became the cornerstone of the 2025 program since it allowed getting the budget approved. Working at 45 merges and 340 pipeline runs per day, every minute of merge-train time represented roughly six engineering hours of blocked time daily. Arithmetic helped us to turn “CI is slow” into a purchase order and made clear our first step – biggest saving first and cheap fixes concurrently. Our first savings would be achieved by using the right infrastructure and, we consciously decided to go for the slower solution. We calculated runtimes of ten to 20 runs on three different Amazon Web Services (AWS) instances: C5, C7a, and C7i. C7a, an AMD-based instance family, finished about 90 seconds faster on average than C7i, but its runtimes varied across a ten-minute range, compared with six minutes for Intel. As a result, we selected the more consistent Intel family of instances and increased the number of workers, which saved us approximately 12 minutes per run for 2.5% more costs. No single change made the difference between an hour and 22 minutes, however. We split the pipeline into two phases such that linting, unit testing, and [security](https://leaddev.com/culture/why-security-everyones-responsibility) scans, which yield results quickly because most failures happen in the first eight minutes, did not make authors wait for the full build cycle. We stopped putting a browser on all test machines if it was only needed by some suites of tests, splitting them into browserless workers and saving around 0.5 GB of RAM per worker. A coverage scan would break the build if the split reduced the coverage in any way. Retrying became a practice that nobody even doubted; as such, we automated the disbelief: any test that had three unrelated merge requests fail within half a day would enter a 24-hour quarantine period, and a [large language model (LLM)](https://leaddev.com/leadership/llms-an-operators-view) costing $100 per month would analyze the error logs. It compares each failure log with the corresponding code change and flags potential flakes after the first failure instead of waiting for the third. The success rate of the merge train improved above 90% from the lowest point of 82%, while 150 wasted retries per month evaporated. The tried-and-true quick wins were not absent either – cache linters, refactor a call to a deprecated metric that cost 120 seconds down to 20 – a total of three and a half minutes saved for three and a half days of work. The bill for all of that increased by $1,680 a month for infrastructure, which is ~10%. The payoff: over 1,300 engineering hours a month, with only 30-50% of blocked hours considered lost, valued at over $65,000 even considering a low hourly rate of $50 an hour. It went from a one-hour process to a 22-minute one, which is a 64% reduction. I declared victory and moved on. That was the mistake. #### Why our pipeline slowed down again The [pipeline was not broken,](https://leaddev.com/technical-direction/how-to-redesign-a-broken-delivery-flow) and we kept succeeding through it. Over the next six months, the test suite on the backend grew from 2,338 to 4,100 tests, while the test suite on the frontend tripled in size, and merges per month doubled. Then came [AI-assisted development](https://leaddev.com/software-quality/7-prompting-strategies-to-sharpen-your-ai-assisted-code), when growth became decoupled from headcount: the pipeline volume on our main repository increased over fourfold at peak, compute capacity grew 7%. All of the mechanisms introduced in the 2025 program were operating, the stable instances, the split workers, the quarantine bot, but the workload had merely doubled around them. The “build and test” question in our quarterly developer experience survey scored a 39 out of 100 in the winter, the lowest score in any category, and the simplest chart we weren’t looking at was a straight line up, it was pipeline volume against compute capacity. By May, the merge train was more than 30 minutes. It was not the long runtime that was the issue per se, but what happened when the train failed. Not only would an occasional failing test delay a single build but cause delays for all the builds behind that one in the queue, which would be canceled and have to be rerun from the start, although the change was unrelated to the failure. Due to the out-of-memory errors and exit-137 errors on the node worker and the infrastructure not being able to handle the load properly, about 4,000 runs were canceled in May alone. The figures were obvious. About a quarter of the runs finished green. Another quarter started to fail because of unstable infrastructure, some tests started to be flaky again because of it. The rest were canceled because of being queued up behind a failing build. #### The second recovery: three engineers, six weeks, reduced costs The next improvement resulted from reworking the way we were using the system all along. We built upon the foundation that was already laid out across the organization and took three focused steps using our three-person engineering platform team in six weeks. The first step that needed to be done was that most of the end-to-end tests, i.e. the slower ones where the product is tested the way customers would test them, were no longer part of the merge train at all. This is what we established as the basis last quarter, with the product teams splitting their [end-to-end tests](https://leaddev.com/software-quality/how-standardize-codebases-across-teams) into two types – the critical ones, which comprised about 15% of all the end-to-end tests and were responsible for the scenarios that we simply could not afford to break, and the deprecated ones. The deprecated tests were slowly replaced by backend and component tests as the test coverage of our backend reached 80%. Meanwhile, the critical tests remained mandatory for the merge train, whereas the deprecated ones were moved to a schedule of running every two hours on the main branch. We had dashboards to check if the critical tests caught bugs before code hit the production. The effect was felt immediately, with an average runtime of the merge train dropping from about 31 minutes to 18 minutes, which is a 42% decrease, and with the ratio of successful merges per failed/canceled ones improving from one-to-three to 25-to-one on the first day. The tradeoff was also quite obvious – any failure of one of the offloaded tests resulted in blocking the deployment until sign-off from the corresponding team. We took another look at how the tests were distributed in parallel groups. Rather than having them equally distributed, we leveraged the production timing data to balance out the distribution and cut the time needed for the longest group of tests by about a third, getting some to 15-17 minutes. Additionally, we have improved our fail-fast strategy and now a train run automatically kills itself a minute after it fails its first job (before that it was up to 70 test jobs). As I’m writing this, the train is running at 15-16 minutes a pop for the last week, and roughly 95% of runs get completed green. It can now handle 130 merges a day or 22 an hour. The median time it takes from hitting merge until reaching the main branch is under 20 minutes. During the worst weeks of May, it would take half a day to complete the same run. Flaky-test loop, meanwhile, is now 100% automated – detect, classify, ticket, and auto-skip – but it is still capturing 20 flaky events a day because flakiness is ‘weather’ rather than a bug that can be fixed once. The bill got smaller too; the 75th percentile run time – the time three out of four runs beat – was reduced from 33 minutes to 20, while CI runner costs were reduced by 40%. Meanwhile, cancelation storms had turned the merge train pool into our biggest CI line item – more than all of our tests machines combined – and reduced its cost by 70%. LDX3 New York is live #### The future continuous integration pipeline Do the math that started off the post with today’s figures, around 60 plus merges per day and 700 runs through the pipeline per day, and the value of one minute of train time is even more than it was in 2025. The mathematics still holds true, but we had just stopped using it. That is what I will do anywhere else because I will treat pipeline time as a product metric and allocate a platform team and/or a product manager who is in charge of meeting a target and a [budget](https://leaddev.com/career-development/rethinking-your-engineering-budget-during-downturn). That is what we currently do. This team is responsible for a 75th percentile 15 minutes target in Q3, and ten minutes target by the end of the year. Do you have an owner and a number for next quarter or just another happy story from last year?, We Halved Our Continuous Integration Pipeline, [MISSING], 01m1y15g2cpx0tnsa4zsq8y1hv.