How Etcd Solved Its Knowledge Drain With Deterministic Testing  ## Metadata - Author: [[The New Stack Podcast]] - Full Title: How Etcd Solved Its Knowledge Drain With Deterministic Testing - Category: #podcasts - URL: https://share.snipd.com/episode/cefe01c5-0b9d-45e2-89b8-14e294b2b265 ## Document Notes ## Highlights - **Maintainer Turnover Broke Testing Knowledge** - As maintainers left, crucial unwritten testing knowledge drained from the etcd project. - That knowledge loss produced a release with critical reliability issues, including potential data inconsistencies after crashes. Transcript: Marek Szakowicz The project has hit some challenges related to reliability and correctness of the project. It took us three years to roll out a new release. And during that time, a lot of maintainers shifted and a lot of maintainers left the project and were replaced with new maintainers. And there was a drain of knowledge. So all the properties that could not be written into the code were lost with those people. So all the procedures, how to test, how to guarantee correctness that was done before, were not done for the new release. So we hit a project that we released a version that had multiple issues that were critical. ([Time 0:04:57](https://share.snipd.com/snip/a22c0127-085e-44f9-96a9-ecf31e27d1f8)) - **Linearizability As The Core Goal** - Etcd aimed to validate linearizability, the 'Holy Grail' guarantee for distributed systems. - Achieving that required custom failure-injection tools and teaching the community how to debug complex scenarios. Transcript: Marek Szakowicz So we invested a lot into preventing it from happening again by building our own framework inspired by Jepson to validate not only basic correctness, but also correctness of distributed System, which is a level above like etcd provides you the highest guarantee of correctness from the system so called linearizability if you make it's basically you can have distributed System that behave should behave like a single note so this is like a golden or grail of distributed systems. And validating this is a very hard problem. So we need to learn all this knowledge. We need to do our own failure injection mechanism. We needed to teach people, the community, how to debug it. ([Time 0:06:10](https://share.snipd.com/snip/09401d8c-cecc-4aeb-9632-3ae60c795c66)) - **Adopt Deterministic Simulation Testing** - Use deterministic simulation testing to make executions fully reproducible and avoid flakey, hard-to-reproduce failures. - Codify implicit properties as assertions so tests reliably catch subtle race conditions and regressions. Transcript: Marek Szakowicz It's very hard to first think you need to reproduce and prove that it's happening. Having that without deterministic means you are throwing darts and try to reproduce it, but you're basically, you need to get lucky to sometimes find some race between multiple components Or multiple logs and multiple separate processes communicating by network to find the bug. So the thermistic simulation testing allows you to linearize everything, so there will be only one execution path, and it'll always be reproducible. ([Time 0:07:51](https://share.snipd.com/snip/64feba27-1f15-4363-b10f-0224344256ec)) - **Stack Failures To Find Rare Paths** - Stack and combine failure injections in test VMs to explore rare multi-failure paths instead of single-step chaos experiments. - This massively increases the chance of finding subtle bugs and raises system reliability nines. Transcript: Marek Szakowicz Yes, this is chaos engineering, but under one VM that makes sure that you can reproduce everything. So if you think about chaos engineering, it's like, it does one action usually, like kills a process, kills application, gets a port or container, or does some network disruption, But it can do it, like, it will do a one action and then need to wait for like system to reheal, like because you cannot like take down 100% of your infrastructure at once. You need to split it into actions. But in deterministic case, you can stack multiple failures because you're still not running on production, you're doing it before you roll out your application. So by stacking the failures, you can find new paths that never happened. So instead of testing things that happen one per 1,000 hours or once in 1,000 chance, you increase it to 1 million. ([Time 0:09:58](https://share.snipd.com/snip/0fea0f74-ddd6-4972-a2b0-90f8fe37f09c)) - **Reproducing Historical Bugs Built Trust** - The team reproduced historical bugs with the new testing, confirming coverage of past failures. - That gave maintainers trust they could catch regressions and ship new releases with higher confidence. Transcript: Marek Szakowicz Like we did full like retests. So like for me, one of the main principles for our own and then later antithesis testing was we need to make sure that the system that we've built is able to reproduce all the historical Things that we knew, that we like all the lessons learned that we knew. So we started from reproducing all the historical issues and in both cases they succeeded wonderfully and we were then able to start testing the newest version. So if we know that we can find, if we know all the bugs that we had before and we know that we can find, we have a trust that those parts of the code base are covered to, that in future will not Be, we have trust that we will catch it again and prevent a regression by contributors. ([Time 0:12:14](https://share.snipd.com/snip/9737da9c-7c52-4fed-abfe-2aeba196b7ab)) - **Testing Skills Go Beyond Standard Suites** - Robust distributed-system testing sits beyond unit and end-to-end tests and requires new skills. - Tools and dashboards are essential to teach contributors how to write and run property-based robustness tests. Transcript: Marek Szakowicz Like main challenge that I'm still like having in the community is teaching people how to do this kind of tests because this is like outside of anything you could learn in university Or in most companies there are not a lot of companies that went so far to get to the same level. So it's beyond unit testing, end-to testing, or smoke testing. It's a situation where you're trying to find one million disk failures that results in your system. One random note returning I know something from one second instead of like instead of current states returning something from a couple of seconds ago or things like that so I think like That there is a lot of challenge in sharing the knowledge so I think we like having my hope is that we having the tools, the dashboards that are built just for that, can help learn people This approach. ([Time 0:13:29](https://share.snipd.com/snip/bc96c614-53e4-4898-8fa6-e28f71f10b69)) - **Use Property Tests And AI For Exploration** - Use property-based exploration to cover many execution paths rather than writing single concrete scenarios. - Consider AI to guide exploration and correlate events to pinpoint failing code lines faster. Transcript: Marek Szakowicz So AI would be interesting to use to make the exploration smarter. This is so that we not only like cover the trivial cases or we stop hitting the same case, we can like explore. And then with this exploration, we can find what things correlate, what events correlate, what not, and this even goes to like granularity of code, line of code. So it would be interesting to see if you know how exactly code trigger a bug and what it correlates with the code. Can AI make even guesses where is the bug and be able to fix it itself. Because we can pinpoint exact lines of code that are needed to trigger the bug. So that's already a lot of data, that for human is still hard to parse. ([Time 0:16:51](https://share.snipd.com/snip/c0daf7c2-ff73-4e8f-be6d-76d7e15dd76b)) - **Protect Core Changes With High-Level Assertions** - With strong robustness tests, safely revisit and extend core features like two-node clusters and witness designs. - Rely on high-level assertions to catch failure modes instead of chasing mythical 100% unit coverage. Transcript: Marek Szakowicz It's something that we haven't touched for a long time, but now we can pretty, we are pretty sure we can like start building new things. And we are talking about like building two node etcd clusters. So for anyone that runs Kubernetes, that's a big thing that you need to run always free no when you run your control plane you need to run free instances because because for high availability You need to have consensus which requires you to have much rich majority so you need odd number of nodes but with with so-called witness nodes which can be as simple like even GCS bucket That only observes as a witness what is happening in the other nodes, we can go down to only two nodes. It requires changes in consensus in all the state machines, all the layers. ([Time 0:18:12](https://share.snipd.com/snip/34b69d70-2e30-4920-bad2-c79bf7dd1d87)) - **Join The Etcd Community To Learn** - Join etcd via GitHub, the Kubernetes Slack channel, or the mailing list to contribute and learn. - Attend bi-weekly community and triage meetings and watch robustness test debugging sessions. Transcript: Marek Szakowicz So on our GitHub page, we have our contact information. So there is a link to our Slack channel, on Kubernetes Slack. There is a mailing list link that you can join, and if you join you will get an invite to the meetings. So we have a bi-weekly meeting, community meeting for the main community, we have a triage meeting, and we have even robustness tests. ([Time 0:19:54](https://share.snipd.com/snip/7ae16d78-aac8-442a-b5ce-d1913cf56eee))