Software Engineering in Production With Cameron Etezadi  🔗 **Source:** [Software Engineering in Production With Cameron Etezadi](https://share.snipd.com/episode/e6e16cc3-bc8f-478d-bdb6-432ce2deda4b) ## Highlights - **Vibe Coding Breaks At Production Boundaries** - Vibe coding produces syntactically correct code but often fails on real-world behavior and scale. - Cameron describes AI-generated code hard-coding a test environment variable so it passed tests but failed in production streaming data. Transcript: Cameron Etezadi I've been in this industry for 30 years, and my favorite phrase from the early days was, hey, it worked on my machine. And I think we've all been there. You three are practicing engineers and probably know this well. And I think Vibe Coding kind of takes us, takes that to the next level, to be honest. It's very easy to produce code that works, especially down a happy path where you achieved your objective long as everything is perfect in the way. It doesn't scale. It doesn't operate. It doesn't run at enterprise scale. You guys know from operational experience, day two is just as important as day one when you're putting out software for millions or hundreds of millions of end users. Vibe Coding doesn't have that yet. And maybe it'll get there, but it makes architectural mistakes that no senior engineer would ever truly make. And they get better over time, but it's not capable yet of understanding the very nuanced things you need to do to run at scale. So coding has become free. That part of it, translating English into Python or Go or Rust or pick your modern language you actually want to work in. Please don't say Java. COBOL. It does it really well, but it doesn't fix the rest of the process. The release, the operationalization, the monitoring, the metrics, there are pieces of that, but that still takes time and effort and cost. And so what we're really doing with the Vibe coding is shifting the bottleneck in the process of software delivery to different points. And this is where the process just backs up somewhere else. I had said that, you know, code that compiles is structurally correct, but it's often not functionally correct. Great example for us is we built, we built a system internally and it used AI. Even 60% now of our check-ins are running with AI somewhere in the process overall and 30% are just done without human touching it. It picked the wrong environment. It hard-coded in the code the test environment and so all the tests passed and when it went out to production, the data wasn't streaming into the database like it should have. So the output was wrong and the test didn't pick it up because the test passed because in the test environment it worked. We would have, as experienced engineers, noted this when we were writing it and said, oh, this has to be a variable. This has to pick it up from the environment. This has to pick it up from Terraform, from config file, from somewhere. ([Time 0:03:00](https://share.snipd.com/snip/7cb95ea9-ba6a-4acc-a4a0-db059fd304a8)) - **Hard-Coded Test Environment Caused Production Failure** - AI-generated check-ins in Cameron's org sometimes hard-coded the test environment so tests passed but production data didn't stream. - He notes 60% of check-ins use AI and ~30% had no human touch, illustrating the real exposure. Transcript: Cameron Etezadi Even 60% now of our check-ins are running with AI somewhere in the process overall and 30% are just done without human touching it. It picked the wrong environment. It hard-coded in the code the test environment and so all the tests passed and when it went out to production, the data wasn't streaming into the database like it should have. So the output was wrong and the test didn't pick it up because the test passed because in the test environment it worked. We would have, as experienced engineers, noted this when we were writing it and said, oh, this has to be a variable. This has to pick it up from the environment. This has to pick it up from Terraform, from config file, from somewhere. ([Time 0:04:47](https://share.snipd.com/snip/cb4c6c65-e80a-409c-b404-1c6949efcf39)) - **Use Models To Critique Models** - Use LLMs as adversarial reviewers and thinking partners rather than single-shot coders. - Cameron runs cross-model evals and prompts models to play 'what if' to surface second-order effects and domain issues. Transcript: Cameron Etezadi I look for, I mean, the structural correctness is there. Like, it compiles, it runs. You know, the best users of AI use it for goal- which is start with the objective that you want. In a way, this is like the ultimate test-driven development like we did for a while. And I think it still behaves that way where you're writing to satisfy the output because AI is really at the fundamental level. Like the technology is just a goal seeking kind of path of gradient descent to your answer when it comes down to it. I'm looking for those second order effects, the what ifs, the what could go wrong, what could happen here. We can predict, you know, poor API calls, badly formatted parameters, buffer overflows, cross site requests, forgeries, all the stuff we've done for a while. But it's those second order effects to my business, to understanding my domain that it's not nearly as good at. And I'm worried about, you know, building, I'm worried about three things. I'm worried about building good code because fundamentally I'm an engineer and I nerd out on building good code. Second thing I worry about though is building a healthy organization that's sustainable. And the third thing is I'm worried about building a healthy business because we're all in this to eat and survive and hopefully make a little money in the process. Erika Eggemeyer I don't know how much code you're reviewing, but I guess theoretically, if you were in a situation where you're reviewing an AI-generated PR, would you use AI to review it at all? Would you do it all manually? Like what would kind of be your checklist? Cameron Etezadi I absolutely love LLM as a judge and those patterns of evals where you're using one agent against another. It turns out agents, as good as they are at writing code, for instance, they're really good at critiquing. It's easy. It's funny. I've spent some time like on the philosophical side thinking about this, which is we're really good at criticizing other people as humans. Like we love to poke holes in things, throw rocks. Turns out when you train a bunch of LLMs on human generated content, they pick up some characteristics. And so I've always felt like maybe they're a little human in that regard and I anthropomorphize them that way. So I like to use other models. I like to use different vendors, open source models, the freebie models. Quinn is a good example. Kimmy, even DeepSeek is super cheap. Like using it to play adversarial roles and question things is really helpful. I've also come to the conclusion in the last, you know, a few months, the more I get it to engage and ask questions and play the what if, it becomes a good thinking partner. And so I feel like the best engineers in the organization are using it as a thinking partner and not trying to just one shot something, but actually working through workflows where there's A really good set of research, a really good set of planning. And that process is iterative. It's like the, you know, it's like the rough drafts we all wrote for papers back in high school where you kind of help your thinking along by questioning your assumptions. And so that's what I like. ([Time 0:07:36](https://share.snipd.com/snip/41575143-3e89-4a8b-b08d-48a7f29eb3a2)) - **Ship Small Experiments And Measure The Right Metrics** - Treat each change as an experiment: release small, observe defined metrics, then iterate quickly. - Cameron emphasizes choosing metric types (latency, throughput, business conversion) and using feature flags or A/B methods to close the loop. Transcript: Cameron Etezadi Think that loop is really important because we're, you know, if we learned anything in the last 25, almost 30 years from agile and extreme programming and all those sort of methodologies, It's all about small incremental changes to try to drive to the right answer. And if you take that for a philosophy that, you know, I'll say this as an ex-Amazonian myself, like that Jeff Bezos had at the start, you know, everything is supposed to be a two-way door If you can. Reversible changes, course corrections, and you spend your time on the one-way doors. There's lots of talk about that, but I think it holds true. I think it holds true with these agents. And so the natural extension of this is when it's very inexpensive to make a change because it's very easy to write the code or not write the code as the case may be. I think this ROI loop of release, observe and iterate becomes ever more important. And we're trying to get things out to production at a pace that's unprecedented. So making those incremental changes, finding your product market fit or your improvement or your goal, and then being able to make small changes of, you know, one degree of freedom At a time is super critical. I think where it's missing is a couple of things. It's understanding what went wrong. So in, you know, agile development, for instance, for years, we said, you know, one line changes or functions that only fit on a page that a human can put it in the head. The context windows for these agents are so much larger now. You can make bigger changes because you don't have to hold it in a human brain. You can hold it in an agent brain, but you still have to understand what you did and why you did it. So I think that understanding is often missing. The right set of observations is often missing. So what am I looking for here? What are the metrics I want to gather out of this bit of code? Is it latency, throughput, or is it a business metric? If I were writing a payments gateway, how many people did I steer toward Visa versus MasterCard? That's also really interesting. Did this change my conversion? So you're almost having to set up an experiment. And I'm a big fan of experimentalism. I think people don't set up those experiments correctly for the right output, and then gather the right data to make the right decisions. And then closing that loop very quickly, knowing where either a feature flag, which we can put in there to steer an A-B test or to run a multi-armed bandit or any type of those type of experiments That you would run, like those aren't in place either. And so it's really a matter of learning where your control points are, how to know what the control point data means, not just that you're getting data, but what does that data mean? And then being able to act on it. ([Time 0:13:05](https://share.snipd.com/snip/a2ca6080-7089-4a08-b43a-e1eabd570e21)) - **AI Turns ICs Into Frontline Managers** - AI shifts the IC role toward frontline management where one person orchestrates multiple agent teams. - Cameron calls this the 'two slice team' model: an engineer managing ~four agent teams rather than a two-pizza human team. Transcript: Cameron Etezadi So I had a little bit of experience, but definitely not a lot. And, you know, the two pizza team concept, I'm sure we've all heard of it. We've all found a squad of, you know, five to 10 people that you could feed with two pizzas. I think that those days are over. I actually think I'm now a believer in the two slice team method, where somebody who can feed himself or herself for lunch on two slices of pizza and controls a team of four agent or four Agent teams underneath them has sort of replaced that engineering two pizza team. But that means the behavior of an engineer has changed from I'm an IC to I'm a frontline manager. And so you have to behave on project management, on work breakdown, like a frontline manager to be successful now. ([Time 0:16:11](https://share.snipd.com/snip/141d2089-cec4-4990-b02d-c26856840416)) - **Interview Candidates On AI Ownership And Debugging** - Design interviews for the AI era: let candidates use AI and require them to explain and own the output. - Cameron's intern loop asked candidates to use the LaunchDarkly SDK and debug intentionally buggy code to prove operational understanding. Transcript: Cameron Etezadi I walked in there and I'll share it here. It's no secret. We told our intern candidates when we selected the ones we'd like to talk to, to please use AI to write the code for their interview. That was, in fact, we expected them to do it, but we told them to use our SDK. We told them to implement a feature and an experiment using our SDK. And then we had them walk through the code that the AI generated to be able to explain everything that was there. And we told them they were on the hook for the output. They had to be the responsible party. And they needed to show the value and that they understood what they were creating with the project. The second interview we did was a debugging interview where I wrote the code. I'm proud of it. I deliberately wrote bugs in there, or at least I'll pretend that I deliberately wrote bugs in there. Maybe it was intentional, maybe it was just my code. But we put a series of different errors in there, and we had them find all the errors and debug by hand. And we thought that was a really important skill to show understanding in the code that was being produced and to be part of that resolution loop of how do we operationalize it. It was our cue for operational behavior of this went wrong in production. You showed up. Can you fix it? ([Time 0:20:21](https://share.snipd.com/snip/f5eb0d75-e208-465c-a782-3fc4ebcb92fb)) - **Agent Workflows Should Mirror Full Engineering Process** - Build an agent-driven workflow that mirrors research, planning, coding, testing, and operational metrics. - Cameron describes agents fetching docs, planning, parallel test-writing agents, style checks, then feeding CI and metrics back for iteration. Transcript: Cameron Etezadi And I touched, I think, briefly on this a few moments ago, which is every project, every idea needs to start with a research phase. What do I have for assets that I can reuse? This is still the DRI principle of don't repeat yourself, or DRI, sorry, the DRI principle. Don't repeat yourself. I can't spell. Don't repeat yourself, right? So we want to reuse what we have because that makes the code more efficient. And then it goes into the, you know, get the background. So my agents will go out there and they check Confluence, which we use as our sort of wiki source of knowledge. All of our GitHubs, including our internal repos, which we have for our tools. What can it find? It looks at Slack for conversations that I might have. We're a G suite shop, which makes it very easy for me to search through documents. So it searches all my documents. And so I get a good research plan of what do I know about the problem I'm being asked to solve. And then the second thing is let's formulate a plan based on that research. And so I've got a planning agent and the planning agent goes out and says, based on all the research, I'm going to plan this. And that becomes a very iterative conversation. That's where, you know, I use my human judgment in the loop to plan out what it's going to look like. And then just go write the code. And of course, while it's writing the code, I still believe in test-driven development. And so I will have another parallel agent writing tests for that code and those agents are communicating, right? And so I get the testing agent to spit something out. And then once it's there, you know, the PRs, the feature flags, the style. You know, we have agents to review style. Does it match the existing code? Is it thematically correct? You know, you don't want a jarring experience of the agent wrote this, and it's completely different. And then, you know, eventually there's the operational metrics. What are we looking for, which are like the output criteria? So the tests run as part of the build-release pipeline, you you know, the CI pipeline, and we see what the results are and we go back and we pass that data back for iteration. For a while. ([Time 0:22:42](https://share.snipd.com/snip/c6496d12-7f15-4a71-b4ac-0e63763e1f79)) - **Run Regular Sharing Rituals To Scale AI Adoption** - Encourage company-wide sharing to accelerate AI adoption using low-friction rituals. - LaunchDarkly runs 'AI Fridays' and company moonshot hackathons; 75 projects with ~50 potentially shippable showed broad cross-functional impact. Transcript: Cameron Etezadi Love what you said too about sharing a bit. We do something we call AI Fridays. I can't take credit for it. It started before I joined the company. But it's truly been a boon for our adoption across the company where folks, I mean, it's a random share for all practical purposes where it's demo wear, but a walkthrough on Fridays of, Hey, this worked for me, or I discovered this new skill or tool or behavior, and here's how I'm using it to make my life better. And it's not just engineers. We held a hackathon internally, we call them moonshots, a couple weeks ago here. And we're a small company. We're 600 people-ish in that scale, so not huge. We had 75 submissions. Some teams did multiple submissions during the week, which we've never had. And we had folks from HR, from customer support, from sales, building apps and delivering them because they'd been enabled through the use of these tools to deliver code. And of the 75 projects, I'm going to say at least 50 of them are things that we could potentially ship with a little bit of cleanup. And the other 25 were either something really cool internally or make great demo wear for a sales or customer visit to show them the capabilities of what we got. I was blown away by the creativity, by the productivity, by the usage and the embrace that I've seen over the last six months from folks who probably, you know, six months ago were digging Their heels in. ([Time 0:28:45](https://share.snipd.com/snip/98d7bb9e-e162-4040-937a-70d3d152f8a6)) - **Control Is Mandatory When Models Are Probabilistic** - Speed without control becomes a liability with probabilistic AI outputs that can drift. - Cameron uses the bowling bumpers metaphor: you must add runtime control planes and drift monitoring because model randomness can cause regressions. Transcript: Cameron Etezadi And, you know, from a launch darkly point of view, we believe speed without control is a liability. It's a shipwreck waiting to happen. It's like giving a 16-year keys to a McLaren and hoping that they drive safely to the prom. It's not going to happen. I think we're all feeling that. Software has become deterministic now. Sorry, has gone from being determin to being probabilistic now. These models are random. And, you know, there's a temperature coefficient on the model, which tells you exactly how random it is or not. We as people love that randomness because we anthropomorphize it and believe that they're human because they sound like they're human because of it. None of us probably went into software engineering because we like probability. We like the computer to do exactly what we told it to do all day long. It's like the perfect, it's the perfect wage slave for us, right? And we have to find a way to put it back on the rails. The best metaphor that somebody described this as in the company to me was when you go bowling, have you ever seen those like kiddie parties where they pop the lane bumpers up on the side? And so you could throw a strike if you're good, but if you're like me, when you bowl, you're going to need those bumpers on the edge to keep the ball going down the track so that you hit a few Pins at the end of it. And I think that can, I mean, our platform is all about that control. I ship something. I test the living daylights out of it. I mean, I started my career in software QA. I do everything I can to put the right thing out there in production. And lo and behold, we go from, you know, 4.8 to 5.0 on the model. And that we're using for an eval in the product to do something agentically, everything changes. Or what worked today in a prompt doesn't work tomorrow in a prompt. You got to have a control plane out there that is managing your releases in production and watching for drift because drift is now happening where it used to be perfect before. ([Time 0:31:04](https://share.snipd.com/snip/a2247155-99ed-44e2-b812-39d8d31fde23)) - **Regulation Forces Human Accountability For AI** - EU regulation is shifting liability onto humans: you must ascribe a responsible person for AI outputs. - Cameron highlights that the law won't accept 'the algorithm did it' and requires accountable humans in the loop. Transcript: Cameron Etezadi Yeah, it's super interesting. And, you know, we can all have opinions about regulations and whether they're good or bad, whether they're doing the right thing or not. But the facts are they're there. And the bottom line comes down to you fundamentally have a human responsible for it. There is liability. And it's no longer okay to say the agent or the algorithm did this. You have to ascribe a human to it. And that's a really weighty responsibility. I think we're all searching for the right balance of tools that give us that confidence that they're doing the things with the best of intentions that I would do. There's a lot more to it, whether you're a frontier model company, a manufacturer, like there's levels of depth to unpack here. But fundamentally, they want to ascribe a human in the loop as responsible. And I think that's right. I can't fire an AI agent. ([Time 0:35:21](https://share.snipd.com/snip/b5196b7f-d652-4d4b-af23-4afbfa1eb2f5)) - **Stack Overflow Snippet Caused Kernel Blue Screens** - Cameron once deployed Stack Overflow code that leaked sockets on Windows, eventually causing blue screens of death. - He traced the issue to user-space code that never closed sockets, wrecking non-paged kernel memory. Transcript: Cameron Etezadi Right. I took out a system once because of Stack Overflow. It wasn't intentional. It was an internal system. Thank goodness. I had taken some code. It was doing a liveness check against another service on the network. And the Stack Overflow code never closed the socket. And leaking sockets in Windows kernel leaks a page of non-paged pool, or it did at the time. So that's memory that you can't swap out of the kernel. And so eventually the machines would just hit the blue screen of death. And we were out of non-page pool. I couldn't figure this out for a while until I realized that it was me in user space doing something that should be fairly innocuous, open a socket, and never got cleaned up. And it just blew the machine out every night. ([Time 0:37:16](https://share.snipd.com/snip/9e9e67e9-31d5-4c8d-a1ec-4b54ad635535))