Inside Oxide  ## Metadata - Author: [[The Changelog: Software Development, Open Source]] - Full Title: Inside Oxide - Category: #podcasts - URL: https://share.snipd.com/episode/8599825c-b030-4992-82ae-bd37d80d6bce ## Document Notes ## Highlights - **Building Firmware From First Principles** - Cliff Biffle built the very low-level firmware and even designed the first PCB prototype himself before hiring electrical engineers. - He started by iterating cheap boards as throwaway MVPs to validate circuits and reduce risk. Transcript: Cliff Biffle The cliff. Don't know why we started having this stuff. What is it you do here, man? I'm responsible for parts of the really low-level firmware on the computer, so everything from the machine turning on up through fans and power management and all of that. Basically all the stuff that happens before the thing our customers think of as the computer turns on. Right. That's all, well, me and my colleagues now. Originally it was me. Is that like BIOS stuff? Or is it before that? Before BIOS. So like from... What's before BIOS? What indeed. Okay. So these big AMD and Intel processors, and even the big ARM nowadays that you see in something like an iPad, there's actually a lot of work that has to go on to allow them to turn on. They need a bunch of different like voltage supplies stable. They need a bunch of clock signals set up and devices set up, and you got to get their flash ready for them before they wake up so they can get their code out of it and so there's almost always A one or more small processors in the machine that are responsible for doing all of that dirty work that nobody ever thinks about because it happens in the moment between like when you Hit the power button and when the screen wakes up yeah so that's our job pretty fast like 20 seconds uh 15 seconds Ours is milliseconds. Adam Stacoviak Okay, milliseconds. To me as a user, it's like 15 seconds. They mentioned PCBs, so you're building your own boards. So you're going first principles on a lot of this stuff. Yeah, absolutely. And you're a first principles kind of guy? Cliff Biffle Yeah, I mean, I think I actually designed the first board we made back before we hired actually qualified electrical engineers. Okay. But it was an MVP or was it a... Yeah, it was just for testing out some of our circuits we proposed for the bigger expensive board because it's nice to make cheap things that if they're wrong you can throw them away. Right. Jerod Santo And so Oxide has this writing culture. I assume that you were kind of attracted to that or kind of helped formulate it. ([Time 0:04:01](https://share.snipd.com/snip/28fe37ad-1ecc-4f32-967e-db820877f660)) - **Writing Culture Scales Knowledge** - Oxide's heavy focus on written documents streamlines interviews and decision-making by forcing clarity up front. - Living documents are updated after work to serve future hires and reduce tribal knowledge loss. Transcript: Cliff Biffle No, it was pretty well established when I got here. Okay. But like the fact that we make people write a bunch of stuff during the application process, the materials and like that whole packet of like, why do you want to work here? What's an interesting problem? Because it really saves time in the interviews. But like, for a lot of people, that's daunting. For me, it was like, oh, you want me to write a bunch of stuff? I like writing. I can do writing. Nice. Yeah. Was that a process you liked, this request for discussion, right? RFDs? Adam Stacoviak Had you done that before? Is this something that was... Cliff Biffle Previous companies I'd worked for had a process around design docs that kind of was similar, but not exactly. We are way more writing focused here. And in particular, most of the docs are living. Like as we learn what we're doing we go back and fix the docs that we wrote when we were done to try to better reflect so that way they can also serve as documentation for the next person that Comes through so it's not perfect but I honestly couldn't really point to what I would change like it's working pretty well as long as you've got people that are invested in the process And that are comfortable expressing themselves in writing. Adam Stacoviak Would you call it document driven? Do you think you start with docs or an idea or not so much a spec but like fleshed out first before you do? I actually feel like ([Time 0:05:41](https://share.snipd.com/snip/515caccc-25c4-481f-b0ab-c1e5a405c461)) - **Choosing Rust For Firmware Reliability** - Cliff found Rust while working in firmware at Google and favored it because it reduced class-of-bug overhead when onboarding mixed-experience teams. - Oxide uses Rust extensively for low-level firmware and most components below the host OS. Transcript: Jerod Santo Okay, so when you're writing on your blog This is about rust right lately. Cliff Biffle Yeah, but back then when they found it since like 2015. Jerod Santo Okay, so how'd you find rust? And why'd you like it? Cliff Biffle So I was working in firmware at Google doing like high-altitude balloon tracking and communication stuff that we were using C and I've been using C since I was a kid because my dad but It's really hard to produce correct software that doesn't contain bugs Particularly on a team with different experience levels working in C And I was bringing people in at intern level, Up through experienced developers and trying to get everybody working together and productive as a team. And just like the problems with a lot of work in process, you can manage it. Like car companies do this all the time, but it takes them a tremendous amount of overhead. And I just wanted some way out of that. So there were a couple of different alternative languages I was watching at the time, and Rust was the one that matured at about the right time and got enough things right to be worth spending Time on. Adam Stacoviak This is 2019, you said? This would have been 2015 originally. Jerod Santo Okay, way back. So you said recently, but that was a decade ago. Yeah, I'm old. Just wanted to point that out. Alright, so you're writing rust blogs, you're writing rust. Here comes oxide. You start working on oxide, writing more rust. Cliff Biffle I think oxide was an interesting opportunity because I don't super care about our product. Ooh, tell me more. I care that it's good. I want to make a good product.. Sure. I'm never gonna buy one of these. You're probably not ever gonna buy one. Jerod Santo I can aspire to buy one. Yeah, you could aspire to buy one. We can set that aside for now. Yeah, we're talking. Cliff Biffle But it's like a fancy McLaren sports car or like a Ferrari. I'm probably never gonna buy one. Right. So this product isn't for me. So I had to find other ways to really get motivated around it. And the main things are, like, this is a team that really wants to try to do things right from the ground up, which I can get behind. Like, that sounds like a hell of a challenge. The team is amazing. Like, my coworkers are amazing. You should talk to more of them. We will. Good. And so, like, you have to look for all the other ways to do this. And sort of the how do you build an engineering org from a team that fits around a small restaurant table to this size and be able to bring in people that don't have relevant experience And be able to bring in people that maybe had a career change, but are enthusiastic and like building a framework in the software, but also in the processes and the documentation to support Expanding the team like that, like that's the thing I got really passionate about. Adam Stacoviak In the firmware world, is it written in Rust or is it written in C? Our stuff's all Rust. Cliff Biffle All Rust. Adam Stacoviak So there's no... Yeah, from the time you apply power... There's like one thing in C, right? Wasn't it like some operating system you have? It's not you, Rust. Yeah, the host... There still is C. Cliff Biffle Illumos Helios is our sort of version of the Illumos operating system, which is descended from SunOS Solaris. That's mostly C. It's also older, pretty well-tested C. It's not new, potentially buggy C. So we think it's a lower risk. Does it change much? It doesn't change that much, although we've obviously had to extend it a bunch, but we've been doing a lot of the extending and changing in Rust. ([Time 0:07:16](https://share.snipd.com/snip/8087d99a-3dd6-4a4a-b141-3ed898c6ff82)) - **Why Hubris Was Born** - Cliff prototyped Hubris after finding Tok didn't align with Oxide's security and operational needs, and proposed building their own OS in 2020. - Hubris now runs on dozens of different boards across racks and tooling. Transcript: Cliff Biffle Yeah so so I tried really hard not to write hubris when I got here they were there's this there's this other operating system called talk it's also in rust okay targets the same sort of Very low level deeply how do you spell that talk is it-O Okay. Like you'd expect. Like TikTok. Yeah. But not like TikTok. Not the platform. Right. The way clocks work. Yeah. The original TikTok. Yeah. They were trying to use TikTok. Sorry. You fried my brain. So they were trying to use Tok when I got here. And like I had some previous experience with Tok. I know the people that wrote it. Okay. So I got in line and tried really hard to make it work for our application, but we just kept hitting areas where they, their design intent and the things we needed didn't really overlap. Like Tock is mostly at the time being written with educational use cases in mind. So they wanted kids to be able, kids, university students who are adults. Okay. I'm old. To be able to like dynamically reload programs on it as they're working and like nice use case stuff like that. But we really don't want that for security reasons. We want any code that runs on this better be what we shipped when we shipped it. So we put a bunch of work into trying to work around that. And then finally in like May of 2020, I think I wrote an RFD that was like, guys, I think we're going to have to do our own thing. And like, here's a rough sketch of how it might look like. And there were enough people here that had been involved in operating systems works before that they all kind of, we pressed our heads together and they said, okay, this might work. Like take a week and see how much we can prototype. And we got a thing working. Then it seemed compelling enough that now it's there's what 64 to 70 computers running it inside every rack all the little service processors you don't think about they're all running Jerod Santo That they're all big or small is hubris in terms of like line count or whatever so it's the core kernel ([Time 0:12:31](https://share.snipd.com/snip/c9bcb19f-f092-4c17-8892-0b6d91e07f29)) - **Tiny Kernel, Wide Coverage** - Hubris is tiny at the core (≈1k LOC) but scales across many device classes, from microcontrollers to service processors. - Multiple Hubris instances per board address distinct hardware roles and security needs. Transcript: Cliff Biffle Is like a thousand lines of code but there's a bunch of other stuff you want to make it useful but uh it runs right now on everything from like sub 50 cent microcontrollers that you wouldn't Even spot on a printed circuit board because they're just a tiny fleck of silicon up through the big service processor that we use to run the oxide rack which is like a it's basically a Computer you would have been really excited to own in 1999 or 2000 but now it's's $3 inside of another chip. And does that job. Adam Stacoviak Are there multiple instances of hubris on a given full-right system? Cliff Biffle Yeah, so every slide has at least two. There's the service processor that is responsible for basically care and feeding of the big AMD chip. Then there's a root of trust that handles security and crypto. That's a separate copy. And then there's, so there's those two on every compute sled. There's two in every switch. There's two in every power supply. And that's everything in the rack. But then a bunch of our manufacturing tools are also running hubris. So all the little boards we plug into a thing to program it or, you know, to interpose in an interface for testing and be like, you know, I need to remove this fan and simulate the fan controller. All of our tools for that are all hubris based. Why so many copies? Adam Stacoviak Is that hard to manage? It's like, yes and no. Multiple updates, different versions, potentially. That's true. Cliff Biffle It has pluses and minuses. So like the SP versus root of trust split which is the main source of the many many copies There has to be at least one of these on each board here any because it's boards got to be able to power Itself on because it might be the first one powering on it's a specialized case that makes sense to be a copy but so one makes sense quite two and the honest reason why two is we can buy one Chip with the features we need to do the service processor we can buy one chip with the crypto security features we need to do the root of trust. We can't get them both in one chip right now. ([Time 0:14:20](https://share.snipd.com/snip/83416645-1788-4742-abbf-acc2fa85715d)) - **When Fans Wake The Office** - Cliff recounts living-office culture moments like sudden fan spin-ups when service processors crash during firmware updates. - He admits sometimes he's the one who wrote the code causing those noisy watchdog reactions. Transcript: Jerod Santo About being here in the building with everybody. Yeah Is every once in a while a fan just goes crazy. This is not a person who enjoys like a literal fan just starts to woo like a What's going on when that happens? Cliff Biffle That's a great question. So, one of two things is going on. Okay. The good thing is, somebody just ran something on that machine that's like boosted all the CPUs. It's like when your laptop starts trying to take off. Sure. So something made the CPUs go really fast, everything ramped up, machine got hot, fan turns on, cools the machine back down. That's sort of the working as intended. What's probably happening here is that something's crashing. So we have a chip on the board that's a hardware watchdog that if it doesn't receive regular instructions on what to do with the fans, it assumes the worst and it ramps the fans up to make Sure we don't overheat. To avoid, okay. This means that if you're doing a firmware update on the service processor, the hubris-based service processor that's responsible for sending those messages, and it's gone for more Than Watchdog, however, the setting is seconds. Yeah. Then the chip takes out, ramps the fans up, and then it wakes up and finishes doing its update, and then the fans go back down. So this is why, I don't know if you saw earlier, but whenever those fans go up, like, I'm like. You're like the Watchdog for the Watchdog. I get yelled at in chat. Because you're one of the people that works out of the office generally. Yeah and like I might have wrote the code that's messing up so. Jerod Santo You better get out of your seat and go do something. Cliff Biffle So frequently I'm like it's not me it's not me this one's not me. Jerod Santo So as being somebody who's regularly in the office but most folks aren't what does Oxcon do for you in terms of your camaraderie with your colleagues or the excitement level? Like, how do you feel about it? Cliff Biffle I get to find out how tall everybody is. Tall? You can't tell that on the computer. That's true. That's true, yeah. Right? ([Time 0:18:30](https://share.snipd.com/snip/f17eb00a-6b43-48da-89b5-c1af014efd9a)) - **Flat Pay, Clearer Culture** - Uniform compensation reduces hiring stress and salary secrecy, making discussions about personal hardship easier. - Cliff values non-hierarchical manager roles and uniform day-to-day pay while recognizing sales incentives differ. Transcript: Adam Stacoviak Had people leave yeah yeah without being you know, what are some of the reasons? Have they been negative or has it just been just mutual separation? Cliff Biffle I actually really like basically everybody who's left. So like for some people, this work environment doesn't work. Honestly, it barely works for me. Like the all remote thing. I actually took this job because I turned down two other offers that were fully remote because Oxide at the time wasn't. Yeah. And this was in February 2020, so you can guess what happened next. Yeah. So, oh well. Literally next. It's a good thing my coworkers are great, but the whole remote thing doesn't work for some people, and sometimes you can't ever really get in the swing of things. Yeah. We've had folks where this turns out to set off other past work trauma. We've all got work PTSD from some shitty former boss. And if things are happening here that's too much like that, you might get freaked out and decide, do you need to leave? Which I totally respect. We've had folks like Arian. Arian joined before me and was involved in bringing me over here. And he just left last year because he's I've been here for five years. I've done, I feel like I've done all the startup-y stuff I can do here. Adam Stacoviak I'm going to go do a new startup. So, just time. Yeah. So, mostly good reasons, not like, this place sucks, I'm out of here. Cliff Biffle I think there, because there's a lot of upside. You can probably find people that think this place sucks. Yeah? Yeah. I'm not totally sure who, but I'm pretty sure you could one person Probably one or two two people. Yeah, I know I know 80. That's pretty it's pretty good odds. It's not too bad Mostly about the uniform compensation stuff. I think it's amazing. It's worked this long It was part of the reason I joined because like I came in all like knives out like expecting to negotiate Yeah, and Steve's like so what we're doing is we give people Stock according to this formula and we pay everybody the same amount of dollars. I'm like, well, that saved me a lot of stress. Like sold. Yeah. Probably the opposite of your time at Google. Yeah. 100%. Oh my God. My only real concern there is that one of the things I really like doing is bringing people in who either don't have a lot of experience in the industry or like are just out of school or just Out of some other job. Right, mentoring them up. Right. And like, are we comfortable bringing in people that are basically interns and paying them pretty good Bay Area salaries? Maybe we are. Yeah. But like, I do want us to be able to bring those people in because that's how we get the next generation of us. That's a good point. But if we can do that while keeping the compensation uniform, or at least fair, that would make me really happy. Yeah. Adam Stacoviak What does it do for your personal ability to show up to not worry about compensation? A lot. As much as you had to before. Oh my god. So like not having to worry that some of my coworkers are getting screwed over by having not like hardball negotiated in their interview. Cliff Biffle So I became a manager at Google of a team that I had previously been on. And at that point, at the next promotion cycle, I was able to see everybody's salaries for the first time. And that was how I found out that there was like $100,000 a year difference in salaries among people at the same level on my team. Wow, that's a lot of money. Yeah, and it was mostly us guys that had the higher numbers. It was kind of crap. So I don't have to worry about that here, which is great. And I feel like people are a little more comfortable talking about both job conditions and also kind of like financial stress. Like some people have been really open about like, you know, my husband lost his job and like, we've got the new kid. And so it's kind of rough right now. And I feel like people are a little more comfortable sharing that because we all know what we each other make. So it's not like you're going to reveal that like, oh, wow, you're being really overpaid and now all your colleagues are mad at you. Right. ([Time 0:22:21](https://share.snipd.com/snip/7862087c-aaed-4879-a334-f5f6ac43b21a)) - **Two Years Toward Self-Service Updates** - Dave Pacheco has worked for two years building Oxide's cell service Update, moving from a support-driven flow to self-service API updates. - The initial MVP focused on recoverability of individual compute sleds before adding full operator-driven upgrades. Transcript: Dave Pacheco Update. Here we go. Adam Stacoviak What is the update on the update? Dave Pacheco Well, let's see. We're working on shipping the first version of what we call a cell service update for the Oxide rack. So today, the process for updating the Oxide system, including the control plane and everything, involves a support process where our support engineers are getting on the system Through a debug interface. And that works pretty well for a lot of things. And it's very simple. But it doesn't work for a lot of customers to have an Oxide person involved in the whole upgrade operation. And so we want that to be something that they can do through the API, just like they can do all the rest of the infrastructure stuff. But that is kind of a big deal because that means that the control plane is driving the update, which means the whole control plane is online during the update. And so, you know, people use different metaphors for this. It's replacing all the parts on the car while you're driving down the freeway, replacing all the parts in the plane while it's in the air or whatever. But it's just a lot of work. You can make that work. How long have you been working on that? I've been working on it for about two years. Jerod Santo How long has Oxide been working on that? Dave Pacheco We needed an update process for our MVP, and so that started before that and finished before that because we launched before that. But that's the process that we use today in the support-based process. Jerod Santo This is like an upgraded rewrite. Dave Pacheco Kind of, yeah. So the idea behind the first version of Update was what we call the minimum upgradable product, which was Mupdate. And so the idea here is that, you know, it's an MVP. We're shipping it really as fast as we can, but there's a lead time between when you deliver software to the factory and when you actually get it at the customer site. And that's a period in which we can continue working on the MVP. And so what we needed was for there to be enough in that first thing that we could update it to whatever software we wanted once we got to the customer site. And that became the minimum upgradable product, which is Muppdate. And that's the procedure that we have today. So the priority there was about having a robust support procedure for recovering the software on any one of our compute sleds. And that's something that we knew we needed even separately from update. And so that's why it came first was this idea that like we need the ability to recover a sled, it's one of our compute sleds, no matter what state it was in, and we can use that to do our initial Updates. And that's what we've been doing for the last two years. ([Time 0:28:50](https://share.snipd.com/snip/1cf9daeb-375a-43b0-a136-2b8b739493ab)) - **Updates Replace Hundreds Of Pieces** - An Oxide update replaces hundreds of components across sleds, switches, bootloaders, and firmware, so updates act on a system-of-systems. - Oxide packages releases into a single large bundle that operators upload and trigger via API. Transcript: Dave Pacheco Yeah, no, that's a really good question because you think of the control plane as like it's the control plane. It's just like one big piece of software or something like that. But actually every single one of our, so we have in a rack, we've got 32 sleds. We've got two switches and we've got a couple PowerShell controllers. Every one of those has a service processor, a root of trust, and the root of trust has its own software and the bootloader software. Then on all the 32 sleds, we also have a host OS, which comes in two parts for historic reasons around bootloaders and stuff. Then we have all the control plane software on top of that, including storage software, which is one per disk. What all this means is that when you update the software in an oxide system, you're updating literally hundreds of components. And, you know, we're kind of doing it one at a time. And you're also going through all these intermediate states where you're running some of the old software and some of the new software. So you're asked, like, what is an upgrade? We're replacing all of the software running on everything in the system. And it's a lot of different things. So big deal then. Yeah. And that list I just gave doesn't even include a lot of stuff that for us gets bundled. Like CPU has microcode. Nix have their own firmware. For the update system, that's simplified because it all gets part of the host OS, but there's a lot of software in the system, and it's updating all of those things. But the whole idea of what we're doing is that operators don't have to think about any of that stuff. So our release process puts together a giant zip file, it's like two or three gigs of data. You download that from us, you can look at it if you want, you can validate it, whatever. But then you upload it to the API and you hit go. ([Time 0:31:51](https://share.snipd.com/snip/51f1e560-a080-498b-b544-b0e86b4a2c8f)) - **Support Air-Gapped Update Patterns** - Design updates to work for air-gapped customers by letting them download releases locally and upload to racks without internet access. - Keep the update flow flexible so operators choose between networked convenience and strict air-gap security. Transcript: Dave Pacheco A good question. You know, it's not something we've been focusing on for the most part so far. It's the kind of thing where the customer is currently going to be responsible for getting that from whatever our download site is, whether that's GitHub to the Rack. So they might get that on their laptop, downloading it from GitHub or something like that, and then they'll upload it to the Rack. Yeah, that part is GitHub. All right. That makes sense. And then uploading it to the rack is over their network. And we've kind of been assuming it wouldn't be an issue. That's a fair question, whether they would consider it an issue. I'm just talking about size, file size. Adam Stacoviak I guess most people can download a couple of gigs pretty easily without it being a major problem. But some people, yeah. It's definitely a lot. I've been talking before. I'm like, I guess if you're... Jerod Santo Most of their customers are not... Adam Stacoviak Yeah, they're probably not having those problems. Yeah. Jerod Santo I'm solving different problems. Dave Pacheco What problems what about air gaps so some people yeah yeah that's really important too and that's what this model works pretty well for air gaps because you're downloading it to you The customer are downloading it to your laptop and then uploading it no question required no connection to the the rack definitely doesn't care it doesn't even know if it's connected To the internet that's fine it doesn't care right you could imagine a nicer experience where where the rack was connected to the internet and could see oh Oxide's just published a new Thing and I'm going to download it. Maybe I only download certain parts that I need or I download it one part at a time. Maybe that helps with some of the bandwidth stuff. But actually, it's not something most of our customers are interested in right now. Most of them are actually more interested in the, I'm really not connected to the Internet and I'm doing this because I care about my security and my privacy and my data. I definitely don't want the Rack talking to the Internet. ([Time 0:33:47](https://share.snipd.com/snip/deef01a7-6b78-4474-9430-ce23e22d8fdb)) - **OS As Both Target And Orchestrator** - Hubris serves both as an updatable component and as the low-level interface used to orchestrate many parts of the update flow. - The architecture creates unique update challenges that bespoke cloud/on-prem providers also solve internally. Transcript: Dave Pacheco Yeah, so Hubris is a sort of, it's an operating system. We use it in a couple different components in the service processor and the root of trust on all these systems, on the sleds, the switches, and the PowerShell controllers. So it's one of the things that we update. It is also true that in order to update everything, we talk to the service, or in order to update much of the system, we end up talking to the service processor, which is talking to that Hubris thing. So we end up using the current version of Hubris through the service processor to be able to update the service processor itself, the root of trust, the root of trust bootloader, and The host OS as well. All the control plane stuff is on top of that and doesn't go through Hubris. Adam Stacoviak Is this a novel problem that you all invented, given your architecture? Dave Pacheco That's a good question. Yeah, a lot of the details are specific to our architecture and pretty novel. It's the sort of thing that I expect cloud providers today have their own bespoke software for. And in fact, large deployments of on-prem stuff will have their own bespoke software to do a lot of this stuff. ([Time 0:35:32](https://share.snipd.com/snip/8b4d16d6-f7af-44ad-beef-521f43249617)) - **Plan-Execute Enables Safe Automation** - Oxide uses a plan-execute pattern: the system computes a desired blueprint before any action, enabling safer automation and previewing actions. - This pattern underpins live features like sled add/remove and eventual non-disruptive updates via VM live migration. Transcript: Dave Pacheco All the stuff we've been doing so far is like the orchestration. It's foundational stuff. So like our system is based on what's called the plan execute pattern, which means that before taking any action, the system generates a new intended state of the world, which we call A blueprint. And then it goes and executes that blueprint. And like all that was really important foundational work for building a system that can be operated autonomously, which is also really important for the air gap thing. Because we can go test all kinds of things that can happen with just the planner part without even worrying about the execution stuff. And then we can go test all the execution stuff given whatever plan we want without having to have gotten a system into exactly that state. And it also lets you do all kinds of things like ask the system, what are you going to do next before it does it? And why are you doing that thing? You know what I mean? So these are really important operational things that we just need to have. That's the kind of stuff that's taken the first two years. So you laid a lot of groundwork. That's right. And so now we're talking like doing what we're calling non-disruptive updates. So this is doing updates without rebooting the customer VMs. We're still rebooting the sleds. So do you move the VMs? Exactly. We're going to live migrate the VMs. And now that's a question. I don't mean to oversimplify it, but that should be just a question of policy, which is like we're flipping a bit in that blueprint that says this sled needs to be evacuated. So let's say you have an oxide rack, 16 sleds. Jerod Santo Is that typical? Yeah, 16. All right. So I got 16 sleds and we need to run an update this is in the new world when this exists and i got each sled has i don't know 30 vms on it okay now we're doing math in our head oh ([Time 0:37:52](https://share.snipd.com/snip/be35a801-0dbe-4e91-b253-b603bbe9bb74)) - **Live Migrate, Reboot, Rebalance** - For non-disruptive updates, live-migrate VMs off a sled, reboot it, then rebalance workloads to preserve availability. - Explicitly expose operator trade-offs (capacity reservation vs. potential pauses) so customers choose update policies. Transcript: Dave Pacheco We're still rebooting the sleds. So do you move the VMs? Exactly. We're going to live migrate the VMs. And now that's a question. I don't mean to oversimplify it, but that should be just a question of policy, which is like we're flipping a bit in that blueprint that says this sled needs to be evacuated. So let's say you have an oxide rack, 16 sleds. Jerod Santo Is that typical? Yeah, 16. All right. So I got 16 sleds and we need to run an update this is in the new world when this exists and i got each sled has i don't know 30 vms on it okay now we're doing math in our head oh he's gonna do a math Problem no i'm not going to and it's coming time to run update so i go put it on my thumb drive off my laptop or whatever plug it into the rack and it's going to run it's going to Live migrate VMs off of a sled one at a time Update that sled reboot it. Mm-hmm and then move some stuff back to that one probably something like that Yeah, that's where actually distribute that load evenly across the other ones in the meantime That's where Dave Pacheco A lot of the complexity does come in with the with the non-disruptive update is first of all how do you like mechanically move these things around like a bin packing problem but then there's Also like how do we make sure that we have the capacity to do that right if we're going to start doing it up if you've totally filled every sled you're running your thing at max right there's No place to put it. You have nowhere to move them. Buy another rack. Well, that's the thing. At scale, people actually don't care about this problem. Because keeping a couple of sleds capacity free when you've got 100 racks is a very small fraction of your cost. And it actually makes sense for a lot of reasons. It also allows you to sustain failures and put that stuff over there. But when you've only got one rack, that might be more of a problem. Then there's the question of how do you create an experience for the operator that communicates clearly what the trade-offs are, but also gets this input from them, which is like, what Do you want to happen? ([Time 0:38:50](https://share.snipd.com/snip/fe73c319-fff5-4bbf-b298-8f4e2cd099f3)) - **Expect External Failure Modes** - Updates can fail for reasons outside the control plane, like unreachable sleds or unhealthy distributed stores, so automation must surface recovery choices. - Oxide built safeties to avoid actions that would reduce CockroachDB quorum during upgrades. Transcript: Dave Pacheco Non-disrupted. Jerod Santo Yeah, undisrupted. With the current iteration you're working on now can you guarantee that an update will finish that's not what i thought you were going to say finish no because there's things outside Dave Pacheco Of our control okay like for example um one of the biggest challenges in self-service update because the control plane is running we have this these intermediate states i mentioned Where you've got new version of software talking to old version of software. And how do you avoid that becoming a problem on our ability to change our own software because you have to do backwards compatibility like forever, right? And one of the ways we've addressed that is to say that there will be an order to the updates. So we will always update, for example, the host OS before we update the control plane that talks to it, because the reverse never happens right now. And so that's fine. But that means that if you're doing an update and one of the sleds is like out to lunch and we can't talk to it, we don't know if we've been able to update it, we can't actually keep going and Update the rest of the control. Jerod Santo Yeah, to wait for that sled. Dave Pacheco Yeah, so we got to tell the operator, look, you either need to what's called expunge the sled, which means to remove it from the control plane and we'll pretend like it's just caught fire, Like it's failed, we've moved everything else elsewhere, or you've got to figure out what's wrong with it and bring it back and uh that would be a support call probably a support call And then the other thing go they just unplugged it and then plug it back in or whatever those sorts of things are always outside of our control um yeah now what do you think i was gonna say I thought you were gonna say an update that explodes that like you start doing the update that was my next question and the control plane is now down and what do you do and like that's not That's my nightmare that's been my fear for the last couple of years. That's why it's taken two years and I'm not done yet. As an example, we've got a CockroachDB cluster that's storing all the control plane database data. We've got five nodes. ([Time 0:41:04](https://share.snipd.com/snip/f488c504-860d-4e99-b0dc-7bd18eaaea27)) - **Catch Incompatibilities At Build Time** - Prevent upgrade breakage by detecting incompatible API or schema changes early via CI and build-time checks. - Use generated API specs and client code to force compile-time failures for incompatible changes. Transcript: Dave Pacheco Sure. But it's a hard problem really because part of what we do, part of what is involved in an upgrade is making backwards incompatible changes to data formats like database schemas and things Like that and once you've done that the old software can't read the new thing so rollback is really not possible so what a lot of software does is they'll have like a point of no return whether Some call it a finalizer or a deferred update or something like that where you basically get the whole thing kind of working before you've committed in that in that way and then you ask The operator like does everything seem to be okay? And then they hit the button and it's like, okay, go. But even then there's like, there's still risk there because whatever it is you're activating by taking that last step hasn't been tested before that. And there's kind of no way to get around that. And that's, that's kind of a future problem for us right now, but it's something we're going to have to deal with. Adam Stacoviak So it is your nightmare. Has it ever been your nightmare? Meaning it's happened? Dave Pacheco Like in my career? Yeah. Adam Stacoviak Well, like specifically for oxide and updates. No, no. Dave Pacheco Any updates gone wrong? No, but we haven't started doing that in production yet, so there's still time for that. How do you do it in production now? We're using the process we used when, yeah, the manual thing when we, so the way the manual process that I was talking about at the very beginning works basically shuts down the whole Control plane, replaces all the software and brings it back up again. Adam Stacoviak How long was that process? It's actually not that, it's shorter than the other way. I feel like that's safer. Why? I mean, I get the whole reason. Downtime, man. Can't have it. Dave Pacheco It's downtime and it's the self-service aspect. Although you could imagine a self-service version that looked more like that. But then the thing is, like, if it's self-service, you're talking to an API. What's running that API while the thing is down? You don't have, there's nothing, right? Yeah. So that's why you kind of, that's why you got to do what we've done. Jerod Santo It's got to be done. Yeah. Adam Stacoviak So downtime is required now. Future is, you said non-destructive? Non-disruptive. Non-disruptive. Meaning the VMs get migrated around versus shutdown, but you still reboot sleds. That's right. You're still rebooting controllers and control planes and stuff like that. Dave Pacheco Yep. But that's not visible to them because we have enough redundancy. Jerod Santo There a world in which that isn't even required? Which part? What if you don't have to reboot anything? Oh, I think that's pretty dicey. Dave Pacheco It's definitely a thing that people have done Where yeah, there's like there's like there's types of updates for like patches of like a running list kernel You kind of like write the New one over here, and then you jump over there. But then the state... Jerod Santo It's just a pointer. Right. Dave Pacheco It is. It's all software, right? That's right. Jerod Santo Like who says we have to reboot, you know? Yeah. Dave Pacheco That is definitely harder to do with stuff like the SP and the ROT. Part of it is that the ROT, part of the ROT's job is to attest to the software that's currently running. Jerod Santo Yeah, if you're going to change it out from underneath it. Dave Pacheco Yeah, then you need to figure out what that means. And then there's also this risk that you're now in a different state than you would have been in if you had actually bounced that thing. So have you created a time bomb for yourself where if that thing loses power and actually does power back on, is it going to do the same thing that it was doing? That's the thing I always... Like bifurcated code paths where you're like, this is the thing we do sometimes. And this is the thing we do other times is totally the kind of thing that results in something failing at runtime. Catastrophic failure. Yeah. All right, fine. Jerod Santo Bad idea. Dave Pacheco There's the thing you asked about, like my nightmare, right? The upgrade just like explodes and we're toast. Another thing that we've done there is try to create a lot of guardrails around the types of changes that we can make to the software so that we know if we're making a change that's going To break things. So for example, if you're changing the database schema, we know that you're changing that and we've operationalized that one, so that's kind of fine. If you're changing like an internal API, we make sure that you're doing it in a way where none of those intermediate states will expose us to a situation where those components don't Speak a common version. And that's something I imagine other organizations do do. I haven't actually really seen that before, but I think it's really important. Because that's the way that I've seen this fail in the past. It's like someone goes and makes a change to the API. They're like, I'm not changing upgrade. So they test everything, everything seems to work. And you go deploy it like it blows up in the middle. It's like we've tested the end point, we've tested the beginning point and we just got unlucky in one of these intermediate states that wasn't tested. And so we've tried to identify the kinds of changes that would cause those problems and then detect those at CI time for us and build time if we can. ([Time 0:43:19](https://share.snipd.com/snip/64624bc7-9ed1-48e2-af5a-9dfefdc3f003)) - **Rust Moves Failures Left** - Rust's type system and compile-time checks let Oxide encode invariants so many upgrade-time errors surface as build or CI failures. - That compile-time safety increases developer confidence when performing large, system-wide changes. Transcript: Dave Pacheco So the big thing for me about Rust that's been, that I really love and I think that's been huge for oxide is its ability to help us ensure things at especially a build time that need to be True of the system. And so that sounds really vague. But what I mean is you can catch so many problems early. And everyone talks about the obvious ones. We talked about this earlier, but everyone talks about the obvious ones like it's the borrow checker will help you find memory safety problems, but it also allows you to create abstractions For the rest of the team that can't be misused. So you can say, I'm creating this thing. Like maybe you represent it with an object and you say, you can't do these two operations concurrently on it. Well, that's the thing that we can enforce in the type system. And like you literally just can't compile the code that would do that. That's awesome. And it sounds so low level, but that's what a lot, like it's extending that same idea and applying those same tools to do it that allows us to say, if you try to evolve the API in a backwards Incompatible way that won't work at runtime, at upgrade time, you might on how you do it, you might get a build failure. You'll at least get a CI failure. I don't know if we have a minute to talk about it. I mean, one of the first things that I built here with Adam Leventhal is something called Dropshot, which basically lets you write an HTTP server and then generate an open API spec from The code. Then we feed that into something called Progenitor, which generates the clients for it. But that alone means that if you make an incompatible change to an API, even before we've done any of the versioning stuff, your client fails to compile now, which is hugely valuable. And that's true, not just because it doesn't have the operation in it, but you passed in a Noom with three variants and it now has four variants and you need to like accommodate for that Fourth one or something like that. Adam Stacoviak I call me enums. Yeah. Yeah. Dave Pacheco So anyway, we ended up extending that. So now like that's how we do this versioning stuff is we have a bunch of these open API specs that are the ones that this thing supports. And then we know if you've changed it because we know if it generates a different thing, then you've changed it incompatibly. So Rust has facilitated all this stuff. And sorry, the last thing I want to say about that with the Dropshot thing is like you've got your rich structured types in Rust and you're like all happy because I'm in Rust land and I've Got my strong types. That carries all the way to the client because of the way it goes. Like Dropshot just takes those types and puts them in the open API spec, and then the client generates faithful things on the other side. So you can basically get that strictness all the way through there. Adam Stacoviak We gush a little bit more on this then. Speak to your confidence in the code you write because it's Rust. I'll just easy shot you that one. Dave Pacheco Yeah, I mean, I'm reluctant to make any bold claims about the correctness of the code I write, lest I immediately walk back in there and there's some horrible thing happening. The fan spin up, that's why. But I'll say this, there's a lot of changes I've been able to make where I'll go work on the code for like four hours, and when it compiles, I know it's already correct. I know I haven't broken anything. ([Time 0:53:16](https://share.snipd.com/snip/d7eaae55-3f87-459e-8df6-44832f497a37)) - **Oxcon Recharges Remote Teams** - Oxcon is a remote-company ritual modeled after Joyent: a company-wide meetup with keynotes and vital hallway-track conversations. - Dave says Oxcon provides crucial face time that doesn't happen in distributed workflows and helps align teams. Transcript: Dave Pacheco Oxcon. Oxcon is awesome. So this has been something we've been doing for a while. And this is something we, it's very similar to something we did at Joyent. These sort of company-wide meetups, engineering-wide meetups. You know, for a remote company, it works really well to be remote basically all the time, but it is also so valuable to have that time in person together to get to... There's so many conversations that don't happen if you haven't scheduled a meet for it. And we have other ways of trying to have those conversations anyway, but it's just really nice to have the FaceTime with people. And to also, like the small talk and like you're going to get dinner and you're just like talking to people about whatever it is. You learn more about them. It's it's a really good time. So it's there's a mix of the structured time that we have. You know, Brian and Steve talking about all the exciting stuff over the last year. It's inspiring for people. It's sort of gets everyone ginned up to talk about everything that we got to go do. And then, you know, there are other good company-wide sessions talking about important projects and stuff like that. But then there's also all this breakout time that I started by talking about, which as with any conference, the hallway track is almost, it's at least as important as the rest. Adam Stacoviak It is the best. Yeah, right. So that's huge. Not really. Hallway track only. Keynotes only. And then hallway track. That's our move. Dave Pacheco This is so much more, there's a lot, it's getting more like mature and professionalized. We never had a stage before. We never had like pro AV before. So this whole week is a little bit of a dream. ([Time 0:57:46](https://share.snipd.com/snip/61b328ca-ce51-4384-b74b-89b0eb89fbfd)) - **Design Ownership Across Product And Brand** - Ben Leonard joined from Pentagram to own and iterate Oxide's visual identity across product, marketing, and hardware. - He constantly tweaks details (even tiny logo weight changes) to keep cohesion across brand and product. Transcript: Ben Leonard Yeah, so I was working at a branding agency for Pentagram. Oxide was my last branding project. I was planning on leaving to work as a freelancer. I'd been at Pentagram for four or five years. I was ready to move on. And it just so happened that Oxide were looking for a designer just as I was leaving. And it felt like a unique opportunity. Because what would happen is you'd work on these brands and then you'd throw it over the fence. And then you'd check in on it two years later. You're like what have they done to my baby they destroyed it my boy my boy um and so it was a yeah unique opportunity to continue it on and i'm i'm kind of i was chatting to you about this before I'm like relentless that i like i i like to fiddle with things and it i can't work it on it yeah i kind of get bored of it as well. So I just want to, I just want to tweak something. So we, I mean, recently, I mean, like we made a modification to the, to the logo that, uh, most people wouldn't even notice it's slightly thicker, but it's something that's bugging me This entire time. Adam Stacoviak Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. And on stage here, it's much thicker. I know. Ben Leonard I saw it right away. Jerod Santo I noticed. Oh yeah. Yeah. Yeah. Yeah. Ben Leonard Yeah. Yeah. Yeah. Yeah. Always kind of, yeah, always filling with stuff. But then the beauty of OXIDE is I get to work across everything. So I'm working across most generally, most of what I do is working on the, on the, on the product. So in my case, it's the web console. So you work in the design system, the UI. I'm also working on the brand and the marketing and the sales assets and all this stuff and industrial design. ([Time 0:59:51](https://share.snipd.com/snip/78aaa219-3e9e-489c-920f-aaaf083898fd)) - **Brand And Product From One Idea** - Ben argues branding and product design should come from the same idea so the product feels like an authentic brand expression. - Oxide's 'old meets new' aesthetic blends nostalgia (terminal green) with modern UI to create cohesion. Transcript: Ben Leonard So the proliferation of SaaS, the product itself became the design. So the UI language of the, of the, of became the design. So linear is a great example of this, which is the design of the product and the design of the creative are one and the same. What that can mean is you have the world of design within product is much smaller. And so what that can mean is you have, everyone's always complaining that all tech looks the same now. And that's probably because product is much smaller. And if your brand looks like your product, then your world is much smaller. So I'm kind of, I'm always figuring out how wide I can go with the branding and keeping it still like it comes from the same world as everything else. And yeah, it's a, it's, yeah the industrial design the creative the product there's so much to there's so much to work on so you have this rock behind it yeah with this rack behind us look Like that at all if you weren't here um it'd look a bit like that i think maybe same shape yeah if i wasn't here it'd be it'd be it'd square, it'd be other size. No, I think... Less green, probably. It might be less green. Or maybe more green. We went back to... I worked with... So we consulted a little bit with my old agency, Pentagram, industrial designers there, and we worked together on that a little bit. One thing that I'd learned from working on a hardware product previously is the first version was, it was a small run that they distributed to people, right? And so they produced this thing, which is initially really magical. And then as soon as they mass produce it, they think, okay, we're going to have to make this cheaper, easier, more practical to make. And then the next version looked really, really bad. Because essentially they were just trying to recreate what they've done and, okay, we can't do this complicated tile system, so we'll just print it directly on, it was a PCIe card, right? And what it was, it was compromised in the worst way. So I think like going into it from the beginning, thinking like, what's are our limitations? We're going to make thousands of these from the beginning. I don't want to do things that just exist in this initial run. And so figuring out what those compromises are. But working with hard, like industrial design, working with hardware is hard. I haven't done very much of it. Um but uh yeah you you're as soon as you're touching materiality you're you're dealing with things are different yeah i mean we were speaking about like color color is the bane of my life Like we have we have the we have painted dry bays we have powder coated metal we have like plastic pieces and um and color matching green i think is yes that's probably gotta be your thing Right yeah yeah and even a little off sometimes oh it's a little it's a lot of all time and some of it's outside your control so you have to think about how to handle that so what you do is you Avoid putting elements next to each other that that might be different yes but like if there's a little bit of separation then visually you can get away with like with a little bit of difference Between those things um we had so one one i mean one example is originally the rack the was like it was a black but it was a slightly bluey black um like bluey as in the cartoon bluey? Adam Stacoviak Exactly. Jerod Santo It was like a bluey black. Ben Leonard Disregard. It was a cool black. Cool as in like color temperature, not like cool black. I thought you liked it. Yeah. But my thought was as soon as we start integrating multiple components, then I'm trying to make sure like the i'm using the exact right black oh because uh and yeah so that's hard you're Trying yeah well i don't know how much of it is me trying to avoid upsetting myself like these things that no one else knows yeah yeah yeah we said why i don't know this is like this is a like Trauma-driven design. Right. You guys have TDD. I have the same. Adam Stacoviak Do you get into the PCB design then, too? Like, how much, or is it simply this, I don't want to simplify what you do, but do you get to step into that world where, like, how is this? Because I think about Apple. Apple has done a great job of, like, branding everything. Ben Leonard Yeah, yeah, yeah. Look at their CPU, right? Adam Stacoviak Their latest M1 chips are the latest M series chips. Yeah. It's a design to everything they do. Yeah. Do you get into that as well? Ben Leonard I mean, Apple's a good example because they're stunning inside as well. A little bit. Occasionally I dip my toe in and I get shouted to leave. So the PCBs are currently there. The PCBs are green. And I think once I came into a channel and I was saying, Hey guys, can we make the PCBs black? I'm thinking, okay, you changed the color of PCB. You just order black ones instead of green. I was told that's not the case. Adam Stacoviak I think it does have impact to like thermo and stuff like that. What did they tell you? Ben Leonard They told me some information, none of which I retain. You're like, sorry I asked. Maybe they said it, maybe they weren't telling the truth, maybe they were just saying it because they were like, yeah, yeah, yeah. Black was cool, I've seen black PCBs. But there are issues with, yeah, there are manufacturing ramifications which mean that it's a bit more complicated than that. I think with the rack, you have to take, there's a balance. We can't go full Apple and just invest so heavily in details which cost money for little benefit. But what we can do is make sure that something which is an item which is underappreciated. I like taking boring objects and boring designs and boring, boring things and making them better. I think it's easy to come into, it's easy to work with Nike and make, and do some beautiful design for something that's already, like, really compelling. But I think it's much better to take something which is, for which design does not use it usually kind of feature and and try and elevate that so oxide is big on the values you know you go Jerod Santo To the website there's values i'm sure you know all the values they inform all the decisions yeah how you act do you have like a design language or value system or something that that drives Your design for oxide that's separate or maybe even congruent with that? Ben Leonard I think there are ideas. So I think my approach to branding is you have one idea which should then filter through into everything and that's how you make a holistic thing. It's not necessarily that this looks like this, it's that it comes from the same place. So with Oxide, Oxide is an old idea, like broad new, right? So it's this, it's like this old crazy idea that you own your own hardware. And so you see that in the design language, you see the ASCII, I mean, ([Time 1:02:15](https://share.snipd.com/snip/cbf205a9-4cad-452f-b446-1f03032c2a98))