Keeping GPUs Ticking Like Clockwork  ## Metadata - Author: [[The New Stack Podcast]] - Full Title: Keeping GPUs Ticking Like Clockwork - Category: #podcasts - URL: https://share.snipd.com/episode/c06c1514-56dc-494b-b383-3859567122f1 ## Document Notes ## Highlights - **Platform To Optimize GPU Communication** - Clockwork optimizes GPU-to-GPU communication across large clusters to boost AI efficiency. - Their FleetIQ platform provides visibility, fault tolerance, and congestion management for training workloads. Transcript: Suresh Vasudevan Yes, so today Clockwork builds a software layer that focuses on optimizing the communication between GPUs in large clusters that are then used for AI workloads. As you well know, AI workloads are among the most distributed, the most demanding distributed applications in history. And a lot of how well the workload performs depends a lot on how effective the communication is between GPUs. And so really what Clockwork focuses on is a set of software building blocks that allow you to get three things that ultimately lead to higher AI efficiency. The first is you have deep visibility into what's happening with your GPU fleet, all the way from the network up to the actual application layer, to be able to detect problems and rapidly Address those problems. That's the first thing we do. The second thing we do is really focus on how do we deliver fault tolerance such that when you have hardware failures, in particular when network links, for example, disappear, how Do you make sure that your AI workloads are resilient to those kind of failures and continue without interruption? This is a particularly big pain point for AI workloads. Frédéric Lardinois Because when there's interruptions, you have to go back to whatever the last checkpoint was, wherever you made it back up the last time, right? Indeed. Suresh Vasudevan I mean, it's crazy. When you look at large-scale GPU clusters, typically availability of those clusters runs in the high 80s to low 90s in terms of percentage of availability of the overall hardware. Contrast this with cloud availability, which is often measured in three to four nines. It's a completely different world. And what's worse is that when a link disappears, you have to, as you rightly said for a break, stop the workload, go back to a checkpoint that may be many hours old, and restart your training All over again. So hundreds to thousands of GPUs really wasting all the compute that they've already performed. That's the second thing we do is address fault tolerance to failures in the network. And then the third thing we do is look at the network that's carrying messages from one GPU to another, and understand when you have congestion, when you have two flows colliding in the Network that might cause things to slow down and avoid those using software. We can do congestion management, quality of service, redirect flows so that they're not contending with each other. So if I step back, really the software platform we built is called FleetIQ. ([Time 0:00:35](https://share.snipd.com/snip/fa9bb550-6a0f-4550-b796-9104954d8b5d)) - **From Clock Sync To Network Telemetry** - Clockwork began as Stanford spinout work on precise software clock synchronization. - That accuracy enabled measuring packet latency and became the foundation for their network telemetry pivot. Transcript: Suresh Vasudevan So, you know, I think if you look at the history of clockwork from 2018 up until 2022, the fundamental building block and the core innovation of the company was, as you rightly said, the Ability to synchronize clocks within a data center, within a cloud, across cloud regions, across multiple clouds, all using software. And being able to get to tens to hundreds of nanoseconds within a data center using software clock synthesis. The first four years of the company was really a small team that acted almost as an outgrowth of Stanford, a Stanford lab. And so it was five or six people. Both the core technology and the use cases we were pursuing were all about clock sync. So, for example, we have some of the Fortune 100 financial companies using us to synchronize clocks, but then use that for timestamping financial records, for timestamping market Data, and so on, right? Right. In 2002, there was this big epiphany, and that also accompanied a funding round by NEA, and the epiphany was... 2022. Sorry, 2022. Frédéric Lardinois 2022. I was like, that was a very different time. Suresh Vasudevan We'll talk about that too. 2022, we had this epiphany that if I have accurately simplised clocks, then it's possible for us to measure the time it takes for a packet to go from place A to place B. And that really became the foundation of network telemetry, right? Accurate latency monitoring for data center applications was really what began in 2022. And in fact, one of our largest customers that we've been public about is Uber. Uber is using that ability to look at cloud VMs and cloud containers, accurately understand what's happening within the network, and is deploying us across hundreds of thousands Of VMs across multiple clouds, right? Now, 24, I think, is the next big jump in our evolution as a company. While accurate latency monitoring is extremely important for cloud applications, it's critical for GPU clusters. And so we started applying the technology to GPU clusters, where we really managed to get extremely accurate telemetry all the way from the network. We were also able to correlate this to what's happening with GPU-to communication. So NICL, which is a communications library, how is the communication between GPUs and the AI training workload itself being impacted by that communication? So, that was monitoring. ([Time 0:03:43](https://share.snipd.com/snip/8d867fcb-4105-4bbc-b27c-d85b04123ea5)) - **Measure First, Then Control Traffic** - Accurate timing plus telemetry enabled Clockwork to add dynamic traffic control to actively reroute flows. - They integrate with NCCL, TCP and RDMA to both observe and control GPU communications. Transcript: Suresh Vasudevan Along the way, we also were able to complement our global clock sync with another really building block technology that we call dynamic traffic control. Because we now know exactly what's happening in your network between GPUs, what we're also able to do is things like redirect flows by intersecting the software layer. So we plug into the communication library that NVIDIA has called Nickel. We plug into TCP communication libraries. We plug into RDMA communication libraries. And when we see congestion, when we see flows contending, we're able to redirect. So the evolution was with clocks, I can measure things. Once I measure things, I can control them. ([Time 0:06:13](https://share.snipd.com/snip/a37a0d00-e841-4634-877e-ce75db7600d8)) - **Automate Failover For Link Flaps** - Detect transient link flaps quickly and trigger automated failover before the application notices. - Restore original connections when links recover to avoid disrupting training checkpoints. Transcript: Suresh Vasudevan Most often you see that cluster availability runs in the high 80s to low 90s, and network link flaps are one of the biggest problems. Now contrast this with a cloud. Frédéric Lardinois Explain what a network link flap is. Suresh Vasudevan Yeah, so typically when you have all of these GPU machines connecting to each other, you're going through an enormous fabric of switches, all of which are interconnected to either Optical connections or copper connections if you're within a rack, but optical most likely if you're going sort of a fair distance as you have thousands of machines, right? Now, what tends to happen is that these GPUs are often, because it's an optical connection, any breakdown in the, there's a whole chain of really microelectronics that are necessary For packets to travel from one end of the optical connection to the other end of the optical connection. What can disrupt this? Overheating because these GPU servers are running extremely hot. And for a brief period of time, the link breaks. You can have dust that comes on top of the actual glass, if you will, and break the connection. You can have cabling mistakes that result in the connection going down. Many of these are transient. And yet, if it's a dust particle that's breaking the connection, unless you wipe the glass or reseed the cable, you've basically lost that link. Now, normally in most cloud applications, protocols, communication protocols and applications have been built to resilient all over. Right, that's the whole point of it, right? Exactly. Unfortunately, with distributed training and GPU to GPU cluster communication, these communication libraries are so optimized for speed and performance, they don't necessarily Build for tolerance into those libraries themselves. And that's really where clockwork comes in. What we are able to do through our monitoring and observability product line is detect rapidly when a link is somehow experiencing a problem. But then we go further. You don't need to click a button or do something to rectify that. When we detect a link flap, we automatically have a watchdog timer that says this has gone on for too long. If I wait a little longer, it's going to propagate all the way up to the application and take the application offline. We automatically institute a failover. Not only that, we also wait to see if the link recovers, and when the link recovers, we recover back the original connection. ([Time 0:08:50](https://share.snipd.com/snip/914e5cfb-1067-4ce0-9974-120478fee242)) - **A Shim That Spans From Kernel To PyTorch** - Clockwork deploys user-space agents on GPU nodes and uses eBPF for TCP flow control and RDMA shims for low-level management. - They also plug into PyTorch to correlate iteration-level training metrics with infrastructure telemetry. Transcript: Suresh Vasudevan Balaji, our founder, is often fond of saying we are a software shim company. And so, we essentially monitor all of the fabric from the edge. So, we deploy agents on each of the GPU nodes that you're running, thousands of machines, or on your VMs in the cloud. And the agent itself for monitoring is a user space agent that's able to have very deep telemetry while still being a user space application on the end. Frédéric Lardinois So not EVPF, not in the kernel. Suresh Vasudevan The moment you want to control the flows for TCP, for example, we have an EVPF module that plugs in and is able to manage the flow of data through the network. So that's an EVPF module. Now when it comes to GPU RDMA networks, we have an agent that talks to the RDMA library called IV-Verbs. And so it's a shim that talks to the IV-Verbs layer. For fault tolerance, we plug into the nickel or rickel libraries that NVIDIA has for collective communications. So really, this notion of dynamic traffic control is up and down the stack. In fact, we're at PyTorch conference today, and we just announced a product that we're demoing, which is able to plug into your PyTorch libraries and instrument what's happening at The training iteration step boundaries. Look at the entire iteration from beginning to end, from data loader to checkpoint, and break that down and correlate that with what's happening in the infrastructure. So that's a user space application that plugs into PyTorch. ([Time 0:11:21](https://share.snipd.com/snip/dc02c7a3-c485-4a94-b080-66670dcf9f27)) - **Root Causes Of GPU Cluster Downtime** - Common causes of GPU-cluster downtime include optical link flaps, GPUs falling off the PCI bus, firmware (panic) errors, and thermal issues. - These failures reduce availability and can force restarts from old checkpoints, wasting massive GPU compute. Transcript: Suresh Vasudevan I would say linked flaps are probably, and in general networking, is probably the top one or two problems that we see from a resilient standpoint. It's also one of the biggest bottlenecks to performance. When you have congestion or you're oversubscribed networks, your network might have on average more capacity, but when you have really high bandwidth flows, they collide, etc. Right? So that's one. But if I were to take some of the other problems, in fact, something that we're working on is how do we extend our fault tolerance that today is able to address network faults all the way To entire nodes failing or entire GPUs. So now break that down and say, what are the other causes of downtime? I would say memory errors are very common in GPU nodes. I would say NVIDIA's GPUs or AMD GPUs often fail to communicate over the PCI bus. People describe this as GPUs falling off the bus. Frédéric Lardinois Yeah. Oh, yeah. Suresh Vasudevan And there's a whole host of reasons why that happens, but that's a very frequent cause of downtime. I would say what's called anxiety errors. So firmware on the GPUs somehow breaks down. Exciting errors, unfortunately, can't be easily predicted, and so they happen all of a sudden. That's a third common cause of failures. Thermal overheating, and therefore the GPU being throttled. So underclocking the GPU, and then sort of over time that becomes hard, and the GPU starts to overheat could be another cause of problems. ([Time 0:14:31](https://share.snipd.com/snip/47e07be7-f6f4-400f-ba91-9de0ff78bf4e)) - **Use Health Telemetry And Standby Nodes** - Monitor hardware health signals like memory errors and temperature to predict failing GPUs. - Provision spare nodes and automate bringing spares into the cluster to avoid training interruption when nodes die. Transcript: Suresh Vasudevan As far as the other faults are concerned, our observability is not just network observability. We are able to observe things like temperature overheating that I don't understand. We're able to observe things like there are too many memory errors that we're starting to see, and that means the GPU is sick, and eventually it's going to die. So monitoring the health of the nodes in the fleet is also something that we already do. But in terms of remediation, as I mentioned, what we're working on is the ability to say when an entire GPU goes offline an entire node goes offline, how do you take a pre-provisioned Spare node, bring the spare into your working cluster, and still leave the AI training uninterrupted, if you will? So that's sort of where we are heading over time as we think about our roadmap. ([Time 0:16:34](https://share.snipd.com/snip/18b5af18-6a89-45fe-ad30-11333b83e1e4)) - **Distributed State Is The Resilience Challenge** - The core challenge in GPU training resilience is capturing and maintaining consistent distributed state across many GPUs. - Understanding distributed communication lets Clockwork design fault tolerance that preserves application state without full restarts. Transcript: Suresh Vasudevan So the first aspect of state is what's happening within that single machine when I'm running lots and lots of further kernels, when my CPU is running certain things. Can I snapshot the state of the machine into a consistent state? And that is a difficult problem, but tractable because it's about single machine and its state. What is a lot more intractable is it's not enough for me to capture the state of that single machine. My AI workloads are always about distributed state. You were talking a little bit about what makes fault tolerance so hard in these GPU clusters, right? And I think you mentioned snapshots. So the thing we have to realize is that when you think about AI workloads, the challenge is about how do you capture distributed states. So if you were dealing with a workload that's just running on a single machine, that single machine itself is complex because you have hundreds of CUDA kernels and CPU processes that Are running at the same time. But at least the problem of how do you snapshot the state of that machine is somewhat tractable. But the moment you realize that that machine, when you snapshot that machine, you also have to figure out how to maintain a consistent state. And that machine may be talking to 100,000 other GPUs. So you have to simultaneously maintain state across all of these GPUs. And that's really where the communication problem becomes really meaningful. The fact that we understand distributed communication is what's allowing us to say, how do we build fault tolerance by understanding the distributed state of the entire cluster and Allowing the application to continue when one of the nodes in the distributed sort of cluster is somehow in a faulty state. ([Time 0:17:55](https://share.snipd.com/snip/6a7b8f15-616c-48a1-84ac-d866dc9c1f6a)) - **Turn Telemetry Into Chat-Based Diagnostics** - Convert observability data into conversational AI assistants to make diagnostics accessible. - Use natural-language bots to surface root causes and recommend remediation from telemetry. Transcript: Suresh Vasudevan It's a very, very good question. We have an exciting project that we are working on. We have such a wealth of data from an observability standpoint. And yet for most customers, the idea that I can basically understand what's going on in my system by looking at all of these dashboards, et cetera, as elegant a UI as we've built, it still Takes a lot of knowledge to be able to really understand what these dashboards are telling you. So something that our founder is extremely passionate about is how do I convert all of this data into spoken language interactions with our customers where we should be able to say, Tell me what's going on in my data center XYZ, where I seem to see that jobs are somewhat slow, right? And so really, that's where our first use case for AI is around how do we take our observability data and manifest all of that data into a very easy to engage with AI bot. ([Time 0:19:48](https://share.snipd.com/snip/058413f0-b2a4-4a11-ae8b-41dc4f0d9378)) - **Supply Constraints Boost Software Value** - Much of 2026 GPU infrastructure demand is already contracted by hyperscalers, keeping hardware supply constrained. - Software innovations that increase efficiency and utilization will therefore be highly valuable. Transcript: Suresh Vasudevan Think that the good news, if you want to think about it this way, the context of investments and the progress of AI, is that given the supply-constrained industry we've been living in, Most of the demand for 2026 is already spoken for in the form of contracts by the hyperscalers with a lot of the infrastructure providers. So on the one hand, you worry about how we overhyped the space. On the other hand, in terms of real revenue and real flow of infrastructure, deployment of infrastructure, most of 2026 is already spoken for by many of the large providers. As a result, a lot of the software innovations that are powering AI, whether it's models making forward progress, and now vision models and multimodal models, all the rage, if you will, And then their use in agentic AI use cases, I don't think that will slow down. That will only accelerate even more. ([Time 0:25:07](https://share.snipd.com/snip/0376b202-30d0-4d7d-81d1-5270b47ea23a))