Confronting AI’s Next Big Challenge: Inference Compute  ## Metadata - Author: [[The New Stack Podcast]] - Full Title: Confronting AI’s Next Big Challenge: Inference Compute - Category: #podcasts - URL: https://share.snipd.com/episode/2b0b5536-355a-496e-80db-699aa0d1e90a ## Document Notes ## Highlights - **Inference as Knowledge Application** - AI inference is when a trained model applies learned knowledge to solve tasks, similar to humans applying skills post-education. - Inference usually requires less retraining and focuses on monetizing existing knowledge efficiently. Transcript: Heather Joslin My understanding of inference in AI, and please correct me if I'm wrong, is that it is when a model takes data and uses it to solve a problem or complete a task. Is that basically correct or how would you define it? Sid Sheth No, it is correct. I mean, I like to typically always give my favorite analogy when I get this question. So if I may talk about that a little bit, but it's not very different from what we human beings do when we get ourselves educated and apply the knowledge we acquire in the workforce right We spend what the first 20 years of our life you know going to school or college you know and educating ourselves and acquiring knowledge we pay sometimes we pay a lot of money to acquire That knowledge but then we spend the next 40 years of our life really putting that knowledge to use and work and essentially monetize all the skills and knowledge that we have acquired In the first 20 to 25 years, right? It's not that our learning stops. When we apply our knowledge, we are learning and fine tuning specific areas. We become domain experts in certain things that we do. And we keep learning in those specific areas. But the general kind of broader skill acquisition and knowledge acquisition no longer applies once you leave school and college, right? So it's not very different for AI. Pre-training is essentially that general acquisition of skills and knowledge for the AI workload or AI model. And then once you deploy the model, which is inference, you're essentially monetizing all that knowledge that has been acquired. And in some cases, you need to fine tune a model for a specific area of expertise. But you don't really kind of have to retrain or pre-train the model like you did, you know, for general skill acquisition, right? In that sense, inference is really very similar to us human beings joining the workforce and applying our knowledge and monetizing it. ([Time 0:02:39](https://share.snipd.com/snip/522f9887-9977-4240-b8a6-6f7182acc3ee)) - **Inference Challenges and Agentic AI** - Major inference challenges involve integrating AI models into workflows and improving user adoption through better interfaces. - Agentic AI will increase compute needs and emphasize low latency for machines communicating with machines rapidly. Transcript: Heather Joslin What are some of the challenges? Sid Sheth I think a lot of the challenges today are around essentially a couple of things I would say. It's actually more software focused and specific challenges. How do you really introduce a lot of these capable models into software workflows? And then what is the killer app or the user interface that typical enterprise users would use to actually tap into those software workflows. Right. So that's, I think, what is probably holding back adoption of broad based inference in the enterprise market. If you look at the way inference is done today, you know, you obviously have tools like Copilot and there's, you know, agent force from Salesforce. And there's quite a few of these tools out there that are already, they've already introduced AI into their workflows, but we still haven't gotten to the tipping point, right, where There is broad-based adoption of these, not just these tools, but, you know, other software and SaaS tools with AI. And one of the reasons is probably because, first of all, a lot of retraining is required in the workforce. You know, the workforce in general needs to become very comfortable, you know, in terms of how they use these tools. I think there is still a lot of human intervention required in terms of how these tools get used. So, you know, if you want to accomplish something, you got to prompt, you know, these models. And, you know, if you don't prompt it right, you may not get the result you want, and you got to reprompt it and, you know, kind of iterate with the model to get to the outcome you want. A lot of that will change once agents come into the, into the workforce. And, you know, with the promise of agentic AI is you can really describe a task at a very high level. And, you know, simple English and agents are smart enough to break down that task and understand the intent of the user. And not just one agent, but these agents could work across silos in the organization across multiple you know different tools and essentially come back with an outcome that the user Is looking for right so it's like almost like a team of agents working and collaborating together to solve a task now when you get into that world obviously you're going to need a lot more Compute and that's where dmatrix comes in because the cost of compute is going to spiral up the cost of inference compute is going to spiral up as you start introducing agentic workflows Into into these organizations and uh and latency uh clearly becomes very very important because now it's not humans talking to machines it's machines talking to machines and you want You know really quick response time, you want a quick turnaround time on the questions you're posing these agents. So latency is going to matter, ultra fast inference is going to matter, right. And that is exactly the problem dmatrix is solving for, where we are able to support multiple users using these agentic workflows, why without sacrificing the latency per user. So typically what happens is that you start batching multiple users. In the case of agents, it could be batching multiple agents, right? What happens is that your latency tends to suffer because as more people start using the application, each user is not getting as much time with the application. So their individual experience begins to suffer. ([Time 0:05:33](https://share.snipd.com/snip/e8ac4704-cd7c-49a1-9815-49b13e735c25)) - **Heterogeneous Inference Hardware Needed** - Inference workloads vary widely; a one-size-fits-all hardware approach no longer works unlike monolithic training setups. - The inference hardware landscape will become heterogeneous, with specialized devices serving different performance profiles and needs. Transcript: Heather Joslin And I want to dig down a little more into what D-Matrix does, but I just want to get a sense from you of why you feel, do you feel that a different kind of infrastructure is needed for inference As opposed to training in AI? Sid Sheth Absolutely. Absolutely. I think I would argue that there is general consensus now that that will happen. The world of inference, if you look at the world of pre-training, it has been kind of monolithic, right? GPUs have dominated, specifically GPUs from one company have dominated the landscape. But as you enter the world of inference, it is not really a one size fits all. You know, there are too many different workloads, each workload with very different requirements. User profiles are very different, right? I mean, you know, you might be a user who cares all about cost, there might be some user who cares all about interactivity, where I really want to be able to interact with the model. That's the latency part I talked about. There might be users who don't really care about either and just care about throughput and, you know, batching, you know, a lot of data and gathering, you know, summaries from all of That data, right? So I think there's just different profiles that users bring to inferencing workloads, right? And it's not like the same hardware or the same computing infrastructure can serve all these needs simultaneously, right? ([Time 0:09:07](https://share.snipd.com/snip/8e8afe1a-c93c-4c15-ae3b-bc67f392fb93)) - **Memory-Compute Proximity Reduces Latency** - Latency heavily depends on the distance data must travel between memory and compute during inference. - Co-locating or 3D stacking memory directly on compute chips drastically reduces data travel distance, improving speed and efficiency. Transcript: Heather Joslin When we met at HumanX, you spoke to me about the connection between distance and latency. How does that apply in handling AI workloads? Sid Sheth I think, I believe when I was talking about distance, I was probably talking about the distance for the data. The data has to travel to get to the compute. Right. And one of the key challenges with inference compute is really about keeping the memory where the data resides as close to the compute as you can possibly keep it. And the reason for that is you are kind of making a lot of trips to memory. You know, when you talk about generative AI workloads, you're generating content that relies on caching data. So all the previous data gets cached. And every time you generate a new token, you are essentially kind of tapping into that cache data to figure out what the next best token needs to be. And now just take that problem in an agentic world. That problem gets multiplied, you know, 10x or 100x, right? So the memory footprint becomes very, very important and keeping that memory close to compute becomes very important, right? So the less distance the data has to travel to get to the compute, the faster your inference is going to be. And the more optimal your inference is going to be, the lower cost your inference is going to be and all that good stuff And that is what Dmatrix has solved for. Essentially, we have created a platform that, from the grounds up, co-locates compute and memory very close to each other. So this way, data has to travel a very short distance. In some cases, it is actually physically co-located with the compute, so it doesn't have to travel at all. ([Time 0:10:50](https://share.snipd.com/snip/54781ae4-57da-495f-9c5e-b05efc173aa6)) - **3D Stacking for Fast Data Access** - d-Matrix uses a 3D stacked architecture overlaying memory chiplets directly on compute chiplets, unlike traditional side-by-side packaging. - This arrangement increases data bandwidth and availability, keeping compute units fully utilized without idle waiting. Transcript: Heather Joslin For those members of our audience who are watching this on video, I remember when we spoke, you took my phone and used your phone to show sort of almost like a pancake kind of situation. Right. So that's, can you elaborate on that? Yeah, sure. Sid Sheth So I think absolutely. And that is one of the key tenets, again, of the Dmatrix platform. We built our platform with chiplets, which are smaller chips. We don't build big monolithic chips. We build smaller chiplets, as they're called, and then we co-package these chiplets into a fabric. And that gives us that elasticity and modularity in the platform. We can always scale it up or scale it down, depending on the customer's requirements. And what that chiplet-based architecture allows us to do is what I talked about when we met at HumanX, which is there's different ways of packaging those chiplets. So you have compute chiplets, you have memory chiplets, and the traditional architectures today, actually pretty much all the traditional architectures today, the way they're Packaged is you'll have the memory, say this is the memory this is the compute and you'll essentially kind of you know try to keep compute and memory as close together right so you kind Of keep these two things very close you abut them uh and you co-package them on the same you know what they call say common substrate and they're kept you know together in our case what We did was we said instead of doing that why don't we just kind of overlay uh the memory directly on top of compute so do a 3d stacking as opposed to a 2d you know go 3d and now the memory is sitting Directly on top of the compute and you're effectively the data is sitting inside this memory and it's raining down uh into the compute which is sitting right underneath it right so now Obviously the surface area is much greater when when you when you package things this way obviously there is more surface area between the memory and the compute and a lot more data can Drop down into the compute. And it is physically, because it is co-packaged this way, it is physically like right next to the compute. So the data doesn't have to travel that much distance at all. And then we can replicate this. So this could be one pair of chiplets and there could be similar, many instances of this that we can sprinkle through the platform. And that is where the advantage comes from in terms of how we are able to quickly access data from memory. And computers, you know, always kept fully utilized. There's never a point where, you know, our computer is running idle, waiting for the data to show up, right? ([Time 0:12:42](https://share.snipd.com/snip/4ef73852-a161-48e0-845a-e7cf923c922c)) - **Fast Inference Boosts User Retention** - Any organization using generative AI inference benefits from ultra-fast inference to provide highly interactive user experiences. - Faster AI responses improve user retention, making latency reduction crucial across industries and cloud providers. Transcript: Heather Joslin Kind of organizations or customers do you think might be the best use cases for this solution as things stand now? Sid Sheth Oh, it's everyone in the world who wants to use generative AI inference and wants really fast generative AI inference. So it would be pretty much all the enterprises, all the big clouds who serve those enterprises, all the Neo clouds who serve enterprises, even the sovereigns who care a lot about highly Interactive inference. And highly interactive inference in general is becoming popular everywhere. And the reason for that is very simple. I think it's pretty obvious to everyone that when users use an application, the faster the application responds back to the user, the more likely the user is to stay with the application, Right? If the whole experience is a very offline experience where the user has to work with the application, the application takes, you know, minutes, maybe multiple tens of minutes to respond, Very unlikely that the user stays with that application. They'll go do something else in the meantime, right? So this ultra, you know, ultra fast inference is becoming a very important ask from pretty much all our customers who want to essentially provide their users with a highly interactive Experience so that they stay with the application that they're building. So I think this is anyone who's using generative AI across the planet. ([Time 0:16:15](https://share.snipd.com/snip/56aca0f2-bd11-456b-bfbe-ad8a2329f175)) - **Reskill Teams for AI Infrastructure** - Organizations must reskill existing infrastructure teams to handle AI-accelerated and heterogeneous compute fleets. - Efficient query routing across geographic and energy-optimized data centers will become key to managing AI workloads. Transcript: Heather Joslin We've been writing a lot lately about the new infrastructure that AI requires. And one question we've been grappling with is who will maintain this new AI infrastructure? Will it require new roles, new skills? What do you see ahead for organizations that are investing in AI-specific infrastructure? Sid Sheth Yeah, I mean, there's going to be a retooling and a reskilling that is going to be needed. The AI infrastructure is, you know, I mean, obviously, you know, the general purpose computing infrastructure that has served us for the last 20 odd years, 20 plus years in the cloud, Right, is going through a massive upgrade as you essentially are augmenting that infrastructure, in some cases replacing that infrastructure with accelerated computing infrastructure. So it's starting with GPUs today. And, you know, over time, as I was stating earlier, it'll go heterogeneous and there'll be other forms of accelerators and, you know, other forms of compute that will be incorporated Into those fleets, right? And, you know, typically the accelerated computing fleets, there is, you know, there's a different set of requirements in that they are highly performance-oriented, highly efficiency-oriented, Depending on which type of application you're trying to solve. There's a lot of orchestration that you have to do at the data center level, because if your fleet is truly heterogeneous and there's different accelerators, you're trying to figure Out in real time, as users are sending queries to that infrastructure, which queries need to be routed to which part of the fleet. And again, the fleet may not entirely be local, right? The fleet could be distributed across the globe. You might have to, in some cases, route queries to a fleet which is sitting maybe many, many thousands of miles away. If the request is not necessarily a latency optimized request, it is more, you know, somebody is just looking to energy optimize, right? If there is a data center that is sitting close to a very efficient and cheap energy source, you might want to route some of those queries there. So there's a lot of orchestration and software that will come in in terms of how you serve the users and serve these queries in real time. Right. So I think there is just there's going to be a lot of retooling and reskilling at the infralayer. I would not expect that it's an entirely, you know, new set of people that have to come in here. It is the existing folks that have to rescale and retool and they can do it. I think it's starting with the hyperscalers because they have a lot of their expertise in-house. But very quickly, there's folks in the NeoClouds and Sovereigns that are acquiring those same skills as GPU infrastructure proliferates into those setups, right? And we'll be beneficiaries. Companies like Dmatrix who are not in the first way will come as a follow-on, will be beneficiaries of that retooling that is already happening and the reskilling that is already happening. ([Time 0:17:45](https://share.snipd.com/snip/831c3af9-e20f-4acd-acdd-117d56b78ce2)) - **Evolving from Planar to 3D Stacking** - d-Matrix's first-gen product is a planar PCI accelerator card using multiple chiplets, already shipping to customers. - The upcoming 3D stacked "skyscraper" product will add floors to increase compute and memory capacity, tailored for reasoning-heavy inference tasks. Transcript: Heather Joslin One last question. What's on the roadmap for Dmatrix for the rest of the year, the next few months? Any hints you can drop of what to expect? Sid Sheth Yeah. Yeah. I mean, nothing that I haven't already said. A couple of things that we are really focused on is quickly, you know, releasing our product that we talked about the 3D capability product. So that is really what the company wants to get out into the market quickly. Our current product is already out in the market um maybe maybe i don't know if there'll be a video uh for your users but this is the accelerator card uh this is our first gen product uh which Is a pci card it's uh you know rated at 600 watts so it can go into a lot of the existing ai infrastructure that is already out there um and you can see these two chips here uh if pop the lid on One of these chips, there'll be four chiplets sitting inside each of these. So there's a total of eight chiplets. And then what we do is we package that into what we call an accelerator unit, which looks like this. And this has two of those cards, the ones that I just showed you. So there's two of those sitting here. And then they bridged uh with what we call a what a you know kind of a dmx we call it a dmx bridge that bridges these two cards together that allows any chiplet on any car to talk to any other Chiplet so they're connected in an all-to configuration so from a software perspective the software treats this as a single unit it's not treated as two distinct units it's treated As a single unit and this is what we ship to customers and it's shipping to customers today and would plug into server infrastructure out there. So this product is already out there and this is not currently a 3D product, right? It is a product that is planar. It's just got that first level. The next product that we're going to put out is going to have the 3D stacking. So then we start building what we call the skyscraper. We'll put the first floor in, then we'll put multiple floors in and keep increasing capacity that way. And the reason we are doing that is we are very excited about what's happening with reasoning or what is called inference time compute or test time compute, which is where models are Beginning to think more. And that is what feeds directly into the agentic era, right? Agentic workflows and agents will be using those reasoning models to think more as opposed to, you know, just training models with more data. And more thinking equals more inference. And so we are kind of, as a company, very excited about that future, which is coming in the next few years and making sure that we are ready to serve that future. Heather Joslin Excellent. Well, and when does that 3D product do to hit the market? Sid Sheth Yeah, we'll be announcing something pretty shortly here in the next few months. ([Time 0:20:35](https://share.snipd.com/snip/266af0c2-4cdc-4f02-a86e-30128565ba5b))