The AI Code Generation Problem Nobody's Talking About  ## Metadata - Author: [[The New Stack Podcast]] - Full Title: The AI Code Generation Problem Nobody's Talking About - Category: #podcasts - URL: https://share.snipd.com/episode/b55c82c9-83bd-4ac8-935f-853d1c04fd21 ## Document Notes ## Highlights - **Founders' Story at Avoca** - Steve Demchuk and his co-founders met working at Avoca building front ends for banks. - They left to start Nitric, inspired by frustrations working within rigid fintech environments. Transcript: Alex Williams Like, what's the story there? Steve Demchuk Yeah, so we met at Avoca, which was a fintech software startup. And really fun one because we were actually building high-end front ends for banks. And the reason we were able to do that is they invented a great server product that you hosted in the cloud to basically take all the transaction data from humans and store it in a safe place. Well, we gave a great experience to people and then that integrated with backend banking. Alex Williams Ah, so you could give that really great experience, but you had all the data on the backend. That's right. Steve Demchuk And imagine this pretty hard to innovate on a bank's backend. So the fact that our technology was transactional, it was hosted in a private cloud. We were able to actually build and use latest technology on top of that. So I met Jai and Tim through that. They worked with the biggest banks in the world. They're those types of people that architects would really want to meet with. And so we had a really successful run there. We were bought by Temenos, a really large global back-end banking business. Tim and Jai actually ran all the front-end applications, about a billion dollars in acquisitions. And I left to go to another startup. And when I left, I looked at them both and said, if you ever start a startup, I'm your first call. And so two years later, I got the call and we started Nitric. ([Time 0:00:47](https://share.snipd.com/snip/a9c997f4-7a4d-4315-857d-27ba3af8d678)) - **Developer Frustration in Fintech** - Developers are frustrated by outdated DevOps tooling and language restrictions. - Banks want modernization but have many business constraints limiting tech choices. Transcript: Steve Demchuk I think the biggest frustration of a really talented developer is not being able to use the tools that they want to use. Alex Williams Not being able to use the tools they want to use. And meaning that their architecture can't support them? Steve Demchuk Architecture can't support them. They're being told to use outdated DevOps tooling. They're not using the language they want to use. So the original problem was actually working with large banks, working with regulated customers who wanted to do a modernization project with a lot of handcuffs. And so everybody would talk to you about, oh, I want to look like Netflix, but the realities of their business of just hosting with one cloud provider, choosing that cloud provider, Choosing the language, choosing which IAC DevOps tool really wasn't that realistic. ([Time 0:02:17](https://share.snipd.com/snip/c6a6717c-9b6f-4eca-b74d-49a7a397c80b)) - **Infrastructure from Code Concept** - Nitric lets developers write app code while it infers and automates infrastructure provisioning. - This approach frees developers from managing separate infrastructure code manually. Transcript: Alex Williams So did they want to provide automation to then take those regulatory concerns into consideration? Steve Demchuk You nailed it. It's actually what they really wanted to do and what they built was just write application code, build the app that you're trying to build. We will look at that app, inspect it, create an infrastructure spec, and then automate building the infrastructure. So ([Time 0:03:07](https://share.snipd.com/snip/f20ac79e-20fb-4d8f-a507-a587bbd57991)) - **Nitric Framework Overview** - Nitric framework abstracts common cloud elements like databases and messaging. - It supports multiple languages, IAC providers, and cloud platforms via SDK, CLI, and server products. Transcript: Steve Demchuk So we built the Nitric framework. The Nitric framework literally is an abstraction layer. So it's taking the common elements of any cloud application. So just imagine you need database, functions, compute, storage, queues, messaging. And we've created that to basically support any language. And we can support any IAC provider. So Terraform, Pulumi, and then we can deploy to any cloud. ([Time 0:03:44](https://share.snipd.com/snip/123b8b8b-45d9-47d9-801a-52fae3a5b8c2)) - **Evolution to Automated Infrastructure** - Developers wanted automation beyond Nitric's initial framework, to avoid writing manual Terraform or CDK. - This led to "infrastructure from code": generating infrastructure specs automatically from app code. Transcript: Alex Williams Layer? Steve Demchuk What was interesting is that when we first built it, when the guys released it, they really thought their first customers were going to be multi-cloud portability concerns. Like, hey, I want to write one code base and deploy it to any cloud. Our original design partners were just really excited that they found a framework that was opinionated and could just put them in some guardrails within AWS or Google. And so the first challenge that we came to was people coming to us and saying, hey, I love how much faster I'm writing app code. But what I don't love is I'm still writing all the Terraform or I'm writing all the AWS CDK or whatever they've chosen. And they basically came to us and said, can we automate that? And so that was, I would say, the second invention of the Nitric framework was, instead of it being infrastructure as code, it was being infrastructure from code. ([Time 0:04:40](https://share.snipd.com/snip/1b4e3c0a-c216-4ac4-a708-20beded349d1)) - **Building Trust with Open Source** - Build trust by open sourcing the framework and showing how it works. - Mimic familiar developer experiences in SDKs to drive adoption and ease of use. Transcript: Steve Demchuk You have to show them. So number one, we're really proud that we started as an open source framework. It's a strategy because we believe tools like this should be extendable. People should know exactly what's going on from it. You have to earn the trust by making sure people know what's happening. So when you create a new category, you're going to have to be a bit patient and you're going to have to solve real problems. So by open sourcing it and being really willing to jump in and help people build out POCs, we do that every single day. So those two things that get the mindshare. And then once people use it, they love using it because it's in their language of choice. ([Time 0:05:49](https://share.snipd.com/snip/c8313ae7-eb0c-4866-8d38-804356c8e47a)) - **Multi-language Support via gRPC** - Nitric uses gRPC to support multiple languages and ensure a consistent local and cloud experience. - SDKs emulate idiomatic syntax of various languages like TypeScript, Python, and Go for familiarity. Transcript: Steve Demchuk Yeah. So we use gRPCs as a... Ah, you do. Yeah, so as long as gRPC supports the language, we can write an SDK for that language. Ah. And then we're inspired. Like, if you're going to be in TypeScript, we're going to use, you know, our favorite TypeScript type of style. Same thing with Python. So whatever Python Go, your language of choice. ([Time 0:06:31](https://share.snipd.com/snip/6880f70e-8228-4fea-bec5-0ff9a914effa)) - **Transitioning to Nitric Safely** - Transition by wrapping existing code (e.g., Express) with Nitric to gain benefits gradually. - Start greenfield projects to gain familiarity, then retrofit legacy apps incrementally. Transcript: Alex Williams Yeah. So it's interesting. Steve Demchuk If you're already been using a common JavaScript framework, like let's say you've been using Express or Koa, we actually have HTTP support so we could just wrap your current code. And then after you do that, see some of the benefits as you're writing that code and get the generation piece coming. Obviously, you can delete a lot of the IAC code as you're doing that. ([Time 0:08:17](https://share.snipd.com/snip/03b29f45-dd31-44d2-a365-30ac4157be85)) - **Balancing Convenience and Control** - Convenience versus control is a core trade-off in infrastructure tooling. - Nitric enforces platform team guardrails, ensuring control while speeding developer workflows. Transcript: Steve Demchuk Yeah, I think the biggest thing is we always talk about convenience versus control. So I would say those people who are really high leaning towards convenience, like want a ton of automation, are really looking for speed, are going to be less resistant to a change of Doing something like Nitric. People who are high control, it's an illusion. They think that they're going to have to give up some control. I think when people hear infrastructure from code, they get really nervous that it's just developers code and then infrastructure just comes out of nowhere. That's not true at all. We are actually building the guardrails from the platform team that they want and then that's what the developers are self-serving from. ([Time 0:09:39](https://share.snipd.com/snip/624675fe-8e16-4899-b969-6c8977f141e0)) - **Biomedical Customer Use Case** - A biomedical customer uses Nitric to enforce serverless architectures per cloud provider with least privilege security. - Automatic tagging supports scanning tools for performance and compliance by default. Transcript: Steve Demchuk So a really good example would be the choices that you're making inside a cloud provider. So I think here being at Google Next, if you're using GCP, there's, I think, over 300 services that you get to choose from. So imagine a platform team trying to come out to their developer audience and saying, hey, what are you building? Oh, I'm building this type of app. Well, within that app architecture, there's a lot of choices at your disposal. So a really common use case, and I'll use one of our biomedical customers, is really the platform team choosing for the different use cases. So, hey, for data capture, we want to use a serverless architecture type. So it's actually the guardrails are saying, when you go to deploy for Google, you're going to be using this serverless. If you deploy to AWS, it's going to go to Lambda. And then it also, at the same time, puts in least privileged security provisions to make sure that the developers aren't storing all that type of stuff. The other thing that we do that's, I think, very underappreciated is tagging every resource that you create so that basically any of the scanning tools that you need for code quality Or performance are there by default. ([Time 0:10:49](https://share.snipd.com/snip/a86d2483-d5c3-416b-9cab-a465d69460c2)) - **AI and Infrastructure Automation** - Nitric complements AI-assisted app code generation by automating infrastructure provisioning. - Guardrails focus on security and architecture while allowing rapid AI-driven creativity. Transcript: Steve Demchuk AI here. So it's been really cool for us to see, being on a show like this where Google just released IDP Cloud, which I think we're a perfect fit for. It's their developer portal technology. That's fantastic. All the new things that are going on with Agentec AI. What we have found is, first of all, our team's had to change a lot as we write the Nitric Core service, as we write that service. We've been using a cursor to actually use Agentec AI to write app code. And what we've had to learn is, okay, so as you're using AI to write app code, how does Nitric add value in that? And that's actually in the infrastructure provision. So thinking it's actually really kind of a great compliment what we've seen is the way that people are starting to code is, yeah, like let's let people be creative, build code quickly, But let's make sure the guardrails are in the right place, which is going to be around security, infrastructure provisioning, making good architecture choices. ([Time 0:12:30](https://share.snipd.com/snip/3314235b-1998-4a95-bb54-c82145a364c8)) - **Deploy AI Models Safely** - Build and test AI models locally with Nitric before deploying to cloud. - Use guardrails to produce production-ready infrastructure for data engineering teams. Transcript: Alex Williams Next step. Where do you all come in? Steve Demchuk First of all, we're just putting out a guide at the end of this week about working with people that are building deep learning models. They're going to want to first be able to build those locally. First of all, Nitric, you can build that locally. You don't even have to choose the LLM at that point. You can start building a model, test it locally, make sure it works. And then at provision times, the first time, now you're going to be making choices of which LLM and incurring a bill. So that's a very typical model. Another one is because we are working with people that have big teams of data engineers, data engineers work a lot on building local models on their laptops and cloud infrastructure Provisioning and cloud architecture. They can learn it because they're super smart people, but it's not their day job. So it's the same thing is making sure we see the architecture patterns they need for the models they're deploying. ([Time 0:13:38](https://share.snipd.com/snip/53c02a78-5008-49b2-81b5-c27484ebaf6f)) - **Single Source for Infrastructure** - Writing separate app code and IAC leads to synchronization problems. - Nitric unifies this by converting app code directly into infrastructure specs automatically. Transcript: Steve Demchuk Yeah. So instead of, I think a very typical pattern for someone today would be, you write app code, but then your platform team is going to have to either write the Terraform or Pulumi or their IOC of choice to support the resource provisioning. And those two things don't really talk very well together. It's usually a people problem, not a technology problem. And then when technology is involved, it's a lot of templates and portals and things that get stagnant. So the way that we're helping in that is it's truly app code being written or being generated. And then that's what's creating the spec that actually turns into the infrastructure as code. So my point being, it's kind of going away from writing two code bases that you're trying to keep in sync. Instead, write your core business app. And we're going to be taking care of the infrastructure creation based off of only the infrastructure you need for that app code. ([Time 0:15:23](https://share.snipd.com/snip/f2f56a13-52c2-4cde-aa65-e1b342296151)) - **Customer-Driven Evolution** - Initially Nitric had its own deployment engine but extended to support Terraform and AWS CDK after customer requests. - Customers want choice on infrastructure as code providers just as they want choice on cloud and language. Transcript: Alex Williams Oh, yeah. Steve Demchuk Again, we used a deployment engine when we first came out. And guess what? As we started taking customer projects, that's when we realized, wow, a lot of people want the same capability with Terraform. So we extended that. Then we got, hey, can you do that same thing with AWS CDK? So what we first thought, maybe we could be opinionated on the IAC provider. ([Time 0:17:37](https://share.snipd.com/snip/9ab902bc-aa2e-4f1d-ac2f-a7ca533753e8))