The AI Industry Is at a Major Crossroads  ## Metadata - Author: [[Decoder with Nilay Patel]] - Full Title: The AI Industry Is at a Major Crossroads - Category: #podcasts - URL: https://share.snipd.com/episode/a72ea17f-7510-41ca-aadc-5b4a9cb93b26 ## Document Notes ## Highlights - **AI At A Crossroads Between Open And Closed** - AI platforms risk becoming new walled gardens that lock users into a single provider and ecosystem. - That centralization threatens users' control over digital environments as incentives diverge from users' interests. Transcript: Kanjun Qiu I think that this is all to be expected. You know, this is kind of where the entire agent ecosystem has been going toward. And the way I think about it is it's kind of like the iOS of AI, where you have this single interface that OpenAI is trying to build that is the way that you as a user get into all of these different Apps through this single interface, ChatGPT. And I think that's a big deal. And I think that there are going to be really major implications on the power dynamics with AI forward from that kind of approach. How so? Yeah, I think right now we're kind of at a crossroads on whether AI becomes another walled garden platform situation, just like the internet has been the last 10 years, or whether we End up having AI that actually democratizes and decentralizes power in the way that the original internet did or the personal computer did. And right now, we're in this kind of platform network effect ecosystem right now where, you know, you have these AI model providers, and they are providing something like this that Lets you integrate with a lot of other apps through partnerships. And what that means is that just like with iOS, Apple, they're kind of trying to lock you into their platform ecosystem. What he calls in-shittification, which is platforms, they'll start out doing great things, you know, being really useful to your users, like Facebook. And then over time, what they try to do is they lock in their users, and then we end up being at their mercy, and we can't exit. And so we're really in a place right now where the question is like, do we control AI or does it control us? Do we own it or do we just rent it from these centralized platforms? And I think there's a risk today of people losing the ability to shape the digital systems that control our lives. And you can see it a little bit in how we relate to our digital devices right now. Like my phone, I feel really aversive to my phone. You know, I need to detox from my phone or my TV or my computer. And that's a really weird relationship to an object in my life. Like if I had to detox from my sofa, then I would get a different sofa. ([Time 0:04:37](https://share.snipd.com/snip/e79fabbb-f887-4b2f-85c6-bf117896d8ee)) - **Modify Software At Point Of Use** - Lower barriers to building software with AI could let users modify tools at the point of use and reduce dependency on designers. - Enabling personal modification could recreate the move from time-shared systems to personal computing freedom. Transcript: Kanjun Qiu I think it's a really hard problem. And the moment in time we're at is kind of where we are in our digital environments today is what I said about our phones or our devices, where we're really renting these apps. We don't own them. Somebody else is making them and we're using them. And that means that, you know, there are incentives where people who make them want to make money off of them. And that's generally okay. The way in which we really get into trouble is these devices, digital systems, they have very fine-grained controls that capture our attention. They notify us. They are embedded in our workflows. And so because other people are controlling these apps, it's also easy for them to start controlling us. And so I think actually right now, today, we have a window of opportunity to really change this dynamic of control. In the past, the reason why we have this dynamic with software creation is because it's been really hard to make software. So you need to hire developers who are really expensive, build software, pay for the development, and then sell it for a profit. But we're starting to get into a world where, you know, as OpenAI Dev Day demonstrated, you can generate apps, generate things using natural language, and these language models are Starting to get really good at writing code. And so in theory, we could go into a world where a lot of people can write code. And a lot of people could not just write code, but take a piece of software and change it to make it fit for themselves. An example of that is like a doctor who's dealing with an electronic medical record system where they have to put in all these fields, it's not suited for them, and a lot of doctors encounter A lot of burnout from this situation. And in theory, they could, in this future world, just have a system that helps them change the software at the point of use. We call this modifying at the point of use. And so I, as the user, know what's best for me. I know what I need. And I can change my digital environment right in the moment so that it suits me better instead of suiting whoever designed the software. However, that requires inventing a few different pieces. Just like how, you know, the moment we're in, it reminds me of the 1960s. In the 60s, people were really excited about supercomputers. They thought, everyone's going to timeshare on supercomputers through these terminals. Like, we're all going to be in terminals, timesharing and using supercomputers is going to be super cool. And then in the 70s, late 60s, a small group of people at Xerox PARC invented the mouse, the GUI, files, folders, Windows, everything that allowed us to make personal computers. And that led to personal computing today. And what that did is it made the technology capabilities more accessible so that people could use computing for themselves and modify their computer for themselves. And I think we have an opportunity kind of to make the same inventions for the AI ecosystem. ([Time 0:08:49](https://share.snipd.com/snip/5104688a-b15f-4b6a-9981-7435446cd36e)) - **App Store Ban Shows Platform Power** - Imbue launched Sculptor, a Mac coding-agent app, and briefly got banned from the Apple App Store during registration. - The team had to pull internal strings at Apple to resolve the ban, illustrating central-platform power. Transcript: Kanjun Qiu Recently, we, last Tuesday, actually launched Sculptor, which is a tool for building software using coding agents. And it's a Mac app. And when we were trying to register with the Apple App Store, Apple banned us, even though we are a legitimate company. And it took us several days and pulling a lot of internal strings at Apple to figure out how to unban ourselves. And that's an example. It's a very trivial example, not huge downside effects, but it's a trivial example of a way in which we have lost power relative to a centralized platform. ([Time 0:13:15](https://share.snipd.com/snip/7e3dbccc-8191-4f40-a7e0-d9443313c1de)) - **Engagement Incentives Drive Sora's Effects** - Sora demonstrates how platform incentives for engagement amplify both joy and attention capture risks. - Ultra-accessible AI video will drive mass use and the same attention harms we already see in social media. Transcript: Kanjun Qiu I think Sora shows what happens when AI capabilities meet platform incentives. It's really interesting. Platforms and incentives you talked about with social media, they optimize for engagement and attention capture. And that's what allows for making money, profit, ads, et cetera. Generally, this is not a huge problem until you take it too far. It's actually a perfect case study when we talk about the shape of tech power. My friends at OpenAI talk about how it's actually cool. They're trying to create a video generation app that creates joy, gives people moments of entertainment and joy. And I think from my perspective, that's a great thing. And anytime you're trying to create joy and also maximize engagement to maximize profit, it is not necessarily, I think it's like very difficult to trust. And so I think what I would expect to happen is that this platform engages a lot of people, gets a lot of users using it and creating lots of videos. ([Time 0:18:37](https://share.snipd.com/snip/c44f5787-25f6-4705-9463-cfcc1ccdbcae)) - **Video Tools Increase Expression And Misinformation** - AI-generated video broadens creative access but also magnifies misinformation risks and attention harms. - Solving trust will require verification systems beyond simple watermarking and platform-only fixes. Transcript: Kanjun Qiu Who couldn't before? They didn't have the tools or the tools were super hard to use. And I think that's really cool. That's really powerful. And on the other hand, like I always say are not bad. Incentives are bad. Like on the other hand, we're in this incentive ecosystem of the internet right now where there's kind of like rampant attention capture as the primary way of making money off of things. And what I would expect is some people will use it to really deeply express themselves, and it's going to be really cool. And other people will use it in order to capture attention and optimize for that. And OpenAI, as the platform, also has an incentive toward capturing attention and increasing engagement so that they have more users. And so I think that the technology itself, Sora generating video is like really, really cool. And yeah, it has misinformation problems. We have to figure out how to do like watermarking or something to verify the reality of something. I think that has to be solved a different way. But the problem with it is that it's being launched into an environment that has rampantly inhumane, I think, incentives, where as humans, we are losing control over our attention. And that really sucks. Like, I don't think that's a future we want to live in. ([Time 0:21:30](https://share.snipd.com/snip/3feb1b47-baf8-498e-a474-59b084529c36)) - **Opaque Systems Trigger Gaming And Inequity** - AI-mediated decisions create 'arms race' behaviors when systems are opaque and lack recourse. - Black-box screening incentivizes gaming and amplifies harms for those with the least power. Transcript: Kanjun Qiu This is actually a perfect example of when AI mediates human relationships. So we're going into a world where algorithms, AI, I use them interchangeably, and algorithms are making more and more decisions about our lives, whether we get a job or whether we get Our mortgage or whether we get approved for something or prisoner recidivism. There's all of these different things where the algorithm is making the decision. And this is like a perfect example of what you'll see in that situation when there's no recourse for the human. So when there's no recourse for the person who's being affected by the algorithmic decision, we will see these arms race effects. And so, yeah, I think this is basically, let's see, I actually built an ML recruiting startup in my last company and saw that when AI systems are black boxes, people will try to game them. And ultimately, I think the real solution is better incentive alignment between the creators of the algorithm and also the users, quote unquote. And also laws, better laws that protect people. It's about power and in a way about control. Like when you have a human making these decisions, you can generally appeal like, hey, I think you misunderstood or something like that. But right now, our legal infrastructure doesn't cover algorithms very well. Like we don't have very many laws that talk about how should we govern the way that algorithmic decision-making affects people. When we have people making decisions that affect people, we do have laws that govern, like discrimination and things like that. And some of those laws also cover algorithmic decisions. But it's not complete by any means. And internally at Imbue, we have this notion we call lawless spaces, where a space is lawless if there are not many laws, like the Wild West back in the day, where you could go steal money From a bank and not get punished, or you could do whatever you wanted. And that's great for innovation and trying things and freedom. But when it's lawless, then you go into this lawless space, you get bonked on the head, there's no recourse. ([Time 0:29:00](https://share.snipd.com/snip/d336bc47-2b0e-4ca3-bdbc-ecbb525d7bc6)) - **Common Source As A Middle Path** - 'Common source' could bridge open and closed models by letting users modify software while preserving commercial incentives. - This model aims to let creators get paid while users retain the ability to adapt tools to their needs. Transcript: Kanjun Qiu So I actually think like going into the future, it's important for most software to be open source and for each individual person to be able to take the source code and to use an agent to Modify it using natural language in a way that feels more intuitive to us. And at Embiid, it's something that we're working on. We call it common source. Right now, we have open source where volunteers make stuff for free. And then on the other side, we have closed source where enterprises use all of this free software that has been built by volunteers and make money off of it, but don't pay the volunteers. And there has to be something in the middle where you can ship your software to other people, let it be open source so that the users, your users can change it, but you still get to make money Off of it. But that doesn't exist today. ([Time 0:35:00](https://share.snipd.com/snip/1aa612f8-8314-4076-937a-142d27712acb)) - **Demand Explainability And Control** - Make AI systems explainable and controllable so people can see and change decision logic that affects them. - Use general models to monitor bias and flag when systems optimize against protected groups. Transcript: Kanjun Qiu Think this is part of why it's important for these systems to be more explainable and controllable, where ideally you could like see what's happening in the decisions and like be able To change it. Today, it's kind of like a black box with some reinforcement loops. Like you're like, yes, no, yes, no. And then you don't know what the black box is learning and what it's starting to index on. I think more general models might actually solve some of this problem. Like the more general the model is, the more it understands about the world, the more it understands like, hey, we shouldn't be optimizing for this, like, we shouldn't be optimizing Against minorities. Or, oh, maybe we could flag if we are ending up optimizing against minorities. Or maybe we could analyze the data at the end and take a look and see what's happening. And so I actually think that we can use these models to make much better, like, you know, the optimistic take is that we can use these models because they give us a lot more intelligence Capability to analyze the effects of what's going on in, say, hiring and what's going on with these algorithms and to change them to our liking. And this is kind of a world, you know, I imagine where we are controlling the AI and we are directing it to create the kind of outcomes that we want to see that's more in line with our intentions. ([Time 0:37:25](https://share.snipd.com/snip/a2c92e50-ca0a-4c0a-8e8a-4536c9c7b459)) - **Use AI To Upskill And Scale Individuals** - Move away from the recruiter-applicant 'arms race' by investing in upskilling and giving applicants AI tools to scale themselves. - Empower people to build personal agents that automate and amplify their abilities rather than competing with opaque systems. Transcript: Kanjun Qiu Ultimately, I think where the value is is to get out of this war of attrition. And it's hard to know how to do that. Part of what's happening is there aren't that many. It's hard to find jobs right now. Maybe people aren't upskilled in the right ways. And so maybe we could direct more of our energy and attention toward upskilling people and trying to teach them how to have the skills that are needed. And with AI systems, there are a lot of creative ways to use AI systems to learn really fast, learn new things really fast. This also goes to how we think about AI, where instead of letting AI automate us out of a job, like somebody else can automate us out of a job today, there is an alternate world where we can Use AI to automate ourselves out of our jobs. And that actually feels a lot better and is awesome. Like I use my agents, you know, I make, I make little scripts and agents to like automate myself out of some of the processes that I'm, I have to do every day. And what that means is a world in which we get all of these like tiny AI systems that are embedded throughout our lives that we're creating and we're controlling those workflows. ([Time 0:39:35](https://share.snipd.com/snip/012e9e4d-8f10-422c-b58c-d67a75b9d139))