Can We Ever Trust an AI Lawyer?  ## Metadata - Author: [[Decoder with Nilay Patel]] - Full Title: Can We Ever Trust an AI Lawyer? - Category: #podcasts - URL: https://share.snipd.com/episode/73aca772-c620-45c9-877e-9dddc2ec772b ## Document Notes ## Highlights - **Robin AI's Practical Use Case** - Robin AI helps businesses by answering legal questions from company policies and past contracts. - An example is employees querying if they can accept gifts like tickets based on compliance rules. Transcript: Jon Fortt What's the latest? Richard Robinson We're building an AI lawyer, and we're starting by helping solve problems for businesses. Our goal is to essentially help businesses grow. It's one of the biggest impediments to business growth is it's not just about revenue. It's not just about managing your costs. It's legal complexity. Businesses dealing with legal problems actually slows them down. So we exist to solve that problem. We've built a system that essentially helps a business understand all of the laws, all of the regulations that apply to them, and also all the commitments that they've made, their rights, Their obligations, their policies. We use AI to make it easy to understand that information, easy to use that information, to ask questions about that information, to solve legal problems. We call it legal intelligence. So that's how we're taking the latest AI technologies. We're taking them to law school and we're giving them to the world's biggest businesses to help them grow. Jon Fortt A year and a half ago, I talked to you and your description was a lot heavier on contracts. But you said, yeah, we're heading in a direction where we're going to be handling more than that. It sounds like maybe you're more firmly in that direction now. Richard Robinson Yeah, that's correct. I mean, we've always been limited by the technology that's available. Before ChatGPT, we had very traditional AI models. Today, we have, as you know, much more performant models. And so that's just allowed us to expand our ambition. And you're completely right. It's not just about contracts anymore. It's about policies. It's about regulations. It's about the different laws that apply to a business. We want to help them understand their entire legal landscape. Jon Fortt Give me a scenario here, case study-wise, on the sorts of things that using your technology, your customers are able to sort through. And then recently, you guys have amped up your presence on AWS Marketplace. So there are a lot more different types of companies that are going to be able to plug in Robin AI's technology to all kinds of different legal landscape. Software and data that they have available. So case study, what's the technology doing now? How is that kind of hyperscaler cloud platform potentially going to open up the possibilities for you? Richard Robinson We help solve concrete legal problems. So a good example is every day our customers, organizations, they want to know whether or not they're doing something that's compliant with their company policies. And those policies are uploaded to our platform. And anybody can just ask a question that historically would have gone to the legal team or the compliance team. They can say, I've been offered tickets to the Rangers game. Am I allowed to go under the company policy? And we can use AI to intelligently answer that question. Every day, businesses are signing contracts. That's how they record pretty much all of their commercial transactions. And now they can use AI to look back at their previous contracts and then use that to help them answer questions about the new contract they're being asked to sign. So if you're doing a deal with the Rangers and you've worked with the Mets in the past, you might want to know, what did we negotiate that time? How did we get through this impasse last time? You can use the Robin platform to answer those questions. ([Time 0:04:08](https://share.snipd.com/snip/ee6c9f47-5650-48de-8479-c0ea1d98c2af)) - **Validate AI Legal Outputs** - Always verify AI-generated legal outputs with original sources before submission. - Lawyers must remain responsible for validating information despite AI tools to avoid errors or sanctions. Transcript: Jon Fortt Are you avoiding the issue of hallucinations? We've seen these mentions in the news of lawyers submitting briefs, arguments to a judge that include stuff that is just completely made up. We hear about the ones that get caught. I imagine we don't hear about the ones that don't get caught. I know those are different kinds of AI uses than what you're doing with Robin AI, but there's still got to be this concern in a fact-based, argument-based industry about hallucination. Richard Robinson There is. It's the number one question our customers ask. And I do think it's a big part of why you need specialist models for the legal domain, because it's a specialist subject area. It's a specialist domain. You need to have applications and people like Robin, who are not just taking ChatGPT or Anthropic and doing nothing with it. You need to really optimize its capabilities for the domain. So in terms of what we're doing, to answer your question directly, I mean, we include citations with very clear links to everything the model does. So every time we give an answer, you can quickly validate the underlying source material. That's the first thing. The second thing is we are working very hard to only rely on external, valid, authoritative data sources. So we connect the model to specific sources of information that are legally verified so that we know that we're referencing things you can rely on. And I'd say the third is that we're educating our customers and reminding them that they're still lawyers. You know, I used to write cases for courts all the time. That was my job before I started Robin. And I knew back then it's my responsibility to make sure every source I reference is 100% correct. It doesn't matter which tool you use to get there. It's on you as a legal professional to validate your sources before you send them to a judge, before you even send them to your client. ([Time 0:11:23](https://share.snipd.com/snip/09f9f0f7-5aa8-4ef5-bc96-f85a0727a50f)) - **Facts vs Truth Challenge** - The gap between facts and truth complicates AI's role in society's understanding. - People may accept or reject information based on beliefs, not factual accuracy, challenging AI's impact. Transcript: Jon Fortt And it seems to me that the two steps at the core of this are first, how do we figure out what's true? And then two, how do we figure out what's fair? And you are a student of a practitioner of debate. We'll get a little bit into that in a bit too. But figuring out what's true, I'm not a professional debater, though I have been known to play one on TV. Figuring out what's true is step one, right? Yeah, I think it is. Richard Robinson It's increasingly difficult because there are so many competing facts and there are so many different communities where people will selectively choose their facts. But you're right. You need to establish the reality, the core facts before you can really start to make decisions and start to debate what you should be doing and what should happen next. And I do think AI helps with all of these things, but it can also make it more difficult. These technologies, they can be used for good and bad. And so it's not obvious to me that we're going to get closer to establishing truth now that we have AI. Jon Fortt I think you're touching on something here that's interesting right off the bat, the difference between facts and truth. Richard Robinson Yes, that's right. It's very difficult to really get to the truth. I mean, facts can be selectively chosen. I've seen spreadsheets and graphs that technically are facts, but they don't really tell the truth. So there's a big gap there. Jon Fortt How does that play into the way we as a society should think about what AI does? AI systems are going out training on data points that might be facts, but the way those facts or those details or those data points get arranged ends up determining whether they're telling Richard Robinson Us something true, right? I think that's right. I think that as a society, we need to use technology to enhance our collective goals. We shouldn't let the technology just run wild. Now, that's not to say that we should regulate these things, because I'm generally quite against that. I think we should let innovation happen to the greatest extent reasonably possible. But we should have a say, as consumers, we do have a say in how these systems work and how they're designed and how they're deployed. I think as it relates to the search for truth, the people who own these systems and who use them, they've grappled with these questions in the past. Search certain questions like the racial disparity in IQ in the United States, you're going to get a fairly curated answer from the Google search. And I think that that is in itself a very dangerous, polarizing set of topics. And I think we need to certainly ask ourselves the same questions that we've asked with the last generation of technologies to this new technology, because that's what it is. AI is just a new way of delivering a lot of that information. In some ways, it's a more effective way. It's going to do it in a more convincing and powerful way. So it's even more important that we ask ourselves, how do we want information to be presented? How do we want to steer these systems so that they deliver truth, so that they avoid bias? It's a big reason why Elon with Grok has taken such a different approach than Google took with Gemini. If you remember, the Gemini model famously had black Nazis and it refused to answer certain questions. It was alleged to have some political bias. It was because I think Google was struggling to answer and resolve some of these difficult questions about how do you make the models deliver truth, not just facts. And they maybe hadn't spent enough time passing through how they want to do that. Jon Fortt Well, I mean, yeah, Grok seems to be having its own issues. It is. Right? It's very much like people, right? Like somebody who swings in one way, you know, has trouble with certain things, and somebody who swings in another way has trouble with some other things. There's the matter of facts, and then there's what people are inclined to believe. ([Time 0:20:03](https://share.snipd.com/snip/f580b8f2-3ea5-43c3-9706-ea1433d5e348)) - **AI Attention vs Truth** - Current AI and social media algorithms optimize for attention, not truth. - Tools like citations and community notes empower users to validate information and combat misinformation. Transcript: Richard Robinson I think a lot of this comes down to validation. Sam Altman said something that I thought was really insightful. He said that the algorithms that power most of our social media platforms, X, Facebook, Instagram, they're the first example of what AI practitioners call misaligned AI at scale. And what we mean by that is systems where the AI models are not actually helping achieve goals that are good for humanity. So the algorithms in these systems, this is before ChatGPT, but they are using machine learning to work out what kind of content to surface for people. And so it turns out people are entertained by really outrageous, really extreme content. It just keeps their attention. I don't think anybody, though, would say that's good for people. It makes them better. It's not nourishing. There are no nutrients in a lot of the content we're getting served to us on these social media platforms, whether it's politics, whether it's people squabbling, whether it's culture Wars. These systems have been giving us information that's designed to get our attention. And it seems like that's just not good for us. It's not nutritious. So I think that on the whole, to date, we're not doing very well in the battle to search for truth because the models haven't been optimized to do that, actually. They've been optimized to get our attention. And so I think you need platforms that find ways to combat that. So to the question of how do systems like us, but also more broadly, how do AI applications help combat this? I think it is by creating tools that help people validate the truth of something. Most interesting example of this, at least in the popular social paradigm, is community notes. Because community notes is a way for someone to say, this isn't true, or this is false, or you're not getting the whole picture here. And it's not edited by a shadowy editorial board. It's generally crowdsourced. Wikipedia is another good example. These are systems where you're basically using the wisdom of the crowds to validate or disvalidate information. In our context, we use citations. So we're saying, don't trust the model. Test it. It's going to give you an answer, but it's also going to give you an easy way to check for yourself if we're right or if we're wrong. And I think for me, this is the most interesting part of AI applications. It's all well and good having capabilities, but as long as we know that they can be used for bad ends or they can be inaccurate, we're going to have to build countermeasures that make it Easy for society to basically get what we want from them. And I think community notes, citations, these are all children in the same family for trying to understand how these models truly work and are affecting us. ([Time 0:28:41](https://share.snipd.com/snip/5f0c5787-a791-4d83-ad76-e0eb18700ba7)) - **Personalized Coaching for Teams** - Coach individuals by focusing on strengths and weaknesses for effective team debate. - Apply this approach to build diverse expert teams in AI product development for higher performance. Transcript: Jon Fortt Yes. Richard Robinson All the time I see the similarities between coaching the England team and running my business, it still surprises me, to be honest. But I think that when you're coaching debate, the number one thing you're trying to do is to help people learn how to think. Because in the end, they're going to have to be the ones who stand up and give a five or seven minute speech in front of a room full of people with not a lot of time to prepare. And when you do that, you're going to have to think on your feet. You're going to have to really find a way to come up with arguments that you think are going to convince the people in the room. So for me, it was all about helping teach them that there's two sides to every story, that beneath all of the information and facts, there's normally some valuable principle at stake In every clash or every issue that's important. And you want to try and tap into that emotion and that conflict when you're debating. You want to find a way to understand both sides because then you'll be able to position your side best. You'll know the strengths and weaknesses of what you want to say. And I'd say as well, final thing, it was all about coaching individuals. Each person had a different challenge or different strengths, different things they needed to work on. Some people would maybe speak too quickly. Some people were not confident speaking in big crowds. Some people were not good when they had too much time to think. And you have to find a way to coach each individual to manage their weaknesses and bring the team together so that they're more than the sum of their parts. And so when we're building software, I see this challenge all the time, right? We, number one, we're dealing with systems that require different expertise. No one is good at everything that we do. We've got legal experts, we've got researchers, we've got engineers, and they all need to work together using their strengths, managing their weaknesses, so that they're more than The sum of their parts. And so that's been a huge lesson that I apply today to helping build Robin AI. I would say as well, just focusing on individuals, because at any given time, especially in a startup, you really need to find a way to put people in the position where they can be in their Flow state and do their best work. ([Time 0:38:16](https://share.snipd.com/snip/2ca05cfb-b81a-497c-8015-07b9be097a25)) - **AI Enhances Communication Prep** - Use AI to prepare, research, and review before and after important conversations or meetings. - AI supports reflection and improvement through transcripts, action items, and coaching suggestions. Transcript: Richard Robinson But I think you can maybe think about that as in-game. But before and after the game, AI can be really powerful. So people in my company will often use AI in advance of a one-to or in advance of a meeting where they know they want to bring something up and they want some coaching on how they can land The point as well as possible. Right? Or maybe they're concerned about something, but they feel like they don't know enough about the point and they don't want to come to the meeting ignorant. Well, they'll do their research in advance. And so I think AI is helping before the fact. And then after the fact, we're definitely seeing people basically look at the game tape. All the meetings at Robin are recorded. We use AI systems to record all our meetings. Transcripts are produced. Action items are produced. Summaries are produced. And people are asking themselves, how could I have run that meeting better? I feel like the conflict I had with this person didn't go the way I wanted. What could I have done differently? And so I think AI is helping there. I'd say final point. We have seen systems, and not much is written about these systems, but we've seen technologies that are extremely convincing one-on So there was a company called Character AI, which Was acquired by Google. And really what it did is build AI avatars that people could interact with. And it would sometimes license those avatars to different companies. And we saw a ([Time 0:42:31](https://share.snipd.com/snip/83a9dc53-9928-464e-9bc1-88cc54fc0cc2)) - **New Problems from AI Usage** - AI increases text production, potentially diluting content quality and decreasing reliance on writing skills. - There's a growing crisis of validation as people tend to distrust extraordinary online claims by default. Transcript: Richard Robinson Well, we're definitely creating new problems. And this is true of all technology that's significant. It's going to solve a lot of problems, but it's going to create new ones. I certainly think that with AI, one thing I'd say point to three things. Number one, we are creating more text and a lot of it is not that useful. So we're generating a lot more content for better or for worse, right? So you're seeing more blogs because it's easy to write a blog now. You're seeing more articles, more LinkedIn status updates, and more content online. Whether that's good or bad, we are generating more things for people to read. And what may happen is people may just read less because it's harder to sift through the noise to find the signal. Or they may rely more on the systems of information they're used to to get that confirmation bias. So I think that's one area AI, at least today, has not solved. It's a new problem is generating incremental text has gotten dramatically cheaper and easier than it ever was. I'd say another thing I observe, second thing is people are losing the skill of writing because you don't have to write anymore, really. You can just tell ChatGPT. You don't even need to tell ChatGPT in proper English. Your prompts can be quite badly constructed and it kind of works out what you're trying to say. And I think what I observe is that people's ability to sit down and write something coherent that takes you on a journey is actually getting worse because of their dependence on these External systems. And I think that's very, very bad because writing to me is deeply linked to thinking. In some ways, if you can't write a cogent sequential explanation of your thoughts, that tells me that your thinking might be quite muddled. And Bezos had a similar principle. He would ban slide decks and insist on a six-page memo because you can hide things in a slide deck, but in a six-page memo, you have to know what you're talking about. I think that's a gap that's emerging because you can depend on AI systems to write. I think it can excuse people from thinking. And then I think the final thing, at least that I would point to, is that we are creating this crisis of validation. So it's no longer possible when you see something extraordinary online. I, by default, don't necessarily believe it. I just assume it might be fake. So I'm not going to believe it until I've seen more corroboration, more validation. By default, I assume things aren't true. And that's pretty bad, actually. It used to be that if I saw something, I would assume it's true. And it's kind of flipped the other way over the last five years. So I think AI has definitely created that new problem. But like we talked about earlier, I think there are ways you can use technology to help combat that to fight back. I'm just not yet seeing too many of those capabilities at scale in the world. ([Time 0:47:53](https://share.snipd.com/snip/bb676923-891b-428d-a578-fd1ce17e90ed))