The Surprising Case for AI Judges  ## Document Notes ## Highlights - **Determinism Would Reduce Disputes** - The legal system should be more deterministic to reduce disputes and improve planning. - Bridget McCormack argues AI can make many routine legal outcomes predictable and efficient. Transcript: Nilay Patel A computer. You can feed it inputs and it'll API access the law and then you'll get some predictable outputs. And I'm always trying to convince people that that's not the case. And just even hearing you talk about the politics of running the legal system underlines for me that the legal system is absolutely not deterministic. Should it be? Because you're the first person I could just straightforwardly ask that question to. Should the legal system be more predictable and deterministic? Bridget Mary McCormack So it absolutely should be, at least in a majority of cases. In fact, if it were more deterministic, we would have fewer disputes, right? It's because it's probabilistic, and I agree with you, it for the most part is because it's run by humans who are, you've met humans, right? They're flawed. It therefore isn't always predictable. If it were more predictable, we would be a more efficient and effective system. We'd avoid a lot of disputes because people could plan their business around what, in fact, the rule was going to be and how it was going to be enforced and how they could, you know, count On it being enforced. That's true for, in my view, you know, most cases for which there is a rule of law and we know, you know, how it how it's been interpreted historically or at least how it's been correctly Interpreted historically by a majority of courts. Are always going to be new frontiers in legal. So, you know, cases where courts are having to decide how to interpret a new statute, courts are going to have to figure that out for the first time. That's not going to be able to be deterministic. I mean, could get better and better, frankly. I think AI could do a very good job at the front end of statutory drafting in making sure there was less ambiguity in statutory terms, I think. So I think AI could impact that. But there are even, you know, still modern questions about historic provisions in statutes and, you know, constitutions, state and federal, that we have entrusted judges to decide. So I don't think it can all be deterministic. I think an awful lot could be, and it would improve the way the law operates. ([Time 0:09:21](https://share.snipd.com/snip/a3c4aebc-2f32-4ff7-954d-63c27750ca2a)) - **AI Makes Parties Feel Heard** - AI systems can enhance perceived fairness by ensuring parties feel heard and understood. - McCormack says iterative agent summaries let parties confirm claims and evidence until satisfied. Transcript: Bridget Mary McCormack Do view it as a significant advantage of an AI dispute resolution system. And we can talk more about when that's appropriate and when it's not. I don't think every dispute should go to an AI dispute resolution system. But when parties prefer it, it was for me at the front end of what we built, which is a very narrow product right now, but I undervalued it and underappreciated it. So at the front end of our AI arbitration process, which is really a series of agents that operate across the process on the back end, even though the parties are interacting with one, The agents take in the party's complaints and their, you know, whatever pleadings they're filing and whatever evidence they think supports their claims. And then a series of agents like parses the claims, the elements of each claim, the evidence that may or may not support each claim, but the parties believe supports each claim and the Legal framework that surrounds it. And then it goes back to the parties and says, here's my understanding of what the claims are, you know, what your claims are party A and yours party B and what the elements are and what The evidence is and what the legal framework is. And did I get that right? And the parties get to say, you did or no, you didn't. You missed this one element or this one claim or the fact that this evidence supports both of these claims, not just one. And then it goes to work again. The agents go to work again until the parties are satisfied that they have been heard and understood. I mean, maybe we could do that in courts, but we would have to spend a whole lot more money. Like, imagine if judges in trial courts or even in, like, appellate courts. In appellate courts, you file briefs and then you wait by your computer for months and months and months to see, like, when some white smoke emerges from the state Supreme Court building And then you get a decision and the decision may or may not have even addressed all of the issues that you raised. You know, that happens all the time. You know, I reviewed 2000 cases a year, 2000 applications a year when I sat on the Michigan Supreme Court. And I can't tell you how many of them were the intermediate appellate court didn't even rule on issue number three. Like they just they just ignored it, you know, and we just want an answer. So imagine if courts could do that. Imagine if, you know, trial judges or appellate judges could pull the parties in and say, here's my understanding. Did I get it right? And then and then we could actually like be satisfied that they did. ([Time 0:28:30](https://share.snipd.com/snip/cb9efcd9-6f18-4121-8797-20c12a9423b1)) - **Start Narrow And Keep Humans In Loop** - Start with narrow, document-only dispute types where evidence is structured. - Use human arbitrators in the loop to review AI reasoning and finalize awards. Transcript: Bridget Mary McCormack It's a website, although I assume we will have an app version of it, and we have an app version of all of our services. You can log in on your phone, but you're logging into the case management system. So really it's two things. We built an AI native case management system for the AI arbitrator to operate on. That effort is time well spent because it will replace our legacy case management system for all disputes within two years. That's just going to be something everybody has an opportunity to benefit from, which saves time and money. But the AI arbitrator is, like I said, a bunch of different agents, probably 20, sometimes more, depending on the complexity of the dispute that operate across the arbitration process. A bunch operate at the front end. We talked about those that are parsing claims and organizing arguments and organizing claims, making sure the party is satisfied. And then there's a bunch of reasoning agents. And those reasoning agents take the summary of the dispute that the parties are satisfied with and start reasoning across it. And then there are agents that do a draft award. There's a human in the loop throughout. In the beginning, it's the parties. They are the human that are making sure that they're heard and understood. Then we have a cohort of construction arbitrators who serve as the human in the loop arbitrator for the reasoning and award drafting. And it's the human arbitrator who ultimately issues the award, and she makes any changes that need to be made so that she's comfortable with the award. ([Time 0:31:32](https://share.snipd.com/snip/60b0957b-1f4b-4b48-8166-388cad2b23c6)) - **Give Domain Experts AI Access** - Equip domain experts with AI tools and enterprise access to discover use cases. - Give all staff access and let domain teams build point solutions before scaling. Transcript: Bridget Mary McCormack Like that's obviously going to have an impact. So when I got to the AAA, we gave everybody enterprise chat GPT licenses. And I mean, everybody, like not just our engineers, but our caseworkers and our legal team and our marketers and our, because I think with any general purpose technology, you need the Domain experts to figure out where it's going to impact them. And we started building point solutions. And so our AI engineers, we're not historic AI engineers, but like everybody else, they learned pretty quickly. And so we do have a very talented set of AI engineers. With this particular product, we built with a partner. We sort of, we did, you know, co-pilots. We built it with Quantum Black. Quantum Black is McKinsey's AI team. ([Time 0:36:32](https://share.snipd.com/snip/8d9a01aa-2325-40b4-a7f6-0f6980dd0c63)) - **Use AI For Early Case Assessment** - Offer AI as an early case evaluation tool so parties gauge merits before heavy investment. - Build an intake that returns likely outcomes from submitted evidence to inform settlement choices. Transcript: Bridget Mary McCormack They want to be able to run it just on their own. They want to put all of their evidence up into it and get an early case assessment about where their case is likely to go so they can figure out whether they even want to spend any time in Arbitration. ([Time 0:40:11](https://share.snipd.com/snip/ae280890-4a0c-4d60-9193-2640f3d7cba2)) - **Rising Self-Representation Needs Tools** - Self-represented parties in commercial disputes are rising and need accessible tools. - McCormack notes a marked increase in self-representation and suitability for AI arbitration. Transcript: Bridget Mary McCormack I follow our case filings day by day. And the self-represented parties in commercial and construction disputes are right now, I mean, it's one month, so who knows, maybe it was a weird month. But January of 2026 was double what it was in January of 2025. So in fact, I think this system is kind of perfect for the smaller party construction disputes. And as you know, there are also lots of those smaller construction projects where people really don't want to have to hire a lawyer to go figure out how to sort it out. ([Time 0:41:41](https://share.snipd.com/snip/dd581e53-d7a4-4532-a23b-7e75ed637c9c)) - **Not All Disputes Belong In AI Systems** - Some disputes (criminal, public-interest) must remain in public courts for accountability. - McCormack argues cases involving liberty or government actions shouldn't be outsourced to private AI systems. Transcript: Bridget Mary McCormack There are also some for which it will never be appropriate. Nilay Patel Where will it never be appropriate? Bridget Mary McCormack I mean, this is just my view, but I believe that criminal cases, cases where the government is accusing you of something and, you know, wants to take your liberty as a result of that accusation, Or cases that individuals or organizations or even businesses bring against the government should happen in public courtrooms and with publicly appointed or elected if you're in State court judges. Those have to happen in courtrooms. Nilay Patel One of the things you and I briefly talked about the first time we met and talked about all this was the idea that there's transparency in the state courts. There's transparency when the government sues you and you need that transparency to build trust. Like you brought up the healthcare industry. There's zero trust in the healthcare industry. And there's a lot you could say about the consequences of the absolute lack of trust in the healthcare industry. If my medical billing goes to an AI agent, do you think that's going to result in more or less trust? Bridget Mary McCormack Yeah, I don't know. It depends on the AI agent who built it, what their audit trail looks like and how transparent they are about showing you what that audit trail looks like. Let me go back for a minute. I said payer-provider disputes. I'm talking about disputes between healthcare, hospitals, and insurance companies. Do you think there's a lot of confidence or trust in insurance companies? Because I don't think so. So when, you know, if they're not at all, that's kind of what I'm saying. Yeah. But I mean, so imagine, you know, disputes between those two that could be resolved instead of taking two years. So somebody is waiting for their insurance company to tell the hospital that they're going to cover their treatment and they're dead before they get an answer. Imagine if we could just resolve those right away, but in a system that transparently shows its work and shows its audits, right? ([Time 0:43:21](https://share.snipd.com/snip/c688abb9-dd41-4923-97c1-29857041a880)) - **Governance Beats Generic LLMs** - Governance, grounding, and narrow domain training reduce AI hallucinations and risks. - McCormack emphasizes transparent audits and academic review to build trust. Transcript: Bridget Mary McCormack Your system has to be governed, trained in, and grounded in the kind of reasoning that you're asking it to do. So it sounds, you know, extremely narrow that we're doing only documents, only construction cases, but there's a reason for that. That's where we could build a governed and harnessed, you know, agentic system. We keep a human in the loop to make sure that before an award issues, there were no hallucinations, even with our governance. And we're going to be very transparent about all of our audits. And that's, I think, really critical to growing trust. I think a lot about whether the frontier models get so good eventually that they can just do this, that you don't need, you know, an institution training and governing an agentic system In a specific kind of dispute because, you know, we move past the age of hallucinations. It doesn't sound like that's imminent today, but, you know, I don't know if it's imminent six months from now. I mean, I didn't know last week that agents were going to be on a subreddit talking about us. You know, I didn't know Maltbook was coming, right? So you can surprise me. For now, I think you have to have governed, transparent, audited systems so people can grow that trust. ([Time 0:50:05](https://share.snipd.com/snip/12ef3e81-10b0-42ab-9979-d9e0c11c0048)) - **Judge Mistakenly Insisted She Attended** - McCormack recounts a probate judge falsely claiming she made statements at a forum she did not attend. - The judge insisted she was present, illustrating human error and bias in the judicial system. Transcript: Bridget Mary McCormack But I was the chief justice of, you know, the Michigan Supreme Court. I've been in that role, like, three years. And we were working on some reforms in the probate court system. And some of the probate court judges, I guess, didn't like them. And one of those probate court judges from a county not that far from where I live got on his listserv and was like just talking trash about me. He said that I was at a forum and I had said a bunch of stuff and like how outrageous it was. Fine. I want people should criticize me whenever they want. The thing was, I wasn't at that forum. My colleague, Megan Kavanaugh, also a white woman with an Irish name, was at that forum. And I know Megan, she probably did say those things. So I called the guy. I said, Judge, you know, you made this claim about me on the listserv to every probate court in the state of Michigan, but I wasn't there. Like, I wasn't at that forum. I have an alibi. And you know what his answer was? His answer was, yes, you were. What do you do with that? Like, I mean, what do you do with that? He won. I was like, well, I don't know. Okay. I mean, you know, what do you even do with that? ([Time 0:52:22](https://share.snipd.com/snip/61bba7d4-5916-44c8-8d10-3dbcac8e8742)) - **Mission And Transparency Counter Client Bias** - Nonprofit mission and due-process protocols can mitigate capture risks from paying clients. - McCormack contends transparent audits and showing work help prove impartiality to parties. Transcript: Nilay Patel In particular the automated system doesn't favor them over time? Because that's a pretty easy outcome to start programming in. Bridget Mary McCormack Super good question. As you know, easier to train a data set than a human. So frankly, you know, and let me step back for a minute, in our consumer cases, again, we do not accept cases unless, consumer B2C cases, unless a business has cleared their clause with Our due process protocols. So we're, again, in a lucky position there, I guess. But back to the, you know, who's in charge, you know, for us, if the consumer feels like the process wasn't fair, that's not going to work. You know, we're just, you know, we're in the business of actually, you know, giving more options to more parties. And if half of the parties in one kind of dispute type, I should say, we mostly do B2B cases, but when we do B2C and they've satisfied our due process protocols, it's critically important To us that both sides feel in the individual case that they were treated fairly. If it's algorithmic, I mean, if it's automated to some extent, even in part, that's just as true, but easier to show our work. We did a lot of training, judicial training, and we do a lot of arbitrator training when human arbitrators decide cases to make sure that parties get equal treatment, fair treatment. But you can de-bias a data set a lot easier than you can de-bias a human. And so when you benchmark and do your audits of your AI arbitration system and you show your work, you can either convince the public that it's treating both people fairly or you won't. ([Time 1:02:16](https://share.snipd.com/snip/433830cd-3d8d-4a3a-ba31-2def0614709d)) - **Agentic Commerce Needs Dispute Design** - Agent-to-agent commerce will create new dispute types that require automated resolution. - McCormack warns agentic contract negotiation needs dispute processes to fix inevitable agent mistakes. Transcript: Bridget Mary McCormack I mean, one other thing that I spend a lot of time thinking about is how many B2B contracts are going to be negotiated and executed by agents. Like Walmart is doing some significant number, has agents negotiating and executing a significant number of their contracts. In a B2B agentic commerce world, and so you probably read these estimates as much as I do, and maybe some people say that by the end of 2027, as many as 40 percent of contracts might be negotiated And executed agent to agent. Sometimes agents will make mistakes, right? Like, I mean, as good as they are, they will make mistakes. What's the dispute resolution process there? And is it just is it is it a on chain process that, you know, maybe it better? Like it's an upstream automated dispute resolution process. I'd like to be in the conversations to figure that out because you need that somewhere, right? Your agentic commerce is only as good as your process for fixing it when it breaks. But nobody's talking to me about that. I want them to. ([Time 1:13:11](https://share.snipd.com/snip/e04adddb-11b6-4d65-9be0-d5a956833ff2))