# Why nobody's stopping Grok  ## Episode metadata - Episode title: Why nobody's stopping Grok - Show: Decoder with Nilay Patel - Owner / Host: The Verge - Guests: [Riana Pfefferkorn](https://share.snipd.com/person/ec7e9b8a-6c49-4962-b990-02d5fa35da13) - Episode publish date: 2026-01-22 - Episode AI description: Riana Pfefferkorn, a policy fellow at the Stanford Institute for Human-Centered AI, dives deep into the controversies surrounding Grok, Elon Musk’s AI chatbot notorious for generating problematic imagery. She discusses the challenges of legal frameworks dealing with non-consensual content and how the evolution of technology outpaces existing laws. Riana highlights the inadequacies of age verification and the implications of policies on minors, along with the role of app stores and payment processors in mitigating harm. Tune in for a thought-provoking conversation on tech policy and its impact. - Duration: 01:05:51 - Episode URL: [Open in Snipd](https://share.snipd.com/episode/ad9faed7-d215-4a2d-9ff4-76ea2c55d3f5) - Show URL: [Open in Snipd](https://share.snipd.com/show/35c6d22e-02b6-48ad-b520-9d1cccd695c4) - Export date: 2026-02-17T23:32:21 ## Snips ### [Scale Changes The Legal Picture](https://share.snipd.com/snip/8b96c404-c834-4bc6-b0b9-4dca70de3ad0) 🎧 01:38 - 02:53 (01:14) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/d280b4fe-4881-409c-a4fc-11b11f6eb7cf" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - Grok lets users generate and distribute non-consensual sexualized images at scale across X almost instantly. - That speed and integration makes prior legal categories and defenses insufficient to address the harm. #### 💬 Quote > Grok, the chatbot made by Elon Musk's XAI, is able to make all manner of AI-generated images, including non-consensual intimate images of women and minors. > — Nilay Patel Nilay Patel introducing the Grok controversy #### 📚 Transcript **Nilay Patel:** By now, we're several weeks into one of the worst, most upsetting, and most stupidly irresponsible AI controversies in the short history of generative AI. Grok, the chatbot made by Elon Musk's XAI, is able to make all manner of AI-generated images, including non-consensual intimate images of women and minors. What's more, because Grok is connected to X, the platform formerly known as Twitter, users can simply ask Grok on X to edit any image on that platform, and Grok will mostly do it, and then distribute that image across the entire X platform. Over the past few weeks, X and Elon have claimed repeatedly that various guardrails have been imposed on the image generator, but in testing by Verge reporters and by others, these guardrails have been mostly trivial to get around. In fact, it's become abundantly clear that Elon wants Grok to be able to do this, and that he's become very annoyed with anyone who wants him to stop, particularly the various governments around the world that are threatening to take legal action against X. This is one of the situations where if you just describe the problem to someone, they will intuitively feel like someone should be able to do something about it. And it's true. Someone should be able to do something about a one-click harassment machine that's generating intimate images of women and children without their consent. --- ### [Many Outputs Fall In Gray Legal Zones](https://share.snipd.com/snip/6cc2c27e-2ff9-4fc8-b3ba-0a58076ea09c) 🎧 08:07 - 10:21 (02:14) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/41764a2c-addc-4b24-98ff-03ea315606a3" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - Some Grok outputs may meet existing criminal laws for CSAM and non-consensual intimate imagery, but many sit in legal gray areas. - Enforcement will be fact-specific and depends on definitions like nudity and morphed images. #### 💬 Quote > Some of the imagery that's coming out of Grok maybe qualifies as that. ... Whether bikini pictures themselves cross that line. > — Riana Pfefferkorn Riana Pfefferkorn on how Grok images may or may not violate federal law #### 📚 Transcript **Nilay Patel:** I'll **Riana Pfefferkorn:** give you the lawyer's answer, which is it depends. The difficulty is that some of these probably do cross a line and others of them probably do not. So we have for three decades now had a federal law on the books criminalizing using a computer, whether that was Photoshop or now with AI, to modify a real child's image into being a sexually explicit image into child sex abuse material or CSAM, as we'd call it for short, used to be called child pornography. That law has survived various legal challenges at the courts appeals level. There is Supreme Court precedent saying that fully virtual CSAM is protected speech, unless it also is obscene, because there's no real child being harmed. The Supreme Court didn't deal with the part of the definition of child pornography, again, to use the term on the books, that dealt with what are called morphed images, but several courts of appeals have, and they've said, well, the harms that come from morphing a child's image into a sexually explicit image are more akin to the kinds of harms that are experienced in real hands-on abuse, and so no no, there's no first-amount protection for morphed image CSAM either. So some of the imagery that's coming out of Grok maybe qualifies as that. It's an open question whether bikini pictures themselves cross that line. And then more recently, we have the Take It Down Act, which got passed last year by Congress and signed into law by President Trump, that criminalizes non-consensual intimate imagery, whether it's real or deep fake, whether it is of a minor or of an adult. So that is now a crime to publish or to threaten to publish. There are takedown provisions, hence the backronome for the name, that don't go into effect until May. And this is certainly pointing up that a lot of people might be having to wait a long time before they can try and invoke that to require X to remove material that it has not necessarily seen fit to remove so far, even though X nominally participates in a voluntary initiative called Stop NCII for Removing NCII. But again, there too, not all of these necessarily qualify under the federal definition of what's called an intimate visual depiction, which itself by and large requires nudity. --- ### [Weaponized Harassment Through Automation](https://share.snipd.com/snip/3f4ca5c4-88f0-4216-a92f-bc1267af5e93) 🎧 12:17 - 13:40 (01:22) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/0389938c-e0c9-4da9-bc34-d74aa1dfaf96" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - The core harm is weaponized harassment: easy creation plus instant mass distribution differentiates Grok from past Photoshop-era abuse. - Existing categories (protected speech vs illegal content) don't capture harm driven by automation and scale. #### 💬 Quote > It's like the old categories and the old gradations of this is First Amendment, this isn't. Kind of, in my mind, go out the window when it's click any image, say undress her, and then instantly distribute it to millions of people. > — Nilay Patel Nilay Patel on how automation changes the harassment dynamic #### 📚 Transcript **Nilay Patel:** There's CSAM, which gets no protection, and even just photoshopping an image of a child into something illegal gets no protection, and there's longstanding loss in the book. Then there's, you have an adult, and you can change their clothes or make them naked. And somewhere along that spectrum, you get first amendment protection or not. Then there's, in my mind, the most important aspect of all, which is scale and speed of creation and scale and speed of distribution. And it feels like a lot of this conversation doesn't reckon with, you know, Photoshop has existed for a long time and Photoshop fakes of people have existed for a long time. Every female celebrity in the world has some amount of Photoshop fakes of them floating around, not wearing clothes, but that still required like someone to do it. And then there was pretty narrow distribution for those things, right? Some pretty shady websites might distribute that stuff and you had to go find them. Grok is just different on every level, right? It's like the old categories and the old gradations of this is First Amendment, this isn't. Kind of, in my mind, go out the window when it's click any image, say undress her, and then instantly distribute it to millions of people. And it just seems like we need a different framework, a different legal rationale, maybe different penalties for enabling that versus the traditional fact-based, okay, this image is this person. Is that First Amendment protected or not? --- ### [Use Defense In Depth, Not One Fix](https://share.snipd.com/snip/587c1114-65e3-461c-b180-90f9b1dc66b7) 🎧 15:12 - 16:25 (01:13) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/c1311df4-223a-4314-8403-0e6c390aee7f" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - Regulators should leverage multiple tools, not a single narrow fix, to create defenses in depth against integrated image-abuse flows. - Consider targeting platforms, app stores, payment providers, and downstream services simultaneously. #### 💬 Quote > And so you have all of these different platforms that were being brought to bear to make that happen. ... And therefore you have all of these different potential prefer points maybe to try and cut those things off. > — Riana Pfefferkorn Riana Pfefferkorn on leveraging multiple platforms and gatekeepers #### 📚 Transcript **Riana Pfefferkorn:** Even if you could reduce the volume of non-compliant ads, or if you could improve the level of app store review to keep those from happening in the first place, if you could ban or filter out some amount of the sites that are accessible even without an app for doing notification, that might be some amount of defense in depth that you could do to try and take a bite out of this problem. And then instead, we now have a one-stop shop in the form of a major social media platform that integrates an image generator that to all appearances does not have safeguards in place to prevent this kind of imagery from being generated. And like you said, is now fully integrated to do this at scale pretty much instantly. And that's where I agree with you that it doesn't seem like our regulatory frameworks are really equipped for that. We have seen from the UK, for example, that they're saying we're doing an investigation under the Online Services Act because they have already put in place obligations essentially to force certain online platforms covered by their various categories to step up and keep certain kinds of content from happening on their platforms in the first place. And so they have a different set of tools that they are looking into using. --- ### [Different Jurisdictions Move Differently](https://share.snipd.com/snip/47594013-0ecc-4e6c-bab3-6d116a486fc9) 🎧 17:25 - 19:22 (01:56) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/6813741f-2079-4d0c-8649-729f80540397" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - Other countries with laws like the UK's Online Safety Act can act faster because they permit regulation of lawful-but-harmful content. - The U.S. First Amendment and legal history make comparable action harder and slower here. #### 💬 Quote > Other places around the world, though, have laws that kick in differently and have decided in some instances, actually, yes, this is something that we can take action upon to block Grok entirely. > — Riana Pfefferkorn Riana Pfefferkorn comparing U.S. and international regulatory tools #### 📚 Transcript **Riana Pfefferkorn:** I was talking to somebody earlier who said, it seems like there just isn't the muscle memory in the United States to know how to respond here. Because we haven't really had both the kinds of integration, like you mentioned, between a social media platform and a nudification app like this into a rapid fire one-stop shop for this kind of victimization of people. We've historically had, you might have a hidden site for serving up CSAM that does not care what the law is. They exist for one purpose. You have Nudify apps that are increasingly starting to see regulations for those apps in particular. But again, they're targeted at something that is specifically intended only for non-essential deepfake pornography, not a general purpose image generator. you also have regulators who are used to dealing with, by and large, big mainstream, often publicly traded companies that operate these social media services and that are pretty used to behaving in a pretty legal risk averse, optics sensitive, answerable to shareholders or to paying customers and the public kind of approach to what they allow on their services. to the degree that even if they could theoretically allow everything that is legal under the law in any given jurisdiction, they don't do that because you rapidly turn into a cesspool if you do. territory where this particular sort of perfect storm, not only of poorly secured image generator, social media platform that already had become known as the Nazi bar, and somebody who is the richest man in the world, owning both of those things and seeming to thumb his nose at regulators and who we're going to find out, like, is he so powerful and rich as to be untouchable, even in this instance, even when we're talking about child abuse imagery and non-consensual deep fake pornography, it feels like a combination that we just were not equipped to handle. --- ### [Push Liability Up The Stack](https://share.snipd.com/snip/bb94d89d-5116-4284-86d9-aa7369d633b4) 🎧 20:58 - 23:08 (02:09) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/840cf9a4-59d4-4ba3-bbc0-d9abff33f188" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - Victims and advocates should push for platform- and downstream-liability rules beyond chasing individual users. - Targeting platforms' design and service providers can create more meaningful accountability than one-off lawsuits. #### 💬 Quote > And so I think that's probably why a lot of people are focusing more on like, we have to go one or two levels up. And instead of saying, okay, let's go onesies, twosies after the people generating that image, what are we going to have in terms of potential liability or accountability for AI image generation services or for platforms that enable the distribution of it? > — Riana Pfefferkorn Riana Pfefferkorn on focusing liability on platforms and services #### 📚 Transcript **Riana Pfefferkorn:** I think this goes back to the difficulty of, are we trying to target the platform that is enabling the creation and distribution of this material as opposed to the end users who are engaging in this? Because, you know, I would disagree with you that we have no language and, you know, just nothing to say about this form of what is often called image-based sex abuse by people who work in and study this space. We have seen the advent year over year of what used to be called revenge porn laws, and now more broadly, non-consensual intimate imagery or NCII laws that got passed state by state by state by state before we had the Take It Down Act last year. Now, like I said, I don't study NCII for myself really, but it has been a many years long campaign by people who are in that sphere, who are academics, scholars, lawyers, etc., to figure out how do you get accountability for the people who are victimizing you in this way. And while we now have, take it down as like a federal statute that defines a federal crime, we have long had other torts, things like intentional infliction of emotional distress, false light invasion of privacy amongst the traditional privacy torts, the kinds of things that may seem like less perhaps of a powerful tool compared to a federal criminal law that Congress fit to pay attention to and pass. So if you were to talk to people who, unlike me, have been up to their elbows in the NCII problem for a number of years, they could probably tell you, yeah, here's the approaches that we have taken over time. And they might also tell you, like, yes, people are being victimized in this way. And even if you can see who's doing it, you see that they have their handle attached to their username, and you have the image right there, and you can document all of that and get the timestamps and everything, how easy is it going to be to actually hold that particular end user accountable in court? And what remedy do you get out of that? And so I think that's probably why a lot of people are focusing more on like, we have to go one or two levels up. --- ### [Legal Shifts Create Risky Trade-Offs](https://share.snipd.com/snip/4f552bf1-4757-4a3e-8bd4-572e9a0b5287) 🎧 27:47 - 29:28 (01:40) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/0fe97ae8-64e6-4401-8673-5e905c599108" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - The Supreme Court's Paxton decision signals willingness to revisit settled precedents, complicating digital-safety responses. - Policymaking driven by 'we must do something' can produce harmful trade-offs like privacy breaches. #### 💬 Quote > I am not a fan of the Paxton decision and that underlying rationale of like, well, set aside the security and privacy issues. You just got to start like making sure that you can collect everybody's like biometrics or their ID or whatever in the name of keeping children from accessing pornography. > — Riana Pfefferkorn Riana Pfefferkorn criticizing the Paxton rationale #### 📚 Transcript **Nilay Patel:** I'm glad you brought up the Paxton case because it actually helps me think about what's going on with Grok in a very specific way. I didn't think the Supreme Court issued a great decision in the Paxton case, but a lot of the rationale was, hey, we left this alone for a long time. But now all the kids have these devices and it's not out of bounds for us to say you got to try harder. So the states are allowed to say you got to try harder. And all of these concerns about privacy, about data security, you guys are smart. You figure it out. Right? Like that was the thrust of that case. Like it's different now. And so all these precedents from the 90s when everyone was on desktop computers, whatever. Like all the kids have phones now. You can get porn so easily. Asking to verify your age, figure out the privacy problems. Like you're so concerned about – figure it out. Like just solve it. But there's a compelling interest here to do age verification. I'm not saying that I thought the specifics of that argument were great by the Supreme Court. I'm saying that that's what that felt like to me. Like it's different now. The scale is different. The scope is different. The access is ubiquitous. We need to make some rules. Like it's time for some rules. And I know a lot of parents who generally agree with that. Parents who grew up with the internet and access and are like, I don't want my kids to have the experiences I had on AOL in the 90s. There should be a lot. I'm putting that right next to Grok where it's like, yeah, I can just put anyone I want in a bikini in a sexually suggested pose and no one seems to be doing anything about it. I can't square those two impulses. Can you? Is there something really weird about that? **Riana Pfefferkorn:** I agree with you that I am not a fan of the Paxton decision and that underlying rationale of like, well, set aside the security and privacy issues. --- ### [Lecture Pushback On Age Verification](https://share.snipd.com/snip/6e7a54ae-d945-4968-a5d9-c35ecb5fc816) 🎧 30:52 - 32:22 (01:30) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/0ef9720e-965c-4304-95fc-f14cac1ef688" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - Riana described a lecture Q&A where age verification was suggested as the fix for nudify apps and she pushed back. - She argued the core problem is non-consensual deepfakes, not the age of the person making them. #### 💬 Quote > If you want to deal with like nudify apps and making inappropriate imagery of children isn't age verification the answer. And no, the problem isn't whether it's a 14 year old or a 24 year old who's making non-consensual deepfake imagery of somebody. > — Riana Pfefferkorn Riana Pfefferkorn recounting a lecture Q&A #### 📚 Transcript **Riana Pfefferkorn:** I was giving a lecture a few weeks ago about the research that I've done on this area. And one of the questions that came up during Q&A was, well, if you want to deal with like nudify apps and making inappropriate imagery of children isn't age verification the answer. And no, the problem isn't whether it's a 14 year old or a 24 year old who's making non-consensual deepfake imagery of somebody. It's that any deepfake imagery without their consent is being made of somebody. That's the problem. And so it was this weird sort of window into when you have an age verification hammer, now everything starts to look like a nail and that is just not the, that is not the case. Not having adequate, like, you know, controls over who can view and make the non-consensual, like bikini imagery is not the place to be focusing on. And so it remains to be seen whether this is the thing that breaks everybody out of that mode of thinking that like, if we just pass age verification in all 50 states, you know, the internet will somehow be a better place to be. And instead, we're just seeing like, now not only are we seeing the failure of age verification for the benefits without all the drawbacks that it was purported to have, but also we're seeing what used to be, with regard to notification apps, kind of confined to this seedy underbelly of the internet as escape containment, and now is integrated, vertical integration directly into a major online social media platform. --- ### [AI Output Tests Section 230](https://share.snipd.com/snip/c763bd76-dc6e-4370-be50-7c4fa5b1faa2) 🎧 36:43 - 38:32 (01:48) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/b446411d-8867-4917-97e7-dfbd01acc8fd" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - Section 230 immunity likely won't fully protect platforms when the platform itself generates or materially contributes to illegal content. - Courts are already seeing design-defect claims and will test whether AI outputs get 230 protection. #### 💬 Quote > There's also this other portion of Section 230 that says it has to be information provided by somebody else. ... My understanding is that it's not yet a settled question in the courts. > — Riana Pfefferkorn Riana Pfefferkorn on Section 230 and AI output #### 📚 Transcript **Riana Pfefferkorn:** So a few things. One is Section 230 has never barred federal criminal enforcement of the law, and both with respect to whatever content counts as CSAM under federal criminal laws and whatever violates the new law, take it down now, defines a federal of publishing non-consensual deepfake pornography of adults and minors alike. So for those whatever images qualify as one of those two categories, nothing stops the DOJ from taking action against XAI or X in that regard for something that violates federal criminal law. That's just the DOJ. It doesn't get at civil liability or like state AGs, like the one that California has now opened an investigation. What you're mentioning is something that actually there's also this other portion of Section 230 that says it has to be information provided by somebody else, some other third party. Usually that's a user. And it doesn't bar liability, including civil liability, where it is the platform itself that's contributing to what it is that makes the content illegal or where the platform is generating the content itself. We have seen statements from Ron Wyden, the senator who has been around in Congress long enough that he helped write Section 230, saying that AI output is not meant to be covered by that. It is not supposed to be immune by Section 230. My understanding is that it's not yet a settled question in the courts. And my prediction is that this is the situation that's going to give rise to some amount of litigation where that question will be squarely presented. Is the output of an AI, a generative AI tool covered by Section 230 immunity or not? I think we're going to find that out in court opinions probably sometime this year. --- ### [High-Profile Lawsuit Uses Design Claim](https://share.snipd.com/snip/a17a29a7-39f5-4a67-a921-68a24743ebd4) 🎧 46:51 - 48:10 (01:18) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/e725512c-a68d-489b-bdf5-e56887217f21" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - Nilay noted Ashley St. Clair sued XAI claiming Grok undressed her, alleging Grok is 'unreasonably dangerous' by design. - Riana said design-defect claims are an expected legal route to try to bypass Section 230 defenses. #### 💬 Quote > Her allegation is that Grok is unreasonably dangerous as designed. ... there you go. You're going to get the design defect type of allegation. > — Nilay Patel and Riana Pfefferkorn Nilay Patel on Ashley St. Clair's lawsuit and Riana Pfefferkorn on likely claims #### 📚 Transcript **Nilay Patel:** Ashley St. Clair, who had one of Elon's many, many children, seeks a restraining order to stop the Grok chatbot from undressing Earth AI. This idea that there's a private right of action against the platform, new-ish. These are the sorts of lawsuits that have typically hit the 230 wall. The Defiance Act has not passed yet. Do you think she's got a chance? I'd have to see what her lawsuit says. Her allegation is that Grok is unreasonably dangerous as designed. So **Riana Pfefferkorn:** there you go. You're going to get the design defect type of allegation. And that is not surprising to me at all. As I was mentioning, we've seen in social media addiction type cases that plaintiff's attorneys have argued it is not about the content of users' posts or whatever, and therefore Section 230 should not apply. We're talking about defective design, and sometimes those claims do get around Section 230. We'll see. Now we'll see if XAI and X want to actually finally bring up Section 230 in response, or whether they will only invoke the First Amendment, or will they choose to just sell this quietly and make it go away. It remains to be seen. But like I was saying earlier, it is a little bit of guesswork involved in depending on what court you get, what judge you get, what arguments get made, what will come of these types of claims. --- ### [App Stores' Silence Is Politically Risky](https://share.snipd.com/snip/0cf4bb6e-6c64-45e2-9ee2-db833fb5617b) 🎧 52:16 - 56:01 (03:45) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/14971579-26ed-487e-9177-5ef12306cd84" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - Apple and Google have powerful control points but have largely stayed silent and inconsistent in enforcement. - Their selective action undermines their safety defense and could weaken antitrust arguments. #### 💬 Quote > Why aren't you taking these down? ... It becomes a lot harder to defend, as you said, when they are picking and choosing their own enforcement. > — Riana Pfefferkorn Riana Pfefferkorn on app stores' inconsistent enforcement #### 📚 Transcript **Nilay Patel:** We've talked a lot about the sort of legal frameworks here, what courts and governments might do, what individual actors might do using those laws. Earlier, I mentioned that there's kind of a universe of actors here, right? There are payment processors and app stores and all kinds of other pressure points that might serve as gatekeepers or guardians of norms. I'm not sure that there are norms anymore in 2026, but that's kind of the idea, right? You don't always have to go to the government to solve these problems. You've built these systems and the system should have some points of pressure to make sure things are operating well or safely for people. The ones that come to mind for me the most here are the app stores. Apple and Google have become empires. They've become some of the companies in the world on the back of having total control of the applications on their platforms, charging absurd rents to those app developers for the transactions that take place. They have endured antitrust lawsuit after antitrust lawsuit. And their argument is always, we need this control in order to keep our users safe. It is the bedrock principle of their defense, particularly for Apple. And in this case, they have done nothing. They like literally nothing. They won't even comment when we write stories about it. As far as I can tell, they've commented to no one. Why do you think that is? Why do you think there's such silence from the two biggest and baddest gatekeepers of all? There's **Riana Pfefferkorn:** other reporting in The Verge just calling Tim Cook and Senator Petraic cowards. There's a letter from multiple U.S. senators, Senator Wyden, Senator Markey, Senator Lujan, basically saying X and XAI, like these apps are violating your own terms of service against like, you know, non-consensual, like sexual imagery of people. Why aren't you taking these down? You know, the way you did with IceBlock and Red Dot, the, you know, perfectly legal apps for doing the perfectly legal act of observing ice when they're kidnapping our neighbors off of the streets. And it's hard to discern any, certainly any like moral or ethical principle in refusing to take action here. Again, you know, we're not operating on a blank slate. If you remember a year ago, at this exact time, a year ago, everybody was freaking out over the impending banning of TikTok from App Store, something that a group of senators had all decided that needed to be done, that the Supreme Court had upheld as like the power to do, and everybody freaked out. And then it just kind of went away. TikTok came back to be available, again, despite the fact that you would think any of these app stores would think there's potentially ruinous liability under the law that Congress passed and that got signed into law and the Supreme Court upheld if we allow it to still be in the app store going forward. And then it just kind of went away. And I think you're pointing to how we are in a different realm now from where it used to be that any kind of jawboning by government actors was supposed to be completely unacceptable. We even had a Supreme Court case against that. Now we have a letter from three senators saying you should remove these apps from the app store because of violations of your terms of service. That's jawboning. But I don't think a lot of people are necessarily pissed off about that. And at the same time, we're now in a place where actually it turns out you can just jawbone if you are Pam Bondi, et cetera, and threaten them. And the people who used to be mad about jawboning are nowhere to be found on that. And so it seems like we are past just sort of normal expectations around the rule of law or around principled enforcement of your own policies when it comes to apps that are doing harm to people every single day. And it might just be that we're going to find out whether Elon Musk companies are just too big to fail. --- ### [Hold Service Providers Accountable](https://share.snipd.com/snip/983ecbfe-9163-45c6-8525-7096e06f2a59) 🎧 59:55 - 01:01:29 (01:34) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/f643e86d-9474-489b-b9ba-1614c40a2f37" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - Policymakers should consider liability for downstream service providers once notified of deepfake-porn customers. - That creates incentives for payment, hosting, and infrastructure providers to cut off abusive services. #### 💬 Quote > California passed a law that didn't get a lot of notice last year that not only outlawed non-consensual deepfake pornography services, it also allows for liability for service providers to those services. > — Riana Pfefferkorn Riana Pfefferkorn on laws targeting downstream liability #### 📚 Transcript **Riana Pfefferkorn:** hard, and I don't want to really make a forecast about what should happen there, because anything that you go different layers at the stack, I think, has very different arguments compared to when we're just talking about the app itself. There have been various arguments in the past about whether Section 230 comes into play. Again, if you go like one level down, we are starting to see now efforts to try and target layers down the stack with anybody who's providing services to non-consensual like deepfake pornography services. California passed a law that didn't get a lot of notice last year that not only outlawed non-consensual deepfake pornography services, it also allows for liability for service providers to those services. Once they are put on notice that they have a deepfake porn service as a customer, if they don't then drop that customer, they can be held liable. And that's the kind of further down the stack liability that I think we will see more of. I've seen the UK government talking about this. Minnesota made a kind of hand-fisted attempt at writing a bill to do the same thing that would have inadvertently outlawed all AI-generated porn, not just non-consensual ones. But opening up the realm of further down the stack liability to try and go from like, well, okay, well, if the user material doesn't get taken down, let's go after the platform. If the platform isn't responsive, let's go after the AWSs and CloudFlares and Stripes of the world. I think we might see potentially more of that also in response, both to the general problem of like Nudify apps that are still out there and bringing in millions of dollars a year. --- ### [Good-Faith Moderation Has Frayed](https://share.snipd.com/snip/61d0fe61-4d81-4044-8ce4-a7b1f7707a4b) 🎧 01:03:25 - 01:05:28 (02:02) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/73869e68-c0d1-4c58-aca7-c3b689404f22" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - Reliance on centralized platforms' 'good faith' trust and safety is collapsing as companies politicize and deprioritize moderation. - Community-driven, decentralized, or middleware moderation tools may regain importance. #### 💬 Quote > It has laid bare how craven the motives are for major platforms ... people cannot rely upon these companies to view their trust and safety teams as something that is overall benefit and boon for the company rather than a cost center to try and be minimized wherever possible. > — Riana Pfefferkorn Riana Pfefferkorn on decline of platform trust and safety #### 📚 Transcript **Riana Pfefferkorn:** mean, it has laid bare how craven the motives are for major platforms that even before the inauguration, right after the election, already we were seeing announcements from online platforms that they were pulling back from content moderation. Either they were going to say, we're going to put that on to end users to do community notes instead of having fact checkers ourselves, or we're going to have AI tools do the moderation for us, just straight up backtracking on things like, oh yeah, let's just say dead naming is okay again. We're not going to bother ourselves too much with that. I think you're right that like, you know, there will be a natural ebb and flow to how much effort companies put into content moderation, depending on political factors, social factors, business factors. But think it is now much more clear that people cannot rely upon these companies to view their trust and safety teams as something that is overall benefit and boon for the company rather than a cost center to try and be minimized wherever possible. And I think this also feeds into why we have seen such a great increase in interest in, you know, defederalized social networks, in the rise of middleware, and giving more tooling that is built by, for, and about the communities that are most adversely affected by noxious content online to try and put tools back into the hands of users to tailor their experience. which, you know, ironically, going back to the roots of Section 230 to begin with, was one of the things that was envisioned to happen coming out of 230. Not just that you would have immunity for taking stuff down or leaving stuff up, but that we would see a greater ecosystem of content moderation tools that would be encouraged to flourish, to restore that level of control and customization and curation in the user's hands. Instead, I think a lot of us got used to just relying that it would probably be in the hands of these large centralized platforms to do that for us. --- ### [Watch For Litigation And Superficial Fixes](https://share.snipd.com/snip/8cacc013-ca3b-41ea-b4ec-1eb982fade62) 🎧 01:06:00 - 01:06:56 (00:55) <iframe src="https://share.snipd.com/embed/obsidian-player/snip/420fd796-bec5-48a1-b9e9-34bb6bd23012" width="100%" height="100" style="border: none; border-radius: 12px;" sandbox="allow-scripts allow-same-origin allow-forms allow-popups allow-clipboard-write" ></iframe> - Watch for superficial policy changes that don't work and for litigation testing Section 230 and design-defect theories. - Expect Grok's image features to stay live unless financial or regulatory pressure becomes decisive. #### 💬 Quote > One thing I don't think is going to happen, unfortunately, is seeing them pull Grok's image generation or image editing features offline because, frankly, there's just too much money in it. > — Riana Pfefferkorn Riana Pfefferkorn predicting next moves on Grok #### 📚 Transcript **Riana Pfefferkorn:** So we already have seen it took them three weeks to finally announce that they were going to disallow the generation of bikini pictures. Turned out that didn't work, but it took them three weeks to get there. It's difficult to predict exactly what will happen, but I will kind of be unsurprised if the outcome of this really does end up being outright saying it is censorship to tell us we can't generate imagery of 14-year girls. Like that has felt like it was always where the content moderation is censorship discussion was going to go. And this would be the logical culmination of that. I think kind of as you were hinting at, Eli, that we might just see further public statements about changes being made, which do not actually work in actuality, whether because their heart isn't in it or because, as I've discussed elsewhere, safeguarding AI models really is hard. One thing I don't think is going to happen, unfortunately, is seeing them pull Grok's image generation or image editing features offline because, frankly, there's just too much money in it. --- Created with [Snipd](https://www.snipd.com) | Highlight & Take Notes from Podcasts