Reality Is Losing the Deepfake War  ## Document Notes ## Highlights - **C2PA Is A Provenance Standard, Not A Panacea** - C2PA is a metadata standard meant to record provenance at creation, not a foolproof AI-detection system. - Its promise depends on universal adoption and robust platform handling, which currently don't exist. Transcript: Nilay Patel The most promising is something called C2PA. In view is that so far, these systems have been almost entirely failures. In this episode, we're going to focus on C2PA since it's the one that has the most momentum. It's a labeling initiative spearheaded by Adobe with buy-in from some of the biggest players in the industry, including Meta, Microsoft, and OpenAI. But C2PA, which is also sometimes referred to as content credentials, has some pretty serious flaws. First, it was designed as more of a photography metadata standard, not an AI detection system. And second, it's really been only half-heartedly adopted by a handful, but not nearly all, of the players you would need to make it work across the internet ecosystem. We're at the point now where Adam Masseri, who runs Instagram, is publicly posting that the default should shift and that you should not trust images or videos the way that you maybe Could before. Think about that for one second. That's a huge, pivotal shift in how society evaluates photos and videos, and it's an idea I'm sure we're going to come back to a lot this year. But we have to start with the idea that we can solve this problem with metadata and labels, that we can label our way into a shared reality, and why that idea might simply never work. Okay, Verge reporter Jess Weatherbet on C2PA and the effort to label our way into reality. Here we go. Jess Weatherbed, welcome to Decoder. Hi. I want to just set the stage. Several years ago, I said to Jess, boy, these creator tools are criminally undercovered. Adobe as a company is criminally undercovered. Go figure out what's going on with Photoshop and Premiere and the creator economy, because there's something there that's interesting. And fast forward, here you are in Decoder today, and we're going to talk about whether you can label your way into consensus reality. I just think it's important to say that's a weird turn of events. Jess Weatherbed Yeah. I keep likening the situation to the Jurassic Park memo, whereas people thought so long about whether they could, they didn't actually stop to think about whether they should be doing This. And now we're in the mess that we're in. Nilay Patel The problem, broadly, is that there's an enormous amount of AI-generated content on the internet. Much of it just depicts things that are flatly not real. An important subset of that is there's a lot of content that depicts modifications to things that actually happened. So our sense that we can just look at a video or a picture and sort of implicitly trust that it's true is fraying, if not completely gone. And we will come to that because that's an important turn here. But that's the sort of state of play. In the background, the tech industry has been working on a handful of solutions to this problem, most of which involve labeling things at the point of creation, right? At the moment you take a photo or the moment you generate an image, you're going to label it somehow. The most important one of those is called C2PA. So can you just quickly explain what that stands for, what it is, and where it comes from? Jess Weatherbed So this is a metadata standard, effectively, that was kickstarted by Adobe, interestingly enough, Twitter as well back in the day. You can see where the logic lies. It was supposed to be that everywhere a little bit of content goes online, this embedded metadata would follow. So what C2PA does is at the point that you take a picture on a camera, you upload that image into Photoshop. All of these instances would be recorded in the metadata of that file to say exactly when it was taken, what has happened to it, what tools were used to manipulate it. ([Time 0:02:40](https://share.snipd.com/snip/1700aa64-5b0d-4172-8990-0093ccade53f)) - **Metadata Can Be Stripped Or Lost** - Platforms and vendors claim C2PA is tamper-proof but practical use shows metadata is easy to strip or lose. - That fragility undermines any confidence users might place in embedded provenance. Transcript: Nilay Patel About the actual label. You said it's metadata. I think a lot of people have a lot of experience with metadata. We are all children of the MP3 revolution. Metadata can be stripped, it can be altered. What protects the C2PA metadata from just being changed? Jess Weatherbed They argue that it's quite tamper-proof, but it's a little bit of an action-to kind of situation, unfortunately, because while they say it's tamper-proof, this thing is supposed To be able to resist being screenshot, for example, by the way. But then OpenAI, who is actually one of the steering community members behind this standard, openly says it's incredibly easy to strip to the point that online platforms might actually Do that accidentally. So the theory is there's plenty behind it to make it robust, to make it hard to remove. But in practice, that just isn't the case. ([Time 0:06:50](https://share.snipd.com/snip/1cdaaf75-0b0a-4aa1-9555-69eaae1faffd)) - **No Single Technical Silver Bullet Exists** - Multiple technical approaches exist (metadata, watermarking, inference) and they can complement each other. - None offers a single definitive yes/no; the ecosystem needs layered methods. Transcript: Jess Weatherbed It's a little bit of a confusing landscape because i think one of the few kind of like tech says that i would say there shouldn't actively be competition and from what i've seen from what I've spoken to with all these different providers there isn't competition between them so much as they're all working towards the same goal google synth id is similar It's technically A watermarking system more so than a metadata system, but they work on kind of a similar premise that stuff will be embedded into something you take that you'll then be able to assess Later to see how genuine it is. Like the technicalities behind that are difficult to explain in a shortened context, but they do operate on different levels, which means technically they could work together. A lot of these systems can work together. You've got inference-based systems as well, which is what they will look at an image or a video or a piece of music and they will pick up telltale signs that apparently may have been manipulated By AI and they will give you a rating. They can never really say yes or no, but they'll give you a likelihood rating. None of it will stand on its own to be like a one true solution. ([Time 0:07:39](https://share.snipd.com/snip/3849a692-653c-4f9d-8116-97d4e31f51bd)) - **Apple's Silence Matters For Trust** - Apple has not publicly committed to C2PA or SynthID despite being a major camera maker. - Their silence matters because device-level signing from Apple would materially change trust dynamics. Transcript: Jess Weatherbed Haven't officially or on record. I have sources that say apparently they were involved in conversations to at least join, but nothing public facing at the minute. There has been no confirmation that they are actually joining the CIA initiative or even kind of adopting Google Synth ID technology. They're kind of very carefully skirting on the sidelines. For some reason, it's a little bit unclear as to whether they're kind of letting their caution about AI generally kind of like stem into this at this point, because as far as I'm concerned, There is not going to be a one true solution. So I don't really know what Apple is waiting for and they could be making a difference. But no, they haven't been making any kind of declarations about what we should be using to label AI. ([Time 0:10:46](https://share.snipd.com/snip/cda453ad-3455-45a0-93eb-1c82b4af396b)) - **New Cameras Add Credentials, Old Ones Don't** - Camera makers like Sony and Nikon have added the standard to new models but don't backdate older cameras. - Photographers won't replace gear each year, leaving many images unsigned. Transcript: Jess Weatherbed Models that have got the system embedded. The problem that they're having now is in order for this to work, you don't just have to do it on your new cameras because every photographer in the world worth their salt isn't going to Go out every year and buy a brand new camera because of this technology. It would be inherently useful, but that's just not going to happen. So backdating existing cameras are where the problem is going to be. We've spoken to a lot of different companies, as you said, like Sony has been involved with this, Leica, all of them, Nikon. The only company willing to speak to us about it was Leica. And even they were very vague on how internally this is progressing. They just keep saying that it's part of the solution. It's part of the step that they're going to be taking. But these cameras aren't being backdated at the minute. ([Time 0:14:00](https://share.snipd.com/snip/614cdc84-2187-42f4-bcb6-ea7c14d6d328)) - **Stock Agencies Could Be Trusted Middlemen** - Stock agencies such as Shutterstock could act as trusted middlemen by embedding metadata on submission. - That approach would help verify important images but isn't widely implemented yet. Transcript: Jess Weatherbed We know that Shutterstock is a member at the minute. The system that you're describing would probably be the best approach that we have to making this beneficial, at least for us, as people that see things online and want to be able to trust Whether protest images or horrific things we're seeing online are actually real, to have a trusted middleman, as it were. But that system itself hasn't been established. We do know that Shutterstock is involved. They are part of the CTPA committee, or they have general membership, so they are on board with using the standard, but they're not actively part of the process behind how it's going To be adopted at a further stage. So unless we can also get the other big players involved for stock imagery, then who knows where this is going to go. But Shutterstock actually implementing it as a middleman system would be probably the most beneficial way to go. ([Time 0:15:16](https://share.snipd.com/snip/a3aba855-3f1a-4b08-b85d-bde87557d22d)) - **Sora 2 Spread Shows Distribution Failures** - Sora 2 videos spread widely without consistent provenance labels, exposing distribution failures. - Platforms often strip or fail to surface metadata, so creator-side labeling doesn't reliably reach viewers. Transcript: Nilay Patel Nobody's respecting the stuff as it travels across the internet. Talk about that. You wrote about Sora 2 videos and they exploded across the internet. This is when it should have not been controversial to put labels everywhere saying this is AI generated content. And yet it didn't happen. Why didn't that happen anywhere? It generally exposes the biggest flaw that the system have and every system like it to its credit. Jess Weatherbed So I would always argue, I don't want to defend CTPA because it's doing a bad job. It wasn't ever designed to do it on this scale, right? It wasn't designed to apply to everything. So in this example, yes, platforms need to be adopting it to actually read that metadata, providing they're not the ones ripping it out during the process of actually supposedly scanning For it. But unless this is absolutely everywhere, it's just not going to go. Part of the problem that we're saying is as much as they can credit it saying it's going to be really robust, it's going to be really efficient, you can embed this at any other stage. There are still flaws with how it's being interpreted, even if it is scanned. So that's a big thing. It's not necessarily that platforms aren't picking up the metadata or stripping it out. It's that they have no idea what to do with it when they actually have it. ([Time 0:16:40](https://share.snipd.com/snip/45cb2eab-36b8-4e70-9784-0793b1134da3)) - **Platforms Admit You Can't Trust Your Eyes** - Adam Mosseri argues we must shift from default trust to default skepticism about photos and videos. - That admission signals platforms expect an era where 'you can't trust your eyes' becomes normal. Transcript: Nilay Patel I'm going to read you this quote from Adam Masseri, who runs Instagram. On New Year's Eve, he just dropped a bomb, and he put out a blog post in the form of a 20 carousel Instagram slideshow, which has its own PhD thesis of ideas about how information travels On the internet embedded within it. But he put out a 20-slide slideshow on Instagram. In it, he said, quote, for most of my life, I could safely assume photographs or videos were largely accurate captures of moments that happened. This is clearly no longer the case, and it's going to take us years to adapt. We're going to move from assuming what we see is real by default to starting with skepticism. This is the end point, right? This is you can't trust your eyes. You can no longer trust a photo. You can't trust a video of any event is actually real and reality will start to crumble. And you can just look at events in the United States over the past month. The reaction to ICE killing Alex Preddy was, well, we all saw it. And it's because there was lots of video of that event from multiple angles and everyone said, well, we can all see it. And the foundation of that is we can trust that video. And I'm looking at Adam Aseri saying, we're going to start with skepticism. We can no longer assume photos or videos are accurate captures of moments that happened. This is the turn. Like, this is the point of the standard. Do you see Masseri saying this out loud about Instagram is the endpoint of this? Is this war just lost? Jess Weatherbed I would say so. I think we've kind of been waiting for tech to basically admit that. I see them using stuff like CTPA, almost kind of a meritless badge at this point, because they're not endeavoring to push it to its utmost potential, really. Even if it was never going to be the ultimate solution, it could have been at least some kind of benefit. And we know that they're not doing this because in the same message, Missouri is describing this like, oh, it would be easier if we could just tag real content. That's going to be so much more doable. And if anything, that would be good. And we'll circle those people. It's like, my guy, that's what you're doing. That is like, CTPA is that. It's not specifically an AI tagging system. It's a where has this been? And who took this? Who made this? What has happened to it? So if we're going for authenticity, like Mozeria is just openly saying, we're using this thing and it doesn't work. But imagine if it did, wouldn't that be great? It's like, that's deeply unhelpful. ([Time 0:23:38](https://share.snipd.com/snip/4c6b5b4a-d54c-43f0-a2d1-23076993ef36)) - **Labels Create Social And Technical Friction** - Labels provoke backlash from creators who feel devalued and from audiences who misinterpret nuance. - Defining what counts as 'AI' is technically and socially fraught because many common edits use AI internally. Transcript: Nilay Patel He says platforms like Instagram will do good work identifying AI content, but it'll get worse over time as AI gets better. It'll be more practical to fingerprint real media than fake media. Labeling is only part of the solution, he says. We need to surface much more context by the accounts sharing content so people can make informed decisions. So he's saying, look, we'll start to sign all the images and everything, but actually you need to trust individual creators. And if you trust the creator, then that will solve the problem. And it seems like you're really skipping over the part where creators are often fooled by AI-generated content, like all the time. And I don't mean that to say like creators as a class of people. I mean, literally just everyone is fooled by AI content all the time. And so if you're trusting people to understand it and then share what they think is real and then you're trusting the consumers to trust the people, that also seems like a whirlwind of Chaos. On top of that, and you've written about this as well, there's the notion that these labels make you mad at people, right? So that if you label a piece of content as AI generated, the creator gets furious because it makes their work seem less important or less valuable. The audiences yell at the creators. And so there's been a real push to get rid of these labels entirely because they seem to make everyone mad. How does that dynamic work here? Does any of this have a way through? Jess Weatherbed I mean, it doesn't. And the other kind of amusing thing is Instagram knows this the hard way. Missouri should remember. One of the very first platform implementations they did of reading CTPA was done by Facebook and Instagram a couple of years ago, where they were just slapping maybe the eye labels onto Everything because that's what the metadata told them. The big problem here that we have isn't just communication, which is the, like, that is the biggest part of it. How do you communicate a complex bucket of information to every person that's going to be on your platform and get them only the information that they need? If I'm a creator, it shouldn't have to matter if I was using AI or not, but if I'm a person trying to see if again a photo is real I would I would greatly benefit from just a easy button or label That verifies authenticity finding the balance for that has proven next to impossible because as you said people just get upset about it but then how do you define how much AI in something Is too much AI? You know, like Photoshop and all of Adobe's tools, they do embed these content credentials. All of this metadata, it will say when AI has been used. But AI is in so many tools and not necessarily in the generative way that we assume it's going to be like, I'm going to click on this. It's going to add something new to an image that was never there before. And that's fine. There are very basic editing features that video editors and photographers now use that will have some kind of information embedded into them to say that AI was involved in that process. And now when you've got creators on the other side of that, they might not know that what they are using is AI. We're at the point where, unless you can go through every platform, every kind of editing suite with a fine-tooth comb and designate, what do we count as AI? ([Time 0:26:19](https://share.snipd.com/snip/5753f730-dc0d-4e39-903c-e19728ac6620)) - **Mixed Incentives Let Reality Erode** - Powerful actors, including government agencies, already publish AI-manipulated images, worsening the reality crisis. - Platforms have weak incentives to fix this because their business models profit from volume and engagement. Transcript: Nilay Patel You heard us talking about how AI labels are falling apart at the distribution level, the social media level, and how some tech executives like Adam Masseri, the head of Instagram, Are trying to wrap their heads around a world where we can't trust anything posted to social platforms. But now I want to shift the focus away from all these people who are ostensibly acting in good faith towards some of the people who are accelerating the chaos on purpose, the White House. That's a lot of mixed incentives for these platforms. And it occurs to me as we've been having this conversation, we've been kind of presuming a world in which everyone is a good faith actor and trying to make good experiences for people. And I think a lot of the executives of these companies would love to presume that that is the world in which they operate. And whether or not the label makes people mad and you want to turn it off or whether or not you can trust the videos of significant government overreach and cause a protest, that's still Operating in a world of good faith. Right next to that is reality, the actual reality in which we live, where lots of people are bad faith actors who are very much incentivized to create misinformation, to create disinformation. And some of those bad faith actors at this moment in time are the United States government. So the White House publishes AI photos all the time, Department of Homeland Security, AI-generated imagery up, down, left, right, and center. You can just see AI-manipulated photos of real people modified to look like they're crying as they're being arrested instead of what they actually looked like. This is a big deal, right? This is a war on reality from literally the most powerful government in the history of the world. Are the platforms ready for that at all? Because like here's the, they're being faced with the problem, right? This is the stuff you should label. No one should be mad at you for labeling this. And they seem to be doing nothing. Why do you think that is? I think it's because it's the same process, right? Jess Weatherbed What we're talking about is a kind of a two-way street that is on the same road. You've got the people want to identify AI slop, or maybe they don't, but people want to be able to like see what is and what isn't AI. But then you've got the more ins thing if we actually want to be able to tell what is real but it unfortunately benefits too many people to make that confusing now but the solution is for Both ai companies platforms that are profiting off of all of the stuff that they're showing how they're making it so much more efficient for content creators to slap stuff in front of You like we're in a position now where there's more online than we've ever seen, ever, because everything is being funneled out. ([Time 0:37:34](https://share.snipd.com/snip/5c7f0500-9618-4890-9439-243b787abc4d)) - **Bans Are Hard Without Reliable Detection** - Few platforms can credibly ban AI content because detection is unreliable and bad actors use private or underground models. - Promises to keep AI off services lack enforceable mechanisms today. Transcript: Jess Weatherbed Small. There's an artist platform right called Kara, which says that they're so for supporting artists that they're not going to allow any AI generated artwork on the site. But they haven't really clearly communicated how they are going to do that. Because saying it is one thing and doing it is another thing entirely. There are a million reasons why we don't have a reliable detection method at the minute so if i in complete good faith pretend to be an artist that's just feeding ai generated images onto That platform there's very little they can really do about it so anyone that's making those statements saying yeah we're going to stand to merit and we're going to keep ai off of the platform How they can't they're like the systems for doing so at the minute are being developed by AI providers, as we've said, or they're at least AI providers deeply involved with a lot of these Systems. And there is no guarantee for any of it. So we're still relying on how humans intercept this information to be able to tell people how much of what they can see is trustworthy. That's still kind of putting the onus on us as people. It's, well, we can give you a mix mash of information and then you decide whether it's reliable or not. And we haven't operated on that way as a society for years. People didn't read newspapers to make their own mind up about stuff. They wanted information and facts. And now they can't get that. Nilay Patel Is there user demand for this? This does seem like the incentive that will work. If enough people say, hey, I don't know if I can trust what I see. You have to help me out here. Make this better. Would that push the platforms into labeling? Because it seems like the breakdown is at the platform level. The platforms are not doing enough to showcase even the data they have, let alone demand more. But it also seems like the users could simply say, hey, the comment section of every photo in the world now is just an argument about whether or not this is AI. Can you help us out? Would that push them into improvement? I would like to think it would push them into at least being more vocal about their involvement at the minute. Jess Weatherbed We've got, again, it's the two-sided thing. At the minute, you can't tell if a photo is real, but also a less nefarious thing is like Pinterest is now unusable, right? As a creative, if I want to use the platform Pinterest, I cannot tell what is and what isn't AI. I can, but a lot of people won't be able to. And there is so much demand for a filter for that website just to be able to go. I don't want any of this. Please don't show me anything that's generated by AI. And that hasn't happened yet. They've done a lot of other stuff on that. But like they're involved with the process behind developing these systems. It's kind of more is the problem that they've set themselves an impossible task. In order to use any of the systems that we've established so far, you either need to be best friends with every AI provider on the planet, which isn't going to happen because we've got Nefarious third party things that focus entirely on stuff like nudifying people or a deepfake generation entirely. This isn't kind of the open AI or the big name models, but they exist. And they're usually what's used to do this kind of underground activity. They're not going to be on board with it. ([Time 0:42:21](https://share.snipd.com/snip/f9416133-a05a-4849-9799-cda6ecd1405a)) - **Pair Technical Fixes With Regulation** - Expect regulatory action to accompany technical tools; policy will be necessary to force standards and accountability. - Combine labeling, metadata, and rules to create enforceable responsibilities for platforms and providers. Transcript: Jess Weatherbed Kind of regulatory efforts there's going to be some kind of legal involvement because it up until this point there have been murmursurs of how we're going to regulate stuff. With the Online Safety Act in the UK and everything now kind of pointing, going, hey, AI is making a lot of deepfakes of people that we don't like, and we should probably talk about having Rules in place for that. But up until that point, these companies have basically been enacting systems that are supposed to help us out of the goodness of their heart, out of the, oh, we've spotted that this Is actually a concern and we're going to be doing this, but they haven't been putting any real effort into doing so. Otherwise, again, we would have some kind of solution by now where we would see some sort of widespread results at the very least. It would involve working together, having widespread communications, and that's supposed to be happening with the CAI, with the initiative that everyone else is currently involved With there are no results we are not we are not seeing them Instagram made a bold effort over a year ago to stick labels on and then immediately ran back with its head between its legs that Is the point of this so unless regulatory efforts actually come in clamping down on these companies and saying okay we actually now have to dictate what your models are allowed to do And what they we're going to have repercussions for if we find out what your models are doing and not supposed to be doing. That is the next stage. We have to have this as a conjunction. ([Time 0:48:10](https://share.snipd.com/snip/44246d4b-3879-45e0-a4c1-7ae3d0d86bbd))