A.I. And the Future of Scams  🔗 **Source:** [A.I. And the Future of Scams](https://share.snipd.com/episode/f01e736b-894d-4244-a84d-2d07e229ac5d) ## Highlights - **Host's Local Voice Scam That Fooled Him** - Andrew Heaton got sucked into a phone scam that sounded local and authoritative, which is why he believed it even though he normally spots scams quickly. - The scammers used an Oklahoma area code and local place names, suggesting either spoofed numbers or AI/filters tailoring voices to his background. Transcript: Andrew Heaton What's the biggest amount of money you've ever been scammed out of? Maybe you got pickpocketed one time, $100. Somebody you gave cash to because they had a sob story, and later you thought about it and they were probably lying. Or maybe somebody got a hold of your credit card and they bought junk on eBay, although usually your credit card company will reimburse you for that. The other day, for at least half an hour, about an hour actually, I got totally sucked into a phone scam, which I'm a bit embarrassed by because I used to be quite adept at not only seeing Phone scams from a mile away, but actively playing with the scammers for sport. When I lived in New York, I received multiple calls from Windows. You know the computer company, Windows? Bad news. They told me they discovered a virus on my computer, but don't worry, I could download their antivirus software and they'd scrub it for me. That scam, I would just draw things out as long as I could. Oh no. Okay, hold on. I need to boot up my computer. And I'd wait five minutes. Oh, wait a minute. Was it my Mac or my PC? I just booted up my Mac. Was it my PC? Hold on. Another five minutes. I could, while making coffee, eating a bagel, and watching old news radio reruns, often waste a solid half hour of the scammer's time before they got angry and hung up on me. When I moved to Austin, I got a phone call from Publishers Clearinghouse. You're never going to believe it, guys. I had won Publishers Clearinghouse. But in order to collect all that money, I needed to pay the tax in advance, inexplicably through CVS gift cards or something. So in that instance, I got very excited and I said, this is amazing. I could really use that money. Is Ed McMahon going to come to my house? The person on the other side who had a heavy Southeast Asian accent said, no, Mr. McMahon is off today, which is an understatement as Ed McMahon, who used to personally show up to houses to dole out giant novelty checks, died in 1992. So yes, he's on sabbatical, isn't he? With that particular scam, I said, all right, I'm going to drive to the CVS right now and get those gift cards. But my mobile phone's about to die. Can you call me back on my office number, please? The guy on the other line said, sure, he'd be happy to. So I proceeded to give him the number for the regional FBI office, and I thanked him and went out and celebrated with a breakfast taco. But this last time, this recent time, I bought it. I totally bought it. I believed the whole thing in no small part because the guys running the scam sounded like they were from my home state, from Oklahoma, not Southeast Asia, not Romania, not an indeterminate Phone center in a country I've never heard of. Macalester or perhaps Purcell or Anadarko. I knew those voices. I knew those places. And I now know, after speaking with today's guest and security expert, that it's not only possible, but probable, that I was, yet again, talking to some guys from a call center abroad, But with one big difference. AI. Voice filters tailored to my address and phone number modulated in real time to sucker me in. Chances are I will never know. We won't know if ([Time 0:00:00](https://share.snipd.com/snip/29ba0e54-611d-479e-9448-a3559513da49)) - **Magicians Teach The Mindset Behind Scams** - Brian Brushwood frames magicians as 'benign criminals' whose skills map directly to classic scams because both rely on deception and narrative misdirection. - He argues current security training focuses on artifacts, not on training humans to think like deceivers, leaving a defensive gap. Transcript: Brian Brushwood In one lane of narrative misdirection. Now, in the case of the stage show, it's for the harmless creation of wonder. But then, I don't know, around what, six years ago when I started doing World's Greatest Con, you notice that all of those skills for a good magic trick track one for one to a classic scam. And then you see similar riffs on it in marketing and other stuff. But what I've noticed recently is a bit of the rules have changed. The meta has changed. The bad guys have robots now. And all of a sudden, all of the training that we've been doing for years and years has been largely training to make awareness of the artifacts of deception, not about the core fundamentals. And so what I am desperately trying to do now in the world of cybersecurity is train the human because all of the tech, that's an arms race. You know, the bad guy's robots versus the good guys' defenses. ([Time 0:05:43](https://share.snipd.com/snip/ce5a38de-2165-46b0-a8e6-02147b8467d1)) - **AI Erases The Old Tells** - AI removes low-fidelity tells like bad grammar and accents, making scams indistinguishable from legitimate communication by providing perfect voices and translations. - That shifts the reliable cues from technical artifacts to human-based signals and relationship provenance. Transcript: Brian Brushwood This has been cooking for a while and it's hit a breaking point. So for whatever strange cosmic reason, I've been following both the rise of AI tools, wondering what they would be used for. And I suspect that long before we have to worry about Skynet or whether or not we're going to turn into WALL-E as a society, what we have to worry about today, today, is that the bad guys Suddenly have robots working for them. And so previously, all the pig butchering scams and the romance scams that I'm sure we'll talk about, those were all powered eventually by humans, slaves, sex slaves inside of tenement Buildings. But now they're able to 10x that reach and they don't have to worry about accents. They don't have to worry about grammar. They don't have to worry about speaking your cultural interests. They've got translation bots for all of that stuff. So all of a sudden, what was very low fidelity information, think about a highly compressed MP3 or a JPEG that's compressed so much that it's barely recognizable. Been teaching for years how to identify those artifacts of compression, but that doesn't matter now because now rich audio comes. There are deepfake voice softwares that ([Time 0:10:11](https://share.snipd.com/snip/75fd1ca1-3eb2-475a-9729-d1c72451d3a5)) - **Hang Up And Verify Using Official Phone Trees** - When you receive a suspicious official call, hang up and call the agency using an independent phone tree or published number to verify identity. - Brian recommends rehearsing this habit and demanding alternative contact vectors rather than trusting the number handed to you. Transcript: Brian Brushwood For example, your phone gets taken away or whatever. Or it could be, in your case, a set of narrative restrictions that bind you to restrict levels of communication. This is the other thing. We talked about how most folks haven't had a reason to be on guard against scams. Even worse, we're coming out of a five-year cycle where we've primed everybody to believe that Zoom calls are a high fidelity form of conversation, which they absolutely are not. On most of my Zoom calls, about halfway through, I ask probably until I said these words, it never occurred to you that I'm not actually Brian Brushwood. And because we haven't encountered enough of that and the deep fake, there's enough legitimate use case, whether it's sales reps wanting to match the ethnicity of the people that they're Selling to, whether it's people who want to put on digital makeup, whether it's people with disabilities who want to present as something other than they are, there's enough legitimate Use cases for all this deep fakery that it's not going away. There is no asking people to turn off the digital paint. So instead, we have to find other ways to have that common ground. For example, one thing that people often say is, oh, I come up with a code word. I think that's a terrible idea because that is hinging your entire trust architecture on one compromisable bit of data. Andrew Heaton After this, I called the courthouse. I called, just make sure I was like, I was rattled enough that I was like 20%, maybe so. But called the courthouse. No, I'm fine. Called my parents, was like, hey, the scam just happened. If somebody calls you and says that I'm in jail, I am not in jail. Do not, you know, be on guard for all of that. My thought process was, we need to come up with something, particularly if the goal of that exercise was to get a voice imprint of me, which anybody could do. I'm on the Internet. I have a podcast. Anybody could make a Heaton voice clone at this point. So were you to get a recording that said, mom and dad, I'm in jail. This is a recording. I can't respond to this, but I need $20,000. Please, please, please. I beg of you, whatever the thing is, right? My thought was, well, we'll have a code word for this is a legitimate exercise. I actually mean this thing I'm talking about and a code word to say I am under duress. Do not believe what I am saying. You're saying no for this. Brian Brushwood Why? Well, mainly because it requires too many layers of abstraction. You've got your parents who, let's say, even when they're running at 100 percent, now they're dealing with a distressing narrative. Let's say they're down to 30 percent of all their brain cycles. And in that, we want them to hold on to a nested set of code words and to remember the code words and it and. Well, meanwhile, narratively, they because it's a cloned voice, all they have to do is insert the forget about that. I've forgotten the whatever. This is important. Now they could just press an emotional lever button because you'll notice like. Andrew Heaton Mom, I'm sorry. I know we've got a code word, but I forgot about it. And I'm going to they're going to put me in jail for a week. You have to do it right now. Yeah. OK. Brian Brushwood Instead, instead, what needs to happen is this is one of the this is why traditionally a lot of companies tend to think of HR as or some people think of HR as morale or lawsuit mitigation. HR needs to be thought of as critical security. You need to put a lot of thought into what happens at your holiday Christmas party. Like if I'm designing holiday Christmas parties, I won't even hire a magician because magicians keep doing the same act again and again. I would hire improv comedians and I would guard very closely the hat that suggestions come out of so that later on two people who were physically there at the same party have a shared reality. For example, right now, let's say you suddenly were not certain it was actually Brian Brushwood that you were talking to or I wasn't certain that it was Andrew Heaton. I would engage in a bit of mobster talk. I would obliquely reference maybe something particularly spicy or private or ask like, oh, man, what was the name of that incredible? It was like a blend. Oh, you know what? Actually, I'll do this right now. There was one time that you violated or thought about violating your no booze rule that year you took a break, but it was a very good exception. Do you happen to remember? And now at this point, you know what I'm talking about, but here's what I want you to do. I want you to answer me in a satisfying way without ever giving the details of what it was. Tell me, mobster talk me so that I know that you're thinking of the same thing. Andrew Heaton Yeah, Brian, I know exactly what you're talking about. We all went out to lunch and our friend had a tasting for some very, very old stuff. That's exactly right. One of them was from the year the Lusitania went down. And I thought, OK, I'll break it for this. This seems like a reasonable thing. And now what just happened? Brian Brushwood Now, that was a robust handshake. There is no conceivable way that an outside entity could get access to enough of those vectors. There's nothing that was posted on social media. Everything was so oblique and it was never a public story. Andrew Heaton Can I ask you, because I might have mentioned that on the podcast, entirely possible, right? As podcasters, we're kind of exhibitionists. So if somebody were really, really dedicated here, could they make an LLM, upload every transcript of every podcast I've done, and then have a Heaton AI clone actually say that to you? Like, is it possible that they could? Brian Brushwood It is, but at some point, you know how, like, technically there's no reason that a starship can't go the speed of light. It's just that as you get to that 99.999, it becomes, you just need all the energy in the universe to go a little bit farther. So at some point it takes all the calories of the sun in order to power enough compute where it's like, it would just be easier for it to be another human who is there. Andrew Heaton There's some, okay. So, okay. In the instance that we were talking about, um, there, you know, if mom, there's a gun to my head, I don't remember the code word, give me the money. The code word's now out the window, right? Presumably under one of these scenarios, my guess is at least presently, we might get to that point where somebody could actively do my voice. I think at the moment you could absolutely do it as a recording. I don't think it's common to- Real time, real time. Brian Brushwood It's either here—it's certainly here today at a state actor level. It's here today at a—if you want to spend money as a corporation level. The only question is, when does it hit folk grade level, as Perry Carpenter puts it? Andrew Heaton That's apparently coming then, right? So, like, brace ourselves for that. There will come a point where I, Andrew Heaton, might call somebody and go, hey, this is so embarrassing. I'm at an airport in Paris. I'm really stupid. I need ([Time 0:30:09](https://share.snipd.com/snip/a6cd9c6b-c8ce-49d5-8c59-198ac3ea1052)) - 59sec Snip Transcript: Andrew Heaton Can I ask you, because I might have mentioned that on the podcast, entirely possible, right? As podcasters, we're kind of exhibitionists. So if somebody were really, really dedicated here, could they make an LLM, upload every transcript of every podcast I've done, and then have a Heaton AI clone actually say that to you? Like, is it possible that they could? Brian Brushwood It is, but at some point, you know how, like, technically there's no reason that a starship can't go the speed of light. It's just that as you get to that 99.999, it becomes, you just need all the energy in the universe to go a little bit farther. So at some point it takes all the calories of the sun in order to power enough compute where it's like, it would just be easier for it to be another human who is there. Andrew Heaton There's some, okay. So, okay. In the instance that we were talking about, um, there, you know, if mom, there's a gun to my head, I don't remember the code word, give me the money. The code word's now out the window, right? ([Time 0:35:25](https://share.snipd.com/snip/9ad24f91-ac65-4f33-a234-0c7c88be633c)) - **Never Do Anything While In State** - Never act while in an emotional state; extract yourself from urgency by hanging up, walking, and consulting someone else. - Brian says strong social networks and a quick external check blunt social-engineering leverage like guilt or panic. Transcript: Brian Brushwood Never do anything while in state. This is good advice for relationships, for business thinking, for financial decisions. If you are feeling any emotion, almost certainly that was caused by an external force, which means you are literally in an altered state of mind, which means you must not trust present Tense you. So you must extract yourself. Andrew Heaton Hang up the phone, go on a 10- walk, resume. Brian Brushwood And call literally anybody else besides that person. ([Time 0:43:11](https://share.snipd.com/snip/c085ccec-4154-4531-81c6-2f13812d8ae9)) - **Create Simple Money Threshold Rules** - Set hard delegation thresholds for money or sensitive actions so you must consult a trusted person before acting on large requests. - Brian suggests simple flowcharts like 'Is this over $X? If yes, call Y' to remove ambiguity in crises. Transcript: Brian Brushwood And then now, you don't have to know whether it's legit or not legit. You don't have to suspect if it's legit or not legit. It has simply hit a level of scale where as a matter of policy, you do not act on your own. ([Time 0:49:13](https://share.snipd.com/snip/5b9117ef-e17c-407f-9285-eabdb1cb9727)) - **Use Long Videos And Encrypted Channels For Proof** - Use higher-fidelity communication when building relationships: long video messages and end-to-end encrypted apps like Signal make falsification harder. - Brian sends minute-long videos to new contacts because generated fakes struggle with longer, natural recordings. Transcript: Brian Brushwood Instead of texts, when I am establishing a friendship, I send long video messages over a minute long because those are difficult to generate falsely. And there's certain expectations about communication modalities, for example, blue text versus green text. And sorry, Android, I'm not trying to be phone sys about it. But the important thing is knowing the provenance from a technological perspective of your communication. Signal, for example, is extremely reliable because it's end-to encrypted. And if it's over a certain length, you can be reasonably certain that it is not a generated video. ([Time 0:53:10](https://share.snipd.com/snip/40903126-9ccf-4add-8837-c35bd5f37cdc)) - **We Will Trust People Over Media** - As deepfakes proliferate, provenance and trusted interpersonal witness will regain value over raw audio/video evidence. - Brian predicts daily life will rely more on who you trust and shared lived experiences than on media artifacts. Transcript: Brian Brushwood But within a provisional status, I suspect that we are going back to Victorian era norms. Did you ever read The Diamond Age by Neil Stinson? No, you've mentioned that. Andrew Heaton I need to read that because it's in the future, but they kind of become Victorian, right? Brian Brushwood That's correct. In the case of that story, it's nanobots creates anything as possible. You can have anything you want. And in that world, well, what is there to fight over in your status games? They just decide out of nowhere. It's like, well, let's just pick Victorian England rules. And that's what they do. We're living in that right now online. And so to the extent that we want to step outside of that, we need to elevate the it's very strange. Pictures and video had a good run as evidence, but now I would trust eyewitness testimony of people who I trust, who would say, no, no, no, that's from the before times. That's from this. That's from that. Andrew Heaton You know, this is now four years ago, five years ago, it would have been 2020. So I keep up with AI, not as much as you, but I keep up with it fairly well. And every year or two, I figure out if I can outsource myself. I'm like, okay, if I'm going to get automated and my job's going to go away, can I at least take a vacation for a year where I get paid to do this before the job disappears? So every once in a while, I'll see, can AI write the show? Turns out so far, AI is not very funny. So I'm good there. I can definitely clone my voice. And 2020, 2021, I played a recording for my dad and he was just like, okay, is this a podcast? And I'm like, no, this is a robot that is pretending to be me. I've never said these words. I didn't even write them. I had ChatGPT write this in my style. And then I fed it through Descript that as a clone of my voice, you're hearing something written by a robot, performed by a robot pretending to be me. And his face just fell. And you could see, Dad is a judge. You could see in this moment, he was like, there's that. Everything just got redefined. Now, like whenever whenever we bring in any audio testimony, I am going to have to have I don't know the specifics on this. We didn't really talk about it, but presumably he's thinking I'm going to have to rely on very specific levels of forensic expertise that can confirm that this is actually a real thing And not a, you know. And yeah, so like you're thinking we're going back to the eyewitness testimony and my word is my bond. And I like Brian says he saw it. I trust Brian as opposed to I saw a video of it. Brian Brushwood Well, and there will be a place for forensic provenance to be proven in a court of law for matters that are of that level of importance. But on day to day stuff, I think that that personal relationships and who do you trust are going to be increasingly valuable. ([Time 0:53:55](https://share.snipd.com/snip/a460ca45-d96c-4820-ae9b-516b55058977)) - **Scambot Fooled Brian's Dad In Live Test** - Brian and Perry Carpenter built an LLM scambot that cloned Brian's voice and tricked his dad into revealing his high school name. - The experiment used a live call guiding the target back to the requested data, showing folk-grade threat exists today. Transcript: Brian Brushwood And then we called my dad and it just said, hey, dad, I'm doing a comedy thing. I need you for comedy reasons to tell me the name of the high school you graduated from. And he gave it. We deceived dad with a scam bot. And that was a zero to one prototype with folk grade technology. You have to assume perfection. So right now, what is deeply satisfying is to see those artifacts. It's a bit like, remember those, the Masked Magician specials where they would show a trick and then reveal how it was done? Deeply satisfying. Unfortunately, after three seasons of that, you were no closer to being foolproof than before. You only had the illusion of learning how things were done. It was deeply satisfying to close the loop, but in no way protected you from being fooled. Likewise, all of these little tips of like, watch for the broken English, watch for this or that. Those are not where the real deceptions going to be coming for you. Andrew Heaton With your dad, that's freaky. There was a Brian Brushwood LLM. Was it acting in real time? So this wasn't somebody had a clone of your voice and they're talking to your dad, but it's going through a filter. It's that there was a Brushwood bot that has your voice and it was talking to him in real time, pretending to be you. Yeah. Brian Brushwood Let me do a, here, I'll play for you the audio of it and you can hear. Yeah, there we go. This is the very beginning right here. Here we go. Andrew Heaton This is a level right now where you and I can create, essentially, a nation-state attack when it comes to deepfakes. Brian Brushwood So for this improv game, I need to know the name of the high school you went to. Just the name of the high school, Dad. Oh, God, I had to. This robot's on a mission to get one key bit of info from my dad. Oh, sorry. But yeah, that's that's the opening of the episode. We built an LLM. We gave it instructions to extract that information. And when dad would start to follow a different narrative thread, it would guide him back. It's like, hey, dad, I'm live on the air. You got to give that to me right now. Andrew Heaton Did you did you call him on your phone or did you call his landline using some other number? Brian Brushwood I called his cell phone from my cell phone. ([Time 0:56:52](https://share.snipd.com/snip/763549b3-c024-484a-84fc-d1ae8e8f344f)) - **Train By Playing The Bad Guy** - Teach people offense by having them build scams in safe exercises so they learn to spot patterns and emotional levers. - Brian ran workshops where nursing students created realistic phishing emails, which dramatically improved their detection instincts. Transcript: Brian Brushwood And I kind of came out of the gate swinging like, guys, it looks like we're doing all this wrong. And to his credit, he said, great, when do we start? So we went to the School of Nursing. Healthcare is the number two most targeted industry when it comes to these scams. And we got permission to take a crop of 80 incoming freshmen to the nursing programs, literal children. And so did four lessons where I said, great, we're all going to play as the bad guy. Here's how phishing emails are made. You're going to do a dead simple one. This is a serious document link and then send it to a hilarious image. And then from that, then it's like, great, now we're going to enhance it. By the end of it, these kids were making weapons-grade phishing emails where they were pretending to be the president of the university with stolen banners, trappings of forwarding Emails, using targets that were specifically targeting the anxieties of other students, their FAFSA, the parking permit, tickets, being late for class, academic fraud, AI. Without any me telling them what to do. I set up incentives and they were owning the task of deceiving their fellow students. And you could see that they were fundamentally altered on the other side of that. ([Time 1:03:20](https://share.snipd.com/snip/47c4b476-081f-4088-980d-b10b44b3444d))