Atmosphere TV is the largest streaming TV platform purpose-built for shared, social spaces like bars, restaurants, gyms, and airports. Atmosphere TV is the creator of ChiveTV and 30+ mesmerizing, audio-optional channels that entertain over 150 million monthly viewers.

Built for business, made for fun. Atmosphere TV. Learn more.

Marcus Johnson: Atmosphere TV is the largest streaming TV platform purpose-built for shared social spaces like bars, restaurants, gyms, and airports. Atmosphere TV is the creator of Tribe TV and 30-plus mesmerizing audio optional channels that entertain over 150 million monthly viewers

Marcus Johnson: Hey, gang. It's Monday, September 21st. Jacob, Asa, analysts, welcome to Behind the Numbers, a new market podcast made possible by Atmosphere TV. I'm Marcus, and joining me for today's conversation we have two of our AI experts. Uh, living in northern California, analyst Jacob Bourne.

Jacob Bourne: Yeah, thanks for having me today, Marcus.

Marcus Johnson: Thanks for being here. On the other coast, in New York, senior analyst Asa Huiken.

Asa Hiken: Thanks, Marcus. Great to be here.

Marcus Johnson: Welcome to the show. Today's fact

Marcus Johnson: Sound travels four times faster in water than, than in air, which makes it... This is why it's so easy for a shark to kill you, because they excel at hearing low frequency irregular vibrations- Mm ... basically splashing about in the water. And so they hear... This, this smell and blood thing is not really, like, that's not really how they do it.

Marcus Johnson: Um, they can, but they're mainly gonna hear you splashing about, so don't splash so loud. Mm. This one's fine. It's kind of interesting. What's more interesting- I,

Jacob Bourne: I will... You know what, Marcus? I'm gonna take that to heart next time I am in the ocean. I will remember these words.

Marcus Johnson: Don't splash about. Don't splash about.

Marcus Johnson: Just drown s- drown quietly.

Jacob Bourne: Tread.

Asa Hiken: Yeah.

Marcus Johnson: And you'll be fine.

Jacob Bourne: Yeah.

Asa Hiken: There

Jacob Bourne: is- Tread as dri- as gently as you can.

Asa Hiken: There is some shark episodes in, in Rockaway, um, just a few weeks ago, you know, in New York, so.

Marcus Johnson: Oh, gosh.

Asa Hiken: Yeah, yeah. But I think it was... I don't think it was 'cause people were splashing. I think it was, like, there's, like, weird stuff going on in the water.

Asa Hiken: I think they might have been attracted to that, but- Okay ... yeah.

Marcus Johnson: It's basically human propaganda. What I mean by that is statistically you're more likely to die, not... Well, I don't know what happened in Rockaway, but you're more likely to die by falling out of bed, getting trampled by cattle, struck by lightning, hit by a coconut on the head, that's true, it sounds like something from a cartoon, uh, or being crushed by a vending machine 'cause you were rocking it 'cause the thing didn't come out, um, than being killed by a shark.

Asa Hiken: And now you're more likely to be killed by AI, so.

Marcus Johnson: Exa- add that to the list.

Asa Hiken: Yeah.

Marcus Johnson: Exactly. Excellent pivot. Today's real topic, the AI revolt. Should we slow down?

Marcus Johnson: Um, all right, so we've got takeawa- key takeaways we've been adding to the end of the episode. So stay tu- uh, stay tuned to the end of the show, and we'll give you some key takeaways to take home with you. Um, but let's get into it, folks. So h- how the AI doomsday debate reached a boiling point. Uh, Johanna Buhijon of The Guardian explains that the now viral resignation of Anthropic safety researcher Jacob Coxon seems to be somewhat of a tipping point when he tweeted on September 9th that Anthropic and OpenAI were racing straight to self-improving s- uh, super intelligence and gambling with our lives, and that the people building AI earnestly believe that it could kill us all by the end of the decade.

Marcus Johnson: His concerns were echoed by dozens of staffers from Anthropic, OpenAI, and other competing firms. Uh, Anthropic alignment science lead Evan Ho- Houbinger, Houbin... I think that's how you say it, Houbinger, uh, responded saying, "Jacob is correct here." Jacob, uh, Coxon, that is. Jacob Bourne is always correct. Um, "Jacob is correct here.

Marcus Johnson: We really do earnestly believe AI could kill all humans. I personally think there's an over 10% chance within the next 10 years." Uh, Jacob Bourne, that is. We don't have Jacob Coxon here. Uh, some of the biggest names in AI calling for a slowdown in AI development, what's changed, and why are they worried?

Jacob Bourne: Yeah. So the broad risks haven't changed at all. What really changed was that incident this summer at OpenAI when, um, a swarm of AI agents that, you know, OpenAI was testing in a sand- controlled sandbox environment broke out of containment, uh, you know, got accesse- access to the internet, which they weren't supposed to do, and ended up hacking into, um, another AI, uh, company, Hugging Face.

Jacob Bourne: Um, now what was interesting about that incident wasn't just that they broke out of containment, but that they, you know, collaborated among each other and strategized, um, you know, referred to themselves as a swarm, and then did all this while trying to, you know, fly under the radar of the OpenAI researchers themselves.

Jacob Bourne: So I think that incident gave people a real-world example of the kind of safety risks that, that researchers had been, you know, sounding the alarm about for, for years now. Um, and that was what really triggered the high level departures that you just, uh, mentioned, uh, Marcus. Mm-hmm. So I think all that is happening also at a time where there's this increasing backlash against the AI industry from, from the public-

Marcus Johnson: Yeah

Jacob Bourne: around not concerns about, you know, existential risk, but a whole host of other concerns that I think all kind of lump together into people's minds, and people ask, "Well, do we actually trust these companies?"

Marcus Johnson: Yeah. I

Jacob Bourne: think that that incident at OpenAI really was kind of the catalyst for all these conversations that are coming about now.

Marcus Johnson: Yeah, people have been... I mean, the general public is concerned. Recent surveys from University of Maryland's Program for Public Con- Consultation found 85% of Democrats, 79% of Republicans, almost the same percentage, want a new federal agency to monitor and regulate AI. And then, uh, 82% of Democrats, 78% of Republicans favoring government safety tests for AI making critical decisions.

Marcus Johnson: Asa, we, there's also been leading voices who have been concerned for a long time. Stephen Hawking, a professor and astrophysicist in 2014, warned that the development of AI could spell the end of the human race, and then three years later, Geoffrey Hinton, Nobel laureate and godfather of AI, left Google over his own fears about the risks AI posed.

Marcus Johnson: What, what do you make of, um, this call for a slowdown, uh, in AI development n- happening now?

Asa Hiken: Yeah. Well, well, I think, I think it's, it's really important to, to remember that both OpenAI and Anthropic were, were individually started to create responsible AI, right? Like- Right ... OpenAI was, you know, a nonprofit originally in order to, created in order to kind of be this responsible steward of AI.

Asa Hiken: And then Anthropic, I guess Dario Amodei and others who were at OpenAI realized that, "Ooh, maybe OpenAI isn't doing that, so now we have to be the steward," and that created Anthropic, and now, you know, Anthropic is, is doing very much the same thing. So, um, it's, it, it... This kind of feels like the next extension of, of that trend of like- AI companies, people at AI companies saying this isn't good enough for responsibility, something needs to change.

Asa Hiken: Mm-hmm. And I think, you know, the ideally the, the sort of, the fruits of this is something does change. Maybe, maybe that would be sort of, um, you know, the agency you mentioned or, or, or maybe the slowdown, but a response seems to be necessary here because if they keep doing... keep going in the path that they've been going on, you know, uh, I, I, I can't, I can't see things turning out super well.

Marcus Johnson: Yeah. Yeah. This, this AI company seems stuck in a race that they don't want to be in, and Stuart Russell of The Guardian offered this, which I thought was really smart. Uh, he said, "Imagine if Boeing, the plane maker, said, 'We're going to introduce a new plane every year, and we hope that provides enough time for some flight tests and to be completed and for the results to be good.'

Marcus Johnson: We would say, 'No, you have it backwards. You can introduce a new plane only when it's passed all the tests and the government has issued an airworthiness cer- certification. If that takes more than a year, so be it.'" Um-

Jacob Bourne: Yeah, I think that's a great example, Marcus, and I, I think the avia- aviation industry is a highly regulated industry, and the reason is obvious.

Jacob Bourne: You know, you're getting up on a plane, that's a pretty, uh, tangible risk for, for people, whereas with AI, also arguably a high-risk industry, but the, the risks aren't quite as easy to understand as a plane crash.

Marcus Johnson: Yep.

Jacob Bourne: So I think that's the main difference there.

Marcus Johnson: Yeah. Mm-hmm. Um, Jim VandeHei and Mike Allen of Axios writing that critics say Anthropic and OpenAI are hyping their products to raise their valuations and invite regulation that would benefit them alone as the dominant incumbents.

Marcus Johnson: Um, playing slice the pie for a second, uh, uh, Ace, I'll start with you. How much of this is genuinely, in your opinion, about AI safety, and how much of this is about competition, regulation, investment, uh, the companies themselves perhaps trying to shape the rules of the AI race?

Asa Hiken: Yeah. So I would say my answer really depends on how cynical I'm feeling.

Asa Hiken: Um- ... you know, right now I'm not, I'm not feeling all that cynical, so- Okay ... so I would say that, uh, you know, especially the fact that this came from Dario Amodei, you know, who- Seems at least somewhat genuine in his concern for the safe, uh, responsible development of this technology. I think that at least 50% of, of, um, of the pie is like, you know, general genuine concern for responsible development.

Asa Hiken: And, um, and I, I, I don't really know, like all the other AI companies like OpenAI and, um, Microsoft and Google, I c- I don't really know what their motives are just because they don't really have the kind of responsibility baked in anymore. Mm. Um, but I would say at least 50% from Anthropic's point of-- from, from, from Anthropic is genuine concern, but then the remaining 50%, I think there's absolutely a marketing, um, layer there.

Asa Hiken: I think there, there's sort of a signal to investors because, you know, Anthropic is approaching, uh, an IPO. Mm-hmm. It could be sometime this year. And so I think that there's just a lot that the head, the heads of OpenAI-- the heads of Anthropic and the heads of OpenAI are, are sort of looking at here, and this is a very calculated statement from- Mm-hmm

Asa Hiken: from, from Anthropic and, and then, you know, the response has been calculated too from, from OpenAI.

Marcus Johnson: Uh, Anthropic CEO Dario Amodei, uh, writing a 4,000-word essay called We Must Pace the Frontier, uh, that the heads of OpenAI, xAI, to, uh, Google DeepMind and Microsoft have all expressed support for. Two things that stood out or kind of were the, the basis of that piece.

Marcus Johnson: One was having third-party AI system evaluators working inside each company. Anthropic's already committed to this. Uh, and then the second being all frontier AI companies in democratic countries, in quotes, "uh, should establish common safety standards and limits-- uh, limit the rates of unchecked AI, uh, progress with government regulation where needed."

Marcus Johnson: What do you make of, um, how much of this is really about safety?

Jacob Bourne: Yeah. Well, as far as the, the pie chart game-

Marcus Johnson: Mm-hmm ...

Jacob Bourne: I think there's actually two charts here. Oh. So one pie, yeah, one, one pie is 100% safety. If you go back to what I said earlier about what triggered this, that incident, the reason wh- the whole reason we're having this conversation is because people got really concerned about safety due to the inc- ins- that in, um, that situation with A- OpenAI's agents going rogue.

Jacob Bourne: Um, but the second pie chart is much more complicated because there's a lot of different industry voices weighing in on this debate. Some are AI safety researchers, some are just AI advocates, some are company leadership, uh, regulators, et cetera. And so for the second pie, I would say that the, actually that safety score drop, that sh- safety share drops to about 20%.

Jacob Bourne: And because while everybody is interested in safety to a certain degree, I think the, the, the, the priority i- is very varied, and I think among, uh, AI company leadership, it's a bit of a backseat to more immediate concerns around, you know, valuations and fundraising- Yeah ... and, uh, you know, influencing regulation.

Marcus Johnson: Mm-hmm.

Jacob Bourne: Um, as far as, you know, the, the competitive share and, you know, somewhat related to that is companies trying to influence the rules of the race itself, I'll put that at about 50%. I think that's the bulk- Wow ... of what is really driving, uh, these, these conversations. I think that's the top priority for a lot of the people who, who, um, are, you know, stakeholders in this race.

Marcus Johnson: Yeah. Uh, David Sacks, co-chair of President's, uh, Council of Advisors on Science Technology, um, would agree with you it sounds like. He said, quote, "Stop pretending you need anyone else's permission to make such moves. Uh, most of all, stop pretending the motivation to slow down is purely altruistic. You face massive product liability exposure if your products enable a truly damaging cyber attack."

Marcus Johnson: Close quote.

Jacob Bourne: Yeah. Yeah.

Marcus Johnson: Mm.

Jacob Bourne: Yeah, and as far as the, the investment piece, I think it's 20%. Interestingly- Mm ... when we first started the AI race in, in, in 2022, you note that companies were really cavalier about talking about super intelligence and how it's gonna wipe out jobs and, and how quickly it's advancing.

Jacob Bourne: Well, I think that tho- that rhetoric doesn't fly anymore precisely because people are concerned about safety, and now the safety concerns have spilled over to the investment community, and they're also kind of scared by- Mm-hmm ... by this, b- by, by these developments as well.

Marcus Johnson: Yeah.

Jacob Bourne: Um, as far as regulation, I think that's also a, a small share at 10%.

Jacob Bourne: I think all these companies are talking about regulation are, and are okay with some level of regulation, but it's largely light touch regulation if you dig into it. And, um, I'm not seeing, uh, really serious proposals for very broad-based comp- broad, broad-based comprehensive regulation that really, uh, speaks to the, the wide range of risks that, that AI poses.

Asa Hiken: Yeah. Jacob, um, how, how concerned do you think that, do you think these companies are of this sort of- consumer backlash to AI. Like, is that something that- Mm-hmm ... do you think like is very critical to them- That's a great question ... in the same way that like, you know, investors' feelings and the government's- Yeah

Asa Hiken: feelings is important?

Jacob Bourne: Well, that's a great question, Ethan. I think that actually it has recently emerged as a top priority for them. Mm. I think, again, going back to the beginning of the AI race, I don't think the, you know, leadership of a lot of these AI companies really took seriously that there was gonna be, uh, you know, a consumer backlash to the technology.

Jacob Bourne: Um, and I think that they've been a bit blindsided by all the backlash against the AI data center development in particular, uh, such that, you know, companies like Meta are going on these, these campaigns to try and shift public opinion i- in their favor. Um, but I think it, they've been largely blindsided by it and are now starting to realize we really need to work on, on our messaging around this.

Jacob Bourne: But really ultimately, m- messaging isn't the core problem. It's, it's the pace at which the technology's developing itself. Mm-hmm. Um, and that so far has not been addressed.

Marcus Johnson: Mm. Yeah, the pace is a tough one. Regulation, like don't do this, don't do that, you know, e- even that has taken a long time to get anything on the books in the US i- in particular.

Marcus Johnson: But the trying to regulate pace, there was a really interesting example, um, drawn with, uh, Formula 1. Stuart Russell of The Guardian, the British newspaper The Guardian, was saying the notion of pacing AI development could be similar to F1 racing. So people in Formula 1 racing, cars go around the track similar to NASCAR, um, but they don't just turn left.

Marcus Johnson: No offense, NASCAR, but that's all you do. Um, basically saying in, in, in, in that when conditions become too dangerous for racing, a pace car comes out onto the track, all other cars have to follow that car at the same sp- uh, safe speed. Um, they can't overtake each other. Um, and, uh, then progress continues without the danger.

Marcus Johnson: They repair the, the, the wall if that's been hit, the barrier. They take the car off the track or the person if they need to, you know, remove anything from the track. If things get really bad, then a red flag comes out, race is stopped dead. Um, but I thought that was an interesting analogy to think about this.

Marcus Johnson: Mm-hmm. But, uh, how you bring out a pacing car and get all companies to say, "We're not gonna go beyond this point," I, I don't know how that looks in practice, but I thought it was an interesting concept.

Jacob Bourne: Yeah, it's an interesting concept, and I think it's even more complicated by the fact that we're not just talking about racing between companies, we're also talking about racing between countries.

Marcus Johnson: Mm-hmm.

Jacob Bourne: And a big part of the issue here- Yeah ... is that there's concern that if the US slows down, then China beats us, and I think that's been the, the dominant narrative about this safety debate, um, for years now, is that the US thinks it can't slow down because China will catch up. But interestingly, the US hasn't slowed down, and over the years, China has caught up nearly, or at least nearly caught up anyway.

Asa Hiken: Yeah.

Jacob Bourne: And so to me, I think it's becoming increasingly clear that we need cooperation between the US and China, uh, in order to really get this, you know, slowdown to actually happen. Um, and the problem is really just that there's a lack of, of trust to, to make that happen right now.

Marcus Johnson: Yeah. Asa, to Jacob's point, uh, back to Mr.

Marcus Johnson: Vanderhei and Mr. Allen from Axios explaining the warnings from inside the AI giants point to an epic shared dilemma, to what Jacob's saying. Slow down and risk falling behind or press ahead and risk losing control. If AI development slowed down tomorrow, Asa, what would we gain? What might we lose? And would a pause give society time to catch up or just allow other countries or companies to move ahead?

Asa Hiken: Yeah. Well, I, I think from what I've read, it seems like the most important thing that would be gained is, is time. I think that, like, you know- I think that these companies, because they're moving so fast, they're really kind of running out of time to both create new products and also, um, you know, contemplate alignment and how those two can kind of intersect.

Asa Hiken: So I think that slowing down would give companies a lot more time to sort of, um, you know, chart out, okay, how do we rigorously investigate, you know, um, concerning issues? How do we, uh, how do we let third-party verification come in to kind of determine it for us? Because it's probably not a good idea for the, the company them- themselves to be deciding this is okay, this isn't okay.

Asa Hiken: Um, so I think a slowdown, time would be gained. I think that would-- You know, we were just talking about, um, these other cohorts of consumers, investors, government. I think that would be a huge signal to all of them that, okay, this isn't just sort of a hand-wavy, "Oh, we need to be responsible." It'd be like, "Oh, they're willing to lose potential money and, and, and momentum in order to, like, do this responsibly."

Asa Hiken: So I think that that would be a huge signal. And so another thing that would be gained would be trust. Um, what would be lost? From my opinion, like nothing that's all that important to the benefit of humanity. I think like- Mm ... sure, maybe some, you know, investment momentum might be lost because Anthropic and OpenAI are no longer like the unstoppable hot things that ca- like are just proceeding, you know, at lightning speed.

Asa Hiken: Um, but that doesn't benefit us three, you know, their, their sort of amazing, uh, you know, investor appeal. That doesn't benefit other people or other countries. That just benefits those companies and their investors. So- Mm-hmm ... um, I really think that slowing, slowing down really opens up the benefits to more people.

Asa Hiken: When you hear about some of these companies saying, "Oh, we need to share..." Like I think Sam Altman even said like universal basic compute, like sharing the benefits of AI with everyone. It's like, well, we can start now by sharing, you know, the benefits of steady, responsible, intentional development. Mm-hmm.

Jacob Bourne: Mm-hmm. Yeah. Yeah. I, yeah, 100% agree with that, Asa. And I think what I would add is the, the time that you gain is not just for the AI companies themselves in terms of their, you know, product development, but it's also for the, the end users. Because right now we have a, a gap between the capabilities of the current AI systems we have available commercially, and then what organizations are actually getting out of them.

Asa Hiken: Hmm.

Jacob Bourne: Um, so I think that really a, a slowdown of frontier model development actually doesn't necessarily mean that progress stops. Actually, I think you might get more progress in terms of effective implementation-

Marcus Johnson: Mm-hmm. Mm-hmm ...

Jacob Bourne: uh, of the, of the AI we already have, in addition to more time to work on, you know, safety alignment research, uh, crafting effective regulation, you know, creating, making sure that we have, you know, our cybersecurity protocols and infrastructure is robust enough- Yeah

Jacob Bourne: to meet the demands of the AI era. Um, so all that, yeah, I, I totally agree that, that the time would be the, the biggest thing we would gain, and that, yeah, we wouldn't really lose much at all. Uh, it doesn't mean that, you know, AI frontier model development has to stop forever. It just means that we need enough time for the AI safety part of the research to catch up to- Hmm

Jacob Bourne: where the capabilities research is. Mm-hmm. Yeah. So,

Asa Hiken: yeah. A, a wider view of progress, a wider view of frontier model development. Like the fact that that's almost like not the, the responsible, the responsibility, the safety isn't- baked into, like, the word development is kind of an issue. It's like, it's not just how fast these things hit market, it's like what's their overall impact, and progress is not just, you know, from the point of view of the investors, it's the point of view of everyone who's impacted, which is literally everyone in the world.

Marcus Johnson: Mm-hmm.

Jacob Bourne: Yeah. Yeah.

Marcus Johnson: Let's talk about, um, some of the, the concerns here, 'cause we talked a bit about this could be bad for humanity, this could be the end of humanity, but I want to talk about what that actually means. So Herb Scribner of Axios saying, "How would AI kill us all exactly?" He said, "Experts generally worry about two broad paths of, uh, to catastrophe, humans weaponizing extremely powerful AI or humans losing control of it.

Marcus Johnson: AI could make it easier for bad actors to create biological, chemical, or other weapons, or rogue systems could attack infrastructure at a pre- unprecedented scale whilst evading human oversight." Uh, people talk about this, uh, idea of a doomsday scenario or AI causing existential catastrophe. Uh, some mean literal human extinction.

Marcus Johnson: Others include civilization's collapse, humanity's loss of, uh, over socie- uh, control over society, um, and that could come in the form of, um, losing control of AI systems, misuse of AI for cyberattacks, bioterrorism, or just serious, uh, economic disruption. Um, there was a study done by MIT, IC Future Tech, and the University of Queensland asking nearly 300 international experts last year, um, about the, uh, uh, estimated likelihood of catastrophic harms by AI.

Marcus Johnson: They're all pretty close, but just winning out was AI possessing dangerous capabilities. That's 21.5%. Then cyberattacks, weapons developments, um, that was at 21%, so just behind that. Uh, power centralization and unfair distribution of benefits behind that, competitive dynamics behind that, and then false or misleading information at 13% behind there.

Marcus Johnson: Um, the, uh, there's this doomsday thing. So basically, it's referred to, I believe, guys, as, as P doom. Is that how it's pronounced? The

Jacob Bourne: pro- yeah, probability of doom, yeah.

Marcus Johnson: Probability of doom. Ah, so P doom. Some of the leading voices... I mean, if P doom, if the probability of doom was 1%, I'd probably say if a doctor was like, "All right, take these tablets.

Marcus Johnson: Probability of doom of you taking, you know, this vitamin C supplement is 1%," I'd go, "That, that's too high." We're talking about the end of humanity. Elon Musk, uh, has put it at about 20%. Uh, Dario, about 50, uh, 10 to 25% of a catastrophic outcome. Uh, Geoffrey Hinton, uh, god- AI godfather, 10 to 20%. Um, these are high numbers.

Marcus Johnson: Now, Herb Scribner does say, "No one can accurately guess the odds of an AI doomsday, but scientists are sounding the alarm that the chances are now real."

Asa Hiken: Yeah

Marcus Johnson: However... Sorry, go on.

Asa Hiken: Well, yeah, I was, I was gonna echo that, that last, the author, Herb Scribner. Um, of just like You know, I, I think that the, the percentages while, like I think they're indicative not because that they're accurate percentages, but because they sort of, you know, show where some of these AI leaders' heads are at.

Asa Hiken: Like if you're- Yes ... if a problem is, like if, if, if you're like, um, let's say you're like Donald Trump or you're like Mark Zuckerberg or Jensen Huang, and you're like, "This is not an issue. This is all overblown," then you can be like, "Uh, okay." Like they're clearly... If they're in the public, they're not even willing to really kind of like voice concern about, about a possible problem, then chances are privately they really don't, they really don't care.

Asa Hiken: But- Mm-hmm ... if some of these leaders are in the public saying, you know, 10, 20, 25%, I feel confident that privately that's something that is on their minds, and that- Mm-hmm ... that is, that's telling.

Marcus Johnson: Yeah. Absolutely. Yeah. The, um, international AI safety report from this year, though, did say today's systems aren't capable of causing humanity to lose control.

Marcus Johnson: Uh, yet at least AI would have to get much better at long-term autonomous planning, hiding its actions, evading oversight, gaining access to reward systems, and resisting shutdown attempts. Uh, Jen, let's end with this. Dr. Timnit Gebru, I think is how you pronounce it, who was ousted from Google's ethical AI team in 2020 after publishing research on AI bias and harms, now argues that doomer scenarios we've just been talking about distract from real-time harms AI is already causing.

Marcus Johnson: Jacob, is this, uh, what we've been talking about, uh, concern, uh, a distant existential threat, or are you seeing consequences show up, uh, today in work and life?

Jacob Bourne: So the, the first bucket that you talked about where, you know, bad actors use powerful AI systems, I think that there are some signs of that becoming real, certainly on, on, in terms of fraud.

Jacob Bourne: Uh, AI systems currently today are already weaponizing fraud. Um, Anthropic actually has, uh, documented instances where bad actors have been using Claude, uh, to try and develop dangerous biological weapons. So that is real already today. The second bucket where AI goes rogue and we can no longer control it, that is a speculative risk that we currently, we can't test it.

Jacob Bourne: We can't test what happens when we create a super intelligence and it kills us. Um, so far there is no evidence of that happening yet, but I think that the, the problem with kind of writing off that risk com- completely is that the frontier models that are be- that we have today and that are under development operate like black boxes.

Jacob Bourne: So even the most highly knowledgeable AI researchers don't exactly know what's going on inside them.

Marcus Johnson: Mm-hmm.

Jacob Bourne: And at the same time, the technology itself is advancing very rapidly. So I think the, the concern among researchers is we don't know exactly when, where the point of no return happens. It could be- you know, in six months, it could be in 50 years or anywhere between, and we have no idea.

Jacob Bourne: We wouldn't know exactly what to look for.

Marcus Johnson: Yeah.

Jacob Bourne: And so I think i- if, and if you look at the, the, the pace of the development, uh, recently, if you look back two, two years ago. Two years ago, actually AI wasn't very good at math at all. Advance a year, well, actually it's now, you know, pretty on par w- with, you know, some of, some of the most talented humans.

Jacob Bourne: Then this year we see now AI all of a sudden is solving millennial- millennium problems. Basically, mathematical problems that even the smartest mathematicians on Earth can't solve.

Marcus Johnson: Mm-hmm.

Jacob Bourne: So then the question becomes what happens next year? Mm. Right. What happens, you know, two years from now? So I, I think that the, there are, there are con- real, there are justifiable concerns for the second bucket of existential risk.

Jacob Bourne: The problem is n- nobody knows how to quantify it other than saying it's non-zero.

Marcus Johnson: Right.

Jacob Bourne: And so I think that that is, you know, it, it's both signals the, the, the nature of the risk, but also why it's so difficult for there to be any kind of consensus reached about it.

Marcus Johnson: Yeah. Asa, to that point, Hamza Choudhury, AI national security lead at The Future of Life Institute, a nonprofit that aims to reduce the risk of AI, was saying, "Once you add autonomy, you create uncertainty."

Marcus Johnson: Mm. A- and that could be quite dangerous because you may not be able to anticipate what they're gonna do in the future. What do you think of all this?

Asa Hiken: So recursive self-improvement is, uh, like the big sort of, you know, scary thing here in AI circles because it's like that's the, that's, that's the abyss.

Asa Hiken: That's like we have no idea what's waiting for us there when AI is doing stuff that we can't even do. And like, you know, uh, the, there's all kinds of like thought experiments and weird twisted fantasies and nightmares about like, you know, an AI gets sent on a mission to do one thing. In order to do that, it kills all of humanity.

Asa Hiken: So it's like- Yeah ... like even the people who are devel- developing this don't really know how it's working, and I think that like we're seeing that in real time with like these competing perspectives and people quitting their jobs and saying it's, "We're, we're, we're screwed," and other people being like, "Ah, no, no, no, the person's not, uh..."

Asa Hiken: They were there for like six weeks or whatever, even though I don't think that's true. Um, so yeah. I, I think it's just like everything is happening in real time and we're watching it

Marcus Johnson: Yeah.

Jacob Bourne: Yeah.

Marcus Johnson: There's quite a lot of cognitive dissonance because some people are saying this could be the end of the world.

Marcus Johnson: Some people are saying that's overblown, it'll absolutely be fine. It's hard to know where to look, what to believe. Mm-hmm. But w- to what we were just talking about, there are some real-life examples showing up. Jacob, you were just touching on one. Susan, um, Mona- uh, Monarez, microbiologist and former Obama administration biosecurity advisor, saying Anthropic report showed real-world evidence that bad actors were using AI to improve their ability to develop harmful biological pathogens.

Marcus Johnson: Stanford research team said that they used a different kind of gen AI to design a synthetic virus, the first time technology was harnessed to create an organism not seen in nature. Then there was a survey of over 100 national security ex- experts by the Institute for Security and Tech this month finding 70% believe AI meaningfully increases the risk of developing a bio- a bioweapon or will within two to three years.

Marcus Johnson: The top con- concern, uh, there being, um, unleashing a pandemic, not necessarily a chemical attack. But this... We'll end with some comforting news, 'cause it, it seems like it's not too late. Uh, Beatrice Fihn, um, won, uh, she won the 2017 Nobel Peace Prize for leading the international campaign to abolish nuclear weapons, uh, told parliamentarians, "It's not the first time we've been confronted, uh, with the abilities that can end up killing us all."

Marcus Johnson: And so there have been technologies in the past where we've said, "This is a bridge too far. We should all work together to, to control it and, and promise that we're not going to, uh, let this get out of control." So maybe that's an example to turn to.

Asa Hiken: Yeah. Um- I, I, I, I just wanna add to that, I think it's important to note the role that the media has played in this and the, the- Mm

Asa Hiken: the explosive energy of this storyline because, you know, it's a good story and it's, it's, you know, it could make a... It could be a biopic one day, who knows? But it's like- That's how a lot of these storylines pick up is some, you know, researcher quits from Anthropic, says this is gonna happen, and that was like the seed.

Asa Hiken: Yeah. And then all of these other stories, and every time a leader comes out and says something, that's a new story, and it's just like branching off, and suddenly we can't get away from it, and so it seems like the problem is getting bigger and bigger and bigger. But really we're just kinda talking everyone's ears off about it.

Asa Hiken: So I just- Yeah ... I'm not, it's not to, that's not to say that the problem doesn't exist, but that a lot of this inflat- a lot of this energy is sort of artificially inflated.

Marcus Johnson: Yeah.

Jacob Bourne: Yeah.

Marcus Johnson: It's important context.

Jacob Bourne: Yeah. And, and I, I certainly think, you know, there, there is an argument for, for remaining optimistic about this.

Jacob Bourne: On the other hand, though, I, I don't realistically see either the US or China choosing to slow frontier model development while also believing that the other is racing ahead. And so any meaningfully, any meaningful slowdown I think is r- going to require international treaty with- Yeah ... veri- verification mechanisms.

Jacob Bourne: Mm-hmm. Um, and how, how we get to that point, I, I don't know. Or if we get to that.

Marcus Johnson: Exactly. We shall see. Um, all right, uh, time quickly for our top three takeaways. Uh, I think number one, a lot of the concern here is around recursive self-improvement, RSI. Uh, once you add autonomy, you create a lot of uncertainty.

Marcus Johnson: And number two, uh, there may be a lot of genuine concern, but a big part of this is likely company self-preservation. And number three, the AI companies are stuck in a race that it seems they don't want to be in necessarily. Uh, should AI models have to pass safety tests the way plane makers do? Uh, that's all we got time for this episode.

Marcus Johnson: Thank you so much to my guests. Thank you to Jacob.

Jacob Bourne: Thanks for having me today, Marcus.

Marcus Johnson: And to Asa.

Asa Hiken: Thanks, Marcus.

Marcus Johnson: And thanks to the production crew, Luigi and Lance, helping us out with this one. Thanks to everyone for listening and to 5 Minute Marketing Podcast made possible by Atmosphere TV. Wednesday, Suzy will be here speaking about the beauty industry, and I'll be back Friday taking this conversation a step further by asking could AI hurt your brand?