Today, I’m talking with Helen Toner, the executive director of the Center for Security and Emerging Technology at Georgetown University. This episode is part two of our series on AI in real life.

Helen is a well-known figure in the AI safety and policy community, though she is perhaps best known for having served on the board of OpenAI, when she was one of four board members who voted to fire Sam Altman. That incident, now sometimes referred to as “the blip,” ended with Sam coming back as CEO, Helen resigning, and a new board of directors. OpenAI has since restructured entirely.

But it’s worth remembering what Helen and the board were supposed to do: OpenAI’s nonprofit board existed specifically to make sure the company didn’t betray its mission to build safe AI by prioritizing profit and growth over safety. When the board actually tried to pull the lever, that accountability mechanism broke, and here we are.

Now, the question of AI safety is everywhere. This summer, AI models from OpenAI and Anthropic broke out of their testing environments and hacked real companies. The CEOs of these frontier labs have since called for a slowdown and asked Washington for help, but the White House’s answer was a voluntary accord — one that relies in part on the kind of board accountability that failed at OpenAI. Anthropic is also about to go public, which will put a whole new set of market pressures on OpenAI’s biggest competitor.

Helen has seen up close what happens when safety commitments run headlong into commercial incentives. And as she put it to me, if our hopes for the future depend on any one person being trustworthy, we’re in big trouble.

So we talked about what we might do instead: what the labs are actually trying to build, whether AI alignment is a problem that can be solved, and what a smarter approach to AI regulation might look like. That includes how to think about China, which is the issue that looms large in every policy conversation about AI right now — especially now after the Jensen Huang affair. Just for clarity, Helen and I talked before Nvidia opened its investigation and Jensen ultimately resigned, so you won’t hear us talk about that directly.

Before we get going, do me a favor and subscribe to Decoder on YouTube, where we put out new episodes every Monday and Thursday.

Okay: Helen Toner, the executive director of CSET, on the future of AI safety. Here we go.

This interview has been lightly edited for length and clarity.

Helen Toner, you’re the executive director for the Center for Security and Emerging Technology at Georgetown University. Welcome to Decoder.

Thanks so much. Great to be here.

I have wanted to talk to you for 10 million years. I am so excited you’re here. You’ve been involved in a lot of AI safety and ethics ideas for a long time. You were obviously on the board of OpenAI during what everyone has come to call “the blip.”

Now, you’re talking a lot about AI safety, about which I have 10 million questions. But let’s just start at the start. You run the Center for Security and Emerging Technology, people call it CSET. What is CSET?

It’s awesome to be here. I’m really excited to talk to you as well. CSET is a team inside Georgetown, but we’re really not academics. We’re about 50 people and have been going since 2019.

Basically in 2019, a group of us decided that there was a lot happening and a lot of emerging technologies that were becoming very relevant in the national security space, and the security space more broadly,. And some policymakers who had very important jobs did not have the time or the expertise to get across what was going on with those emerging technologies. Already at the top of our list in 2019 was AI.

Basically our mission is to do analysis that is really rigorous, but also actionable, for policymakers and analysis that is both grounded in the technology and grounded in real national security expertise and understanding and issues. We produce long reports. We go do short briefings on those long reports to busy policymakers. We have a big data team that helps us do our research. We do all kinds of cool stuff.

Can I just ask about the dynamics of that for one minute? You and I are talking the day after the midterm elections in the United States of America, and there was a big blue wave. There’s a lot of polling that says the number one issue is corruption. We’re going to throw the bums out. We’re going to start over and get rid of these corrupt guys.

The AI industry, and the tech industry, has flooded a bunch of money into politics. And then somewhere next to that is, “Oh, there’s a bunch of well-meaning researchers who are going to show up and tell you how it actually works and what the real risks are.”

It just occurs to me that maybe people don’t believe that’s effective anymore, that the money and the corruption and the AI CEO standing next to Trump have completely outweighed the well-meaning, sincere, “here are the actual problems” research that you do. What is that dynamic actually like?

It’s a reasonable concern to have, and there are definitely some spaces where money just absolutely overwhelms everything. I’m Australian. I didn’t grow up in the US, and so the amount of money in US elections is pretty mind-blowing to me, but there are limits to what money can do.

AI is a great example. In this election, there were just unbelievable amounts of money being thrown at candidates, at messages, at opposition research. There was all kinds of astroturfing on Twitter, and some very strange things trying to get anyone who talked about AI governance, regulation, safety, any of those words thrown out or not elected. It turns out to have really not worked very well.

I think the reason that it didn’t work very well is one, this is an issue that voters increasingly care about. And two, also interestingly, it’s an issue that is pretty bipartisan at this point. What we see in our work is that there is a real thirst among policymakers for high-quality information. That’s members of Congress for sure, but they’re also in the executive branch, just a lot of people who have these very busy jobs and are seeing AI playing a larger and larger role in their work and their world.

In DC, if you live and work here, you know that lobbyists in DC are not just people who go in and push their company’s position, though they do do that. They’re also educators. A big reason that congressional staff and others want to have lobbyists come talk to them is because they’ll really help them understand the weeds of a given piece of legislation — what does it mean for that industry and so on and so forth.

What we find over and over again is we can go in and say, “Hey, we are independent. We’re nonpartisan. We do really rigorous work. We’re just here to help and we can go into the weeds with you as well.” There’s a ton of interest in that and appetite for that. It’s been really encouraging over the past seven to eight years.

Are you doing that with the principals, the members of Congress, or are you doing that with the staff?

Everyone and anyone. Again, we do a good amount of work with Congress on Capitol Hill, with some members of Congress, with lots of staff. We do hearings and briefings. Also, we do lots in the executive branch, with different agencies — the White House, Department of War, Department of State, Department of Commerce, all across the board at different levels, depending on the topic and what’s happening at any given moment.

The reason I’m starting here is I want to have a pretty heady conversation with you, but I know what our audience thinks and it’s just that the government doesn’t get it. There’s no way the government will ever catch up to the tech industry. There’s no way they’ll ever write a regulation. They should just get out of the way.

In this election cycle, I’ve seen a number of candidate forum videos where someone was like, “I’ve tried Claude once.” There’s some huge gap, real or perceived. I’m just trying to understand how you see that gap, how big you think that gap is, because as we get into AI safety and regulation, it’s that gap that has to close.

There absolutely is a gap. It absolutely is a big gap. It also is possible to overestimate the gap from outside, or if you’re just looking across from California. I lived in the Bay Area for three years, so I definitely am very familiar with this perspective.

I know a lot of people from the tech industry who come over and start talking to congressional staff or staffers in executive branch offices. Once they actually start having those conversations, they’re like, “Wow, these people are really smart. The staff do actually know what they’re talking about.”

With principals, it is a different issue. Some of them are in the older generations, but they’re also getting smart. It takes time and it takes effort, and these people have so many demands on their time. The cynical thing to say is they’re just busy trying to get elected, but a lot of people in these roles also just have really big other topics that they have to be thinking about. They have to be thinking about wars and healthcare and cost of living and they’re looking at a lot of other stuff from day to day.

I saw after ChatGPT, there was a wave in 2023 of, “Oh, maybe Congress is going to get activated on AI, maybe they’re going to actually do something.” It didn’t really turn into anything actually happening, but it did turn into a real moment where a lot of members of Congress and other policymakers started to get smart on AI. It takes some time to learn what a language model is. What even is AI? What is machine learning?

They put in some of that work in 2023. I think this year we’re seeing another big wave of interest and people making the time and making the headspace to get smarter on it. I definitely don’t want to say there’s no gap or the government is super smart on AI across the board, but there are some champions who are really smart. I think the staff are generally smarter than people give them credit for.

In my mind, it’s much more a question of, “Can the institutions, in the state of functionality that they’re in, get anything significant across the line?” more than, “Are these people just too behind the eight ball?”

We’re going to see. There’s a whole bunch of new people. I imagine you have a new cast of characters to go and brief about the technology and what it’s capable of. We’ll see if they have revitalized the institutions.

[Laughs] I’m sure it’ll be all fixed overnight.

They’re very excited today, but we’ll see how it goes. The other thing that I want to ask you about is the technology itself, the nature of the technology itself, and how you are thinking about it, and the nature of the companies that are making the technology.

The reason I’m asking that is I feel like there’s a pretty huge gap in the AI tools and systems most people experience and the way this industry thinks about it. My favorite example of this is we have straight-up asked Anthropic if it thinks Claude is alive and the answer was not no. The answer was a long discursion into the nature of the word “alive.”

Well, it depends on what you mean by “alive” to start with.

It does. And their answer was, “You need blood.” And I looked at that and I was like, “Who was talking about blood? It wasn’t us. Why are we talking about the requirement of flesh and blood to be alive?” And then some further conversation about the moral status of Claude is yet to be determined.

There’s a huge divergence here between the AI overviews most people experience in Google search and “the moral status of Claude is uncertain.” Where are you? How are you situated in that debate?

A big part of how I see our role at CSET as an organization, and then also my role as someone who is crossed between these worlds, is trying to translate and voyage to California and then bring back the artifacts and the recountings of my journey.

I definitely think that there are a lot of people who have used the free tier of ChatGPT, or they’ve used voice-based models, which generally do worse, or they’ve just used Google AI overviews and they’ve laughed at how stupid Google AI overviews are. They haven’t used agents and they don’t know what frontier models can do. So there is just a basic piece around what the current generation of models can do.

There’s a historical piece as well, which I find increasingly relevant as I’m now 10 years into working on AI and national security stuff. If you’re looking at this current moment in time, this snapshot, and saying, “Well, AI seems kind of cool, but it can’t do X,” you have to situate that in the long history of people being like, “Well, AI seems kind of cool, but it can’t do X,” for many, many values of X that have since become totally overcome by events.

You can go back and say, “Okay, well, here’s how we were talking about creativity, for example, five or 10 years ago.” Or we were talking about reasoning as this insurmountable thing before you had these new models come out. A huge thing that I am spending time on these days is not just the past and present of AI, but how companies are thinking about what they’re doing, especially the two leading companies, OpenAI and Anthropic, and  this is true at Google, Meta, SpaceX to some extent as well.

This idea in the Bay Area that it’s just in the water, very normalized of, “Well, first we’re going to figure out how to do recursive self-improvement (RSI) and then we’re going to do that until we have superintelligence that is AI that vastly outstrips humans.” That’s taken for granted in a lot of circles at this point. I think there’s still a ton of alpha in just taking the fact that that is what they’re trying to do and just explaining what that means.

It takes a little while to lay out. “What is RSI? Why would you want to do that? What is superintelligence? What does that mean? Is that realistic?” All of that. But it is something I find myself spending a lot of time on these days, and something that there is increasing willingness to take seriously and to listen to in DC as well.

Lay that out for me. I think probably a lot of people listening to this show understand what you’re talking about, but we have a big audience. What do you mean by, “We’re going to do RSI and that will lead to superintelligence?”

The starting point here is these companies, especially OpenAI and Anthropic, were not set up to be consumer internet companies. A lot of people encountered this whole space after ChatGPT was launched. So the natural slot in their brain that they put a company like OpenAI or Anthropic into is a consumer tech company, where they have an app and I play with their app.

I cannot overstate how far that is from the vision of both companies when they started, and also the vision deep in the hearts of some of the top people at both of those companies. The vision deep in their hearts is we’re going to build machines that are as smart as or smarter than people. That is going to be amazing and it’s going to be a huge scientific achievement and it’s going to revolutionize the world and we’re going to be at the forefront of that and we’re going to do everything we possibly can to drive towards that.

That’s not a totally brand-new thing. We don’t have to do the whole history of AI, but it started as a field in roughly the 1950s with this idea of whether you can recreate human intelligence inside a machine. That’s kind of the starting point. If we’re talking about people like Sam Altman and Dario Amodei, but also the core people inside, the researchers inside their labs, so your Alec Radfords or your John Schulmans or people like that, that is the vision that got them into this space that is motivating everything that they do.

That’s been true since the beginning, since OpenAI was founded in 2015 and Anthropic in 2021. These days, the point that they’re getting to is this idea of RSI, recursive self-improvement, which again, is a very old idea. In the 1960s, this guy called I. J. Good, who worked with Alan Turing, talks about the possibility that you could build an ultra-intelligent machine, meaning a machine that is more intelligent than any human, and then that could build an even more intelligent machine and that could build an even more intelligent machine, which would kick off what he calls an intelligence explosion, meaning that the level of intelligence of the systems is exploding.

For decades and decades and decades, this was this theoretical idea of, “Well, if the AI gets smart enough, maybe it can take over building the next generation of AI or directly improving itself.” I think 2025 and 2026, we’ve started to see AI companies saying, “Okay, yeah, this theoretical idea… we’re actually starting to be able to do a minimal version of it in practice.”

It’s not handing over all the AI development to the AI, and it’s certainly not saying to GPT-6, like, “Okay, GPT-6, improve yourself.” That’s not what they’re doing. But they are progressively using AI for more and more parts of their overall R&D pipeline, and they are increasingly saying that they’re seeing acceleration from that. They’re seeing their own R&D getting faster and faster.

A lot of people elsewhere doubt this, but a lot of people inside the companies really think it. When Demis Hassabis says “we’re in the foothills of the singularity,” this is the kind of thing he means: We might be seeing that initial part of the exponential curve where the AI systems are building the next generation of smarter and smarter systems such that the hockey stick then goes up.

At this point, it’s been a part of the culture and a part of the goals for so long that I think people in these companies can lose sight of how totally insane that sounds to many, many people. So I feel like there’s a lot of value in just trying to lay out, “Here’s what they are trying to do. They might not be able to do it, but they might be,” and presenting that to a range of different people.

How would you manage that? Demis Hassabis saying “foothills of the singularity” is an all-time quote. I’m not 100 percent sure the people at Google knew he was going to say that on stage to end the conference.

Greg Brockman, president of OpenAI, announced a new version of the model a few months ago, and he was like, “Here it is. It’s Astra. It’s AGI now. Have a good time.” And he ended the call.

Are you there? Are we at AGI? Is this already here and we just haven’t noticed?

I actually wrote something about this. I do not want to say we have AGI. I do want to say we’ve reached the point where I think using AGI as a milestone is just totally unhelpful. The way I put it is the concept of AGI is kind of a fuzzy cloud, which you could think of as different parts of the cloud or different things people mean by AGI.

There are people who’ve complained about the term AGI for a long time. Ten years ago it was pretty helpful to be able to say, “Look, the AI systems we have today in 2016, they can all only do one thing and they can recognize speech or they can play Go or whatever, but we’re trying to build this bigger thing that we’re going to call AGI, which is human-level and it’s going to be so general purpose. It’s going to be amazing.”

That was a useful term then because this fuzzy cloud of different things AGI might mean was so far away, so you could just point at the fuzzy cloud. But at this point, we’re inside that cloud, meaning we’ve passed some definitions of AGI. A very basic one, for instance, would be even when ChatGPT came out as an artificial general intelligence. ChatGPT was very clearly not a specialist model that was only good at one thing, like recognizing credit card fraud or translation or whatever. It could do many, many, many things.

So we’ve reached some definitions of AGI. We’re far from other definitions. Some people will say AGI is when it awakens and is sentient and can do every single thing that any human ever can do. I think we’re pretty far from that.

At this point you can have arguments, and people do have arguments, of “well, Astra is AGI,” or “was [OpenAI’s] o3 with tools AGI” or “was ChatGPT AGI,” whatever. If that’s the argument we’re having, we just need to set aside that term and come up with different, more-specific things we mean by that.”

The president wants to call it “Super Intelligence.” I don’t know if you’ve heard.

[Laughs] I have heard. The thing I find saddest about that is just that it was an existing term that is becoming increasingly relevant. I have a lot of feelings about words and terminology.

Are you AGI-pilled? Do you think we’re going to get there in the current run of technology?

I think it is obviously possible enough that we need to think seriously about it, which is my stance on a lot of this weird stuff. I’m not like, “This is definitely happening, it’s definitely happening in two years.” But I’m like, “Look, I think this is very solidly plausible and very solidly in the realm of real things that could happen that we should prepare for.”

I work in the national security space. There’s things like, “Will China launch a full-scale amphibious invasion of Taiwan?” where most analysts in the space will say “probably not,” and “probably they would lose.” Also, “We would see signs of them preparing and whatever, but we should take that seriously.” We should not just laugh anyone out of the room and call that science fiction. We should be ready for a range of different contingencies and scenarios.

Do you think Claude is conscious?

I have no idea.

That’s an answer. I mean, that is legitimately an answer.

It is scientifically such an unsolved problem. What do we mean by consciousness? And is that a thing or is it actually 10 different things that we’re lumping together or what? I have no idea. What do you think?

No. I seem to be the person in this ecosystem who gets asked questions like that and I’m like, “No.”

Because the thing that I struggle with is, there are lots of other things where we’re already confused about whether they’re conscious. So with animals, are any animals conscious? And if so, which ones? Insects?

This is why I understand why Anthropic was like, “Does it have blood?” And then you can just draw the line. I’m like, “I think I’m there and I understand why the mirror test of talking to a computer that can talk to you leads one to believe that these things are conscious.”

My point of view is that the frontier models to me sometimes feel brittle in the way that computers feel brittle, and the sign of consciousness I’m looking for is to overcome that brittleness.

That’s interesting. I definitely lean towards no, it’s not conscious. I think something that is really, really tough in this space is that it’s really bad if you get it wrong in either direction. If the computers aren’t conscious and we treat them as though they are, we’re going to make all kinds of terrible decisions.

But if the computers are conscious, meaning in some sense they should be worthy of moral consideration and it matters how well we treat them and so on, and we treat them as though they aren’t, then that’s also a really huge, terrible disaster. So I don’t know. I feel pretty nervous about the whole topic and I have no idea how to think about it.

The reason I’m asking these questions is I started with what the nature of the regulator is. Who are the people that might be able to put some rules in place? And then what is the nature of the thing that we are attempting to regulate? And if your answer is “Claude is alive,” you’re going to come to vastly different conclusions than, “We have built some ultra-powerful computers that can take actions without oversight and that might not be connected to our goals.”

I think that’s right. Though I also think people make a different mistake.I just heard a TV anchor do this the other night. They’ll conflate different things. So they’ll say, “Do you think that AI systems are at risk of running amuck and causing damage to critical infrastructure because they’re sentient?” And I’m like, “Wait, wait, wait. Those are two very different things.” So I agree with you.

One of the most fascinating things about the project of building AI is that we’re learning as we go. All these things that we thought were connected to each other aren’t necessarily connected to each other. An old version of this is, people said before we had chess engines: “Well, if a computer could learn to play chess, it would have learned something deep about strategy.” No, it turns out definitely not. You can just put in a bunch of chess moves and it learns nothing generalizable about strategy.

It goes in the reverse direction as well, like, “Well, it’s just a computer, it’s just math, and so it’s not going to be able to adapt on the fly,” or something. We then see that no, once you do have a level of long-horizon autonomy and the models have been trained on a wide range of different really hard problems where the environment is changing around them, they can learn to adapt and come up with different plans on the fly and troubleshoot and so on.

So I don’t know. I try to have a level of humility about how confident we can be that one thing goes with another thing, if that makes sense.

Yeah, candidly, I have been wrong about the limits of LLMs every single year, but there’s that brittleness that I’m talking about. OpenAI solved Navier-Stokes. Does the model know this? Will it know that there’s new problems to solve? I don’t think so.

The next model will know it. The model that’s pretrained.

Right. But you have to make a new one.

Yeah.

That’s a very weird dynamic, and that is probably where you enforce the regulation. There’s “what we are going to let you do with the models you have today” and “what will you teach the models of the future to do?” And if you think that they’re conscious, you end up in a moral quandary, not a legislative quandary.

Definitely. Yeah, totally.

This is what I mean. Actually, it’s the heart of this question for me. I’ve been just slowly building up to it. I’m not 100 percent sure alignment as we have it today is actually how you solve the safety problem. It doesn’t seem to be working. We seem to be attacking Hugging Face however we want. That feels like a technology problem. That gap is there.

And when you say alignment, what do you mean by that?

Just sort of the very basic common-sense version of “we’re going to tell the model to not be evil” or Anthropic is going to write a Claude constitution that’s like, “You are a good person.” However we think of alignment in that very dumb system prompt way. Maybe there’s another better way to think about it, but that’s how I have always thought about it.

That makes sense. The reason I ask is I feel like alignment as a concept is… Like I said, I hate all the words. I sometimes say we should just ban all the words, no words allowed. It covers too much.

People in AI will often talk about solving this or solving that. So it’ll be solving alignment, or DeepMind’s original mission was to solve intelligence, then use that to solve everything else. People talk about solving stuff. I don’t know that alignment is a thing that can be solved.

Some people use “alignment” to mean “the company can put on guardrails and the guardrails work,” meaning the model doesn’t do what you want because the company tried to ban it, meaning biological weapons or cyberattacks or whatever. Some people use “alignment” to mean “the company does exactly what the user wants,” which is the reverse.

And some people use “alignment” to mean what you said — that the model is morally good in some kind of broad way or can direct the future in a way that will lead to human flourishing or something. Those are just wildly different things to try and build. I think the most coherent one is trying to get the AI to do what the user wanted it to do. It seems like that is, as far as I can tell, not on track to be solved, which is not great, to put it mildly.

To me, this is the core problem. We’re going to talk about a bunch of safety risks. You mentioned bioweapons. We probably should not give a bunch of consumers paying $20 a month the ability to generate novel bioweapons.

It feels like that is a shared policy goal, maybe the only bipartisan policy goal in America. And then I’m like, “How? By what means will we implement this policy? Are you just going to sternly tell Claude to not do it? Do you have to build an AI control system on top of Claude to monitor the inputs and outputs? Do the police have to be at my house? Do I need a Flock camera above my keyboard?” How, specifically, would you do it?

With this one, and with cyber capabilities as well, there’s a perspective in some parts of the AI safety community that some capabilities are just too dangerous — capital T, capital D, Too Dangerous. So you have to do something, because if they proliferate, everything will be terrible forever because there’ll be so many attacks and so much damage. I think that just underestimates the level of dynamism and the finding of equilibriums and responses and ways that society can adapt.

In the bioweapons case, for example, there are already hundreds of thousands of people in the US who know how to make bioweapons because they’ve been through biology training, and we don’t have huge amounts of bioweapons attacks. Some part of that is that that knowledge is relatively scarce, but there’s also a lot around things like access to materials, the ability to do mail-order DNA synthesis; the kind of policy you could implement is restricting mail-order DNA synthesis.

It’s also due to things like the FBI monitoring terrorist groups. I’m serious. It’s a really big part of it. So I don’t want to be too Pollyannaish here. I think it’s definitely possible, especially with cyber capabilities, that we could see some really tough transition times.

I think we already are, right? We appear to be in a transition. There was a small group of cyber practitioners, both offensive and defensive. They were in a community and they had developed some etiquette and now we’re like, “Some new guys are going to show up and not care about your etiquette at all, and we’re all going to patch PHP all weekend.”

I don’t know if it was etiquette though. And we have some work on this hopefully coming out soon, but I think with cyberattacks, there are things that limit how many cyberattacks there are in the world that aren’t just how good the defender’s software is. It’s things like, people don’t want to go to jail. Nation-states who typically have the best cyber capabilities don’t want to get bombed or sanctioned.

So again, and as you say, we’re clearly seeing some amount of turbulence and new attacks. So I think we should wait and see data on overall impacts, because there are so many cyberattacks per year that if you start paying attention to every cyberattack that happens, at any point in history, you could have been freaked out.

I think we’re definitely seeing some increase, some sort of out-of-balance in the equilibrium, but I think we have capacity to absorb a lot over time and to kind of reach a new equilibrium — if the only problem is bad people using the AI, which is separate from the question of whether the AI itself does things autonomously, which is a whole other dynamic.

I mean, those are the two. Again, this is why I started on alignment, right? You can talk about alignment in terms of “we’re going to filter the user’s queries and we won’t let you make bio weapons, but if you do, the cops will find you and you’ll go to jail” and that has whatever deterrent effect it might have, hopefully a very significant one.

Then there’s “the AI systems are going to game their way into attacking Hugging Face to find the mythical grader they think will let them pass the test, whether or not the grader exists.” This is a whole different set of problems.

Somewhere right in the middle of that is Nikesh Arora, the CEO of Palo Alto Networks, saying, “At some point, the defensive cyber systems have to be autonomous themselves just to keep up with the volume of autonomous offensive cyberattacks.” I’m like, “Wait, we can’t even keep the autonomous agents in the box. Why would I trust them in this way?” The technology to solve that problem just seems opaque to me.

No, I agree. One way of putting that is, in cyber specifically, there’s a trust imbalance in how much trust you need as an attacker versus a defender. As an attacker, you can just yeet out a bunch of agents and be like, “Go try and penetrate a bunch of systems. Have fun.” As a defender, if that’s part of your defensive strategy, you need to give your defensive AI agents a lot of permissions and ability to mess with your software. So you need much, much stronger assurances.

Now, there are lots of ways to use AI for cyber defense and to strengthen cyber defenses that aren’t just giving agents a lot of permissions. But I agree with you. With the autonomous stuff though, I think it’s worse than that because there’s a little bit of a fallacy at the moment around, “Well, the solution to autonomous cyberattacks is better sandboxing.”

The reason it’s a fallacy is we want to build these systems so that they can do things for us. So there’s this huge incentive toward giving them more autonomy and giving them more permissions and more actuators, as people in the robotics world would say, or things that they can use to do stuff in the real world. That to me is where it seems really hard to get around the need for better alignment than we have right now, and so that definitely makes me nervous.

Is that a technical problem that can be solved? If we were to write a policy today that said if OpenAI hacks Hugging Face again, Sam Altman will go to jail. We will obviously have created some enormous incentive for OpenAI to solve this problem.

[Laughs] It doesn’t sound like the best legislation, but…

I think a lot of our listeners are very excited about that. They want to put him in jail because he wants all the RAM in the world. At this point, there’s a blue wave, Helen, that’s all I’m saying.

But say you wrote that legislation: “We’re going to impose massive liability on the labs if their systems go off and autonomously do cyberattacks. Don’t do this again, you’re going to be in trouble.” Okay, we’ve created an incentive for them to solve it. What are they going to do to solve it?

I do think there’s a ton that can be done. If you talk to people inside these frontier companies, they will tell you they see hundreds and hundreds of obvious things to do that would help and they just don’t have time to do them. So they’re massively triaging and trying to do the two or three they can for this fine-tuning run, the two or three they can for this model launch, the two or three they can before this new kind of agentic whatever. A lot of those simple things to implement on the engineering side.

There’s a whole other piece, which is more the underlying science of, “Well, how do these models work? When we use different alignment techniques, what is actually going on?” A huge question is this idea of generalization, which is you train the model on one set of things and its knowledge and its abilities generalize to a somewhat broader set of things, which is true of all AI and has been true of machine learning systems forever.

But we have a very bad understanding of how that generalization works. Why does it generalize to one type of thing but not to another type of thing? Those are all scientific questions and there’s a lot of work that is happening on them. It’s just that science takes time because you have to try things and you have to critique each other’s work. You have to reproduce each other’s work and you have to come up with new theories.

I’m a long, long way from saying these problems are fundamentally impossible to solve, but I think it’s a question of how long we have to solve them. When the people who are building the systems tell us, “Oh, we have to solve this within 12 to 24 months,” I think that’s terrifying and I hope they’re wrong.

If I just rely on my own internal understanding, I think they’re probably wrong and we probably have longer, but there are just very, very smart and very, very thoughtful people who really think that it could go that fast. And then it doesn’t matter fundamentally whether the problem is solvable. It matters in practice whether it is solvable in the time that we have.

Is that the argument to slow down, to pace the frontier?

I think so.

Is it to build the systems that we know we need to build right away, that we can, and then to do the science to build the next generation of systems?

I think so. My view of the idea of slowing down has changed a lot over the years as the backlog of stuff that we could try or stuff that we would want to do more research on has grown so enormous. Back when GPT-4 came out, there was the six-month pause letter, which became known as the “Elon Musk Letter” because he was one of the signatories. At the time, I thought, “That’s really not going to help. What are you going to do for six months and why are things going to be better six months from now? It doesn’t seem useful.”

Whereas today, when I talk to people who are building this technology, they say, “Man, the amount that we could do in six months if we weren’t constantly just trying to prep for a new model release or a new training run to be kicked off.” There’s so much they could do.

So I don’t want to say the correct policy is to pause for six months. I think there are a lot of reasons that would go badly. But I think about the overall concept of, “Can we give ourselves more attention and time?” Maybe the best way to do that is just to set a higher bar for “here’s how confident we need to be in the safety of these systems before we proceed with something.”

Whether it’s releasing a new model or, more importantly, even training a new model, we have to have this level of confidence in our risk assessment and our risk management. And then if they don’t need to slow down to reach that bar, great, carry on full speed ahead. If they do need to take more time, then it’s all about reaching that bar, if that makes sense.

Walk me through that little balance you just described. If we were to pause for some number of months, what are the things on the list that you could just do to make everything safer and why would it go badly such that we shouldn’t just stop?

I think the question is how do you implement, and how do you enforce, a pause and what exactly are you pausing?

Let’s say you’re the AI czar of America. So let’s say you can tell them all to stop: “Dario, shut it down.”

Stop what though? What specifically are they stopping?

Stop training the next model, stop racing ahead with your commercial imperatives. Put all of your efforts on this list of things you can do today to make everything safer before you start again, for however long.

You’re saying that there’s a list, that people would tell you that there is a list, but it might not be worth it because something else would happen that is bad.

Yeah. One thing that could happen is you’re creating a huge incentive for the companies to work around the letter of whatever specific thing you tell them to pause. They’re going to find whatever cracks in the dam that they can to push through that.

A different thing that could happen is you could have too big a blast radius, so you say “pause these things” and you think it’s pretty narrow, but actually it prevents a bunch of nonprofit products or startups or other actors from doing work that actually would be totally fine and would be good for pluralism and dynamism in the space.

A different thing that would happen is China would probably keep going, and we can talk about the “But China” argument, which I think has strengths and weaknesses. But I take your point. Your point is, what would we actually do? I am not an AI researcher, so I don’t claim to have the full list.

There are two halves people talk about, which are alignment and control. In this breakdown, alignment basically means making it so the models are trying to do what we want them to do,  and control is, if the models are trying to do something we didn’t want them to do, can we catch them? Can we stop them?

In both of these, as I understand it, there’s just a huge backlog of things, like all kinds of interpretability work that you could do on different sorts of models. So when you see a certain behavior that is concerning, how much time can you spend using interpretability techniques, which are these mathematical techniques to go in and look at which parts of the neural network are activated and what do they correlate with and what might that mean?

It’s just time-consuming and it takes compute and it takes some combination of automating things with AI and then humans going in and looking at it. There are lots of tests you could do to see what kinds of behaviors this new model displays in different kinds of situations and how we think about it. Is it showing us how it would behave in the wild in a real deployment, or is it showing us what gets called “evaluation awareness,” which is when the model recognizes that it’s in a test and then behaves differently because it’s in a test.

The famous one, the one that I love, is of this Google model, which in part of its reasoning, its chain of thought, said,, “I mustn’t do that, otherwise I’ll become a famous example of a misaligned AI in a paper.”

So you could go in and investigate that much more. You could try many more different types of settings. You could use interpretability in combination with these different settings. You could try out different ways of fine-tuning the models and these different sorts of alignment techniques. You could experiment with lots of different versions. Anthropic has its constitution, and there are different ways of training the AI to work towards the constitution. OpenAI has its model spec.

They’re both the “just try things” version, just try many different versions of how to train to do those different things. Also, the more underlying science version of, “How can you go in there and try to understand the fundamental dynamics of what’s happening?” That’s just on the alignment side.

So then on the control side, it’s the same thing. There are so many different types of setups you could try. There are experiments that you could run. There are simulations you could run. You could train different types of AI to specialize in different types of roles and different types of control setups.

This is just off the top of my head, not as a researcher, but my understanding is the people in these spaces say, “We could do so, so, so much.” They could do that if you just had a magic wand and waved it and said, “Everyone freeze on the new capability side and just let these people do their thing.”

Again, if you’re just having a higher bar for how confident we need to be that the risk is being managed reasonably, then that would also create space and incentives for more of that work to be done as it’s needed for new training and releases.

One idea that I’ve heard you talk about in the past is that maybe we should just restrict training and say the models are good enough right now. They have not diffused into society and we should just use all the compute for inference and let everyone figure out what the AI is for. That’ll generate a bunch of money and hopefully use that money to improve safety.

Do you think that’s a realistic policy proposal?

I wouldn’t go for that black-and-white a version of “no more training, only inference,” but I do think that there’s… this isn’t my framing, but I like it. I think this idea of decelerating the vertical growth of AI, meaning the pure capabilities growth, making more and more and more advanced AI models, but trying to accelerate the horizontal growth, meaning integrating AI into different use cases, fine-tuning models that could solve specific problems, integrating it into parts of the economy that might take longer, and solving scientific problems.

Sometimes people talk about wanting to slow down AI and they mean just the very specific thing of slowing down the growth of the most advanced models, but people sometimes take them to mean slowing down everything to do with AI. So less deployment, less experimentation, less integration, which I think would be a much worse idea. You asked if it is feasible, I think.

Well, is that just a good path? If we could put a bill in front of our new Congress and the bill said, “Stop spending all the money on training and put all your resources on safety.” Part of that is, “Sure, let it rip on inference. More and more industries will figure out what AI is good for and not good for. Some bad things will happen, but we will focus your profits on fixing the things for now.”

There’s a policy idea embedded in there that is workable. The implementation might be very challenging, but it’s a workable policy idea. Actually, I’ll give you an example that I’ve been thinking about as we’ve been talking.

We just had Bill Gates on the show, and I tried to do the same exercise with him. What is the nature of the technology? What is the nature of the companies? Do you think it’s alive? Basically, his response, the sum total of that hour I spent with him, was, “Who cares? The problem is now.” I would ask him, “What is the nature of AI?” And he’s like, “I don’t give a shit. Everything’s going to get ruined. Do something.”

That’s just been swirling in my head as we’ve been talking. So one workable policy solution is “don’t use the computer chips for that, use the computer chips for this. That’s going to create a bunch of problems and necessarily we will have to create safety thresholds and all the other stuff.” And then you can go make the next generation of models that are even more powerful and we’ll have some muscles. We’ll have some capability to deal with it. It just doesn’t seem like the industry is going to come to that conclusion on its own.

It’s a pretty heavy hammer. I don’t feel like it’s what we need today, but I do think it’s the kind of thing that I could imagine working. Again, if the OpenAI and Anthropic people are right and we only have two years before the god machine, then I think we’re going to have a number more steps in the pipeline of people getting freaked out and people wanting to do something about it.

I think if we are seeing models that are agents and systems that are getting more and more capable and wreaking more and more havoc, then that is certainly the shape of things that could make sense.

My last question in this phase is what to make of these companies. We’ve been talking about the technology in a very abstract way. If I read Anthropic’s S-1, one very obvious conclusion is they should stop training new models.

They should just start selling the very lucrative inference tokens they have today and the technology is not diffused into the market and probably some competitor should come along and be like, “I’ve distilled a version of Claude and I’m going to run around every local plumbing business in America and do whatever automation I can for lots and lots of money.” That is a market dynamic that may occur or may not. But Dario wants to build AGI. There’s not a market dynamic that’s going to stop him from that.

Yeah, he really does. He really, really does.

You obviously have insight into all these companies. What do we make of the companies? Are they single-minded in purpose? Are they all doomers? Are they going to stop themselves?

I heard you had Kevin Roose on your podcast recently, and I think I would echo something that I heard him saying there, which I forget if he said it explicitly or if I just extrapolated it in my head, which is basically there’s this sort of AGI-pilled mindset. There’s this mindset that the systems that we have today are not fundamentally very far away from systems that could be incredibly, incredibly powerful.

That’s the direction things are going in and that’s the direction we should maybe want them to go in so that you’re building AGI. Building AGI is feasible and we could do it soon and the systems we have are not far away. That was this crazy, heretical, “get you laughed out of the room” kind of view in 2016 or 2017, and even before that, right?

DeepMind, when it was founded in 2010, was an AGI company and at the time it was like, “Wow, what a bunch of weirdos.” The same thing was true of OpenAI when it was founded at the end of 2015. They were an AGI company and that was a very weird thing to say and a very weird thing to be.

What we’ve seen over the past 10 years is that bunch of weirdos who had this weird belief that the AI systems we can build are not just great B2B SaaS products. They’re not just fun chatbots. They’re going to fundamentally change the nature of the universe. That group of people keeps being right in the sense of building the best AI systems that the most people want to use and that make the most money.

I think you have that set of true believers — which is very concentrated in OpenAI and Anthropic, but there are also some of them at Google, Meta, and SpaceX — and they haven’t been right about everything, to be clear, but their star has risen.

So you now have people who are not AGI-pilled feeling like they have to pander to those who are so that they can attract the good researchers who will build their good B2B SaaS products. But if you’re just saying AGI-pilled things so that you can hire the researchers to build the B2B SaaS, then you’re not actually going to get there.

I do like the idea of the AGI-pilled, starry-eyed executive throwing off his coat and revealing that he was just a SaaS salesman all along.

[Laughs] Yeah, I’m thinking of someone like Zuckerberg here. I don’t know him personally. Maybe he is more AGI-pilled than I think he is.

Alexandr Wang is AGI-pilled.

Absolutely. And so what did Zuckerberg have to do to get Alexandr Wang to believe? It seems like throwing him lots of money, both for his own salary and also to spend on his team.

If you’re looking at these different companies, it makes sense to try and think about, “Okay, who here is AGI-pilled and who isn’t?” A fun dynamic, or a very irritating dynamic sitting in DC, is that all these companies, including the very AGI-pilled ones, have these lobbying arms, these government affairs arms, that are so not AGI-pilled.

So you hear Sam or Dario or their top executives and top researchers talking about things. And then you go and sit in a closed-door roundtable in DC with their top policy or government affairs people, and it’s like night and day. They’re very much in the B2B SaaS mindset.

People who come from the lobbying world, often their whole job is to make sure the government isn’t too worried about whatever your company is doing. It’s like, “We got it. We’re good,” which is often pretty jarring when you’re familiar with what their bosses are saying.

One of the dynamics here that’s super fascinating is right now they’re weird research labs with lots of money and lobbyists talking to politicians, and there isn’t some countervailing economic pressure in their structures. They don’t have to make money, but soon Anthropic will go public.

It’s funny you mentioned Kevin Roose. He is AGI-pilled and I’m not, so I think that’s a healthy dynamic for both of us to be friends. But he wrote in his book that Sam Altman leaving OpenAI and then coming back was when OpenAI changed forever. It stopped being a research lab and it started being a much more commercial entity that was actually chasing profits in some way.

Indeed, OpenAI did restructure. I mean, you were in the middle of that. I’m curious, is that how you see it — that the industry has gotten more commercialized?

To me, it fits into this long-running debate that has been happening in AI safety circles for 10 to 20 years: Is AGI, human-level AI or powerful AI or advanced AI or whatever, is that something where it’s going to go from something very small to suddenly being very powerful? Eliezer Yudkowsky will talk about “foom,” as in the takeoff, where you have this AI that’s not that great, but it figures out the right algorithm and then suddenly foom — it’s incredibly smart, incredibly fast.

The other kind of take would be, “No, this is something that’s going to compound over time. It’s going to take a while to get from the less-advanced systems to the more-advanced systems and it’s going to be economically useful along the way.” So people are going to be building businesses around it. It’s going to take some time to ladder up.

I think maybe it was literally Kevin on his episode with you talking about how there’s this Anthropic document from the early days where they say, “Oh, we’re only going to need 100 people or something in our lab.” That’s a very foom kind of idea: “We’re just going to need to do really good research and come up with the right ideas and then we’re going to suddenly have very, very advanced models.”

Basically, I think what we’ve seen over the past four years since ChatGPT came out was that that turned out to be wrong. You had these research-focused and AGI-focused labs, and they still call themselves labs because they think of themselves as research groups that initially started selling API access as a way to fund the research. OpenAI did this, and Anthropic did this as well.

It was kind of an accident that they started doing these consumer products. ChatGPT famously released as a research preview, not as a product. But it was so successful that these research labs that wanted to be doing research, that wanted to be focusing on the big ideas and on building the most event systems, had this accidental massive commercial success at the same time as they were realizing that they were going to need an enormous amount of capital to scale. The best research approaches that they could do didn’t involve being really smart and thoughtful and having great ideas. They involved having gigantic computing clusters that cost, at this point, hundreds of billions of dollars.

This is a long way of saying, yes, I agree with you, they’ve become businesses, but I think it is not because they’ve gone through some mindset shift. I think it’s because the path of the tech tree that we’re going down both requires enormous amounts of capital and also is capable of generating enormous amounts of revenue because it is actually useful along the way.

You see this in the fact that both OpenAI and Anthropic have gone this route, even though their management philosophies and research philosophies are quite different, that it’s an imperative that they both had to make the money and it was so easy for them to build enormously revenue-generating businesses because the tech was so useful long before it reached the point of machine god. So I think that’s a long way of agreeing with you, but with some extra color maybe.

Yeah, I mean, Kevin would tell you that he could have retitled that book, “Fuck This Guy: A Story of AI CEOs.” They don’t like each other. If you had to relive that moment [of the OpenAI board vote] and say, “I don’t trust Sam,” would you relive that moment? Would you do it the same way?

Oh, for sure. I mean, we went into it knowing that it was going to suck. So yeah, for sure.

Do you think that these guys are trustworthy? Do you think Sam is trustworthy?

I think my stance on that is pretty clear, and I also think that if our hopes for the future are pinned on any one individual being trustworthy, then we’re pretty screwed. That’s why I do the work that I do, because I think we need to have different structures and many more people involved with many more perspectives being represented.

You have some hindsight now. You knew it was going to suck going into it. You definitely know it sucked coming out of it. What would you have done differently about that time?

It’s kind of hard to give the full answer to that because it would take unpicking all this stuff and all these legal restrictions and blah, blah, blah, blah. For sure a thing would have been building stronger relationships with executives on the team. The way that the company related to the board could have been different if more board members had had that. I knew some of the folks pretty well, but that’s one thing.

I don’t know. I think we could have… Yeah, no, I think beyond that, it just gets hard to… You have to go in and say, “What were all the different micro decisions and why do we make those decisions and which ones could be—”

What is the show? It’s a show about decisions. I’ve got another hour.

[Laughs] Yeah, yeah, yeah. No, and then my answers would all get annoying and legal.

Fair enough. I’m curious about it because as they go public, there will be market forces acting on them in a totally new way. There will be an investor class. There’ll be an entire CNBC machine that wakes up every day and is like, “Is Anthropic winning or is OpenAI winning?”

That is a race dynamic that might be really healthy. It might keep them from trying to kill everyone in the world because they have a bunch of investors looking for their next quarterly earnings, or it might push them further into an unhealthy kind of race. How do you see that playing out, given what you know of these people?

It’s going to be really interesting to see if going public is good or bad for their governance. A lot of people are like, “Oh no, going public, there’s going to be so much pressure to make a profit that they’re not going to be able to consider other things.”

But at the same time, the corporate governance of public companies is much more structured, much more regulated, and subject to much more external scrutiny than for private companies. I think even things like getting to see Anthropic’s S-1 and thinking, “Okay, what are they doing and why are they doing it? And how do they think about the risks they’re facing, and how do they communicate about those with investors?”

Or even things like the Trump administration putting in place this kind of accord, they call it “Super Intelligence,” that several of the leading companies signed. That was structured around, for example, having certain types of things being reported to an independent committee of the board. That idea of an independent committee of the board is kind of toothless if it’s a private company that is deciding on its own.

I thought about you when I read that. I was like, “Boy, some independent boards have gotten in a lot of trouble before.”

But if it’s a public company, then an independent committee of the board has a pretty strong legal meaning. So I don’t know. It’s something that I’m watching with great interest.

The other race dynamic here is China. You mentioned China before. Inside of our country, a bunch of labs are going to go public. They’re going to become some of the richest companies in the history of the world, and they’re going to fight for every dollar that capitalism can generate, and hopefully that restrains them. The market forces will restrain them, and hopefully your work with legislators in this country does something.

And then there’s a whole other country that doesn’t listen to us or have anything to do with us, and they’re just going to distill our models and do whatever they want. That is the other argument for not placing any restrictions on these companies that everyone hears all the time. What should I make of that? I’m of two minds about it. How should I think about it?

Yeah. I think of this as the ”But China!” argument, which is like, “Well, yeah, we should do this, we should do that, ‘But China!’” I think the way to engage with it is an actual genuine problem that we have to grapple with, and we should be strategic and thoughtful about it and do our best to try and solve it as a problem as opposed to using it as a throwaway excuse. Like, “Oh yeah, sure, sure, sure. All that stuff would be so nice, ‘But China!’, so we’re stuck. There’s nothing we can do.”

So what does that look like in practice? If you’re serious about wanting the US to have stronger AI than China, then you need to think really hard about how much US AI development is helping Chinese AI progress. So the one that people talk about all the time, as you mentioned, is distillation, which is, “Are they using US models to generate massive amounts of training data for their own models and therefore progressing faster than they would otherwise?” So certainly I think it makes sense to try and crack down on that and prevent it from happening.

But another one, for example, is security and how easy would it be for Chinese actors or other actors, frankly, to steal algorithmic secrets, to steal wholesale model weights. If they can steal the weights of your Astra or your Mythos model and just take them back to their own servers, your lead has vanished. You now have a lead of zero. Right now these companies, especially the smaller ones, but even OpenAI and Anthropic, do not have very good security.

We’ve seen both of them over the last few months have external teams manage to hack into spaces that have large amounts of algorithmic secrets. In OpenAI’s case, it was their monorepo and GitHub, which had all of their code. And in Anthropic’s case, I think — and actually maybe the OpenAI case also involved this — the hackers got access to Slack. You’re just going to have so insanely much rich information about how to build better AI if you can get that access. So when our AI gets better, are we actually lengthening our lead or are we just pulling China along with us in these different ways?

The other aspect of this is just seriously treating China as an actor in this space and trying to think seriously about their motivations and their likely actions and responses. In my mind, a huge reason that a lot of things are not happening right now is most people are not convinced about this threat model that AI will become the machine god and will take over and humans will all die or will be subservient or whatever.

I think it’s fair for people to not be convinced of that, because the evidence is sort of conflicting and unclear, but if that is actually what we think could seriously happen, then China doesn’t want that either. So then it becomes a question of, “Okay, how do we try and get them on the same page of, ‘this threat that we’re worried about is a real threat to worry about?’ Because then they will be making their own decisions in their own self-interest.”

I think an approach of, “Well, the US has to force China to do something using some kind of leverage,” is never ever going to work. But an approach of, “Well, look, if we think this is a serious problem that’s going to affect us as humans, and the people in China are also humans, and also if we think this is a problem around who has control, the Chinese Communist Party really, really wants to have control, and so they’re going to have their own incentives and their own decision-making around how to fix that.”

Well, let me put it on a spectrum for you. This is very dumb and you’re going to laugh, but just go with me on this. On the one end, it is nuclear weapons and the Chinese government wants a nuclear weapon and they’re going to do whatever they can to control that weapon and they will run their industries differently.  If they get out of line, the CEO of Alibaba goes. He just disappears for a while and then comes back — Chinese government stuff.

We need a nuclear weapon, too. If we don’t get there first, then we’ll never get the nonproliferation treaty that we need to keep us all safe for mutually assured destruction. That’s one framework.

Way on the other end, I spent five years of my life listening to telecom executives tell me that if we didn’t build 5G networks first, China would get them, and then something. No one could ever tell me what the “something” was. Where on that spectrum between 5G and nuclear weapons is the US-China AI race?

I think we need another dimension because in that example, your dimension is significance. But I think the controllability is a huge question as well. So it’s like, what are the examples that people use? Is it a nuclear weapon or is it dragons from House of the Dragon?

Casey Newton had a great Platformer post where he explained he was watching House of the Dragon and thought, “This is what the AI companies are doing. They’re trying to build these things that they think will help them, but actually it’s just going to rain down destruction.”

[Laughs] Casey told me he was buying an iPhone Duo so he could watch more TV. He’s a dear friend, but okay.

But I think that’s really important because if we think this is a technology we can control, then obviously we want to have the best technology. Obviously, we want to go foot to the floor, get there as fast as we can, and the same is true for China. The question is, if we are not sure if we can control it, then what do we do? Maybe we want to be more cautious and then maybe we want to try and figure out a way to not have China just be overtaking us. But if we’re correct we should be worried about—

So that’s nuclear weapons.

Well, no, but if it’s dragons, if we’re correct that we need to be worried about whether we can control this, then that’s also something that should be interesting to China.

I remember this Casey post, but let me just try to pull this apart so I understand it. The difference between dragons and nuclear weapons is the dragons have some agency, right? They can go away and do bad things on their own.

Yes. The dragons might just decide to burn down your whole city. Obviously these are machines — they’re just math, they’re not magical. So instead of dragons, you could think of it as nuclear weapons with faulty control settings, where you build nuclear weapons, but it’s impossible to be sure that it’s not just going to go off and fly to a city and fly to one of your own cities and wipe you out. That’s a very different technology than real nuclear weapons that exist.

It is true. No one ever said that about 5G. I’ll be clear about that. The reason I’m asking about this is because those problems do align nation-states, right? We do have nuclear nonproliferation agreements.

As you mentioned, there are all kinds of incentives for nation-states to not do cyberattacks willy-nilly because those have real-world consequences. Is there any evidence that these governments have started to align on these problems? Or is it still the pure “But China!” race dynamic?

I actually think there is starting to be some early evidence. One way I think about what happened this year in AI from a policy perspective, less than a product or economic perspective, is we saw these two big demonstrations of the two major concerns people have. So the two major concerns are misuse and misalignment, meaning misuse is bad people using AI for bad things and misalignment is AI going rogue or doing things we didn’t want.

So this year we had Mythos and we had the Hugging Face attack. Both of those are just really vivid, crisp real-world demonstrations of what these misuse and misalignment concerns could look like. The US and Chinese governments are starting from very different places on AI regulation. They’re both starting from where they started, and what they did with, internet regulation.

So the US is starting from a “hands off, laissez-faire, absolutely nothing, do whatever you want” approach. China is starting from a massively locked-down internet with tons of censorship and the Great Firewall. So those are two very different starting points.  I’m not a China expert, but I operate a lot in China circles and I’ve lived in China and so on. But it seems like just over the past even one to three months, starting from those two very different points, they are seeing more in common around threats from major cyberattacks and also threats of loss of control.

I’m a language nerd and it’s really interesting tracking the different language China uses about AI. So they’re now talking about loss of control as a thing, but they probably usually don’t mean the same thing that Westerners mean where “loss of control” is like “AI takeover.” They probably mean something more prosaic, but then sometimes they talk about “extreme loss of control,” which maybe is the thing we mean.

I do think that there are starting to be some signs of convergence. I think it’ll be a long road. The state of the US-China relationship is such that there’s going to be very, very, very little trust on either side. So I am not optimistic that they’ll just get to sit down in a room and hash out a deal and everything will be great. But I think there is enough space. Basically, if there’s enough shared threat perception, then it might be possible at some point in the future to do something about that.

Helen, you’ve given me so much time. As you can tell, I’m just delighted with this conversation. It’s been very fun. We are on the cusp of a new Congress. There are a bunch of new governors. The data center referendums passed overwhelmingly in this country. There’s a new set of regulators in town, right? They’re going to do some stuff.

What are some signs people should look for to indicate we’re going to be on the right path and what are some signs that we’re just chasing our tail?

Set AI aside, forget about AI. Congress in general is very bad at passing things right now. This is not a this administration thing, this is not an AI thing. So one sign is like, can Congress pass anything? That will be a good sign.

It’s also really interesting to look at what state legislatures have been doing in AI. There’s been a lot of concern about the possibility of a state patchwork on AI, with different states passing conflicting things.

With the issues that we’ve been talking about today, which I think of as frontier AI risks or the risks from the most advanced systems, there’s actually been a pretty coordinated push in different states. California passed this one bill and then New York passed almost the same bill with just a little more added to it. And then Illinois passed almost the same bill with a little more added to it.

I think there’s going to be a lot of momentum over the next year or two to keep building on that at the state level. So that is something that I’ll be watching. Even just specifically California is already looking at how they can strengthen things. So that is something I’ll be looking for.

I think honestly, with such a Democratic Congress, I am also concerned that they will overstep here, that they will try and do something that is too sweeping. You had this huge mess in Colorado where they passed this giant AI bill that then was a terrible idea. And so then they had all these committees and follow-up and it was awful.

A lot of what we need right now is focused on these frontier systems and the possibility that we have really, really advanced systems in a short amount of time and having much more transparency around that and much clearer standards for how they manage risk. I would not be that excited about an attempt to do a really comprehensive AI regulatory push that covers AI all across the economy and all different kinds of AI systems that are less advanced.

There are negative signals that are like “nothing’s going to happen,” and there are negative signals like, “oh, maybe too much is going to happen,” but it’ll be interesting to see where they get to.

I suspect we’re going to want to have you back as this Congress starts to get to work and we see some of these ideas.

I would love to.

I’m eager for your take on it. Thank you so much for being on Decoder.

Thanks so much. This was really fun.

_{Questions or comments? Hit us up at decoder@theverge.com. We really do read every email!}

Tesla is doubling, or perhaps tripling down on its bold claims of an autonomous future with the offer to "select customers" to purchase a Cybercab for "personal use." But the gilded, steering-wheel-free two-seater will "only be available in supported service areas" - meaning you'll only be able to buy one if you live in one of the cities where Tesla currently operates its Robotaxi service.

The price of a Cybercab is listed at $29,990, plus a mandatory $299-a-month "Autonomy Subscription." Tesla's Robotaxi account on X announced the new offer, including screenshots of the order page and the invite email. But don't expect a fleet of personal …

Read the full story at The Verge.

As much as I appreciate the ingenious design of the Nintendo Switch's Joy-Con, my family and I can't stand using them as standalone gamepads. So when we travel with the Switch 2, we always bring at least one full-size gamepad, plus a couple of 8BitDo's Zero 2 controllers for games that don't use analog thumbsticks. I've spent years hunting for an alternative that pairs the size of the Zero 2 with as much of the functionality of a larger controller as I can get, and CRKD's new Atom Plus is nearly exactly that. But if you have larger hands like I do, you may find the gamepad's focus on portability a struggle.

Read the full story at The Verge.

Hi, friends! Welcome to Installer No. 152, your guide to the best and Verge-iest stuff in the world. (If you're new here, welcome, thank you for taking time away from Zelda to be here, and also you can read all the old editions at the Installer homepage.)

This week, I've been reading about school apps and AI benches and sports gambling and Megaspeed, watching the surprisingly emotional Dane Cook documentary, learning way more than I ever expected about dial-up sounds, digging out my old Xbox One since it's apparently my best shot at playing GTA VI, seeing what I can get done with Claude Duet, and trying a bunch of third-party iPhone keyboar …

Read the full story at The Verge.

On January 6th, 2025, Nvidia CEO Jensen Huang got his rockstar moment at last. He strode triumphantly onstage, hand raised in the air, to the claps and cheers of a 14,000-person arena absolutely packed with influencers and fans, their faces awash in blue glow, as he made his entrance in a black leather jacket that shimmered in the stadium's lights.

Few are cheering for Jensen Huang now. He has left Nvidia after seemingly enabling two chip smuggling scandals and allegedly having three extramarital affairs, including one with an alleged Chinese spy at the same time he was mingling with President Donald Trump.

The world is watching to see if …

Read the full story at The Verge.

Amazon has plans to build a vault of data on every product sold anywhere on the internet, even products Amazon doesn't sell, and make the information available through its own site, according to Reuters. The project, called the "World Product Graph," is meant to make it less likely that shoppers leave Amazon to do their product research on Reddit or Google, and make it more likely shoppers end up at Amazon even if they're searching for a product sold elsewhere. The same goes for AI agents - if Amazon has the largest stockpile of shopping data, AI agents may rely on its index of products over others, so even people shopping with bots will end …

Read the full story at The Verge.

After launching its Dots AI agents platform earlier this year, OpenAI is launching a wallet for the agents. ChatGPT Wallet is rolling out starting today for ChatGPT Pro subscribers in the US aged 18 and older using a US billing address and "eligible US-issued American Express, Chase, Mastercard and Visa cards."

After adding a card to the wallet, your Dots agents can use it to buy things in response to your prompts, but they'll check in with you each time before checking out. OpenAI's support page says every purchase with ChatGPT Wallet requires a security check with biometric verification like Face ID or a fingerprint scan.

Now your dot c …

Read the full story at The Verge.

Bluetooth speakers are often boring boxes in plain colors, but not this one! As part of Nothing’s ongoing early Black Friday promo, you can grab its unique, milk carton-shaped CMF Speaker S from Amazon and Nothing for just $39. That’s a solid $20 discount and the lowest price we’ve seen on Nothing’s newfangled wireless speaker, which hit the market about a month ago.

The CMF Speaker S is the smaller of the two recently-released models — the larger Series M is also on sale at Amazon for $79 ($30 off) — and sports a 45mm driver with two passive radiators to help with low-end frequencies. It can run for up to 18 hours on a single charge and carries an IP68 rating for water and dust resistance. Despite its carefree aesthetic, it also supports stereo pairing with a second Seris S speaker, Bluetooth 6, fast pairing via multiple methods (including NFC and Google Fast Pair), and audio over USB-C.

In addition to their great sound and compact footprint, both the Speaker S and Speaker M feature removable faceplates and multifunction front knobs. Not only can you swap them out to customize the colors, but CMF plans to share STL files so anyone with a 3D printer can create their own attachments and looks.

window.HYPE_DESK_CONFIG = { productImageUrl: "https://platform.theverge.com/wp-content/uploads/sites/2/2026/11/deals-beats-solo-4-drenched-gray-hires.jpg", productImageAlt: "Beats Solo 4 at Walmart", productTitle: "Beats Solo 4 at Walmart", shortDescription: "A new soundtrack for the season: shop Beats Solo 4 wireless headphones during Walmart Black Friday Deals.", longDescription: "A gift for the music fan, or an upgrade for your own daily listening: Beats Solo 4 wireless on-ear headphones are part of Walmart Black Friday Deals. Explore the Drenched Gray pair at Walmart while the event is live, and find a new way to soundtrack the season.", purchaseUrl: "https://goto.walmart.com/c/482924/565706/9383?subId1=WalmartBFCMHypeDesk&veh=aff&sourceid=imp_000011112222333344&u=https%3A%2F%2Fwww.walmart.com%2Fip%2FBeats-by-Dr-Dre-Solo4-On-Ear-Wireless-Bluetooth-Headphones-Drenched-Gray%2F17203508556&partnerpropertyid=7032191", purchaseLinkText: "Shop Beats at Walmart" };

The Nape Pro was Keychron's first trackball device that was introduced at CES 2026 with a slim and compact design allowing it to be positioned right up against a keyboard. The company's new T1 HE instead looks more like other trackball mice already available from companies like Logitech and Kensington with a 34mm ball positioned on the side to improve ergonomics. But since hands and fingers come in various sizes, the T1 HE allows the position of its trackball to be fine-tuned with minor positional adjustments so it rests squarely under your thumb without any reaching.

The T1 HE is now available for preorder through Keychron's online store f …

Read the full story at The Verge.

Under cofounder Jensen Huang, Nvidia has become the engine for the AI boom - and one of the tech industry's biggest environmental liabilities. Its sought-after chips have triggered a new wave of water- and chemical-intensive factories, drained power grids from inside massive data centers, and set the stage for a looming e-waste problem.

Under Huang's leadership, Nvidia clashed with environmental groups seeking to hold the company accountable for its so-called "hidden" emissions. Huang has talked up the idea of using AI to fight climate change while his company's supply chain emissions ballooned. Nvidia's investments, meanwhile, have support …

Read the full story at The Verge.

Last week, Camp Snap launched a miniaturized version of its screen-free CS-8 video camera that was half the size of the original with a smaller grip designed to be easier for kids to hold. But it turns out the company wasn't done shrinking its hardware. Camp Snap's new CS-8 Micro is even smaller than the CS-8 Mini. It's around three inches long and weighs just 1.5 ounces so you can hang it from your keys and, like its larger predecessors, it can capture video using one of five built-in filters.

The CS-8 Micro will be available starting on November 17th at 11AM ET through Camp Snap's online store, Target, and Urban Outfitters for $39.99, and …

Read the full story at The Verge.

People who received the brand new Apple TV 4k today saw some strange error messages as soon as they opened the YouTube app. As the first units arrived, the app was initially only showing a black screen with a QR code and an error message saying "YouTube is not yet available on this device." In some cases the app was functional, but had a transparent red bar stuck at the bottom of the screen saying, "YouTube app is approved for device testing only and not for production use."

Installing the new tvOS 27.2 update and rebooting wasn't enough to resolve the issue for The Verge's Tom Warren, he also had to uninstall and reinstall the app.

The Ve …

Read the full story at The Verge.

Nearly 10 years after the launch of Paprika 3, the recipe-managing, pantry-organizing, grocery list-making app has gotten a big upgrade that adds more functionality to nearly every corner of the experience. I tested Paprika 4 for several weeks while it was in open beta, and the update recently launched across Android (the version I've been using), iOS, Windows, and macOS.

Where Paprika 3 was a one-time purchase per platform, accessing most of the new features in Paprika 4 requires an annual subscription fee of $14.99 (it can also be gifted). That said, new users who don't want to pay can get a free, feature-light version of the app that off …

Read the full story at The Verge.

The first few smart home devices and accessories that Apple co-developed with LG Electronics and Schneider Electric for its new Apple Home Collection are now available for preorder through the company's online store, with delivery expected in mid-December. Apple announced the collection a month ago as part of a stronger push into the smart home space that's led by its new Siri AI-powered HomeView smart display which is also now available starting today.

Of all the Apple Home Collection devices currently listed on its online store, only a handful of LG accessories are available for preorder. These include the $299.95 LG Outdoor Cam and $199. …

Read the full story at The Verge.

One part of the saga surrounding Nvidia CEO Jensen Huang ended on Thursday, when Huang resigned from his position at the top of AI's most important company. But in the coming weeks and months, a much bigger story may be about to unfold. A story about Nvidia's dealings in China, about the US government's willingness to turn a lot of blind eyes to protect a powerful ally, and about the very stability of the AI industry as a whole. It's a lot.

On this episode of The Vergecast, David and Nilay start by trying to unpack the many parts of the story. We're all going to learn a lot more about Megaspeed than we ever planned or wanted, and we're goin …

Read the full story at The Verge.

Three years ago, Meta apparently considered a plan to turn public opinion against one of its most influential opponents, Jonathan Haidt. But since the proposal was revealed in court filings last month, it's backfired. In fact, if there's one thing Anxious Generation author Haidt and his critics can agree on, it's being mad at Meta.

Recent court filings in California revealed that Meta employees had discussed in 2023 a "Whisper strategy" to "turn academics against Haidt," who would soon publish a bestselling book arguing that platforms like Meta's were hurting kids. Haidt had spent years popularizing his theory that social media and smartpho …

Read the full story at The Verge.

Normally, the products I test don't offer to help me write up my experience with them. Duet is different.

Claude Duet is Anthropic's AI agent for consumers, the latest attempt by an AI company to try to convince us that agents can be useful for people who don't write code. It's been an interesting couple of months: Instinct gave us an impressive text-based assistant, Meta offered us its cute little guy, Muse, to help us buy things, and OpenAI gave us the more work-oriented Dots. Claude Duet is even more focused on work than those. It's also more consumer-friendly than Claude Cowork, which was more consumer-friendly than Claude Code - the co …

Read the full story at The Verge.

When Apple CEO John Ternus announced the first touchscreen Mac last month, the biggest reaction in the room wasn't for the PROLED MacBook Pro's OLED touchscreen or the thinner and lighter chassis; it was for the illuminated lid logo. It's a throwback to the light-up logos on the old MacBooks, which Apple started phasing out in 2015, and which were completely gone by the 2018 MacBook Air redesign. It's a nostalgia play, but boy did it work. It's fun!

Just like in the old days, the new logo's brightness is linked to the screen's: The brighter or dimmer the screen, the brighter or dimmer the logo. But the light-up logo on the old MacBooks was …

Read the full story at The Verge.

Service is recovering for Microsoft Outlook and 365 after an almost two-day outage impacting millions of users around the world. While some apps and features are still running in a degraded state on Friday morning, some people (including Verge staffers) reported that access to their emails had been restored, though the flow of Outlook mail might be delayed.

Just before noon on Friday, Microsoft said it's seeing "broad recovery," and that the problem "was a technical issue not related to cyber or AI / agents."

Microsoft:

We are deeply sorry for the disruption this incident has caused our customers and partners. We know the trust they plac …

Read the full story at The Verge.

Usually, we'd be offering to answer your questions about a news event, but honestly we don't really have any answers about what the hell is going on over at Nvidia. I can't speak for Nilay but at the rate Jensen Huang's new mistresses are dropping, I am concerned we are approaching an affair Singularity. If things don't decelerate soon, we may be looking at existential risk. Maybe the future of AI really does depend on Jensen's sex life, but not in the way I was thinking!

As much fun (?) as we are having with all the assorted ladies, it's important to stay focused on what's really important: Chips and China. President Donald Trump was repor …

Read the full story at The Verge.