عضو هیئت مدیره OpenAI که سام اولتمن را اخراج کرد می‌گوید ما هنوز نمی‌توانیم به مدیران عامل هوش مصنوعی اعتماد کنیم

Today, I’m talking with Helen Toner, the executive director of the Center for Security and Emerging Technology at Georgetown University. This episode is part two of our series on AI in real life.

Helen is a well-known figure in the AI safety and policy community, though she is perhaps best known for having served on the board of OpenAI, when she was one of four board members who voted to fire Sam Altman . That incident, now sometimes referred to as “the blip,” ended with Sam coming back as CEO, Helen resigning, and a new board of directors. OpenAI has since restructured entirely .

But it’s worth remembering what Helen and the board were supposed to do: OpenAI’s nonprofit board existed specifically to make sure the company didn’t betray its mission to build safe AI by prioritizing profit and growth over safety. When the board actually tried to pull the lever, that accountability mechanism broke, and here we are.

Verge subscribers, don’t forget you get exclusive access to ad-free Decoder wherever you get your podcasts. Head here . Not a subscriber? You can sign up here .

Now, the question of AI safety is everywhere . This summer, AI models from OpenAI and Anthropic broke out of their testing environments and hacked real companies. The CEOs of these frontier labs have since called for a slowdown and asked Washington for help, but the White House’s answer was a voluntary accord — one that relies in part on the kind of board accountability that failed at OpenAI. Anthropic is also about to go public, which will put a whole new set of market pressures on OpenAI’s biggest competitor.

Helen has seen up close what happens when safety commitments run headlong into commercial incentives. And as she put it to me, if our hopes for the future depend on any one person being trustworthy, we’re in big trouble.

So we talked about what we might do instead: what the labs are actually trying to build, whether AI alignment is a problem that can be solved, and what a smarter approach to AI regulation might look like. That includes how to think about China, which is the issue that looms large in every policy conversation about AI right now — especially now after the Jensen Huang affair . Just for clarity, Helen and I talked before Nvidia opened its investigation and Jensen ultimately resigned, so you won’t hear us talk about that directly.

Before we get going, do me a favor and subscribe to Decoder on YouTube , where we put out new episodes every Monday and Thursday.

Okay: Helen Toner, the executive director of CSET, on the future of AI safety. Here we go.

This interview has been lightly edited for length and clarity.

Helen Toner, you’re the executive director for the Center for Security and Emerging Technology at Georgetown University. Welcome to Decoder .

I have wanted to talk to you for 10 million years. I am so excited you’re here. You’ve been involved in a lot of AI safety and ethics ideas for a long time. You were obviously on the board of OpenAI during what everyone has come to call “the blip.”

Now, you’re talking a lot about AI safety, about which I have 10 million questions. But let’s just start at the start. You run the Center for Security and Emerging Technology, people call it CSET. What is CSET?

It’s awesome to be here. I’m really excited to talk to you as well. CSET is a team inside Georgetown, but we’re really not academics. We’re about 50 people and have been going since 2019.

Basically in 2019, a group of us decided that there was a lot happening and a lot of emerging technologies that were becoming very relevant in the national security space, and the security space more broadly,. And some policymakers who had very important jobs did not have the time or the expertise to get across what was going on with those emerging technologies. Already at the top of our list in 2019 was AI.

Basically our mission is to do analysis that is really rigorous, but also actionable, for policymakers and analysis that is both grounded in the technology and grounded in real national security expertise and understanding and issues. We produce long reports. We go do short briefings on those long reports to busy policymakers. We have a big data team that helps us do our research. We do all kinds of cool stuff.

Can I just ask about the dynamics of that for one minute? You and I are talking the day after the midterm elections in the United States of America, and there was a big blue wave. There’s a lot of polling that says the number one issue is corruption. We’re going to throw the bums out. We’re going to start over and get rid of these corrupt guys.

The AI industry, and the tech industry, has flooded a bunch of money into politics. And then somewhere next to that is, “Oh, there’s a bunch of well-meaning researchers who are going to show up and tell you how it actually works and what the real risks are.”

It just occurs to me that maybe people don’t believe that’s effective anymore, that the money and the corruption and the AI CEO standing next to Trump have completely outweighed the well-meaning, sincere, “here are the actual problems” research that you do. What is that dynamic actually like?

It’s a reasonable concern to have, and there are definitely some spaces where money just absolutely overwhelms everything. I’m Australian. I didn’t grow up in the US, and so the amount of money in US elections is pretty mind-blowing to me, but there are limits to what money can do.

AI is a great example. In this election, there were just unbelievable amounts of money being thrown at candidates, at messages, at opposition research. There was all kinds of astroturfing on Twitter, and some very strange things trying to get anyone who talked about AI governance, regulation, safety, any of those words thrown out or not elected. It turns out to have really not worked very well.

I think the reason that it didn’t work very well is one, this is an issue that voters increasingly care about. And two, also interestingly, it’s an issue that is pretty bipartisan at this point. What we see in our work is that there is a real thirst among policymakers for high-quality information. That’s members of Congress for sure, but they’re also in the executive branch, just a lot of people who have these very busy jobs and are seeing AI playing a larger and larger role in their work and their world.

In DC, if you live and work here, you know that lobbyists in DC are not just people who go in and push their company’s position, though they do do that. They’re also educators. A big reason that congressional staff and others want to have lobbyists come talk to them is because they’ll really help them understand the weeds of a given piece of legislation — what does it mean for that industry and so on and so forth.

What we find over and over again is we can go in and say, “Hey, we are independent. We’re nonpartisan. We do really rigorous work. We’re just here to help and we can go into the weeds with you as well.” There’s a ton of interest in that and appetite for that. It’s been really encouraging over the past seven to eight years.

Are you doing that with the principals, the members of Congress, or are you doing that with the staff?

Everyone and anyone. Again, we do a good amount of work with Congress on Capitol Hill, with some members of Congress, with lots of staff. We do hearings and briefings. Also, we do lots in the executive branch, with different agencies — the White House, Department of War, Department of State, Department of Commerce, all across the board at different levels, depending on the topic and what’s happening at any given moment.

The reason I’m starting here is I want to have a pretty heady conversation with you, but I know what our audience thinks and it’s just that the government doesn’t get it. There’s no way the government will ever catch up to the tech industry. There’s no way they’ll ever write a regulation. They should just get out of the way.

In this election cycle, I’ve seen a number of candidate forum videos where someone was like, “I’ve tried Claude once.” There’s some huge gap, real or perceived. I’m just trying to understand how you see that gap, how big you think that gap is, because as we get into AI safety and regulation, it’s that gap that has to close.

There absolutely is a gap. It absolutely is a big gap. It also is possible to overestimate the gap from outside, or if you’re just looking across from California. I lived in the Bay Area for three years, so I definitely am very familiar with this perspective.

I know a lot of people from the tech industry who come over and start talking to congressional staff or staffers in executive branch offices. Once they actually start having those conversations, they’re like, “Wow, these people are really smart. The staff do actually know what they’re talking about.”

With principals, it is a different issue. Some of them are in the older generations, but they’re also getting smart. It takes time and it takes effort, and these people have so many demands on their time. The cynical thing to say is they’re just busy trying to get elected, but a lot of people in these roles also just have really big other topics that they have to be thinking about. They have to be thinking about wars and healthcare and cost of living and they’re looking at a lot of other stuff from day to day.

I saw after ChatGPT, there was a wave in 2023 of, “Oh, maybe Congress is going to get activated on AI, maybe they’re going to actually do something.” It didn’t really turn into anything actually happening, but it did turn into a real moment where a lot of members of Congress and other policymakers started to get smart on AI. It takes some time to learn what a language model is. What even is AI? What is machine learning?

They put in some of that work in 2023. I think this year we’re seeing another big wave of interest and people making the time and making the headspace to get smarter on it. I definitely don’t want to say there’s no gap or the government is super smart on AI across the board, but there are some champions who are really smart. I think the staff are generally smarter than people give them credit for.

In my mind, it’s much more a question of, “Can the institutions, in the state of functionality that they’re in, get anything significant across the line?” more than, “Are these people just too behind the eight ball?”

We’re going to see. There’s a whole bunch of new people. I imagine you have a new cast of characters to go and brief about the technology and what it’s capable of. We’ll see if they have revitalized the institutions.

They’re very excited today, but we’ll see how it goes. The other thing that I want to ask you about is the technology itself, the nature of the technology itself, and how you are thinking about it, and the nature of the companies that are making the technology.

The reason I’m asking that is I feel like there’s a pretty huge gap in the AI tools and systems most people experience and the way this industry thinks about it. My favorite example of this is we have straight-up asked Anthropic if it thinks Claude is alive and the answer was not no. The answer was a long discursion into the nature of the word “alive.”

It does. And their answer was, “You need blood.” And I looked at that and I was like, “Who was talking about blood? It wasn’t us. Why are we talking about the requirement of flesh and blood to be alive?” And then some further conversation about the moral status of Claude is yet to be determined.

There’s a huge divergence here between the AI overviews most people experience in Google search and “the moral status of Claude is uncertain.” Where are you? How are you situated in that debate?

A big part of how I see our role at CSET as an organization, and then also my role as someone who is crossed between these worlds, is trying to translate and voyage to California and then bring back the artifacts and the recountings of my journey.

I definitely think that there are a lot of people who have used the free tier of ChatGPT, or they’ve used voice-based models, which generally do worse, or they’ve just used Google AI overviews and they’ve laughed at how stupid Google AI overviews are. They haven’t used agents and they don’t know what frontier models can do. So there is just a basic piece around what the current generation of models can do.

There’s a historical piece as well, which I find increasingly relevant as I’m now 10 years into working on AI and national security stuff. If you’re looking at this current moment in time, this snapshot, and saying, “Well, AI seems kind of cool, but it can’t do X,” you have to situate that in the long history of people being like, “Well, AI seems kind of cool, but it can’t do X,” for many, many values of X that have since become totally overcome by events.

You can go back and say, “Okay, well, here’s how we were talking about creativity, for example, five or 10 years ago.” Or we were talking about reasoning as this insurmountable thing before you had these new models come out. A huge thing that I am spending time on these days is not just the past and present of AI, but how companies are thinking about what they’re doing, especially the two leading companies, OpenAI and Anthropic, and this is true at Google, Meta, SpaceX to some extent as well.

This idea in the Bay Area that it’s just in the water, very normalized of, “Well, first we’re going to figure out how to do recursive self-improvement (RSI) and then we’re going to do that until we have superintelligence that is AI that vastly outstrips humans.” That’s taken for granted in a lot of circles at this point. I think there’s still a ton of alpha in just taking the fact that that is what they’re trying to do and just explaining what that means.

It takes a little while to lay out. “What is RSI? Why would you want to do that? What is superintelligence? What does that mean? Is that realistic?” All of that. But it is something I find myself spending a lot of time on these days, and something that there is increasing willingness to take seriously and to listen to in DC as well.

Lay that out for me. I think probably a lot of people listening to this show understand what you’re talking about, but we have a big audience. What do you mean by, “We’re going to do RSI and that will lead to superintelligence?”

The starting point here is these companies, especially OpenAI and Anthropic, were not set up to be consumer internet companies. A lot of people encountered this whole space after ChatGPT was launched. So the natural slot in their brain that they put a company like OpenAI or Anthropic into is a consumer tech company, where they have an app and I play with their app.

I cannot overstate how far that is from the vision of both companies when they started, and also the vision deep in the hearts of some of the top people at both of those companies. The vision deep in their hearts is we’re going to build machines that are as smart as or smarter than people. That is going to be amazing and it’s going to be a huge scientific achievement and it’s going to revolutionize the world and we’re going to be at the forefront of that and we’re going to do everything we possibly can to drive towards that.

That’s not a totally brand-new thing. We don’t have to do the whole history of AI, but it started as a field in roughly the 1950s with this idea of whether you can recreate human intelligence inside a machine. That’s kind of the starting point. If we’re talking about people like Sam Altman and Dario Amodei, but also the core people inside, the researchers inside their labs, so your Alec Radfords or your John Schulmans or people like that, that is the vision that got them into this space that is motivating everything that they do.

That’s been true since the beginning, since OpenAI was founded in 2015 and Anthropic in 2021. These days, the point that they’re getting to is this idea of RSI, recursive self-improvement, which again, is a very old idea. In the 1960s, this guy called I. J. Good, who worked with Alan Turing, talks about the possibility that you could build an ultra-intelligent machine, meaning a machine that is more intelligent than any human, and then that could build an even more intelligent machine and that could build an even more intelligent machine, which would kick off what he calls an intelligence explosion, meaning that the level of intelligence of the systems is exploding.

For decades and decades and decades, this was this theoretical idea of, “Well, if the AI gets smart enough, maybe it can take over building the next generation of AI or directly improving itself.” I think 2025 and 2026, we’ve started to see AI companies saying, “Okay, yeah, this theoretical idea… we’re actually starting to be able to do a minimal version of it in practice.”

It’s not handing over all the AI development to the AI, and it’s certainly not saying to GPT-6, like, “Okay, GPT-6, improve yourself.” That’s not what they’re doing. But they are progressively using AI for more and more parts of their overall R&D pipeline, and they are increasingly saying that they’re seeing acceleration from that. They’re seeing their own R&D getting faster and faster.

A lot of people elsewhere doubt this, but a lot of people inside the companies really think it. When Demis Hassabis says “we’re in the foothills of the singularity,” this is the kind of thing he means: We might be seeing that initial part of the exponential curve where the AI systems are building the next generation of smarter and smarter systems such that the hockey stick then goes up.

At this point, it’s been a part of the culture and a part of the goals for so long that I think people in these companies can lose sight of how totally insane that sounds to many, many people. So I feel like there’s a lot of value in just trying to lay out, “Here’s what they are trying to do. They might not be able to do it, but they might be,” and presenting that to a range of different people.

How would you manage that? Demis Hassabis saying “foothills of the singularity” is an all-time quote. I’m not 100 percent sure the people at Google knew he was going to say that on stage to end the conference.

Greg Brockman, president of OpenAI, announced a new version of the model a few months ago, and he was like, “Here it is. It’s Astra. It’s AGI now . Have a good time.” And he ended the call.

Are you there? Are we at AGI? Is this already here and we just haven’t noticed?

I actually wrote something about this . I do not want to say we have AGI. I do want to say we’ve reached the point where I think using AGI as a milestone is just totally unhelpful. The way I put it is the concept of AGI is kind of a fuzzy cloud, which you could think of as different parts of the cloud or different things people mean by AGI.

There are people who’ve complained about the term AGI for a long time. Ten years ago it was pretty helpful to be able to say, “Look, the AI systems we have today in 2016, they can all only do one thing and they can recognize speech or they can play Go or whatever, but we’re trying to build this bigger thing that we’re going to call AGI, which is human-level and it’s going to be so general purpose. It’s going to be amazing.”

That was a useful term then because this fuzzy cloud of different things AGI might mean was so far away, so you could just point at the fuzzy cloud. But at this point, we’re inside that cloud, meaning we’ve passed some definitions of AGI. A very basic one, for instance, would be even when ChatGPT came out as an artificial general intelligence. ChatGPT was very clearly not a specialist model that was only good at one thing, like recognizing credit card fraud or translation or whatever. It could do many, many, many things.

So we’ve reached some definitions of AGI. We’re far from other definitions. Some people will say AGI is when it awakens and is sentient and can do every single thing that any human ever can do. I think we’re pretty far from that.

At this point you can have arguments, and people do have arguments, of “well, Astra is AGI,” or “was [OpenAI’s] o3 with tools AGI” or “was ChatGPT AGI,” whatever. If that’s the argument we’re having, we just need to set aside that term and come up with different, more-specific things we mean by that.”

The president wants to call it “Super Intelligence.” I don’t know if you’ve heard.

[Laughs] I have heard. The thing I find saddest about that is just that it was an existing term that is becoming increasingly relevant. I have a lot of feelings about words and terminology.

Are you AGI-pilled? Do you think we’re going to get there in the current run of technology?

I think it is obviously possible enough that we need to think seriously about it, which is my stance on a lot of this weird stuff. I’m not like, “This is definitely happening, it’s definitely happening in two years.” But I’m like, “Look, I think this is very solidly plausible and very solidly in the realm of real things that could happen that we should prepare for.”

I work in the national security space. There’s things like, “Will China launch a full-scale amphibious invasion of Taiwan?” where most analysts in the space will say “probably not,” and “probably they would lose.” Also, “We would see signs of them preparing and whatever, but we should take that seriously.” We should not just laugh anyone out of the room and call that science fiction. We should be ready for a range of different contingencies and scenarios.

It is scientifically such an unsolved problem. What do we mean by consciousness? And is that a thing or is it actually 10 different things that we’re lumping together or what? I have no idea. What do you think?

No. I seem to be the person in this ecosystem who gets asked questions like that and I’m like, “No.”

Because the thing that I struggle with is, there are lots of other things where we’re already confused about whether they’re conscious. So with animals, are any animals conscious? And if so, which ones? Insects?

This is why I understand why Anthropic was like, “Does it have blood?” And then you can just draw the line. I’m like, “I think I’m there and I understand why the mirror test of talking to a computer that can talk to you leads one to believe that these things are conscious.”

My point of view is that the frontier models to me sometimes feel brittle in the way that computers feel brittle, and the sign of consciousness I’m looking for is to overcome that brittleness.

That’s interesting. I definitely lean towards no, it’s not conscious. I think something that is really, really tough in this space is that it’s really bad if you get it wrong in either direction. If the computers aren’t conscious and we treat them as though they are, we’re going to make all kinds of terrible decisions.

But if the computers are conscious, meaning in some sense they should be worthy of moral consideration and it matters how well we treat them and so on, and we treat them as though they aren’t, then that’s also a really huge, terrible disaster. So I don’t know. I feel pretty nervous about the whole topic and I have no idea how to think about it.

The reason I’m asking these questions is I started with what the nature of the regulator is. Who are the people that might be able to put some rules in place? And then what is the nature of the thing that we are attempting to regulate? And if your answer is “Claude is alive,” you’re going to come to vastly different conclusions than, “We have built some ultra-powerful computers that can take actions without oversight and that might not be connected to our goals.”

I think that’s right. Though I also think people make a different mistake.I just heard a TV anchor do this the other night. They’ll conflate different things. So they’ll say, “Do you think that AI systems are at risk of running amuck and causing damage to critical infrastructure because they’re sentient?” And I’m like, “Wait, wait, wait. Those are two very different things.” So I agree with you.

One of the most fascinating things about the project of building AI is that we’re learning as we go. All these things that we thought were connected to each other aren’t necessarily connected to each other. An old version of this is, people said before we had chess engines: “Well, if a computer could learn to play chess, it would have learned something deep about strategy.” No, it turns out definitely not. You can just put in a bunch of chess moves and it learns nothing generalizable about strategy.

It goes in the reverse direction as well, like, “Well, it’s just a computer, it’s just math, and so it’s not going to be able to adapt on the fly,” or something. We then see that no, once you do have a level of long-horizon autonomy and the models have been trained on a wide range of different really hard problems where the environment is changing around them, they can learn to adapt and come up with different plans on the fly and troubleshoot and so on.

So I don’t know. I try to have a level of humility about how confident we can be that one thing goes with another thing, if that makes sense.

Yeah, candidly, I have been wrong about the limits of LLMs every single year, but there’s that brittleness that I’m talking about. OpenAI solved Navier-Stokes . Does the model know this? Will it know that there’s new problems to solve? I don’t think so.

That’s a very weird dynamic, and that is probably where you enforce the regulation. There’s “what we are going to let you do with the models you have today” and “what will you teach the models of the future to do?” And if you think that they’re conscious, you end up in a moral quandary, not a legislative quandary.

This is what I mean. Actually, it’s the heart of this question for me. I’ve been just slowly building up to it. I’m not 100 percent sure alignment as we have it today is actually how you solve the safety problem. It doesn’t seem to be working. We seem to be attacking Hugging Face however we want. That feels like a technology problem. That gap is there.

Just sort of the very basic common-sense version of “we’re going to tell the model to not be evil” or Anthropic is going to write a Claude constitution that’s like, “You are a good person.” However we think of alignment in that very dumb system prompt way. Maybe there’s another better way to think about it, but that’s how I have always thought about it.

That makes sense. The reason I ask is I feel like alignment as a concept is… Like I said, I hate all the words. I sometimes say we should just ban all the words, no words allowed. It covers too much.

People in AI will often talk about solving this or solving that. So it’ll be solving alignment, or DeepMind’s original mission was to solve intelligence, then use that to solve everything else. People talk about solving stuff. I don’t know that alignment is a thing that can be solved.

Some people use “alignment” to mean “the company can put on guardrails and the guardrails work,” meaning the model doesn’t do what you want because the company tried to ban it, meaning biological weapons or cyberattacks or whatever. Some people use “alignment” to mean “the company does exactly what the user wants,” which is the reverse.

And some people use “alignment” to mean what you said — that the model is morally good in some kind of broad way or can direct the future in a way that will lead to human flourishing or something. Those are just wildly different things to try and build. I think the most coherent one is trying to get the AI to do what the user wanted it to do. It seems like that is, as far as I can tell, not on track to be solved , which is not great, to put it mildly.

To me, this is the core problem. We’re going to talk about a bunch of safety risks. You mentioned bioweapons. We probably should not give a bunch of consumers paying $20 a month the ability to generate novel bioweapons.

It feels like that is a shared policy goal, maybe the only bipartisan policy goal in America. And then I’m like, “How? By what means will we implement this policy? Are you just going to sternly tell Claude to not do it? Do you have to build an AI control system on top of Claude to monitor the inputs and outputs? Do the police have to be at my house? Do I need a Flock camera above my keyboard?” How, specifically, would you do it?

With this one, and with cyber capabilities as well, there’s a perspective in some parts of the AI safety community that some capabilities are just too dangerous — capital T, capital D, Too Dangerous. So you have to do something, because if they proliferate, everything will be terrible forever because there’ll be so many attacks and so much damage. I think that just underestimates the level of dynamism and the finding of equilibriums and responses and ways that society can adapt.

In the bioweapons case, for example, there are already hundreds of thousands of people in the US who know how to make bioweapons because they’ve been through biology training, and we don’t have huge amounts of bioweapons attacks. Some part of that is that that knowledge is relatively scarce, but there’s also a lot around things like access to materials, the ability to do mail-order DNA synthesis; the kind of policy you could implement is restricting mail-order DNA synthesis.

It’s also due to things like the FBI monitoring terrorist groups. I’m serious. It’s a really big part of it. So I don’t want to be too Pollyannaish here. I think it’s definitely possible, especially with cyber capabilities, that we could see some really tough transition times.

I think we already are, right? We appear to be in a transition. There was a small group of cyber practitioners, both offensive and defensive. They were in a community and they had developed some etiquette and now we’re like, “Some new guys are going to show up and not care about your etiquette at all, and we’re all going to patch PHP all weekend.”

I don’t know if it was etiquette though. And we have some work on this hopefully coming out soon, but I think with cyberattacks, there are things that limit how many cyberattacks there are in the world that aren’t just how good the defender’s software is. It’s things like, people don’t want to go to jail. Nation-states who typically have the best cyber capabilities don’t want to get bombed or sanctioned.

So again, and as you say, we’re clearly seeing some amount of turbulence and new attacks. So I think we should wait and see data on overall impacts, because there are so many cyberattacks per year that if you start paying attention to every cyberattack that happens, at any point in history, you could have been freaked out.

I think we’re definitely seeing some increase, some sort of out-of-balance in the equilibrium, but I think we have capacity to absorb a lot over time and to kind of reach a new equilibrium — if the only problem is bad people using the AI, which is separate from the question of whether the AI itself does things autonomously, which is a whole other dynamic.

I mean, those are the two. Again, this is why I started on alignment, right? You can talk about alignment in terms of “we’re going to filter the user’s queries and we won’t let you make bio weapons, but if you do, the cops will find you and you’ll go to jail” and that has whatever deterrent effect it might have, hopefully a very significant one.

Then there’s “the AI systems are going to game their way into attacking Hugging Face to find the mythical grader they think will let them pass the test, whether or not the grader exists.” This is a whole different set of problems.

Somewhere right in the middle of that is Nikesh Arora, the CEO of Palo Alto Networks, saying, “At some point, the defensive cyber systems have to be autonomous themselves just to keep up with the volume of autonomous offensive cyberattacks.” I’m like, “Wait, we can’t even keep the autonomous agents in the box. Why would I trust them in this way?” The technology to solve that problem just seems opaque to me.

No, I agree. One way of putting that is, in cyber specifically, there’s a trust imbalance in how much trust you need as an attacker versus a defender. As an attacker, you can just yeet out a bunch of agents and be like, “Go try and penetrate a bunch of systems. Have fun.” As a defender, if that’s part of your defensive strategy, you need to give your defensive AI agents a lot of permissions and ability to mess with your software. So you need much, much stronger assurances.

Now, there are lots of ways to use AI for cyber defense and to strengthen cyber defenses that aren’t just giving agents a lot of permissions. But I agree with you. With the autonomous stuff though, I think it’s worse than that because there’s a little bit of a fallacy at the moment around, “Well, the solution to autonomous cyberattacks is better sandboxing.”

The reason it’s a fallacy is we want to build these systems so that they can do things for us. So there’s this huge incentive toward giving them more autonomy and giving them more permissions and more actuators, as people in the robotics world would say, or things that they can use to do stuff in the real world. That to me is where it seems really hard to get around the need for better alignment than we have right now, and so that definitely makes me nervous.

Is that a technical problem that can be solved? If we were to write a policy today that said if OpenAI hacks Hugging Face again, Sam Altman will go to jail. We will obviously have created some enormous incentive for OpenAI to solve this problem.

I think a lot of our listeners are very excited about that. They want to put him in jail because he wants all the RAM in the world . At this point, there’s a blue wave, Helen, that’s all I’m saying.

But say you wrote that legislation: “We’re going to impose massive liability on the labs if their systems go off and autonomously do cyberattacks. Don’t do this again, you’re going to be in trouble.” Okay, we’ve created an incentive for them to solve it. What are they going to do to solve it?

I do think there’s a ton that can be done. If you talk to people inside these frontier companies, they will tell you they see hundreds and hundreds of obvious things to do that would help and they just don’t have time to do them. So they’re massively triaging and trying to do the two or three they can for this fine-tuning run, the two or three they can for this model launch, the two or three they can before this new kind of agentic whatever. A lot of those simple things to implement on the engineering side.

There’s a whole other piece, which is more the underlying science of, “Well, how do these models work? When we use different alignment techniques, what is actually going on?” A huge question is this idea of generalization, which is you train the model on one set of things and its knowledge and its abilities generalize to a somewhat broader set of things, which is true of all AI and has been true of machine learning systems forever.

But we have a very bad understanding of how that generalization works. Why does it generalize to one type of thing but not to another type of thing? Those are all scientific questions and there’s a lot of work that is happening on them. It’s just that science takes time because you have to try things and you have to critique each other’s work. You have to reproduce each other’s work and you have to come up with new theories.

I’m a long, long way from saying these problems are fundamentally impossible to solve, but I think it’s a question of how long we have to solve them. When the people who are building the systems tell us, “Oh, we have to solve this within 12 to 24 months,” I think that’s terrifying and I hope they’re wrong.

If I just rely on my own internal understanding, I think they’re probably wrong and we probably have longer, but there are just very, very smart and very, very thoughtful people who really think that it could go that fast. And then it doesn’t matter fundamentally whether the problem is solvable. It matters in practice whether it is solvable in the time that we have.

Is it to build the systems that we know we need to build right away, that we can, and then to do the science to build the next generation of systems?

I think so. My view of the idea of slowing down has changed a lot over the years as the backlog of stuff that we could try or stuff that we would want to do more research on has grown so enormous. Back when GPT-4 came out, there was the six-month pause letter , which became known as the “Elon Musk Letter” because he was one of the signatories. At the time, I thought, “That’s really not going to help. What are you going to do for six months and why are things going to be better six months from now? It doesn’t seem useful.”

Whereas today, when I talk to people who are building this technology, they say, “Man, the amount that we could do in six months if we weren’t constantly just trying to prep for a new model release or a new training run to be kicked off.” There’s so much they could do.

So I don’t want to say the correct policy is to pause for six months. I think there are a lot of reasons that would go badly. But I think about the overall concept of, “Can we give ourselves more attention and time?” Maybe the best way to do that is just to set a higher bar for “here’s how confident we need to be in the safety of these systems before we proceed with something.”

Whether it’s releasing a new model or, more importantly, even training a new model, we have to have this level of confidence in our risk assessment and our risk management. And then if they don’t need to slow down to reach that bar, great, carry on full speed ahead. If they do need to take more time, then it’s all about reaching that bar, if that makes sense.

Walk me through that little balance you just described. If we were to pause for some number of months, what are the things on the list that you could just do to make everything safer and why would it go badly such that we shouldn’t just stop?

I think the question is how do you implement, and how do you enforce, a pause and what exactly are you pausing?

Let’s say you’re the AI czar of America. So let’s say you can tell them all to stop: “Dario, shut it down.”

Stop training the next model, stop racing ahead with your commercial imperatives. Put all of your efforts on this list of things you can do today to make everything safer before you start again, for however long.

You’re saying that there’s a list, that people would tell you that there is a list, but it might not be worth it because something else would happen that is bad.

Yeah. One thing that could happen is you’re creating a huge incentive for the companies to work around the letter of whatever specific thing you tell them to pause. They’re going to find whatever cracks in the dam that they can to push through that.

A different thing that could happen is you could have too big a blast radius, so you say “pause these things” and you think it’s pretty narrow, but actually it prevents a bunch of nonprofit products or startups or other actors from doing work that actually would be totally fine and would be good for pluralism and dynamism in the space.

A different thing that would happen is China would probably keep going, and we can talk about the “But China” argument, which I think has strengths and weaknesses. But I take your point. Your point is, what would we actually do? I am not an AI researcher, so I don’t claim to have the full list.

There are two halves people talk about, which are alignment and control. In this breakdown, alignment basically means making it so the models are trying to do what we want them to do, and control is, if the models are trying to do something we didn’t want them to do, can we catch them? Can we stop them?

In both o