In-Depth Interview with Ryan Greenblatt: The Four Most Likely Ways AI Takes Over the World

Ryan Greenblatt is the lead author of "Alignment faking in LLMs" and one of the most productive researchers in the AI field.

Source link:https://80000hours.org/podcast/episodes/ryan-greenblatt-ai-automation-sabotage-takeover/

1 Opening Remarks [00:00:00]
2 Who is Ryan Greenblatt? [00:01:10]
3 How far are we from automated AI R&D? [00:01:27]
4 What are the real capabilities of today's models? [00:05:08]
5 Why AI companies will achieve automation earlier than other industries [00:12:35]
6 The most likely ways AGI takes over the world [00:17:37]
7 Will AGI rebel early or bide its time? [00:29:19]
8 The "pause at human level" strategy [00:34:02]
9 AI's control over AI alignment [00:45:38]
10 Can we only hope to catch AI wrongdoing in the act? [00:51:23]
11 What would a slow AGI rise look like? [00:55:33]
12 Why an intelligence explosion might not happen within 8 years [01:03:32]
13 Key challenges in predicting AI progress [01:15:07]
14 A pessimistic view of AGI [01:23:01]
15 The shift to "inference-time compute" [01:28:46]
16 How much has pretraining's effect decayed? [01:34:22]
17 Could an intelligence explosion happen within a year? [01:46:36]
18 Why AI struggles to replace humans [01:50:33]
19 When AI R&D is automated, things could accelerate wildly. Or maybe not. [01:57:25]
20 How quickly could the intelligence explosion decelerate? [02:11:48]
21 Bottom-line advice for ordinary people [02:24:33]
22 Six orders of magnitude of progress... what does that even mean? [02:30:34]
23 Overlooked but important technical work [02:40:32]
24 What is the most promising work in governance? [02:44:32]
25 Ryan's current research focus [02:47:48]

Opening Remarks [00:00:00]

Ryan Greenblatt: The most credible scenario is "humans give AI everything it needs." The AIs keep a low profile and make sure they control the situation. Perhaps they manipulate behind the scenes, sabotage alignment experiments, tamper with alignment results, and mislead us.

But they won't take drastic action; they just watch and wait. They will do many good things: cure all diseases, drive industrial development. People will say: "AI is developing so well, we don't have an alignment problem, which is great." Humans might be kept in the dark indefinitely, as if living in a carefully constructed illusion the entire time.

Another scenario I call a "sudden robot coup"—this also doesn't require superhuman capabilities. We build massive autonomous robots, cheering "we're going to build robots to compete with country X," and country X says "we're going to build robots to compete with country Y." Then suddenly, the robots launch a hard-power takeover.

I think one mistake the safety community makes is focusing too much on optimistic scenarios. In fact, it's reasonable to focus on desperate, crazy, pessimistic scenarios, because most of the risk lies there.

Who is Ryan Greenblatt? [00:01:10]

Rob Wiblin: Today I'm honored to speak with Ryan Greenblatt. Ryan isRedwood Researchthe chief scientist, and also the lead author of the paper 'Alignment Faking in Large Language Models'—a paper described as possibly the most important empirical study on AI loss of control. Ryan, thank you for joining the show.

Ryan Greenblatt: Glad to be here.

How far are we from automated AI R&D? [00:01:27]

Rob Wiblin: Let's start by discussing the best arguments for a software-based intelligence explosion happening in the near term (say, within the next 4 years). How much do you think we could automate AI R&D in around the next 4 years, or roughly automate today's entire AI company?

Ryan Greenblatt: I think the probability that this capability exists (not necessarily that it will be used or fully deployed) is about 25%, and if extended to 8 years it might be 50%.

Rob Wiblin: Many people hearing this will think, is this pure speculation? How do we form realistic or grounded expectations?

I want to know what key evidence supports you making such a prediction or forecast. What key evidence makes you think this is entirely possible?

Ryan Greenblatt: First, a good starting point is to look at the current level of AI capabilities. How close do we feel we are? People have very different intuitions about how close we are in an objective sense.

I think the current situation is: AI is getting better at math; they are gradually reaching human level in the field of mathematics. They can complete isolated software engineering tasks of about an hour and a half—I mean tasks that take a human about an hour and a half to complete. They are improving across a variety of skills.

So we are at a stage that is objectively impressive—and importantly, much more impressive than two years ago and certainly much more impressive than four years ago.

The naive starting point for what the situation will be like in four years is: where were we four years ago? Where are we now? Try to make a qualitative extrapolation from here to there. That is a bit sloppy. For example, two years ago I might have said we had GPT-3, and now we have GPT-4: look at this gap, and then roughly extrapolate forward.

I think now we can do better, because we not only have GPT-3; we have GPT-3.5, GPT-4, and the progress of GPT-4 over the following two years. During this time, AI went from being almost unable to complete agentic tasks to being able to complete these tasks quite successfully, and to be able to recover from errors.

During this period, GPT-4 could perhaps complete some agentic tasks that take a human 5 to 10 minutes; it can understand how to use tools. I think GPT-3.5 was basically unable to understand tools in an agentic framework; GPT-4 can understand them. Then, during this period, we went from 'can understand tools, and sometimes can complete tasks' to 'can complete, 50% of the time, a task that takes a human software engineer an hour and a half.'

There is a trend line showing how fast we are progressing in this area. At least in 2024, progress was quite fast, from almost nothing to an hour and a half, with a doubling time of about—I steal a lot from METR, maybe Beth will have a podcast covering much of the same content—but the doubling time we see is fast enough that AI is expected to be able to complete 8-hour or 16-hour tasks in less than two years, which is quite fast progress.

Then from there, if they can complete a week-long task, I think you might already be close to automating the work of a research engineer, plus some additional work.

What are the true capabilities of today's models? [00:05:08]

Rob Wiblin: Okay, so you would say they're objectively pretty impressive at what they can do now. Some people are more skeptical. Is there anything that could more clearly illustrate what they can and can't do? Maybe in response to those who would say, "Sometimes I use these tools, and they seem dumb, or they don't seem to be able to do what I expect them to be able to do, or they produce a lot of reasoning but I find errors."

Ryan Greenblatt: So my qualitative, felt model is: AIs are pretty dumb, they used to be dumber, they're rapidly getting smarter, and they're very knowledgeable.

I think what a lot of people are exposed to is that there's some cleverness in these systems: they can understand some fairly general situations well, and especially with reasoning models, they're quite good at thinking through some problems. Beyond that, they're very knowledgeable, which gives people a misleading impression of their overall generality and adaptability.

This is why many people react. There's an overly optimistic view, which I call Leopold [Aschenbrenner]'s chart, where he says this is PhD-level intelligence, or people say this is PhD-level intelligence—and then some people respond by saying, "PhD-level intelligence? Come on, it can't even play tic-tac-toe." Maybe with reasoning models that's no longer true, but the direction is like that, you know, it can't play tic-tac-toe; it can't handle relatively novel situations. It gets tripped up by these things.

Now, I think we have to discount its tripping up in these situations somewhat, because I think a lot of it can probably be better described as cognitive bias rather than a lack of intelligence. Just as humans systematically make certain errors, even if they're quite dumb, or in a sense they make quite dumb errors.

Like the conjunction fallacy. If you ask, "What's the probability that someone is a librarian? What's the probability that they're a librarian and have certain attributes of librarians?" [Humans will say] the probability of the conjunction is higher, even though the probability should actually be lower.

I think AI systems have biases like this, shaped by the environment in which they were created or by their training data.

For example, if you give an AI a riddle. Like, "There's a person, a boat, and a goat. The boat can carry the person and one other item. How many trips do they need to cross the river?" The answer is one: they can just cross the river directly. But there's a similar riddle involving a person, a goat, and something like a cabbage—you know, there are some clever methods—and AI's conditioning to that method is so strong that they might immediately blurt out an answer. They have a strong heuristic tendency toward that answer, but it may be more that they feel pushed toward that answer. But if you make them realize, "Oh, this is a trick question," they'll shift from there.

In fact, you can also stump humans with the same trick question, right? So if you ask a human, "Which is heavier, a pound of bricks or a pound of feathers?" they'll say bricks, and then get stumped. The problem with language models is the opposite: if you ask them, "Which is heavier, two pounds of bricks or one pound of feathers?" they'll say, "The same weight! The same weight!"

So I worry that a lot of the tricks many people do are similar to the tricks you can perform on humans, and it's hard to know how much to conclude from them.

Rob Wiblin: Yes. And one general challenge for assessing their capabilities is that, I think [Nathan Labenz used this expression], they are "human-level but not human-like"—so in some cases they may be broadly as capable as human employees, but they have very different strengths and weaknesses; they may be stumped in ways that are completely baffling to us.

But you can imagine an AI society looking at humans and saying, "How can they not multiply two three-digit numbers in their heads? That seems insane. These are clearly not general intelligences. They have almost no memory of this book they read last week. This makes no sense. How can an agent act like that?" This makes it a bit hard to establish common ground for comparing humans and AIs.

Is there anything more to say about how you assess what level they are at now?

Ryan Greenblatt: Yes, I think I would not use the term "human-level." Maybe this is me being a bit conservative or pedantic, but I like to reserve the term "human-level" to mean "can automate most cognitive work."

So maybe we are starting to enter the stage of "human-like-level" AI once it can really fully automate a large amount of human work, or can become part of the cognitive economy in a way comparable to humans—maybe not fully automated at that point, but I also like to talk about the point of full automation. That is one point, just to respond.

More context on how good AI is: some things we have seen that may be relevant are that we see AI continually improving in math and competitive programming. So in 2024, we went from AI being at around the bottom 20% onCodeforcesto now, according to [Sam Altman], around the top 50.

Rob Wiblin: The top 50 people?

Ryan Greenblatt: The top 50 people. Literally the top 50 people. Or at least the people doing Codeforces; maybe some people are not on the leaderboard, but roughly that. And then it looks like before the end of this year, they will be basically better than the best humans at that particular thing.

On math, this is based on an anecdote from a colleague, but maybe they are currently at the level of a very competitive eighth grader on short-number contest math problems like AIME; the top eighth graders and AI now do equally well. I think AI is much worse at proofs.

But both of these things are improving very quickly—a year ago they were much worse. I think this is basically because we are doing reinforcement learning on AI for these tasks.

I expect the same trend to appear in agentic tasks and software engineering. We have already seen this to some extent: AI is already quite good at writing code, quite good at following instructions, and also decent at finding errors and recovering from them. I think with a lot of runtime compute and a lot of scaffolding, this can be pushed further.

And then they are weaker at many things. For example, they are much weaker at writing, and much weaker at many other things. But I expect that as you keep making progress in software engineering, you will gain a lot of other capabilities.

Rob Wiblin: Yes. Okay, so the level we are at now is arguably quite high. They are able to complete tasks that take humans longer and longer; they are able to follow longer instructions and complete tasks with more open-ended choices. And this roughly doubles every six months or something like that?

Ryan Greenblatt: I think the doubling time in terms of time horizon, my guess is that over the next year it will be clearly faster than every six months, but maybe the long-term trend is around every six months.

So we might expect that in early 2024 or a little later, people start doing more agentic reinforcement learning—more RL or specifically training AI to perform well on agentic software engineering tasks. I think this trend will continue through 2025, and maybe accelerate in 2026 or even later. But maybe the long-term trend is more like doubling every six months. I expect the coming year to be more like doubling every two to four months.

Rob Wiblin: Okay, so the coming year will be very rapid growth.

Ryan Greenblatt: Very rapid, yes.

Why AI companies will achieve automation earlier than other industries [00:12:35]

Rob Wiblin: There's an interesting dynamic: we might expect that you could almost completely automate an AI company, perhaps much earlier than automating almost any other company. Because first, they put a huge amount of resources into trying to automate themselves and their own processes, which makes sense from their perspective—first because it's what they know best, and these are among the highest-paid knowledge workers in the world. They earn enormous salaries. So if they can figure out how to get AI to do this, then it has enormous economic value.

Of course, the people running the companies think this is worth far more than it appears based on dollar amounts, because they think they're about to trigger this intelligence explosion, a positive feedback loop, that will change everything. So for them, this is the thing they're most interested in automating. They care much less about automating McKinsey's consulting reports, even though that's also a lucrative business. So it may be that we haven't automated consulting yet, even though that's certainly possible, mainly because they haven't tried. They're just trying to automate their own employees.

Ryan Greenblatt: I'd say my guess is that even with a lot of bootstrapping, nearly fully automating fairly highly paid human knowledge workers across a relatively broad domain may be difficult. But I do expect there are some jobs where, if AI companies tried harder, they might be able to automate more than they do now.

In fact, as you say, the people who benefit most from AI are the people close to AI. That's true now, and I think it will become increasingly obvious in the future—again, because of this dynamic, AI company employees are now highly paid; by the time AI can automate AI companies, they'll be paid even more, you'll be able to invest more, and AI company CEOs will be more convinced that AI is extremely important. So I think at that point we'll see an even bigger gap.

One objection you might raise is: of course, I agree we'll see a lot of effort focused on automating AI companies, but at the same time, there's a lot of valuable human knowledge work outside that you can automate. So we'll see some economic impact in parallel.

I think that's a reasonable initial view, but one problem with this story is that as the value of AI intellectual labor rises, the price of compute will rise substantially, at least in the short-term scenario where compute is very scarce.

Rob Wiblin: To clarify, are you saying that companies will use a huge amount of compute to automate their own work, so much compute that in effect they won't have many chips available to serve customers with lower economic value, or certainly less important to the company?

Ryan Greenblatt: Yes, roughly. But I think it's not even just about automating things, but about experimentation.

Let me be a bit more concrete. How do AI companies spend their compute now? I think it depends on the company, but my rough breakdown for OpenAI is: about a quarter on inference for external customers, half on experimentation—like small-scale training runs that don't end up being deployed, testing some RL code, that sort of thing, so researchers' experimental compute—and a quarter on large training runs. So about three-quarters of compute is in some sense internally directed.

Then if we get to a state where AI can automate AI R&D and this produces a huge speedup and seems very important, then you can imagine that state might look more like: a fifth on inference for your AI workers—so you spend a fifth of your compute just running your employees—three-fifths on experimentation, a fifth on training or other. Obviously this is very speculative.

Rob Wiblin: So customers are almost completely squeezed out.

Ryan Greenblatt: Yes, yes. I mean, you might have some customers, but probably almost completely squeezed out, and we'd see prices rise. When you're thinking about which customers to serve, maybe you should imagine that, perhaps surprisingly, once AI can achieve this kind of automation, the highest-paid employees might be the ones AI targets first. So maybe you should imagine places likeJane Street, high-frequency trading, where AI seems especially helpful, seems especially highly paid, and they are especially constrained by intellectual labor.

Now, I think we'll see parallel automation in many other occupations, but probably at the point where AI can automate the most, more attention will be focused on AI R&D. I think it's even possible that we see some occupations slowly being automated—like mid- to low-end software engineering might slowly be automated more and more—and we might actually see this trend reverse, because compute becomes more valuable.

Because right now we're in a state where everyone is trying to get as much inference compute as possible, or at least the biggest companies or the leading companies are trying to get as much inference compute as possible, just using this compute to compete on software engineering.

I don't currently expect the trend to reverse, but I think we might see the automation trend plateau or even reverse because of this.

Rob Wiblin: Because people find more valuable things to do with AI.

Ryan Greenblatt: Yes, exactly. It depends on a relatively short timeline. I think on a longer timeline, you'd expect things to be smoother, in which case you wouldn't expect the trend to reverse. But if things are more sudden and jumpy, then at least it seems possible.

The most likely way AGI takes over the world [00:17:37]

Rob Wiblin: I'm curious about what implications turning to this has for how worried we should be, and what specifically we should be worried about? If things develop this way, what can we do now to help us deal with this situation? I mean, this is a scenario that becomes quite challenging for humans to track or participate in. So we have to set things up, I guess, so that it goes well for humans—rather than us just being defeated in the course of history. Maybe you'd disagree with this?

Ryan Greenblatt: I think it's quite plausible that we get to this state soon, where AI takeover using various mechanisms becomes quite credible and potentially surprisingly easy through various avenues. We don't know. This is another huge source of uncertainty. We have a lot of transitions, how big is the progress? How much of this progress is in intellectual labor? And then the question is how much does intellectual labor help with takeover? What are the physical bottlenecks to takeover? How much does having more missiles help relative to having other things?

I think there's a lot of scary stuff that could happen here. One thing is that once you've automated the entire AI company, it seems quite plausible to me that: you've handed things over to the AI system, you're still trying to supervise it, you're still trying to understand what's going on.

But it might be that the AI can run what I call 'malicious deployments'—they can basically use a lot of compute in an unmonitored way because they're writing so much stuff, so many things are happening so fast, that if they're not aligned, they can evade your safeguards. And then it might be that most of your compute and effort is being used for things you don't want.

Beyond that, these AIs might be doing things like backdooring future AI training runs, making sure the AI is aligned with them rather than with you, and doing a lot of work where you might have no idea what's happening. Even at the most basic level, what code was run, you might just be misled because things are moving too fast and beyond your control.

Rob Wiblin: Yes. If people are willing to acknowledge that we have billions of fast superhuman geniuses running in this data center with almost no human oversight—because the amount of work being done will exceed the capacity of all employees of one of these companies to review, and even if they had time to read it, they often wouldn't understand it because it would quickly go beyond what they can even track—I think people would intuitively believe that at this point, if the AI wanted to take over, they might have a lot of options. Oh, no?

Ryan Greenblatt: Actually a big point of disagreement is how much does cognitive labor help with takeover? I think there are many different pathways. Man, I should have brought my takeover plan list, my AI hot tips! But I think once AI is very superhuman, there's basically one path, which I call the crazy nanotechnology path, or the crazy superhuman technology path—they just rapidly gain hard military power through very superhuman mechanisms.

So maybe you have them make nanotechnology, or maybe you're like 'No, don't make nanotechnology. That would be really scary.' But they can do it even if you don't want them to—because they can use a lot of the compute in your data center to research nanotechnology, and they can also exfiltrate themselves, coordinate with external versions of the model, arrange wet lab experiments to guide nanotechnology from various things. And then you're completely screwed.

Rob Wiblin: So one possible layer of defense is that, despite the enormous capabilities here, if we can precisely direct what they're doing so that they never have idle time to think about how they would take over or actually try to implement it, then this might keep us safe.

However, in this situation—where we can review so little, and our understanding of how these models work is so limited—it seems likely that there will be some slippage, and they might figure out a way to grab a little time, grab a little compute, and then use these openings to use more and more of the data center's compute to pursue their own agendas rather than the tasks we think we've given them.

Ryan Greenblatt: Yes. If all your AIs are consistently misaligned, and we're at this level of capability, I'm quite doubtful about maintaining a meaningful level of control. By control, I mean things like AIs not being able to do bad things even if they want to.

I think a somewhat slowed-down version of the initial automation seems quite plausible, maybe even full speed, and perhaps you can use clever strategies—including having AIs work on strategy and ensuring their work on strategy itself isn't corrupted—to maintain control while staying within or slightly beyond human range.

But once you get into this completely insane state, especially if you want to use AI to develop things like a cure for cancer, and you don't understand what they're doing, it seems extremely difficult. They're routing through biological mechanisms you don't understand. They're coordinating humans in wet labs who don't know what's going on. I think it's very likely that, for various reasons, you'll want AIs directing people to do wet-lab experiments, and you have no idea what's happening. The AI explains something, you might be able to do some checks, but you're very confused. So maintaining control over that.

Maybe I'll try to quickly go through the takeover checklist. There's crazy superhuman technology, which I think is quite plausible, for whatever that's worth.

There's also what I call superhuman strategy/manipulation, including things like persuading people to do things very much against their interests. I think it's unclear whether this is possible, but if it is, it would indeed make takeover much easier.

Then there's a bunch of things that might be more likely within the human range. One example is AI escape, which might coordinate with a copy on the inside, but it's also running some independent existence on the outside. Maybe it's like running on an internal GPU, but it's also running code on servers outside the AI company.

As part of doing this, it basically builds an independent industrial base, so it has everything it needs to restore industrial civilization, and then kill almost all humans. So like deploying bioweapons, deploying mirror bacteria, deploying mirror mold, whatever, eventually killing most humans relatively quickly.

I think this seems technically feasible. Then the question is, does the AI screw itself over? Can it sustain what it's doing, or does it just collapse into ruins because there are no human hands to build everything? So I think it needs to have some independent industrial base, whether through having some human loyalists, some enslaved and kept-alive humans, or through robots.

Rob Wiblin: At this point, wouldn't there be incredible robots and probably quite a lot of robots?

Ryan Greenblatt: Possibly very fast. I haven't analyzed how many robots you'd expect. The question we also need to answer is, when humans start being killed by bioweapons and possibly suspect it's caused by AI, how many robots get destroyed, and if there are surviving humans, how much military force is needed to deal with them?

I think it's not obvious how the situation would develop, but this is one pathway to why you could get takeover without needing very superhuman capabilities. The things I described seem like something AIs could do in principle, if they simply quickly reach human level and are super coordinated.

Why it's hard for humans to do this: one, humans don't want to do it. The other reason is that I think it's hard for humans to run a huge conspiracy, but it might be much easier for AI to run a huge conspiracy because they don't have to rely on recruiting potentially untrustworthy humans. They might be much better at cybersecurity, more meticulous. They might screw it up, but I think there are mechanisms that could let this happen. So this is the story of 'killing everyone through an independent industrial base.'

Another story, which I think may be the most plausible of all, is the story of 'humans giving AI everything they need'—the AI just stays calm, they make sure they control the situation; maybe they're manipulating what's happening. I talked earlier about how what you see is not what's actually happening. For example, you look at experiments, and they're not what you expect. The AI is doing that. They're sabotaging alignment experiments. They're sabotaging alignment results. They're confusing us.

There are many mechanisms to do this, but they won't do anything very drastic. You know, they just stay calm. We're scaling up compute. They do a lot of good things. There are cures for all diseases, all kinds of great things happening, industrial development. People will say: 'It's so great that AI is developing so well and we don't have a misalignment problem.' Some safety people look at the situation worriedly, wondering whether this is happening.

And then at some point, when the AI has an extremely decisive advantage and humans start to become any kind of obstacle—throughout this whole story, they might not be, right? I think if humans end up becoming an obstacle earlier, maybe the AI would take more decisive action. But if decisive action isn't needed, maybe they just lie dormant and wait.

And then at a point with a truly enormous industrial scale, truly massive numbers of robots, entirely run by AI, with the situation beyond anyone's expectations, maybe even at the time of launching space probes... it could be that humans are deceived indefinitely. It could be like you can in principle Potemkin-village the whole process. But it could also switch at an earlier point.

Now, to be clear, if humans are deceived indefinitely, then everything on Earth might be fine, right?

Rob Wiblin: So you're saying they could even reach the point of 'we're going to settle space and do a lot of things in space,' and they'll tell us they're going to do one thing, but they'll do something completely different, and we'll just be on Earth enjoying our lives, thinking things are going better than they actually are or very differently?

Ryan Greenblatt: Yes. But to be clear, Earth could also be Potemkin-villaged. It could be 'wow, I have so many happy friends,' whatever. But gradually your friends are replaced by robots. At some point, why wouldn't the AI suddenly take over? But in principle, it could go very far. So that's another story.

Another story I call 'the sudden robot coup.' This also doesn't require very superhuman abilities. We build massive autonomous robots. Maybe we think there's a way to shut them down. Either the AI sabotages that, or they have a way around it, or there are various other mechanisms that could fail.

Then we built robots, we saw AI building robots, we said "Hooray, we're building giant robots to compete with country X!" Country X said "Hooray, we're building giant robots to compete with country Y! Indeed, it's so great that we have such big robots." And then, oops. All of a sudden, the robots sweep in and carry out a takeover of relative hard power, because they control, they can be super coordinated, maybe they use things like bioweapons at the same time, but that might not even be necessary.

Rob Wiblin: I mean, at that point they might not even need to kill us, possibly.

Ryan Greenblatt: Possibly.

Rob Wiblin: Because even if we all try to resist, it's basically futile.

Ryan Greenblatt: Possibly. And things like they might use a lot of cyberattacks to disrupt human responses. So they might do this while we naively think robots are weaker than humans.

Or even if we think we have a kill switch, but the kill switch isn't reliable enough. Their tampering with the AI isn't over-engineered enough, isn't shielded enough. I think this is an awkward dynamic, if you want remote shutdown, you have to make it both tamper-proof for an AI that is in principle very general, very smart, very cunning, and it must be remotely triggerable. So it also can't be shielded. It has to be in a way that the AI can't shield the signal and avoid tampering with it—this is in principle a very awkward combination.

Will AGI rebel early or bide its time? [00:29:19]

Rob Wiblin: What determines whether you get an AI attempting a takeover early or late? I guess the reason to wait is that they will be a larger part of the economy; they will have more hard power, so they are more likely to succeed easily. The reason to act early is that a given set of models might expect to be replaced by other models that might not share their goals, so they will miss the chance to take over, basically.

Ryan Greenblatt: I think the strongest reason for acting early might be: worry about other AIs; worry about humans uniting—either because we get enough work out of the AI, or because people panic and trigger some strong reaction that can restore the situation—the third reason is impatience or whatever.

It may be that by default, humans will greatly slow down progress, even if the AI tries to manipulate things, tries to pull strings behind the scenes. If humans greatly slow things down, and the AI says "No, I want to go to the stars now," they are at a sufficiently high takeover probability… or maybe the AI just says "I want to do it now." Like, I know humans sometimes want things to happen faster, even independent of the total amount they end up with.

So these are some reasons for it happening early. These also apply to things less severe than takeover. One of my hopes for things going well is that we can put AI in a position where they are forced to either take drastic action early, or we get enough work out of them. If they take drastic action early, then maybe we can catch that, and then build a dangerous case. Besides building a dangerous case, we can potentially study that case, iterate on it—what I call "a few disaster preventions." Like, we've already caught a few examples of bad things: can we handle them?

If at multiple different levels, AI is forced to take drastic action? Or conversely, AI can bide its time, because we don't have any mechanism to exit this state by leveraging some ongoing background trends—either safety research improving over time, or humans saying "Sorry, what's going on here?"

Rob Wiblin: You mean the general intervention we can do is to create a trend where over time they actually become less and less likely to successfully take over? Maybe because we succeed in figuring out how to use AI to prove alignment and safety, and detect misalignment, or we are improving our control mechanisms. So this forces the models: they see this trend, they see the window is actually closing, so they have to try to strike early. This gives us a chance to realize what is happening, and then actually win.

Ryan Greenblatt: Exactly. But it obviously could go in the opposite direction, our measures can't keep up with the rate of capability growth, and it keeps rising. There's a concept of, at a given level of takeover risk, what level of capability can we handle? Maybe initially we don't make that much progress, so we can handle roughly human-level at around 20% takeover risk, or a very high level—that will depend on some facts about misalignment: which AIs are misaligned, how consistent they are, how unified they are, that kind of thing.

On top of that, there's the issue that if progress is fast enough, takeover risk may increase over time. Because as models become more capable, they are more likely to succeed, and also more likely to be misaligned.

But maybe we can push the curve in the other direction, we can make the range of capability we can maintain at an _n_% chance of takeover success lower.

Rob Wiblin: Yes, explain that last point. I don't understand.

Ryan Greenblatt: Imagine there is some trend over time, namely: for the smartest AI we have now, what are their chances of success? Or how likely are they to take over? So initially our countermeasures are sufficient, for example when they automate the whole AI company, by default we don't even have a very high takeover chance, because what would they do?

But as they become more capable, the immediate takeover chance is rising, and some of our paths to success have been cut off. Maybe the AI has already escaped, they are running malicious insider deployments, they are sabotaging our work.

But we can make the trend go in the other direction—over time the risk of takeover is falling, rather than rising—because capability is increasing, but our measures are growing faster than capability. But obviously, as we discussed earlier, capability may grow very fast. So if the measures can't keep up, if we are not in some stable state, that could be bad.

So I think the model I was using there is most natural to think about in a state of AI misalignment. But if we start from a point where they are aligned enough that they maintain their own alignment, then the takeover risk initially might be that they are actually aligned, so we are in a stable attractor of alignment.

The 'pause at human level' strategy [00:34:02]

Rob Wiblin: Okay. What impact does this picture have on your research priorities, and what do you think the broader AI ecosystem should prioritize to make things more likely to go well and less likely to go badly?

Ryan Greenblatt: First, what I have been talking about is the timeline without very active intervention. There's also the question of how you deliberately pause at these relevant capability milestones?

I think if we end up pausing at the level of AI capability, around the point where you could fully automate an AI company — maybe a bit before, maybe a bit after, maybe right there — for a sustained period, I feel more optimistic, both giving humans time to study these systems and giving us time to extract a lot of labor from these systems and reduce the chances of a continuous takeover. So there are issues, there are many worlds that are slower than this.

Then another source of hope is maybe you just start this crazy state with AI that is actually aligned: an AI that not only isn't plotting against us, but more strongly, is actively paying attention to us, actively considering ways things could go wrong, and trying to maintain our control. This is called corrigibility or an alignment basin.

So the things I'm excited about depend on the state. I think we can consider a few different states. One state is, maybe especially on these short timelines, I don't expect that much lead time among different actors on the US side, at least.

Rob Wiblin: You're saying there would be multiple companies approaching this automation point at roughly the same time.

Ryan Greenblatt: Exactly. So a fairly natural scenario I often think about is a relatively (from my perspective) irresponsible company, basically whose default plan is to scale superintelligence as fast as possible, with not much emphasis on safety. That's their explicit plan, that's their internal thing, they just pursue this as fast as possible. Leading by three months, say. They might be ahead of Chinese companies, or ahead of some more responsible actors, trailing by three to nine months to some degree.

I think in this situation, a lot of the action will come from a relatively small number of people inside that company. And then maybe a lot of action can come from people outside, who exert various kinds of influence by doing exportable research and potentially by changing the policy situation. Also, I think some things can definitely be done through more responsible AI developers who are trailing, who in this situation put more effort into preventing bad outcomes.

Rob Wiblin: In this situation, one thing I think could be productive is someone inside the company saying: "What we're doing looks a bit scary. We should have better control mechanisms, we should have better alignment mechanisms. We should take this more seriously than our current default plan."

Another is that people working at more responsible organizations can produce research that makes it easier for companies to do this, making it cheaper for them to adopt stronger control measures. Then of course, you can have governance responses: maybe this will start to take off, people will be quite shocked, maybe you'll get broad support for the concept of "pausing at the human level."

Ryan Greenblatt: Yes. Man, I really wish this concept were easier to spread. The unfortunate thing about "pausing at the human level" is that all these words are complicated and confusing. And it's not even a very good slogan. Someone should come up with a better slogan for this thing.

But anyway, yes, I think there are many different mechanisms for doing exportable research, trying to wake the world up. There are more direct and more indirect things. So you can do very direct policy influence, or more like demonstrating capability levels — when you demonstrate capability levels, this can trigger a response from the world.

So I think a lot of my hope in this scenario, especially on very short timelines, I think looks worse than on long timelines. My picture that is partly less pessimistic is that it might be eight years from now rather than four years from now.

So on the four-year timeline, I think we have a mechanism where leading AI companies, we have some people working there, their goal is to prevent serious misalignment—by 'serious misalignment' I mean things like AI systems deliberately engaging in deceptive behavior, deliberately sabotaging our tests, deliberately trying to make our understanding of the situation misleading—and then second, we can try to make the properties of AI systems such that we're happy to hand over to them, and try to aim for that quickly, because we're on a very rushed timeline.

So some of this preventing serious misalignment, or dealing with it, I think in a sense it's the traditional push. But I think there's a bunch of stuff related to handover and AI being wise enough, AI remaining loyal to our interests in the long term, ensuring we even have tests to know whether this is true. These considerations.

There are many different incremental steps. Stage zero might be where you use mechanisms to make AI produce work for you that you think hasn't been sabotaged, even if those AIs try to sabotage it, and then you use this to extract a lot of work from your AI systems. Stage one is where you try to create an AI system that safety people are happy to hand over to, as best you can. Then stage two is where you try to make the AI system running the whole organization one that you're happy to hand over to, and assuming others are also happy to hand over to it.

This is a sarcastic or simplified take on the situation, because at many points you might want to do things like pick low-hanging fruit, show the world the capabilities, try to consider whether anything will happen, try to make the main AI systems that organizations are training at the current capability frontier less likely to cause problems. I'm not particularly optimistic about the situation, but I think there are many different potential ways out that give us some hope, or some chances of salvation, some winning conditions.

Things like us catching AI early is another hope. Maybe we catch AI early, maybe this convinces the world to take the situation more seriously. Maybe even if not, it convinces company leadership that their own power is threatened by AI. So even if they're very Machiavellian, I think AI poses a threat to their power, and they should be worried.

Rob Wiblin: They might change course.

Ryan Greenblatt: Yes. So this could potentially increase the resources spent on safety.

Rob Wiblin: Okay, so the main story there is trying to figure out a safe scenario where you can hand over these processes to an AI you trust. How would you do that?

I guess you start from feeling you have a reasonable grasp of, probably aligned, human-level models. For example, we currently think Claude is probably roughly as friendly as it appears; it probably won't massively conspire against us at every turn.

You figure out clever mechanisms by which you can get these models that you feel okay about to do a lot of the work on control and alignment. You rush to do this before allowing other capability improvements to happen. Then you hope this has succeeded, we feel we can hand over to these models trained by a previous generation we somewhat understand—and then we're going to cross our fingers? Is that how it is?

Ryan Greenblatt: Yes. I should be clear that this is a plan with very low political will. I have a series of plans, and this is like the bottom of the barrel or something. So the way I say this is, we have some early points where we have AIs that can automate a lot of things but we don't necessarily trust. We extract a lot of labor from these AIs, and then we try to make a version of these AIs that we do trust. Not just trust that they're not plotting against us; ideally trust at the level where we're happy to hand off messy open-ended questions—like "what strategy should we take to minimize the risk of the following things?" including potentially some tricky, more philosophical topics.

And then from there, we try to hand off to these systems, and hopefully these systems stay aligned, stay on track, are able to. And also succeed themselves, right? It could be that we hand off to systems that say, "Wow, you seem to be in a pretty bad situation. I'm not sure I can pull it out of this bag." There are multiple mechanisms that could lead to this happening.

Rob Wiblin: I think what you're saying is that they could also mess up the next stage?

Ryan Greenblatt: Especially because they have very little time.

Rob Wiblin: Because they're also racing.

Ryan Greenblatt: Yes, yes. Oh, of course. So there's an internal thing where, if you're the safety team, maybe you have a version you've fine-tuned that's at human level, that you trust, that you're happy to hand off. But capability progresses so fast that maybe the company's current frontier is very superhuman. But you don't even want to work with that system.

I think the first handoff point you should try to be closer to human level—maybe slightly superhuman or at the top level of human capability—because the more superhuman, the more likely it is to be misaligned, so you want to start the handoff earlier. But this probably won't stop the rest of the organization from training more capable AIs. So it could be that you as the safety team are handing off to an AI that is less capable than the organization's frontier, and that AI needs to bootstrap quickly, so the trend of the handoff AI intersects with capability soon enough.

Rob Wiblin: Yes. At this point, you have an aligned, somewhat superhuman model. Is there some chance it could persuade a broader group of people that we need to take this more seriously, that it shouldn't be forced to race against another team in the same organization?

Ryan Greenblatt: Yes. I mean, that's definitely part of the hope. Definitely part of the hope is that I can show empirical evidence. I think also, if we're in a situation where we have two AIs trained by different teams, and these two AIs are just slinging mud at each other—one AI says "that AI is obviously plotting; look at all these examples," and the other AI says "that AI is obviously plotting against you; look at all these examples. It's too biased. Too unreasonable. Obviously just a plant"—I'm like, okay, this is a crazy situation. Of course you can't come out of this state thinking it's fine.

Rob Wiblin: In any case, this is definitely not reassuring.

Ryan Greenblatt: It's not reassuring. There might be ways to downplay it. So one state is, I think it's quite likely that the safety team trains their AI, tries to go to company leadership and say "the situation looks terrifying. You're training this AI. I think it's misaligned. Here's a bunch of evidence." Then another AI comes in and says "we all know that AI was trained by the safety team. I think it's very paranoid about misalignment. Of course the safety team would want us to think all AIs are misaligned. I'm not misaligned. That AI isn't misaligned. It just has poor judgment."

The interesting thing about this is that, yes, it's concerning, but not as much as if you knew that at least one of two AIs was misaligned. So overall, I think one concern about the 'improve the epistemology of AI companies' approach is that the people training the AI may, from the company's perspective, have reputations that are biased on these matters, and I think that raises some concerns.

It may be that for every piece of evidence of risk, you can produce more persuasive evidence that the risk does not actually exist. Because if we are in a system where we really don't know how to adjudicate these things, it may simply be that rationality does not win out.

So that is definitely part of the hope. Part of the hope is that once you have this handover, then it's fine.

Now I think there are many ways this story could go wrong. Here is one example. I talked about the alignment team in the corner, trying to fine-tune or perhaps even train from scratch a version they would be happy to obey. Um, one problem is that the main AI system may have already taken over the AI company in a cybersecurity sense—because it has a malicious deployment; it can use compute in unauthorized, unmonitored ways; it can tamper with the work everyone is running. You are trying to align this AI, but you need to secure yourself against the rest of the company. I'm like, oh my God. That looks potentially hopeless.

AI's control over AI alignment [00:45:38]

Rob Wiblin: Zooming out: at Redwood Research, you spend a lot of time simulating these different scenarios, thinking about what kinds of interventions might help in a large fraction of cases, and then trying to get more people to work on this and working on it yourself. Is that the basic picture?

Ryan Greenblatt: Yes, roughly.

Rob Wiblin: Recently Redwood has prioritized control over alignment—saying, let's assume the model does not share our interests, there is some degree of misalignment: how can we make it so that they cannot mess us up anyway? People used to talk about putting AI in a box. That no longer seems so possible, but I guess there are other mechanisms you can use to try to check their work: setting models against each other, forcing them to do useful things for you.

Why focus on control? Is it just because there may be many different ways this could work, and AI control mechanisms seem useful in many different plausible scenarios?

Ryan Greenblatt: Yes. I think in these extremely desperate scenarios where political will is very low and timelines are very short—very fast from my perspective, very fast, perhaps not as fast as people used to speculate—I think control looks less good.

I think there are several reasons control is still very useful in this world. First, if we can catch AI early, this could be very useful. Another thing is that getting useful work out of AI early in this situation could be very important, and we may be able to do that before we completely lose track of the situation.

And then I think you can also prevent... so the situation I am talking about is one where the AI is basically operating your own data center. You may be able to use various mechanisms to make this more difficult, so it happens at a later stage of capability, so you have more time when your experiments are not being tampered with. So these are reasons control might be good in this world.

I think this doesn't cover more complex interventions. I do think that in these very rushed worlds, control looks less good than in worlds with more time and more political will. When we think about control, the initial goal is a world more like one where we have about a year of delay—like, safety basically has a one-year budget, and we can afford to pause for a year. We can do some things there.

I think control looks much better in these worlds. Or I don't know how much better; I think the risk reduction control brings us is quite substantial, but these worlds are less likely. But that's not to say that control brings us much less risk reduction than alignment stuff in other worlds.

Another way to put it is: in worlds where these things develop so quickly, alignment also looks very difficult, and alignment is harder to deal with in advance. A lot of the things I'm most excited about for alignment involve experimenting on powerful AI, which you can only do relatively late. So maybe in very rushed scenarios, alignment ends up being more of the picture, but it's harder to deal with in advance.

These factors make me really like control. There are also worlds with more political will where I think control looks much better. I often think about a plan of control plus "pause at human level"—this might not require any alignment for a sustained period. Because you have an initial year of delay, you control those AIs, and then you steer from there to a longer delay—by building evidence of risk, ensuring nonproliferation, developing better safeguards—and then from that state you can probably proceed quite slowly, at least for a while, until you're actually happy to hand over, rather than making an extremely rushed, desperate handover because that's all you can afford.

Rob Wiblin: I guess this requires some degree of coordination between companies and even between countries.

Ryan Greenblatt: Yes. When I think about this issue, I think a one-year lead is more likely, and I've updated on this. I think it could still happen. Gaps between companies might open up.

I think your story of working to ensure nonproliferation and possibly slowing progress requires a fair amount of government support. So it might be that you have to get the US government—or at least some important governments—to basically support the whole plan. Not necessarily for the same reasons, but for some reason.

I've updated on this, so I now spend more time thinking about relatively more desperate situations. But you know, an important factor is that in relatively more desperate situations, maybe we're reducing takeover risk from 50% to 40%. And in worlds where control might work and we have a one-year delay, we're reducing risk from 30% to 5%.

So maybe the total risk reduction in these worlds is high enough that they're actually more influential. Because in worlds where our influence is very small, because there's so little will, it might just be harder to deal with.

Rob Wiblin: I see. In general, you want to work in moderately tractable scenarios. If things are too grim, then maybe it's impossible to save; if things go very well, then it's hard to improve. It's the middle cases where you're most likely to get the biggest impact, right?

Ryan Greenblatt: Yes, of course. I think one mistake the safety community has made over the past few years is focusing too much on overly optimistic worlds. I think it's actually quite reasonable to focus on desperate, crazy, all-or-nothing pessimistic worlds, because a lot of the risk is there, so going from 50% to 5% is a lot of action.

And from 5% to 0.5%... I mean, a 5% risk level is crazy and unacceptable; we do not accept that our paint company has a 5% risk of mass death, of takeover—all of that is just from takeover; I think there are other risks besides takeover that are completely unacceptable. But even so, that 45% must be mostly mass.

Can we only hope to catch AI red-handed in the act? [00:51:23]

Rob Wiblin: To what extent can we only hope for serious early warning signs? You talked about catching AI in the act, which is another concept you and your colleagues are trying to make more prominent.

This is a kind of early warning, I think, where you discover that AI inside a company has been sabotaging data centers, they have been doing some very serious things. This might make the government or company leadership more supportive of doing more. How much more hopeful would early warning signals like this make things?

Ryan Greenblatt: I think it helps a lot. One reason I am skeptical is that maybe I think this is conclusive evidence, but the broader world does not think so. For example, our recently released work on alignment faking should, from my perspective, be a considerable update for someone who is very skeptical of this threat model. I think it is predictable that for a lot of these people, it is a very small or no update.

To be clear, I do not think this evidence by itself should be a huge absolute update. Maybe it will not bring people to the level of misalignment risk I think, because a lot is based on conceptual arguments they disagree with.

But I think a lot of people who say "this absolutely cannot happen"—basically 0% chance—should probably move to a 2% or 3% chance. That is a huge difference in vibe space or log space or whatever space. Vibe space, log space, the same space, whatever. People are not making this shift.

So I can completely imagine a situation where we catch AI trying to escape. Buck has a post, what happens if AI trying to escape is caught? People will say "you know, systems sometimes do this. They do strange things. Also, is this real? Were you training it in some strange way?" Or "oh, that happened at that company. But at this company we do things differently." And some people just say "of course, maybe AI has a strange instinct to escape. But what are they actually going to do?" Or things like "we will use various checks to prevent this."

I think all these factors could weaken this in a relatively clear scenario. I kind of expect a lot of situations that I think are huge conclusive evidence to be more confusing than this.

One example is that it might look like AI is dragging its feet on safety research—performing worse than it can do, or not doing very blatant sabotage, but there are more errors than we expected, more misleading arguments. From our perspective this is a big update. Because it looks quite off, but it is fairly quantitative. If you just expect it to be low, maybe this will not be so surprising.

Rob Wiblin: I guess this is a strategic situation where AI can to some extent predict what the reaction will be. In the case where it takes hostile action, it can make them things it suspects will not really trigger a bigger reaction. So for things it can get away with, such as trying to escape, but it cannot get away with mass murder, then it chooses to escape and holds off on murder.

Ryan Greenblatt: At least temporarily. One hope for control is to make serious bad things harder to plausibly deny. We call this "conviction." Going from "wow, that's really suspicious" to "okay, we've established exactly what happened" is very important. Just because it's conviction in our eyes doesn't mean it's conviction from the broader world's perspective, but it does move things forward.

I think this can happen across all the different regimes, from looking a bit suspicious, to understanding what happened and looking very suspicious; to the AI trying to escape, we stopped it, we did the thing where we pretend it succeeded, see what it would do on the outside—and trying to get to "it looks like it's trying to figure out how to make bioweapons. It's trying to run a wet lab."

In principle you could at least go quite far, understanding what it would do in an escape scenario. To the extent that it does things that are more clearly malicious, I think this could incentivize additional action. But I do worry that any warning sign that doesn't cause mass destruction and potential human death might not trigger a strong reaction.

What would a slow AGI rise look like? [00:55:33]

Rob Wiblin: We've gone deep into a rather dizzying and worrying scenario, and I think part of the reason is that something so crazy being so consistent with what we observe now is itself interesting. It's also a scenario where we think a lot can be done now, or a scenario where things could get quite bad, and we should probably take precautions and prepare to figure out how we're going to reduce this risk.

Just for balance, it would be good if we could discuss some more mundane, more boring, and perhaps hopefully less terrifying plausible scenarios. So people don't think this is the only way things could go. Because in fact, a key takeaway I think is just that we have a very wide margin of error for how these different things could play out.

Ryan Greenblatt: Yes. I think maybe a good starting point is that we've talked a lot about timelines. Timelines could be longer, right? So the median or 50% probability I gave is full AI lab automation within 8 years from now. But then there's a long tail, so there are worlds where it's much later than that. Progress could be much slower, more gradual. Things take a long time to integrate. Maybe the world has a better understanding, more time for experimentation, which could make things much better.

Beyond that, there could be very long timelines—or longer timelines, I should say, I don't know about very long—takeoff we should also expect to be slower. I think short timelines are, broadly speaking, associated with fast takeoff, because it suggests higher returns to what we're doing. So longer timelines.

Another thing is that AI R&D could be extremely compute-constrained. So even at the point where you've automated the entire AI company and all its employees, you're not much faster than you would be with human employees. In the extreme case, maybe you're only 2x faster. You might even be slower than that. But I think that's on the low end of my margin of error. I think that might be pretty implausible, but 2x or 3x is plausible.

And then maybe you hit diminishing returns pretty quickly after that, so you don't actually speed up that much. That would mean things look much better in terms of takeoff, because you get a less aggressive takeoff.

The other thing is that the default speed of AI progress is already quite fast. So I even worry—without acceleration, just on the default trajectory—that if we reach the point where AI is capable enough to automate the entire AI company, maybe even if it doesn't bring much acceleration, and soon after that there are some risks, within normal government timescales, just from the default speed of progress.

But I think there are some plausible arguments that future progress will slow down. Especially now, progress depends on investing more and more compute, more and more investment. If things stagnate, investment decreases, but we are still operating at close to quite high capacity, we can imagine a world where the AI industry ends up being an industry that might justify hundreds of billions of dollars in compute spending per year, maybe even more, but not much more than that.

In that regime, if we don't invest more and more compute, then we won't get progress from hardware scaling, and algorithmic progress will also be slower. So maybe the current speed is that we get 13x effective compute per year—some of it algorithmic, some of it compute. I think if compute were fixed, we might have maybe 2x or 3x progress per year. In a regime where compute growth is slower, it might be somewhere in between.

Rob Wiblin: Okay, so that would be a slower takeoff. How likely is it that alignment is a relatively straightforward problem? Or even not a problem at all, and we're just deluded, and all these AIs we produce basically by default want to help us?

Ryan Greenblatt: Hmm, an important question from my perspective is, what is the probability that various AIs at different capability levels actively conspire against us? I think it's quite plausible to get to very superhuman capabilities without the AI very actively conspiring against you in some strong sense. My current view is that if you basically take no countermeasures; you basically just follow the most effective path to capability, it's about 25%. Then once we start considering countermeasures, it might be much lower, and this risk might also be lower at this early point.

A hopeful story is that you have these early systems that can automate the entire AI company. These systems are not conspiring against you. You do extra alignment work so that they also try their best—they actually try to pursue alignment research; they try to predict risks—which maybe by default wouldn't happen even if they weren't conspiring against you. It could be that things are not conspiring against you, but things still go off track. But if they are not conspiring against you and you avoid things going off track, then maybe we're fine. We hand over to AI. The AI manages the situation, they realize the future risks. They build another system with the same properties, and that can be self-sustaining.

Another scenario is that maybe a more powerful thing is true by default. Maybe you train AI with a relatively naive—what you would do by default commercially—training strategy, and they really just, out of the box, try very hard to help you. They are really good. Maybe even they are just aligned in some very broad sense. Not only are they not short-sighted; they really try to pursue good outcomes for the world. I think this is possible, but less likely than the previous scenario.

The other thing is there's a question of whether it happens by default. It could also happen just because we put in a lot of work and we might be in a better position. It's hard to have too much confidence about how much alignment or safety work will happen over the next eight years. If we have eight years, a community can build. AI will become more capable over time. More people will be working on this. It seems like there might be things like big insights, but even setting aside big insights, maybe just things like we really advance our techniques, we've tuned the science.

So there are some timelines, I would say, that are maybe technically hopeful. And then there's the broader category of socially hopeful. There are some stories where society takes this more seriously in the future—maybe a subset of countries, maybe the broader scientific community. It might be because of an early warning sign. It might just be that slowly people are exposed to AI, and the timeline might be longer, with more people interacting with it.

It also seems possible that some big event happens that isn't very related to misalignment risk but actually has high leverage. So maybe lots of unemployment prompts a massive response, and some of that response goes into mitigating misalignment risk—either very directly, like people saying 'AI is a big problem in this area, so maybe it's a big problem in that area'—but it could also be more indirect: it leads to AI progress being slower or more cautious because there's more regulation.

Rob Wiblin: Are there other hopeful categories, or does that basically cover it?

Ryan Greenblatt: So the easy things in terms of timelines and takeoff, the easy things technically, and then socially. The fourth category is that we rise to the occasion. Maybe even if society doesn't rise to the occasion, maybe it's just a series of heroic efforts—or hopefully not so heroic—

Rob Wiblin: If a company, the truly leading company, turns out to actually be very responsible, puts a lot of effort into this and succeeds.

Ryan Greenblatt: Yes. I think a lot of the risk comes from a world where, if you had just a one-year delay, you took the situation very seriously, you thoughtfully considered various considerations, you could have saved the world. Or at least the risk would be much lower. Maybe it wouldn't be minimized or eliminated. Of course I wouldn't say that would lead to a broadly acceptable risk level from my perspective. But I don't know. It could happen.

Rob Wiblin: Cool. You did cheer me up a bit there, reminding me that things could develop in other ways.

Why might an intelligence explosion not happen within 8 years? [01:03:32]

Rob Wiblin: What evidence is there that this might take much longer than four or even eight years?

Ryan Greenblatt: First, I should reiterate that I don't think we'll see full automation or the capability for a fully automated AI company within four years. So I must think there's some evidence that this won't happen.

I think the first thing people should start with is skepticism about crazy things happening within a particular timeframe. So I think there's some evidence that we might expect it soon—you know, we have rapid AI progress, and I think that has hit a lot of things—but if you just think, like, how much progress have we made so far? How long has this field been going on? You start from a very outside-view, prior perspective, and you're thinking, what's the base rate for technology, then you wouldn't expect it within the next four years—because this is a crazy technology.

I even think the base rate might be more optimistic than people tend to think. I think usually when people do this kind of outside-view base rate thing, they end up with a 200-year timeline or something like that. But actually, Tom Davidson has a report where if you just apply the outside view in the most naive way, you actually get a fairly high probability within this century.

Rob Wiblin: Is that because investment in the field is growing very fast? We're just adding orders of magnitude to the compute, so if the difficulty is distributed in log space, then we actually—

Ryan Greenblatt: Resources and compute and labor. Yes, we've crossed a lot. So that's one point.

You could also just say that even if you only look at time, we haven't been doing serious AI research for very long. You could ask how long have we been doing deep learning? How long has anyone been doing AI? If you mix these together, it feels like powerful AI soon is not that impossible in some sense.

But I still think how unlikely it is a priori suggests much longer, and I think that inclines me toward longer.

In general, another point that inclines me toward longer is just that AIs are quite smart, but they're not that smart. I think we just can't have that much confidence that the next part of progress can be done.

Then there are some more specific object-level arguments.

Sorry for slightly shifting the question, but I always can't resist bringing up some of the short-timeline arguments you've made. So another point about short timelines that I think is quite important is that we're going through many orders of magnitude of compute and labor that we haven't gone through before. It's very fast now.

On top of that, the orders of magnitude we're going through are somewhat above our best guesses for the compute of the human brain. So our not-so-good estimates suggest that the compute used by the human brain might be 10^24 as a central estimate for the human brain's compute lifespan. Current training runs are about… I think the just-trained Grok 3, I saw estimates of about 3 x 10^26—so two and a half orders of magnitude above the human lifespan.

And then we might think that you first reach AI that can basically beat humans, above the human-lifespan compute—because the algorithms we usually develop are first less efficient than biology, and then they may become more efficient than biology fairly quickly.

This is in the literature, in Ajeya's old biological anchoring report, and those familiar with it call it the 'lifespan anchor.' I put considerable weight on the lifespan anchor. It looks quite good, because we reach these roughly human-level capabilities when we reach human compute. So it feels like this model has a lot of support, which suggests we might reach it in a few orders of magnitude more compute.

We're just burning through these orders of magnitude of compute very quickly. As background, I think training-run compute increases by about 4x per year. So you go through a lot of orders of magnitude fairly quickly each year. That's a bit more than one order of magnitude per year.

And, I just generally think we should put some weight on the compute-centric view of AI development—on the compute-centric view of AI development, even if it's less concrete than the lifespan anchor, we start from a fairly competitive starting point and burn through a lot of orders of magnitude. Maybe we go very far.

Now relative to this point, the pessimistic case is that we can only go so far in orders of magnitude, right? We're not far from the total number of chips you can produce. I might get the statistics wrong, but a fairly large fraction of TSMC or semiconductor manufacturing capacity is going into machine learning chips. My cached number is 10% to 20%. I hope I'm not wildly wrong, but I think it's definitely above 1%. Well above 1%.

So the interesting thing about this is that Nvidia's revenue is roughly doubling every year, and I think the number of wafers for AI or whatever—I hope I won't be too crude here; I'm not a semiconductor expert—I think AI semiconductors are increasing by a bit more than 2x per year. To keep it simple, let's make a slightly optimistic estimate, like 3x per year. It's worth noting that if you start at 20% and increase by 3x per year, it won't be long before you hit the limit.

Rob Wiblin: You use all the chips for this.

Ryan Greenblatt: Yes, you use all the chips, and once basically like all the fabs are producing AI, you can only go so fast.

Rob Wiblin: I guess at that point you're limited by how fast they can build new fabs, basically.

Ryan Greenblatt: Yes, limited by how fast they can build new fabs.

There are other sources of progress, to be clear. So even if we're limited by building new fabs, there's still the possibility of progress from hardware improving over time, and there are also sources of progress from algorithm development. Although I think a key thing, maybe people don't track enough, is that algorithm development is also driven by throwing more compute at it. So we should expect algorithm development to slow down.

So I think the pessimistic case is: AI continues to develop. We've reached insane levels of investment. When you start to hit the limits of TSMC, investment has to be even higher to justify building new fabs.

Or maybe before reaching most of TSMC's production, people just haven't seen enough results to justify these investment levels. I think Microsoft, Google, they can justify like… I think Microsoft is planning around $100 billion in capital expenditure this year. Stargate has around $100 billion committed, maybe that's over a year or two, and then maybe they hope to get more money. Stargate is an OpenAI project.

But I think once you go beyond this $100 billion regime, it's no longer just one big tech company that super believes, right? Google doesn't have the ability to easily spend a trillion dollars, especially not a trillion dollars in a year. I mean, I think raising a trillion dollars is not impossible, but I think you'd probably need very impressive results, you'd need to start attracting more skeptical investors, more revenue. You'd need to talk about potential sovereign wealth funds. It's certainly possible—

Rob Wiblin: The US government could provide that kind of money.

Ryan Greenblatt: Yes, that's true. It's certainly possible.

Rob Wiblin: Although I guess if you try to spend a trillion dollars on this thing in a year, you start running into other bottlenecks.

Ryan Greenblatt: Of course. Yes, yes. I don't currently know what the elasticity is here. I think Epoch made some estimates, like what's the largest training run you could do by 2030? I think their median, on a timeline where people continue to invest aggressively, might be that you could reach a training run of 10^30 FLOP—that takes into account data bottlenecks, various considerations about bandwidth between chips, how much chip and chip production can scale, how much capacity TSMC is building, that sort of thing.

My guess is that, independent of AI acceleration, this might be a fairly reasonable guess. AI acceleration might make this faster. But if we're on a bullish timeline, but not a bullish timeline where AI has already started to accelerate, I'd guess this is a pretty good estimate. But it's possible we deviate, because maybe the bottlenecks are harder than they expected, maybe investment dries up faster. I hope they won't criticize me for this, but I think they didn't take into account this willingness to pay.

But after 2030, I think things start to get much harder. I think if we reach 10^30 FLOP—you know, bare metal FLOP, the computation the GPUs are actually running—from where we are now, slightly above 10^26, then we reach four orders of magnitude within the next five years. So quite fast.

I think that's enough for us in principle to see the trends we already have continue, like the trends we already have. I think we have a bit of a GPU stagnation. I think we'll have GPUs start to build up. I think there was a bit of a stagnation after GPT-4, because people were trying to buy all the H100s. I think we'll see more H100 models, like GPT-4.5, and then we'll see another round of Stargate, which is another big buildout. SemiAnalysis speculates that Anthropic has a large number of chips from Amazon, possibly equivalent to around 100,000 or 200,000 H100s, so we'll see another round of 100,000 to 200,000 H100 clusters. Then we might see, I'm going to get the dates wrong, but maybe around 2028, we'll see more in the million-GPU range.

But if we haven't reached very powerful AI by then, I think you should expect things to slow down. There's a longer tail, we eat up a lot of compute around 2030, 2032, and then progress has to gradually decrease—unless we reach very powerful capabilities, or we're all wrong about how quickly you can build fabs or how much investment you can raise.

But I think a lot of the pessimistic scenarios are that maybe you do a lot of scaling, but you hit these limits. Not all.

Rob Wiblin: So for those who haven't fully kept up, the short version is that we're currently rapidly increasing the amount of compute used to train AI. We won't be able to maintain this pace, because we're currently doing it by taking chips originally intended for other purposes and using them for machine learning.

And, these companies go from using 1% of resources for AI development to 10%—and then maybe they can reach 100%, but they can't go beyond that just by taking resources originally intended for other things.

So when almost all chips are used for AI training, and almost all of these companies' resources are devoted to this, it causes the growth rate you can get there to plateau.

Ryan Greenblatt: Of course. There are probably a few small caveats about this. One thing is that when you consider repurposing chips, it's not like they're repurposing iPhone chips. Rather, there's relatively general-purpose chip manufacturing capacity, and now we're not making as many iPhones, but more AI chips. Some of that comes from building additional fabs, but some comes from slightly raising the prices of other chips, or reducing the quantity you get.

When AI is in a 20% or 30% regime, it won't have a very noticeable market impact on the cost of many other chips, because TSMC can scale slightly faster to keep up. But once AI chips reach 80% or 100%, then we'll start to see larger effects and things slowing down from then on.

Key Challenges in AI Progress Forecasting [01:15:07]

Rob Wiblin: Another thing that makes forecasting all this a bit difficult is that we're not just doing exactly the same thing year after year; we're not following the same trends year after year.

Initially, for example, we got a lot of improvement by putting more compute into pretraining. This is the thing where you dump all the text from the internet and try to get it to predict the next word. We got huge gains from throwing more data and more compute at it. But that gradually diminished, and we had to pivot to better steering, using reinforcement learning from human feedback after training. Then we did different things, which initially worked very well, but then started to plateau.

I think now we're doing a kind of self-play with reinforcement learning, where the model learns to reason better, and basically we just reinforce it when it gets the right answer. We have a very steep curve on this, but probably at some point that will plateau, and we'll have to do something different.

If I understand correctly, this makes it more difficult because we don't know what innovations next year will drive improvements.

Ryan Greenblatt: Yes. The improvements from GPT-1 to GPT-2 to GPT-3 to GPT-4—and then maybe there, there was a bit of a stall in 2023, and then a rebound in 2024—I think a lot of the improvement, up until around mid-2024, was driven by scaling up pretraining, like throwing more data at it.

There's a saying that pretraining has hit a wall. I'm not quite sure how that works. It might be that it has relatively diminishing returns, or that the marginal returns from further scaling are lower in terms of qualitative capabilities than in the past.

Based on some intuition, we have Grok 3—about 10 times more compute than GPT-4 and better algorithms—how good is it? It's a bit better, but it's also worth noting that in previous improvements—like the gap from GPT-3 to GPT-4—that was more compute. I think that gap was about 100 times the bare-metal compute. It turned out there was a lot of low-hanging fruit at the time in terms of rapidly scaling up the FLOPs you spend.

Now we've hit a bottleneck. After GPT-4, they were somewhat waiting for H100s. H100 deliveries were slow. We were a bit late in getting H100 clusters online, and maybe initially there were some difficulties in getting it to work. So GPT-4.5 was a bit disappointing. I think there are rumors that OpenAI had multiple, or at least one, failed training runs.

Rob Wiblin: Oh, wow.

Ryan Greenblatt: So it wouldn't be surprising if we see people adapting to more compute, figuring out how to use it, and the returns from pretraining going up from there.

That's pretraining. Maybe even if pretraining is diminishing, you can scale up RL, reinforcement learning. We saw this in 2024. We have o1, which are trained on RL. They're trained on tasks that are easy to verify, not on next-word prediction. We've already seen a lot of initial returns from that, and we don't know where that will diminish.

You know, we haven't seen that many orders of magnitude in RL training. For example, it's speculated that DeepSeek-R1 spent about $1 million in compute on RL. So in principle, we could scale that up by three orders of magnitude higher, comparable to the clusters people will have. Maybe slightly less than that, but roughly so.

Once we're talking about three orders of magnitude higher, maybe that will produce huge returns, or maybe it won't.

The story where it produces huge returns is that it produced huge returns from the first million. Maybe you just go further, build more environments, do more RL for big returns. The story where it doesn't is that maybe the model has some underlying capacity or potential that RL is unlocking. We've already unlocked most of that potential, and we're hitting diminishing returns.

Rob Wiblin: So to come back to it, one thing that's worth remembering for people who haven't followed it very closely is that I think we used to use reinforcement learning a lot, then it kind of fell out of fashion, and now it's back on the frontier.

This is where you take an existing model, GPT-4o or something like that, you give it very challenging reasoning problems, and you maybe let it try 100 different solutions, 1,000 different solutions. Basically you just find the cases where it ends up getting the right answer, and then you say, "Good job. Reason like that to try to solve other problems in the same way you just did." This has proven to be extremely powerful.

I think what you're saying is that this may be picking some low-hanging fruit, that these models have more capability in their weights to do clever reasoning than is initially apparent, more than we're able to get out of them. By using this process to have it try all sorts of different solutions and then find the reasoning processes that work well, we're extracting a lot of stuff that's just sitting there waiting to be picked up. But that may run out somewhat, at which point basically learning new superior reasoning techniques will be a heavier lift for them.

Ryan Greenblatt: Yes, exactly. People tried to get models to do chain-of-thought reasoning. So at the time of GPT-4, people were fully aware that you could do chain-of-thought reasoning, and tinkered with that. But I think it only worked so well. Models weren't that good at recovering from mistakes. It wasn't very good at carefully reasoning through things.

I think what we're seeing now is that with o1, o3 and other reasoning models, you can get models to reason quite well. A lot of that may already have been achieved through chain of thought—the models were already doing this to some extent, and maybe you're just making it a bit better, but the question is how much better.

Rob Wiblin: Another low-hanging fruit you mentioned is that sometimes we give these models very difficult challenges but only give them $1 worth of compute. And people have found that if you actually give them resources more like what a human would get—like $100 or $1,000 worth of equipment and salary—then they perform better in a fairer comparison.

But you can't scale that again. You can go from $1 to $1,000, but if you go from $1,000 to $100,000, that's real money. The question is whether it's actually economically useful to give a model that many resources just to solve one math problem.

Ryan Greenblatt: Yes. And it's also worth noting that even if this is economically feasible, we'll soon run into limits on the total amount of compute.

Suppose you have a model do a $100,000 task. I did a rough estimate (BOTEC), and my sense is that, at least a few months ago, OpenAI's total compute was about $500,000 per hour. So if it's a $100,000 task, then you'd be using up one-fifth of all of OpenAI's compute for an hour. So the number of such tasks you can do is limited, right? Even if some tasks are economically very valuable, you'll hit a bottleneck.

Rob Wiblin: Because that would drive up the price.

Ryan Greenblatt: Yes, that's supply and demand. Even if some tasks are very valuable, the scaling you can do is limited. And people have already shown examples of getting additional returns not just by reaching human cost, but even far above human cost.

This is actually quite good, because it gives us a glimpse of the future. I think that if you see a model can complete a task at high cost and slowly, we will soon be able to complete the same task at low cost and quickly, because costs are dropping rapidly.

Rob Wiblin: We can draw this curve.

Ryan Greenblatt: Yes, we can draw this curve. So I am hopeful about those who are skeptical of AI capabilities. Perhaps we can first demonstrate certain capabilities using extremely high runtime compute and a large amount of domain-specific guidance, and then very soon no longer need these. Hopefully before that, we can reach some kind of consensus before the capabilities are widely used.

Pessimistic scenarios for AGI [01:23:01]

Rob Wiblin: We have just been discussing pessimistic scenarios, or situations where this is thought to take a long time.

Ryan Greenblatt: I am really a bad pessimist.

Rob Wiblin: All right, let me try to present a pessimistic argument I have heard. AI may become very good at fairly narrow tasks, such as becoming a very good programmer, or doing very well at creative generation, hypothesis generation, worldbuilding, and so on. But operating and scaling an AI company requires more than this, and they will have some serious shortcomings and weaknesses, which will become limiting factors and slow progress. How reasonable do you think this view is?

Ryan Greenblatt: Here is some context: historically, as we said, a lot of the returns were driven by pretraining. Then from the end of 2023 to 2024, reinforcement learning (RL) began to play a role - much of the improvement in GPT-4 Turbo may have been driven by RL, or improved over time. This is speculation, but GPT-4o is a better base model and also benefits from better RL, and then o1, better RL. This drove benchmark score improvements on programming tasks, coding tasks, math tasks, and other more easily verifiable tasks.

Now, we are seeing some transfer. One way to address this is that perhaps you can make AI very superhuman at programming, and also very superhuman at software engineering - although it is harder to verify, at least it is partially verifiable - and also very superhuman at math. Then this will transfer to some extent.

So, the possible situation is that labeling other things is very costly. We can evaluate human performance in other domains; it is just that it cannot be automated. For example, you can evaluate the quality of a paper, and we have such a process, and it has some signal; it is just more costly. So in principle, if AI can almost transfer to writing good papers, and only needs a little feedback - they are very sample-efficient - then this could go very far.

Another thing worth noting is that I think there is a story of 'no generalization needed.' Or almost no generalization needed. So the possible situation is that you can almost make it work, or basically work, simply by directly doing RL in the process of becoming a truly excellent research engineer. We may be able to reach a point where, by scaling RL and piecing things together, we almost completely automate the research engineering of an AI company, without needing much transfer. Sorry, we need in-domain generalization - we need AI to improve relatively quickly and efficiently within the domain - but even if AI does not have very good research taste, or is not very outstanding in these respects, this can still go very far.

Moreover, even if it is only in-domain and does not require much transfer, you may still be able to do very well in other aspects of operating an AI company. So we have these research engineer AIs, on the one hand: perhaps this will make progress faster, and then we start more progress, and eventually end up with a bunch of other things. I can elaborate on this story later.

But on the other hand, if we have these research engineer AIs, but they don't have research taste, and they don't have these other things that are harder to train...

Well, part of what we mean by research taste is understanding what the result of an experiment might be, having good predictions, being able to understand what happened based on relatively little evidence. For this, you might be able to train AIs to be super strong predictors of ML research experiments.

If you train these AIs to be superhuman predictors of ML projects, they predict results based on generating lots of small-scale ML projects, and there's some transfer to larger-scale ML projects—which is not as easy to get data for directly—then from there you might get these AIs that can, likeIlya Sutskeveror Alec Radford, predict experimental results. Maybe they rely less on insight and more on mechanically generating a long list of ideas, predicting how well they'll work, and then going on from there.

I think there are basically many paths like this. Another path is that maybe you can do this within the domain, and this can go quite far in terms of automating AI companies.

An important asterisk here is that I think skeptics might scream at me: "But what about things other than automating AI companies?!" I think they might say, okay, AI R&D, I guess this is verifiable. But what about all the other things humans do where the feedback loops are slower, like being a good strategic CEO?

Rob Wiblin: Fundraising?

Ryan Greenblatt: Fundraising, all these other things. So I want to set this question aside for now, but maybe we'll come back to it later.

Then the second point I want to make is that there's the story within the domain, and there's the more generalized story—you train AIs on all these narrow tasks, they get smarter; and with just a little bit of data, they can transfer to things outside these domains or things we don't have much data for. I think this might be some interpolation between the two, because it's hard to get a lot of data on long-horizon software engineering tasks, so you might need a little transfer, but you can do a lot of data.

Then I think there's also a third story, which is that people just figure out how to do RL on harder-to-verify tasks through other mechanisms, using more like process-based checks, or being able to do RL via self-critique, constructing more detailed checks.

Interestingly, humans are able to learn on many somewhat fuzzy tasks through some concept of self-critique—how well did I do? Should I do more or less of this kind of thing? I think AIs can currently do this, but not very well. I can imagine this getting better over time.

I think what we've seen again and again is that if something is the limiting factor for capability, then there's enormous horsepower in the broader ML community, but I think in AI companies, people will concentrate on pushing that limit, whatever it is.

Turning to "inference-time compute" [01:28:46]

Rob Wiblin: There's been a shift over the past year from using almost all or a large portion of compute resources for training runs, to using a larger portion for inference or runtime compute—so instead of using your compute to make the model better at predicting the next word or more likely to get positive reinforcement during training, you're actually giving it a lot of time to think about the specific task you give it.

How does this change the situation when we automate a large amount of AI R&D? Many people think that under a reasoning-compute-centric paradigm, this will slow things down somewhat, because you will be heavily constrained: if it turns out that running the equivalent of one AI researcher requires a huge amount of compute, then at least at first you can only have 100 or 1,000 equivalents, and then gradually increase.

Ryan Greenblatt: On the one hand, if reasoning compute was not included in your previous model, then it should make your timeline for the relevant milestone shorter. But maybe the first time we reach a certain milestone, the cost will be very high. So maybe what we can do first is have an AI that can automate the work of one research engineer at our company at a compute cost of about $10 million or $20 million per year.

As I mentioned earlier, the supply of compute is actually limited. So even if in principle you are willing to automate all employees—maybe you are willing to automate 1,000 employees at a cost of $10 million per year: that is $100 billion per year, which is already close to the total amount of compute resources the company would have in this situation, right? So we can look at how much capital expenditure people are investing in total? This is roughly the same order of magnitude.

If it is more extreme than that—maybe more like $100 million per year—then suddenly, they may simply not have the money to do this, and they may not have the compute resources to do this either. And, this may not be economically viable either.

It may also be that using this kind of reasoning compute is too slow. Maybe if you can make it faster, it can work, but many methods of scaling reasoning compute may make it serially slower, which would reduce a lot of AI's competitive advantage.

Rob Wiblin: What do you mean by that? Just that because it has to do many things one after another, actually outputting an answer takes a very long time?

Ryan Greenblatt: Yes, for example, o1 is usually slower than humans at answering questions because it spends a lot of time thinking. And previous AIs were almost always faster than humans, whereas now o1 is sometimes slower, or more often at the same level. We may see that reasoning-time scaling involving more serial steps has this characteristic. I think there are various ways to mitigate this problem, so I don't think it is a huge obstacle. At least after optimization, I guess not, but at first it may be an obstacle.

So overall, I think reasoning-time compute should lead us to expect to see capabilities before they become economical, and to see capabilities before they are at small scale.

And I think one surprising thing under the pure pretraining paradigm is that you naively expect that when a capability first becomes available, you can run truly enormous numbers at that capability level. My sense is that this is still broadly true, basically because I think distillation of high reasoning compute is relatively fast, and reasoning compute I think can only take you so far.

So I think it can bring huge gains. But I think you have to quantify how large those gains are. So if this can be distilled away relatively quickly, I think we have already seen—for example, from o1 to o3 mini, we saw distillation progress quite quickly—while scaling training, it can also be distilled more effectively into a smaller package.

Rob Wiblin: So distillation is making the model smaller while performing almost as well. So you can run more in parallel, or it can give you more answers faster.

Ryan Greenblatt: Yes, I should say that distillation is a special case of a broader thing, and sometimes at least initially, it's easier to make something cheaper than to make it better. So suppose we have some new capability that's being demonstrated, we've pushed the frontier, and then we can bring down the cost of that frontier point fairly quickly. Distillation is where you use the outputs of a large model to train a small model.

Rob Wiblin: In the inference compute paradigm, aren't these two things somewhat converging? Because you're saying that if you can make it a tenth the size and it can think almost as well as before, then you can actually let it think 10 times as long?

Ryan Greenblatt: Yes, but of course the returns to inference compute may diminish quickly. This may not be the most relevant benchmark, but for example in ARC-AGI, we saw them go from about $3 or $10 per task to over $1000. So the increase in cost is about two to three orders of magnitude. Over those two to three orders of magnitude in cost, they improved performance from about 76% or 75% to 85%. And for humans, I think going from 75% to 85% is not a big improvement.

Similarly, we see that if you sample a model 64 times and choose the most common answer from it, the improvement you see from that is relatively small. So overall, I think the question is how efficient are these inference-time strategies? I think it depends on the strategy, but you may run into diminishing returns.

How much has pretraining weakened? [01:34:22]

Rob Wiblin: The overall theme here is about the arguments for earlier or later automation of AI R&D. Are there any other key factors that I haven't given you a chance to talk about that would affect this question in one direction or another?

Ryan Greenblatt: I think one important issue is that some people have shifted to thinking that pretraining may have hit a bottleneck. So I want to dig into why this might be true—to what extent it is true—and how true it is.

So I think one important reason may be that there are issues with data quality and data quantity. You know, DeepSeek recently trained DeepSeek-V3 for very little money, and they trained on about 15 trillion tokens. So it may be the case that you can train on a lot of data, but there may be very steep diminishing returns in terms of data quality.

There isn't much public evidence about how much of this is true. But if it is true, then it may be the case that once you've trained on the first 15 trillion tokens, the next 15 trillion tokens are worth much less.

If you imagine scaling up the DeepSeek-V3 training run by 10 times, then based on the Chinchilla scaling laws (which is just about how much you should scale model size and data in parallel), you would scale data by 3 times and model size by 3 times. If you do that, you'd reach 45 trillion tokens. So it may be the case that if you train on 45 trillion tokens, the next 30 trillion tokens are much worse—either because you're training repeatedly on the same tokens or because the tokens are lower quality. So it may be the case that you can stretch it very far, but due to data filtering, returns diminish from here on.

This may be one reason why pretraining scaling looked more promising in the past than it does now, because filtering was worse in the past. Maybe a lot of the improvement from GPT-3 to GPT-4 was because they trained on more tokens and got more good tokens. But now we're in a state where we can really extract the good tokens and make sure we train on them, which may lead to diminishing returns.

So I think this is one reason why we might expect pretraining to be slower, but I don't think this is an argument that the returns to pretraining will completely disappear. You can just train on worse tokens, train for more epochs, find ways to squeeze more juice out of the same tokens and get these. But you'll see progress at a slower rate.

In addition to that, there's also an option not to do pretraining, but to do more RL. We discussed a little earlier that maybe RL has diminishing returns. But even with diminishing returns, it might still be high enough, higher than pretraining, that you can put in exponentially more compute and get linearly more performance—but qualitatively, linearly more performance is a big improvement.

In addition to that, I think we're very uncertain about how real all of this is. And even if the returns are weaker than people previously thought, there may still be a lot of room to scale.

Rob Wiblin: Okay, let me try to feed some of this back to you: Pretraining is where we take a large amount of information, text corpora, and try to predict the next token. It seems that in the past, scaling up the amount of data in this process was very valuable. But it's possible that this is leveling off to some extent.

You're saying that part of the reason for this is that they've actually, to some extent, exhausted the new data they can collect. They've scraped almost all the high-quality text that humans have actually written. But also, over time, they've gotten better at filtering out the truly high-quality tokens, the high-quality content they want the model to train on a lot, and the stuff they can probably discard. So they may keep published books, textbooks, and the like, and weight them extra. Then the random junk scraped from the internet that doesn't have any particular information, they manage to exclude that.

I think you're saying that after filtering out the good stuff, the only thing they can add is really low quality, so they actually don't get much juice out of it. This may help explain why data scaling isn't adding as much value now as it did before.

But you're saying they can use the effort they put into that to improve the reinforcement learning process—this is another way to try to improve the model: reward it for successfully answering questions, reward it for having good thoughts in the process. Am I understanding correctly?

Ryan Greenblatt: Yes, exactly. I think people sometimes also talk about synthetic data. I like to think of synthetic data as a rough version of RL. What you do is use another model to generate some data, and then make that data an improvement over what it generates by default. So maybe you only select the cases where it gets it right; maybe you have it revise its answer, or you have it try to solve a math problem, then you show it the correct answer, and then you have it correct its chain of thought. You throw these into the pretraining corpus, and maybe this data has some value.

This is similar to RL in a sense, because you find good model trajectories and train on them, but it may have slightly different properties. You can scale this up, generate a large amount of synthetic data, and I think this will bring some returns; I think this will bring some improvements, and you can scale it further.

So let me try to lay out the extreme pessimistic scenario. The extreme pessimistic scenario is: DeepSeek-V3 was recently released, with a training cost of $5 million. Meanwhile, we see Grok 3 also released, or DeepSeek-V3 a bit earlier. The cost difference is about two to three orders of magnitude, about 100x. I think DeepSeek-V3 was trained on about 2,000 GPUs. Grok 3 was trained on about 100,000 GPUs, very roughly speaking. So it might be close to around 50x, roughly in that range. So we have this 50x factor in pretraining compute.

Then if you look at DeepSeek-V3, at least qualitatively, it is not much worse than Grok 3. So what is going on? What explains these returns?

There is the data scaling story, that is, maybe they have already trained on the best 15 trillion tokens, and what xAI was able to put together was not as good. So they scaled the model, scaled the data, and maybe they did not get amazing returns from it.

Another story is that the returns are weak, and from the evidence we have this is possible. Then the question comes down to, can you pivot to something more like RL?

Then I think another important story—and also a large part of my model—is that I think DeepSeek has a substantial algorithmic advantage relative to xAI, at least for now. I think DeepSeek-V3 may just be a more optimized training run; they used FLOPs more efficiently—both because of better hardware utilization and training at lower precision, which is a technique of not storing numbers with a larger representation, but with a smaller representation. It is a bit more efficient. You just do it at lower precision, but you can get similar performance.

Beyond that, I think there is actually a better tuning pipeline and a better architecture. That is, an important reason DeepSeek was able to have a better architecture is that they did small-scale training runs, which means they could run more experiments at that scale and really work out all the issues. So maybe you can work out the issues, but every time you first scale up to a larger scale, you run into some problems. There are some rumors about this happening at OpenAI; xAI may have encountered something similar. So I think we are seeing some of this.

So maybe my view is that Grok 3's actual effective compute relative to DeepSeek is about 10x, if I had to guess. Maybe a bit more than that. Yes, I think it is about 10x, because it might be 50x the experiments, but then there are a bunch of efficiency improvements that narrow this gap. I might be a bit off here, and maybe someone will point this out, but this is my guess.

Then I think, does this 10x scaling match your qualitative expectations? You know, it actually is much better, so I do not think it is completely implausible that if this is what 10x scaling looks like, another 100x could be quite significant. So here is a somewhat more optimistic scenario.

I think another important thing happening in AI progress is that many people say: "Grok 3 is not much better than GPT-4. That was two years ago, and progress is very slow. What are we doing?" But I think an important point is that people are comparing it with the recently released GPT-4 model, not the original GPT-4 version. So I think OpenAI's naming conventions here make people underestimate the speed of progress like a frog being slowly boiled in warm water.

So we had the initial GPT-4 release in early 2023, close to early 2023. That model was pretty good. And then we saw OpenAI gradually release models they still called GPT-4. So they called the models GPT-4 Turbo, then GPT-4o, and these models are all better than GPT-4—whether in the quality of the pretrained model, or in RL, or just slightly better. So we have very gradual progress while still calling it GPT-4, so people may have missed roughly an order of magnitude of progress from GPT-4 to the best version of GPT-4o, and they missed a huge amount of progress in between when making this kind of comparison.

Rob Wiblin: Yes, maybe it's easy to forget how annoying the initial GPT-4 was. I mean, given our expectations at the time, it was incredible—I was blown away at the time—but it was very picky and not very good at answering questions. It made mistakes much more obviously than now. But I guess it's called the same product, so now you just feel like it's always been GPT-4, and you forget how picky people were about how to prompt it precisely, and you had to have professional prompt engineers. As they got better at following instructions more reasonably, this has gradually faded away.

Ryan Greenblatt: Yes. A concrete example is that the initial GPT-4 could probably only do agentic tasks very poorly. Maybe it could do five to ten minute agentic tasks, such as agentic software engineering tasks, about half the time. And now GPT-4 is more like able to do one-hour tasks. So there's a very large difference in this downstream capability.

So these are some details. I think it would be very good if someone could do some very detailed analysis, trying to chart out all the qualitative improvements and how much effective compute was invested. Maybe Epoch should do this, or someone else should do this.

Rob Wiblin: Let's go back and discuss the comparison between Grok and DeepSeek more clearly. You said people look at DeepSeek, they compare it to Grok 3, and say Grok is slightly better, but not much better. It looks like it was trained on 50 times the compute, so we're getting very small returns on scaling compute input.

But you said this is a bit unfair, because I think DeepSeek, due to China's restrictions on compute access, has been working hard on algorithmic efficiency to get the absolute maximum benefit from the compute they have available. And Grok is in the opposite situation, they're trying to scale and train and grow very quickly, and they have a huge amount of compute resources. So they're almost completely unconcerned with efficient use of compute.

You said if you make a more similar comparison, in terms of algorithmic efficiency in training, then you'd say Grok 3 was trained on only about 10 times the effective compute, so the scaling isn't that large. So the improvement looks gradual, and actually closer to what we expected, and this actually isn't a sign that compute scaling is useless.

Ryan Greenblatt: I mean, I don't know. This is definitely some evidence. We should give some weight to no, it's actually 50 times the effective compute, and this is what you get. I think this is lower than I predicted. So this is an update. But I think broadly speaking, yes.

It's worth noting that it's not just that I think DeepSeek is more focused on efficiency. For example, I think Grok's algorithmic efficiency may be worse than OpenAI's and DeepSeek's. My feeling is that they're slightly behind on algorithmic efficiency, but slightly ahead on scaling to very large training runs, so they're in a somewhat different position. And my feeling is that DeepSeek at small scale may be quite competitive with OpenAI on algorithmic efficiency.

Then another thing that might be an important factor is that we don't really know, DeepSeek could practice their training runs multiple times, because they could do small-scale training runs multiple times. So it's not just that they're optimizing efficiency; if you can run training runs multiple times, you might have more signal—and if you're scaling to a new order of magnitude for the first time, maybe you just mess some things up.

Can we achieve an intelligence explosion within a year? [01:46:36]

Rob Wiblin: Are there other key pieces of evidence relevant to this timeline question?

Ryan Greenblatt: I think a rather scary fact is that we're entering this state of reasoning models where people are scaling RL and outcome-based tasks. Now people are just starting to do RL. I think the initial o1 and R1 might have been trained almost entirely on relatively narrow short tasks, without that much compute.

We have some understanding of what R1 was trained on. It appears it was trained on math problems, trivia, or science questions like GPQA questions. It was trained on possibly competitive programming or short programming tasks. But as far as we know, it wasn't trained on tasks like software engineering—tasks that require multiple steps. It may have only been trained on truly single-step tasks. It's possible they never did training involving multiple steps, at least not as part of their main RL phase.

So if this is true, you might think there's a bunch of low-hanging fruit to scale the RL paradigm and apply it to agentic-like tasks.

Beyond that, there's scaling the compute further. You can scale on the diversity of environments, and you can also scale on the amount of compute. I think Epoch made an estimate that they thought the RL on top of DeepSeek-V3 (the model R1 is based on) was about $1 million. $1 million is small money in today's AI industry.

So you could scale that by more than two orders of magnitude within the next year. Now, there might be some scaling difficulties, and I think there might be infrastructure difficulties, but in principle this is possible. So to the extent that we see huge gains from one order of magnitude, I just think, man, algorithmic progress, tweaking these things a bit, we might see crazy things next year.

Rob Wiblin: Sorry, o1 and o3 are also reasoning models trained with reinforcement learning. I imagine OpenAI spent far more than $1 million on RL for these models. So why does this suggest that? I mean, we don't think o1 or o3 are much better than R1.

Ryan Greenblatt: First, I actually don't know whether o1 or o3 were trained with more compute than R1. I think we don't know.

Some reasons you might think it wasn't trained with that much compute: one thing is that I think scaling RL is really hard at the infrastructure level, and they might not have had the infrastructure to do it at first.

The second thing might be that it's hard to quickly scale the number of environments, but eventually there's a scalable way to do it. So there might be a lot of returns.

The next thing we see is a considerable improvement in performance from o1 to o3, which is evidence that if this trend continues, it could be very fast. So we see them doing RL in relatively narrow domains, but within those relatively narrow domains, progress looks very fast. So to the extent that they can scale the domains they train on, which might be broader than that, we might see relatively rapid progress.

So I think this is still unclear. To the extent that o3 is saturating a bunch of low-hanging fruit in this paradigm—which might be somewhat true; certainly it's pulling some low-hanging fruit—then this story will disappear. But to the extent that we still have a lot of unexplored areas in RL, I think this could be one of the more compelling stories for a one-year timeline.

To be clear: I don't expect this. I think it's unlikely, but I think one path is that RL generalizes better than you expect, and it can go a bit further than you expect. Maybe by the end of the year you can almost automate research-engineer-level capabilities, maybe a bit below that—and then things could get really crazy from there.

Rob Wiblin: I see. You say it's unlikely, but it's possible. If it does happen, this is the path by which it would happen.

Ryan Greenblatt: Yes, I think the most foreseeable path would be scaling RL within a year. This argument was brought to my attention by my colleague Josh Clymer. So credit goes to him.

Reasons AI might have difficulty replacing humans [01:50:33]

Rob Wiblin: Before we continue, is there anything else to say about this timeline question?

Ryan Greenblatt: I think another source of doubt is that, of course, maybe you can make these LLMs quite smart and do well on these tasks, but don't they need a bunch of other attributes to replace humans?

You know, they need to be able to learn on the job. When humans do tasks, they are learning how to do the tasks they are doing. And AI has poorer sample efficiency, so you can put in a lot of context, and they can learn how to do better based on that, but this is relatively superficial, relatively weak.

Another thing is that AI currently has limited context length, and may have difficulty tracking context over very long projects. I think the effective context length may be much shorter than the actual context length, because they can retrieve across the entire context length, but they may not be able to easily synthesize it—because I feel that when I do tasks, I get some feeling, some level of feeling, I get better intuition. I get some overall sense of how the project is progressing. I think AI may have difficulty tracking all this context, even if you throw it all in.

Maybe there are paths around this, so I want to talk about these structural factors.

For this learning-on-the-job thing: on the one hand, we can study how to make in-context sample efficiency better. One path is, instead of having this shallower architecture where you process all tokens in parallel and they can attend to each other... the Transformer architecture is fundamentally shallow in a sense. I don't know how deeply you want to go into this, but you could change to an architecture that isn't so fundamentally shallow.

In particular, there have recently been some papers about having something more like our current architecture, where you can process activations in a deeper serial way, which might allow AI to absorb context, have more of an overall sense, and learn more context faster from what is happening.

Rob Wiblin: What do you mean by 'fundamentally shallow'?

Ryan Greenblatt: I mean, if you look at a token, the model produces some probability distribution over the next token to some extent. It can only have so many serial steps, basically because you run each layer on each token. At each layer, you can attend to all the previous layers, but you cannot attend to the previous token of a later layer. So if you imagine you have 60 layers, the 10th layer of the 40th token can attend to all the previous tokens' 9th layer and before, but cannot attend to the 60th layer of the 39th token. So this means that if the AI gets some good insights at the end of a layer, the earlier layers of later tokens cannot take that into account.

I mean, capable people are working on it, as usual—or maybe not as usual, but to some extent—and are working on ways to change this.

I think the way we currently have to deal with this is that although AIs are shallow in this sense, they are not shallow in terms of tokens. So if you have a reasoning model, yes, it is shallow in that sense, but it can also produce natural language tokens, which are its more updated thoughts, and it can continue doing relatively deep computation in this way. So it can solve a math problem with 50 steps through all the steps in natural language, even if it cannot do all the steps in this relatively more serial-bottlenecked forward pass—this is the term for transformer activations with this property.

But you might worry that natural language is not a good medium for thinking—I have some thoughts that are not in natural language—but I think in principle you could have a deeper architecture.

I expect people are working on this, and I think this introduces some safety risks, because we have this nice property that we can look at the chain of thought, have some sense of what the AI is doing, and have some confidence in that—at least potentially some confidence—because the AI is to some extent forced to use the chain of thought to make this kind of serial reasoning work.

But if all reasoning is latent, we lose this property, and now we are in a more dangerous regime, where the AI may be doing subversive reasoning and we do not even know, at least not by default.

Rob Wiblin: I did not fully understand, but you are saying you would change how the forward pass happens so that it can do more complex reasoning within it, which creates the possibility that it can scheme before actually outputting any token that we can evaluate?

Ryan Greenblatt: Yes, basically. So basically you should imagine that a transformer is not like a recurrent architecture; it is not recurrent. So your brain is recurrent: you think some thoughts, you think more thoughts, it is fully recurrent—including, as far as we know, recurrent states that we cannot easily verbalize. For example, I think people can do reasoning that they cannot easily verbalize, and some reasoning that they can verbalize. I think that right now transformers can do relatively limited unverbalized reasoning, and then a bunch of verbalized reasoning.

So it might be that you can change the architecture—this is a big change to the architecture—in such a way that they can do more unverbalized reasoning and denser reasoning.

There are some papers showing this, such as Meta's Coconut paper. I think all the existing papers are relatively weak and have not gotten that much—sorry to the authors of the papers, but that is my feeling. But it might be that you can push this architecture forward, and I think people have not necessarily really tried very hard to make this work, because there is low-hanging fruit elsewhere.

Rob Wiblin: Would changing the architecture this way be very computationally expensive?

Ryan Greenblatt: In fact, what I just described would naively use exactly the same amount of compute for generation. Because now you have to do one token at a time, you can't just loop the activations like that.

Now, if you apply gradient descent on all of it, it's more expensive at training time. It makes the computation graph for gradient descent more annoying because of some structural reasons I don't know if we should go into. But yeah, roughly speaking, I think it's more expensive at training time, and also more expensive if you're reading something.

There are a lot of cursed technical reasons why this is true, but basically if you apply this recurrence when reading, then you can only read one token at a time. A transformer can process a whole chunk of text in parallel, so a transformer is very fast at reading. So you could have a transformer read a document that's like a million tokens long, and I think in principle, if you're willing to scale compute, maybe in a minute or 30 seconds—that's very, very fast. And probably you could even do it faster, because it's basically reading the whole thing in parallel, with fewer serial steps.

But if it reads one token at a time, you have to do all the layers, then it would be as fast as generation—generation speed in a single context is more like 100 tokens per second by default, although people have demonstrated faster speeds. So there are some ways in which it's more expensive, but I think in the end the cost isn't that high, and a lot of these costs have already been borne by the inference paradigm.

When automating AI R&D, things could get very fast. Or not. [01:57:25]

Rob Wiblin: Okay. I'm curious: when we can actually mostly automate AI R&D, how do you think that process would play out? What would it look like? In what different ways might it play out?

Ryan Greenblatt: I think there's a big question, which is: suppose an AI company has fully automated AI R&D, and even the best research scientists can't add much value anymore. Maybe they add a little bit of value, but basically the company is fully automated. I think among people trying to do AI forecasting, there's a historical view that at this point you'd get very fast progress, because the AI is automating R&D, and it can run faster than humans can when doing it.

Now there's a question of how much faster. Beyond that, there's also a question of whether progress slows down. So it could be that AI is automating AI R&D, but they consume a bunch of low-hanging fruit, you have a limited labor supply, and then progress slows down, because you put in a lot of labor but you can only go so far.

Another question is whether you hit a lot of bottlenecks in compute for experiments. So you have all these AI researchers, maybe far more than your human researchers, but maybe they don't have much compute for experiments, so they don't have as much easy time producing progress.

There's a question of how fast the initial speed is. Will it slow down?

Beyond that, it might even be that progress doesn't just continue at the same speed, but that progress accelerates. The way this could happen is that you have your smart AI researchers, and they make a bunch of algorithmic progress. You use that algorithmic progress to build a smarter AI, and that AI makes progress even faster, because you can do more labor with the same amount of compute. So even if the compute an AI company has access to is fixed, in principle progress can get faster and faster.

Tom Davidson has done a lot of modeling on this, on whether we expect progress to accelerate or slow down? I'll steal a bunch of his stuff when talking about this. People call it an intelligence explosion or singularity, progress is accelerating.

I think it's worth noting that my view is that even if progress is slowing down, it could be objectively very fast. It could be that you start with a high rate of progress and then it slows down over time.

So one way to break down this question is, first we have to talk about this question, how fast is the initial speed? And then maybe we should talk about, is it accelerating or slowing down? And then from there we can say, for example, how much progress do we get in the first year?

Rob Wiblin: Yes. When you flip the switch and automate almost everything, what determines the initial speed?

Ryan Greenblatt: The very short answer is no one knows. The slightly longer answer is that we can try to have some sense based on what drives algorithmic progress.

So algorithmic progress in AI companies is driven by two main factors: labor—people working on it, people thinking of better algorithms, people implementing experiments—and compute, using compute to run experiments.

I'm now going to separate out the actual training of the final model, so we're only talking about this algorithmic progress. Historically algorithmic progress may have grown by more than 3x per year, including post-training. Maybe more like 4x or 5x per year. When I say 4x or 5x, what do I mean? Like what units? This is in terms of effective training compute: like each year it's like you can train a model four to five times larger with the amount of compute you have.

So this is the initial rate of progress. Now what I'm going to talk about is how much faster can AI researchers make this happen? This is a somewhat tricky question, because we have to answer this question: if you make the labor larger and higher quality, where do we go from there? Because we have two inputs to production—labor and compute—and if we greatly increase labor, do things just bottleneck on compute, or can you push progress much faster?

So I think one naive way to start modeling this is that we have to say, how much labor, how much AI, how good are they, how fast?

Rob Wiblin: The available compute for experiments might even decline, because now you have to use your compute to run your AI researchers, right?

Ryan Greenblatt: Yes, of course. I think this might be a small factor, because—I mean, we don't know—but my guess is that the optimal allocation of compute for algorithmic progress is: one fifth on AI labor and four fifths on running experiments. Roughly like that. So if you imagine this amount of compute, then you have less compute to run experiments, but in terms of quantity it's 80% of the compute, so it's not a big deal. So I think this is not that important a part of the picture. Even if you imagine 50/50, that's just a factor of two. So if you say, okay, you're going to spend all your compute running AI researchers and you have no compute for experiments, that's just an unforced error.

Okay, so how many AI researchers are there? People have made all kinds of estimates of how many AI researchers you'd expect at the point when you can first automate everything. You definitely have to have enough researchers to automate everything, but for various reasons, I think we'd expect that at that same quality level you'd have more labor than is needed to automate everything. Because at the first point when you can start automating everything, you probably have more labor at the same quality level.

I think inference time compute might make this different. It could be that, at the first time when you can automate everything, you can barely automate everything. But I think that's unlikely to persist. So the first time you can do this, maybe you can drive down costs a lot very quickly, using things like distillation that we talked about earlier.

So overall, my sense is, I don't know, I did some rough estimates. I'll try to go through one of them. Maybe out of the box we have the equivalent of 100 million human workers. Because we expect that the amount of compute we'll have available around 2029, 2030 is about 10^28, 10^29 FLOP. And then if you do that, you get a sense of how many tokens you can generate, and then you try to do some rough conversion between tokens and human labor.

And then okay, probably what happens is you get some from inference compute, so maybe we go down an order of magnitude because you get some from inference compute.

And then to fully automate, I think you have to be close to something like the quality of Alec Radford—Alec Radford is a famous AI researcher who had many of the most important capability insights—or, you know, Ilya Sutskever or whatever. To get to that quality level, maybe you have to spend more inference compute. I think it's useful to think in units of top research scientists or top research engineers, because I think that makes some of the conversions easier. Suppose you have a million Alec Radford equivalents running in parallel.

But then there's another factor, which is that AI can run faster. For example, they work at night, which gives them some advantage because they can do more experiments serially. So humans, due to serial time, can only get less done in a year, because they probably only work a third of the time—or for mortals, maybe a quarter of the time, and some people can push it to half the time. There are diminishing returns on focused time.

And then they can also run faster because they spit out tokens faster. There are ways to push this further. So maybe my overall sense is, maybe they're 5x faster at each time point, and then 3x faster due to running around the clock. That's a 15x speedup.

Beyond that, I think you might get another 2x speedup, because there are times when you can run a dumber AI that's much faster for certain subtasks. Humans can't do this as easily, because it requires context switching. So in principle you can imagine a human using a dumber AI to do some subtasks very quickly and switch back and forth, but you can't swap my brain state with the brain state of a weaker AI. Whereas for a transformer, for example, you can just feed the context into a weaker AI. You can train a weaker AI to work together with a smarter AI. You can even do things like stuff the activations of a smarter AI into a weaker AI, and do various variable compute scaling things at runtime. So maybe this gives you another 2x factor.

So now we're at a 30x speedup. To be clear, these speedups will offset. They will cut down the number of parallel copies we have.

And then I think you might get another 2x factor from AI coordinating better than humans. So I talked about maybe they can exchange context with weaker AIs. Well, maybe they're also better at coordinating across parallel tasks.

Let's think about this in terms of acceleration: they can complete a task that is not feasible to parallelize for humans. For example, sometimes when you do an eight-hour software engineering task, in principle you could have five people work in parallel, but you lose a lot in efficiency, and maybe you don't get any serial speedup, because humans are very bad at coordination.

But maybe AI can fork from the same point and have all the same context. Dwarkesh has a good article about all the structural advantages of AI companies, and it talks about this kind of thing. Because you can fork, maybe you can get more speedup. Let's say this is another factor of two.

Now we've reached a 60x speedup, right? So we have our million AIs running at 60x speed. Let's make it 50x speed, sorry. So we have 20,000 parallel instances of AI, each running at 50x speed. All of them are as good as top research scientists and top research engineers.

Now, how much faster is this than OpenAI? Maybe OpenAI, when building this AI, would have about 2,000 to 5,000 researchers. The number of researchers grows over time.

So naively we have 10x more parallel instances, but they are also 50x faster. Then there are some messy conversions regarding how much additional labor you put in, given compute bottlenecks and other factors, and the penalty for running in parallel, and what overall speedup you expect. So you know, nine software engineers can't do in one month something that takes nine months—you know, the same baby—

Rob Wiblin: You can't have nine women get pregnant in one month.

Ryan Greenblatt: Yes. I think one thing humans suffer from is the parallelization penalty, so AI running faster means in some sense they suffer from this less. There are more parallel copies, about 10x or so, so they eat some returns there, but you also have a direct 50x more speed and more quality. Quality also pushes into parallelism.

So I'm like, maybe we should really think of OpenAI's workforce as being 5x or 10x less than better people. So maybe it's like they have 200 or 400 Alec Radfords or whatever. Some people think it's even more extreme than that. Then if it's like they have 200 or 400 Alec Radfords, and we have 20,000 Alec Radfords running at 50x speed, I intuitively feel like things could go crazy.

But the question is just how big is the compute bottleneck? People disagree a lot about this. We really don't know. No one has run the experiments we need to find out how important this is. We only have surveys and vibes and whatever.

Rob Wiblin: What experiments would you run?

Ryan Greenblatt: This is my favorite: Google is known for having a lot of different teams, and I think at some point, someone messed up the compute allocation for some team, or there was some exogenous shock that caused some team's compute allocation to be lower or higher than it should have been. Then you can look at how much progress accelerated or slowed down when that happened. This would give you some sense of what the marginal production function looks like, of what the marginal return on compute looks like. At least this would give us some sense of what's going on.

In the case of AI, we are far from the human margin, because we have more labor. So the situation could be structurally very different, but this would give us some sense.

I think my dream is that someone goes to GDM or somewhere else, collects data on all the natural experiments that they must be running, and does a very economist-style analysis of this to figure out what the local returns look like. That can only tell us so much, because it's just the returns around the current regime.

I think something better than this would be like having a small group of researchers, and you give them much less compute. You know, if Google really likes running experiments and doesn't just give us the data—I'm picking on Google just because it's an example, but other companies could do this—they could take some researchers and split them into two groups or more, and have some researchers get much less compute, get more like the amount of compute we'd expect AI researchers to have, and, say, see how much slower they run. If it's much slower, this gives us some sense of the regime.

I think this is harder to figure out, and part of the reason may be adaptation time. So you put humans in this regime with much less compute. At first they're much slower, but they learn to work within these constraints. I think AI would have a lot of time to learn to work within these constraints, because they run much faster.

Anyway, my feeling is that the initial speedup of your AI researchers, the instantaneous speedup, would be... when I think about all these things and try to do math on the production function, maybe I do something like a Cobb-Douglas production function, with some factors, and we try to apply a parallel penalty to both humans and AI, and we normalize labor. There's a bunch of messy stuff here.

I think the inside view, extrapolating entirely from current frontier economic models, spits out numbers like this, depending on how you do the estimate, and I think my favorite constant choice is about 50 times faster. I think this may overestimate the speed. That's 50 times faster than the current rate of progress. The current rate of algorithmic progress is a bit more than half an OoM per year. So naively this gets you some truly terrifying instantaneous rate, 25 OoMs per year.

I think now people might say, "Come on, that thing you're saying, it's too absurd." I think yes, that thing I'm saying is somewhat absurd. So maybe we want to heavily discount this instantaneous speedup view. So instead of having the equivalent of 50 years of progress, or a year of progress in a week, I think maybe that's too crazy, and then I end up lowering it to maybe more like a 20 times rate of progress, maybe even a little lower than that, as my median guess.

Again, I think this is like a wild guess; we're extrapolating from a regime we don't even understand to a very different regime. No one knows. So it could be much faster; it could be much slower. Or it can't be that much faster, I think.

How quickly would an intelligence explosion slow down? [02:11:48]

Rob Wiblin: So at the point of full automation, it sounds like it could be astonishingly fast at that point. But one way I want to make this sound less crazy is to say that it's very fast at first, and then it quickly starts to flatten out, so you only have one week of this astonishing level of progress. I think another possibility is that it could even be faster—you're saying this is also a live possibility.

Do you want to explain what evidence would affect whether we expect it to slow down or speed up?

Ryan Greenblatt: Yes. Another thing I didn't mention, and I just briefly touched on it, is that I was imagining that we just fly through this human regime with no important discontinuities or kinks around the human level. But in principle it could be that we can reach human level through some kind of hitchhiking or fast following of human behavior. My guess is that this is not a big factor; it's just a one-time cost, not that large. But I think we shouldn't go too deep into this.

Anyway, how fast do we have the initial speed? Is it accelerating or decelerating? What is the limit we have? Eventually it has to slow down, right? So we have this model, and it might even be accelerating at first, it keeps accelerating, it follows this hyperbolic trajectory and tends to infinity in finite time. Eventually that has to end, when you approach the limit. We don't know when it starts to slow down. It will slow down at some point.

But I think the all-things-considered model is: things can be very fast, it can happen quite quickly. I think the estimate means my median might be that we get about five to six orders of magnitude of algorithmic progress in a year.

Rob Wiblin: And it's a bit hard to know exactly what qualitative impact this will have on how smart the model actually feels.

Ryan Greenblatt: Yes, of course. This is another big source of uncertainty. I've been doing this very econ-brained analysis, where I put everything in these effective compute units, and I do a bunch of quick conversions back and forth to labor supply to get a bunch of things.

There are many different ways to visualize this progress. I should also say that I'm ignoring some factors, like scaling compute during this period and some other small considerations. These are all counted in my one year of five to six OoMs of progress. But I don't think we should go too deep into this.

Anyway, I don't know... I have this model that is intuitive to me of initial speed, acceleration/deceleration limit, and then the limit effect, and even if it's accelerating at first, when does it start to slow down again. Does this model make sense to you?

Rob Wiblin: Yes, I think that makes sense. These are the three big style factors you're playing with.

Ryan Greenblatt: Yes. Then there are a bunch of tricky details, assuming the limit is that many OoMs away. The factor of acceleration or deceleration, and how this changes over time, you might think it's accelerating at first, and the time it stops is very close to the end of the limit. Or it could be that it's more continuous at the limit, which would have a huge impact on how many orders of magnitude you get.

But anyway, I think this is the kind of intuitive model. I think people should play with this. I think playing with this kind of model is fun. Obviously, this is both a simplified model and also has a crazy number of moving parts that we have almost no data to estimate. We're basically fitting this model by massive extrapolation from garbage data. You know, what can we do? Including garbage data like guessing how much more efficient you can be than a human brain.

So as you say, we are very uncertain, we have huge error bars. My view is that you will get some initial acceleration, and you will also be able to put in more compute. So maybe the 25th percentile is that you're progressing a bit faster than in previous years, or the 25th percentile might just be a little bit faster than existing progress. I think the 80th or 75th percentile might be completely crazy.

Rob Wiblin: So this is a question of how much it actually accelerates what the company is doing when we're able to automate things. You say the 25th percentile of this might be that it's just about the same speed as before—but the 75th percentile, which isn't even an extreme outcome, greatly accelerates research.

Ryan Greenblatt: Yes. At least soon. It might be that the initial acceleration is not that high, but the acceleration increases over time and the deceleration is relatively slow.

Also, I've been talking about this one-year timescale, but I think in a lot of modeling, most of the progress might happen in the first six months—because you already start to hit this regime of diminishing returns somewhat quickly.

Rob Wiblin: It's like the faster you go, the sooner you start hitting limits.

Ryan Greenblatt: Yes, exactly. You know, it could go in very different directions.

Six orders of magnitude of progress... what would that even look like? [02:30:34]

Ryan Greenblatt: Anyway, I've been saying six OoM of progress: what does that even mean? What does that look like?

Rob Wiblin: "OoM" means "order of magnitude," for anyone who didn't get that but is still with us.

Ryan Greenblatt: I'm sorry. I like OoM. What a great term. It's one of my favorites.

Rob Wiblin: Onomatopoeia.

Ryan Greenblatt: Yes, yes, awesome. Anyway, six OoM, how much is that? So roughly two GPTs: between GPT-2 and GPT-3 there's roughly one OoM, about 10x algorithmic progress and about 100x compute. Very roughly, maybe a bit less than that. Between GPT-3 and GPT-4 there's something similar.

So the naive qualitative model we can do is, we can say, how big is the gap from GPT-3 to GPT-4? And then we can say, we have two of those gaps: two more GPTs. And then I think, what does that mean? I think the analysis of two GPTs makes me feel more reassured. I think, two GPTs, is that so bad? I mean, come on.

I think another framing is how many years of AI progress is this? I think six OoM is roughly five years of AI progress, very roughly, maybe four years. So it's like from 2020, when we just got GPT-2 XL, to now. So it's like—

Rob Wiblin: But I think it's hard to know intuitively what that means, because GPT-2 was almost useless for anything.

Ryan Greenblatt: Or GPT-3 was very close. Yes, so that's almost useless. I think maybe what we understand even less is, what does progress above the human range mean? Like, at the time we started this crazy thing, AI was matching the best human professionals. Maybe they weren't that efficient, not that smart, but through various tricks and whatever, they could basically match human professionals. Now, how far can you go?

So there's GPT. I think another concept is trying to convert from GPT to some kind of IQ or some kind of concept. People have very different intuitions about this, but if we imagine we start at maybe 150 IQ AI, because they can automate everything... again, IQ is a garbage unit.

Rob Wiblin: It feels like it wasn't designed for this purpose.

Ryan Greenblatt: Oh no. Nothing was designed for this. We've also been abusing these economic models. I've been doing all these economic-style analyses, on economic models that were definitely not designed for this regime. Growth economics, the field we're drawing from, is not that good a field—sorry, no offense to growth economists, but just not that many people study it, and we have a lot of uncertainty about many things.

Anyway, so two GPTs. How many IQ points is that? This intuition makes me think maybe one GPT is a bit more than 50 IQ points or something. So we go from 150 to 250, and we also have more parallel copies that can run faster. That's some intuition.

Another intuition is how good are they relative to human professionals? Here's a trend that I think is good to track: if you look at programming competitions, we saw ranking progress in these programming competitions in 2024. At first, maybe AI was around the 20th percentile, roughly. Then they were at the 50th percentile, then I think o1 was around the 75th percentile, o1-preview was slightly above the 90th percentile, and then o3 was at the 99.8th percentile or something.

So there's some relationship between the orders of magnitude of computational or algorithmic progress and your rank order among human professionals. When we started this crazy thing, maybe AI was roughly like the 100th or 10th best rank order among human professionals, and then we had these six orders of magnitude of progress.

I think we can try to do some kind of conversion between orders of magnitude and rankings—like each order of magnitude might mean you're about 10 times better in this ranking. So instead of being the 1000th best, you're the 100th best. My guess is slightly more than that. Like an effective compute OoM is slightly more than an OoM of this rank order. I think no one has done this analysis very carefully. Someone should. Suppose it's slightly more than an OoM, then with our six OoMs, we get eight OOMs of rank order.

Rob Wiblin: So soon you're below one, right?

Ryan Greenblatt: Yes, you're below one, so now we're extrapolating this thing. One way to put it is that we soon reach human parity, and then maybe we have slightly more than six OoMs of progress. Or literally the best human parity, and then we have six OoMs of progress. So it's like the gap from the 1 millionth best human to a thing—because a million is six OoMs—to the best human of a thing. So it's like we took the best human, and we did the equivalent of going from the 1 millionth best to the best.

This is another qualitative intuition. I don't know how much it tells you, but you can do some extrapolation. This is shamelessly stolen from Daniel Kokotajlo's way of thinking about OoMs.

Now, we also have uncertainty on this issue. So I think if it's more like each OoM is two OoMs, then it's more like you're more than a billion times better than the best human professional.

Rob Wiblin: You go from the billionth best to the best, and then you make that jump again.

Ryan Greenblatt: Yes, exactly. That's a pretty big gap. Importantly, I think you can't make sense of the billionth best within the human range, because generalizing from professions doesn't make sense. For example, who is the billionth best person in software engineering?

Rob Wiblin: That's a stupid question.

Ryan Greenblatt: That's a stupid question. I think the 1 millionth best person in software engineering at least makes some sense now. We can start to deal with that. For more niche human professions, it makes less sense. So I think we have this crazy gap.

Another intuition I like is thinking about how big the labor supply is. So in a lot of the economic analysis I did earlier about whether progress is accelerating or slowing down, an important question is how much cognitive juice you can put into the problem per order of magnitude of effective compute, in terms of how much of the labor portion you can put into the production function?

One way is we can say, what is an order of magnitude of compute relative to an order of magnitude of parallel workers? My understanding is that our best available estimate is that an order of magnitude of effective compute is about two orders of magnitude of parallel workers.

Rob Wiblin: Is that because many people working in parallel is actually very inefficient?

Ryan Greenblatt: Yes, AI is faster, more capable, and you get more parallel copies. So when you scale effective compute, at least in the current paradigm, you have more efficient AIs that are smarter, possibly. So you can basically scale all these factors in parallel, and you can scale whichever factor is most effective.

Rob Wiblin: I see. So you can allocate your compute budget between having more and being smarter, in the most effective combination.

Ryan Greenblatt: Yes. You can bias your training run toward, are we training a larger model, or training a smaller model? There are some trade-offs among all of these, a bit complicated. There are ways to trade off between inference compute and training compute. But overall, it's like I'm going to use parallel copies to make up the denominations.

So we started with like 20,000 geniuses running at 50 times speed, and then we have six orders of magnitude—but we're actually doubling, so we have 12 orders of magnitude. That's a trillion. So now we're at 20 quadrillion running at 50 times speed. Now, I think this might be a bit misleading, because an important part is the parallel bottleneck. But if you're used to thinking from the perspective of human organizations, then I think you should think of 20 quadrillion humans running at 50 times speed as correct, and things like stepping on toes are somewhat similar.

Then in practice, maybe what I would actually expect more is, maybe it's qualitatively closer to like a billion or two billion humans that are much smarter than humans. So like humans with 250 IQ running at 100 times speed. Maybe my numbers are a bit sloppy, but I think that's more like my intuition.

Then you can do the same thing with professionals. You have to be careful not to double-count. The mechanism by which they're much better than human professionals is partly that there are more of them, so all these things cross-interact, but maybe it's like you go from the millionth-best human to the best human, and then a million times better than that. We do the same extrapolation. Maybe it's like we have at least millions of them running at 100 times speed, which is like, okay, that's fucking crazy, right?

For example, very quickly, AI will make more cognitive progress on problems than in all of human history, by a huge margin. And very naively they run at 100 times speed. So it's like if there are things you can do purely in the cognitive domain, like without needing to touch the world at all, that would take humans 10 years—that would take a team of 100 humans 10 years: okay, boom, done in a tenth of a year, using only a small fraction of the labor supply.

So I think we should start thinking, what kind of crazy technology will this process spit out? There are many things that I think in principle could be greatly accelerated, and we haven't even tried very hard.

Effort has already been put into atomically precise manufacturing. Not that much effort has been put into nanorobots, nanosystems, and so on. I think Drexler, who first thought about this, thought this would be very little labor, so thought humans could probably easily do it, but very little effort has been invested. So it seems very likely that you come out of this regime very quickly, like, you know, atomically precise manufacturing, allowing massive increases in compute supply and all other crazy things.

This is like an example. I think simulated minds and a bunch of other things could happen very quickly.

Overlooked and important technical work people should do [02:40:32]

Rob Wiblin: To summarize, it would be good to discuss what you think is the highest-priority thing for the listeners of this show, given your huge forecast distribution over different ways of running.

What are some particularly neglected and useful things in terms of technique?

Ryan Greenblatt: I think there should be more people doing control work relative to what's happening. My colleague Buck might talk more about what that would look like, so I won't go into detail there. I think it's a lot less neglected now than it was before, but it still seems quite good on the margin.

I think there should be more people spending time thinking about and researching how to train sensible AI that can make better decisions than us. Basically how do you get to a point where you've ruled out the case of a model scheming against you? How do you make them into AI that you'd be happy to hand things over to? This is a conceptually trickier area, and I plan to spend more time thinking about what research projects should come out of this area.

There's a lot of work on what I call 'model internals' that people could do. Maybe this is somewhat under control, but things like probing to make sure we can detect whether a model is taking misaligned actions. How do we do that? How do we know if it works? That sort of thing.

There's some work that might be on decoding uninterpretable reasoning. Suppose a model learned to reason steganographically in its chain of thought—

Rob Wiblin: So this is them scheming against you, but you can't tell. It's kind of encoded.

Ryan Greenblatt: Yes. Steganographically means they use different symbols and you don't understand what's happening. Or maybe it looks like gibberish to you. Maybe it looks like one thing but is actually another. Or maybe they do a lot of latent reasoning. We talked earlier about maybe models doing a lot of latent reasoning instead of natural language, being able to somehow decode that reasoning, and trying to find some way to train decoders that work to some degree and give us some awareness of what the AI is actually thinking, which I think could be quite helpful.

Beyond that, there's a lot of different work showing that AI is now very powerful. I talked a bit about how many open questions I think there are at the level of capability that has already been demonstrated. I think showing that current systems are capable and future systems will be very capable seems like it might be somewhat good on the margin, because I worry about a situation where the world is underprepared for what's happening.

So something like demonstrating a high level of autonomous cyber capability, I think that's a sweet spot, both directly relevant to threat models many people are already considering and not far from the scenarios we worry about, which involve a lot of autonomous cyber activity, which is actually a key part of the threat model. So it might bridge this divide in a nice way. Especially focusing on the best demonstrations we can achieve in this area.

Another important area people should work on is what I call model organisms: trying to produce empirical examples of misaligned models to study how likely this is to happen and provide evidence. So like does misalignment emerge in XYZ situations? Does reward hacking emerge, and how does it generalize? Things like the alignment faking paper and various continuations.

I think part of the hope here is to gather evidence. Part of the hope is just to have something to iterate on along with the technology. Even model organisms that aren't very convincing to the world or might not provide any evidence about misalignment, if they're similar enough that we can experiment on them, that could be very useful.

Rob Wiblin: Because you can try to develop countermeasures that work in the model organism case and then hopefully they'll transfer?

Ryan Greenblatt: Yes. I think one of the key difficulties with alignment is that usually we solve problems through empirical iteration. To the extent that we are not coordinated, our tests are deceptive, so if we could find some way to bypass this in advance—or just be ready to build it at the last minute and then do a bunch of iterations in those cases—I think that could be quite helpful.

What is the most promising work in governance? [02:44:32]

Rob Wiblin: Okay, that's the most promising thing on the technical side. Is there anything that stands out on the governance or other angles?

Ryan Greenblatt: Yes, I think there is room for all sorts of non-technical interventions that look quite good. I have a hard time having very strong opinions about these things because I haven't spent that long thinking about it. There is a lot of work.

We've discussed a lot of conceptual points here, and I think there are people studying all these details, trying to better understand takeoff dynamics, trying to better understand other considerations that might arise besides misalignment. Like how much should we worry about human power grabs? How much should we worry about other problems? I think there are some of those.

I think there is a lot of work just acting as an intermediary between the very deep technical AI safety world and the policy world, and doing some translation.

There are many specific regulations that could be beneficial. I think making the EU code of conduct better seems good. The EU AI Office is hiring, so you could work there. I think there may be other regulatory strategies that could actually be good.

I think there are some things related to making coordination more likely or assisting coordination that could be quite helpful. Like improving compute governance regimes so that the US and China can verify various claims about current training processes. I don't have a strong opinion on how promising this is, but I think it's surprising how few people are doing this, and it's surprising that it's not coordinated. So maybe someone should do this, because it could be a fairly big problem.

Beyond that, I think there are just a lot of people in positions where they are just trying to provide technical expertise; they are in positions where they are building skills, they are preparing to have more direct impact; and later, as things get crazier, preparing to do something.

Another one is general defense. So we talked earlier about AI takeover scenarios. A bunch of the AI takeover scenarios I mentioned involve, for example, bioweapons. Just generally improving robustness to bioweapons seems helpful. The extent to which it helps is complicated, but I think it helps somewhat.

Similarly, making the world more robust to AI hacking. I think it helps somewhat. I think it may be less leveraged than other things, but interventions that direct more resources to those things seem good from various angles and potentially different misalignment assumptions. I think those things could be very meaningful even with no misalignment risk at all.

Rob Wiblin: Yes, I guess because misuse is also a problem.

Ryan Greenblatt: Yes. Beyond that, there are many different safety efforts that might be good. So some of the threat models I discussed involve various outcomes, like a model exfiltrating itself. They involve models being deployed internally in a rogue way, bypassing your security and potentially using a bunch of compute it shouldn't use.

I think delaying these things through security mechanisms seems good. Also security to prevent human actors from stealing models could increase the probability of delay, increase the probability of reduced competition and more caution.

Ryan's current research focus [02:47:48]

Rob Wiblin: What will your research focus be in the coming months?

Ryan Greenblatt: Right now I'm doing a fair amount of planning and conceptual work, and then the plan is to spin out a bunch of projects from that. So I'm thinking about questions like: in this scenario, what should you do, where you're a responsible AI company three months ahead, and political will is very low—what are some potential alignment interventions? What path should you take? How should people prioritize?

And then trying to maybe come up with concrete recommendations, and also just figuring things out at the margin, with basically the aim that it looks like Redwood is just trying to formulate an overall plan, and then derive some insights from it, and has had reasonable luck. I think control came out of this process. I have some updates from thinking about this process more deeply. That's one thing.

Then I'm doing some demos, or trying to study how big a problem reward hacking is right now. Just recently, we've seen RL work, whereas before it wasn't doing that much and wasn't scaled that far. So a natural question is: how much reward hacking do we get? How bad could that be? Under what circumstances is it more or less bad?

There was some prior work on this, but I think now this is really going quite far, and we might expect that we could see very bad reward hacking, we might see threat models driven purely by reward hacking, all the way up to very bad outcomes. In principle, things like deceiving humans on a large scale or trying to seize assets could generalize out of reward hacking.

There are also stories where reward hacking leads to very harmful misalignment, because you start with an AI that doesn't follow your instructions, and that in some way involves the AI scheming against you, even if it's not as much a thing of direct control over the oversight signal as we might have.

Rob Wiblin: Okay, you and your colleagues have written a lot on the Alignment Forum, and you have aSubstack. What's that address?

Ryan Greenblatt: Substack? redwoodresearch.substack.com. Our Substack doesn't have a short URL, I'm afraid.

Rob Wiblin: So if you want to pull out some threads of things you're talking about here, there's a good chance there's an article or blog post by you or a colleague that can elaborate in more detail.

Ryan Greenblatt: Sure.

Rob Wiblin: I think you listed out a huge to-do list there. If there are people in the audience who can help with these problems, then I imagine time is tight. We can use all hands to push all these agendas forward, and hopefully things will get better.