Dwarkesh Podcast
AI researchers debate how close we are to recursive self-improvement
New episode with John Schulman , Beren Millidge and Charlie O’Neill .
I got together with some of the most insightful AI researchers I know who are at the openish companies, because I wanted to hear the details of what's actually happening at the frontier and what comes next.
Watch on YouTube ; listen on Apple Podcasts or Spotify .
Sponsors
Antithesis helps you trust your code. As agents generate more and more of your software, the bottleneck shifts from your engineers actually writing code to verifying it. Antithesis does that testing for you. Ron Minsky, who co-leads Jane Street’s tech group, told me that Antithesis was able to help his team shake out bugs in software that had already undergone heavy review. If you want to see how it fits into your development process, go to antithesis.com/dwarkesh
Grok Bot has been a great way to hand off tasks. My team uses it as a producer: whenever my editor posts a rough cut of an interview in Slack, Grok Bot opens the transcript on its own computer, matches my notes to the exact moments they refer to, and uses a file of my preferences to suggest edits. Then it sends me its top clip candidates so I can review everything from my phone, which saves my editors from sorting through hours of footage. Try Grok Bot for yourself at x.ai/bot
Jane Street just launched its most ambitious competition yet: design a protocol-emulator ASIC. Basically, if you have a chip you want to test outside of a live system, you should be able to connect it to your design and have it simulate realistic traffic. Jane Street wants general-purpose, reprogrammable designs that can work across multiple protocols and remain useful as new ones emerge. The most novel submissions will actually get taped out, and the winners will receive a physical copy! The competition is open until January 18, 2027, and teams are encouraged. To get started download the template code at janestreet.com/dwarkesh
Timestamps
(00:00:00) – Steelmanning the case against RSI
( 00:18:39 ) – What’s driving the Chinese labs’ progress
( 00:28:06 ) – How will automated AI researchers be trained
( 00:33:51 ) – Will long-horizon RL elicit AGI?
( 01:00:33 ) – How much progress is explained by data?
( 01:18:03 ) – Why is RL working so well?
( 01:24:54 ) – Move 37 and entropy collapse
Transcript
00:00:00 – Steelmanning the case against RSI
Today, I’m chatting with three of my AI researcher friends from whom I learn a lot every time we talk. They also happen to be at somewhat open-ish labs and companies, so you guys can actually say things on the record. I’m joined by Beren Millidge , who is the CTO of Zyphra , which is developing open source models. John Schulman is the chief scientist at Thinking Machines , previously a co-founder of OpenAI, and led the RLHF work that led to ChatGPT. And Charlie O’Neill is head of model training at Baseten .
The first question I have: If we’re in 2036 and we don’t have billions of crazy superintelligence s running around that have radically transformed the world, what is the most likely reason that doesn’t end up being the case? Other than exogenous political shocks, or there’s a war, or they ban AI or something. What is the most likely technical reason that 2036 isn’t a crazy alien superintelligence world?
There’s been a classic thing, almost like Moravec’s paradox , where we think of the AI as, “If it can do this, it’s going to be amazing.” If it can solve these hard maths problems, if it can win at chess, blah, blah, blah… Then it solves these things, and it’s not that impactful. Obviously, it’s somewhat impactful, but not everything.
If somehow that continues, and there’s never the true spark of generalization that occurs, I think that could lead to the AI just being extremely good at everything that people put into a benchmark or put into an environment. But there’s still some persistent sim-to-real gap which is somehow blocking everything. I think this is unlikely. We do actually see this kind of generalization even from RL in practice already. But if it is just ridiculously hard to generalize meta-learning, plus we don’t solve continual learning and it’s just super hard and impossible… This would be my default scenario in that case.
I agree with that. Humans have a lot of advantages over models now. Each time a new model comes out, it’ll catch up in some of these areas. But you end up getting bottlenecked by the places where the model is weaker and where it has worse judgment, or the models can’t check themselves well enough.
There’s this cycle that keeps repeating where a new model comes out and people are blown away and they’re like, “This is it. This is AGI.” But then they use it a bit, and it starts to feel dumb after a month or so. That cycle just might keep going. It’s hard to predict how many times it’s going to repeat.
Right now, you don’t get explosive growth in capabilities because you still get bottlenecked enough when you’re trying to do research and engineering. Even if the model can write way more code than a person, it doesn’t make you 100X more productive. So maybe there are just more of these cycles than we would expect.
For me, it’s a question of how far off the global optimum of “a learner you could have on a chip” is from the transformer + RL, basically the current recipe. People imagine that once you have an agent which is better than all humans at AI research, even if it’s 0.1% better than all humans, then the fact that you can run hundreds of thousands, if not millions, of these in parallel — and you can run them much faster as chips speed up — is going to outweigh every other bottleneck. You’re eventually going to hit this very fast takeoff with regards to self-improvement .
I could imagine that if we continue along the trajectory that we’re currently on with that paradigm, where it’s basically self-attention, RL, scaling up RL environments … Think about what happened with Moore’s law . We had this very nice straight line and that held for a really, really long time. But there were so many discrete discontinuities and innovations that had to happen to keep that scaling law going . The same thing has happened with LLMs . We had this pre-training scaling law , and then that was hitting diminishing returns. Then we came up with RL and solved that, and then we got this new diminishing returns curve to hit that made it keep looking like a straight line going up.
So if it requires another one of those discontinuities to solve, I’m not sure that the current method of training LLMs with these RL environments, even RSI -targeted RL environments, would be able to discover that discontinuity. If not, we’re probably going to hit this asymptotic curve.
But do you think the discontinuity will be harder than anything that’s come since 2012?
If we had the answer to that, we’d kind of have the ability to implement it. But maybe we should distinguish between a discontinuity which adds to the current paradigm, which is cumulative — there’s something beyond the RL that we have to discover, and maybe they’re capable of connecting the dots in that straight line — or, again, how far off the global optimum are we? Do we have to go back and throw out gradient descent and neural nets in general? I don’t think, if you continue to scale up the current paradigm, an LLM, no matter how many LLMs you’re running, is necessarily capable of discovering that if it’s too far away.
The only hope really is if deep learning just can’t get us to an AI which can at least dominate human research and human development, including the human ability to come up with new paradigms and so forth. Or, I don’t know, maybe humans would also never have discovered the next learning architecture. But to the extent humans could have discovered it eventually… But it just seems like… If you just look at the progress that’s happened since 2012 till now, and you just continue that on —I know it’s just been powered by huge amounts of compute scaling and so forth— it would be weird if it just didn’t get to the point where it could dominate humans, at least in R&D, especially over the next few years.
Ryan Greenblatt was on the podcast recently. He made this point that I’d be curious to get your thoughts on. You could imagine, as AIs get more and more capable, that they’re capable of making progress on simulations which incentivize getting better at not only AI R&D, but at science generally. This is a thing that all the labs are targeting and many startups are targeting.
Another intuition pump is if you look at the Elo score of chess bots since the ’80s. There’s a very linear increase in Elo over time. But there’s this huge discontinuity as they cross the human range, from human experts always winning against AIs to human experts never winning against AIs, as this linear increase in Elo happens.
I agree with your point that so far, AI capabilities have not been that big of a deal in terms of their end economic impact in the world. But that is because they’re slowly rising in Elo relative to humans.
I agree it would be very surprising. The only way for this to not happen is if, as you said, it somehow asymptotes just before. Because we’re already pretty close, in my opinion, to where we’ll start crossing the human Elo score. So we’ll need to asymptote before that. That’s the only way — in this scenario you pose where somehow we’re sitting here in 2035 and everything is normal — for this to happen, I think. The only other way is there’s some dramatic regulation on AI. This is what I see as the most likely way for this scenario to happen, actually, rather than a technical thing.
I think there’s different kinds of research. There’s research in the autoresearch style where the objective is already specified very cleanly and you’re optimizing that objective. I think everyone is picturing that if we continue along this path of making pre-training loss go down and making our environments have the reward on them go up, that’s going to lead to improvement.
But maybe what Ryan is talking about is this much more open-ended type of science which is required for paradigm shifts, where we can’t specify the objective, and the AIs are definitely not able to specify that objective either. We have to be really, really careful about how we specify objectives for any of these things.
Maybe your point is that the nature of the breakthroughs that have happened since 2012 is that we have found… In 2012, people weren’t saying… I’m assuming, I don’t know, you guys were there. Or at least John, you were there. But I was not.
Actually, John, I’m curious for your wisdom of the ages, or wisdom of being in the trenches way back when. Presumably, a big breakthrough was realizing that next token prediction is the… You wouldn’t have thought that the nanoGPT speed run is the thing to be optimizing for in 2014. But now that we have come to this new paradigm, you would think to do a speed run on that and have AIs get really good at that.
But maybe there’s a next inner loop to optimize that the AIs wouldn’t anticipate. There’s an outer loop of revenue or something that eventually should be strong, but it’s a very slow outer loop.
In fact, I remember in the early OpenAI days having the intuition that just minimizing log loss wasn’t going to get you to intelligence. Because the important bits are accounting for such a small fraction of the loss that it was going to be overwhelmed by noise. So just training a language model on next token prediction wasn’t going to learn the interesting things you want it to learn. We needed to craft better objectives that would put more emphasis on the important things.
You can make all sorts of arguments for this. You could say, “Oh, humans probably don’t learn how to model everything in our environment. Most people can’t create a photorealistic reproduction of some kind of scene they’ve looked at. So we must need a better objective.” But then it turned out that it just worked anyway.
As you were pointing out, the inner loop, even in current AI research, of post-training benchmarks or whatever, doesn’t necessarily translate into what users like.
Oh, yeah. The whole field relies a lot on generalization and it’s very hard to predict when you’re going to get generalization, or when you’re going to get some kind of out-of-distribution generalization . We know that if you train on the task you care about, you’re going to do better. But the most important advances are often types of generalization that we have no right to expect.
For example, from just pre-training on this very naive next-token-prediction objective to various tasks of interest that require understanding of the input in some deep way, or learning some skill from pre-training that’s very rare and not heavily represented. Then also generalization from these verifiable tasks to less verifiable ones, this is also a type of generalization that there’s no reason a priori to expect.
This is an interesting question, because one intuition pump you could have for why you would see some sort of singularity very rapidly — without even scaling up the inputs to AI progress that are not just AI labor — is that before every single 7-figure experiment you run, you spend an equivalent amount of compute on AI labor. So you just have automated versions of you guys spending a century thinking about what is the optimal experiment to run, doing small-scale ablations , developing literally a century’s worth of theory, going back even before deep learning.
Before you decide what experiment to run, you’re doing extremely optimal setting up of the experiment. Then you do a century of thinking after the experiment is over, where you’re analyzing what happened and what the next experiment to run is.
If you think hard enough, you probably could have expected some of these things beforehand. There is probably some very clever way to do a small-scale experiment that’ll let you build the theory that then will generalize to the large-scale experiment. So I would expect that we’re nowhere near the ceiling of how well you can do research.
I would imagine a future where AI is doing a lot of analysis and theory building, spending a comparable amount of compute to the amount that you’re spending on the experiments themselves, doing various kinds of analysis and building a theory around what we’ve seen so far.
I think there are really concrete examples of this when the objective is well specified. All thinking can do is update your posterior based on the bits that you’ve gotten since you formed your prior. You can’t gain any new bits from just thinking. But when the objective is well specified and there is this data sitting around, I imagine there will be this big speed-up in the current paradigm we’re in.
A good example of this is if you got an AI to think about the Kaplan scaling laws . An AI at this point would have noticed, “Oh, they’ve just taken these intermediate checkpoints and didn’t account for the annealing , and so this is wrong.” That would have been caught years earlier. We would have cut off a year or two of progress just from that observation from an AI.
Again, once the objective is well specified, which is lower pre-training loss or whatever, there are many, many good examples where if you just thought about it a bit more, you would have been able to cut down significantly on things that you’ve done. So muP , and how learning rate scales with model size, and realizing that model width is important in that as well. I feel like you can really back out a lot of these things and cut off a lot of low-hanging fruit. I would imagine a 10x speed-up if our thing is just, “Maximize the objective we’re currently on.”
But I don’t see how that generalizes at all to coming up with the right objective in the first place. Just thinking doesn’t necessarily buy you the right objective in the first place.
I think this is really the key question for any kind of very rapid RSI from current AIs. How well can AIs generalize to learning their own objectives? To have any kind of self-propelling automated loop, we need the AI to propose objectives, optimize them, figure that out, propose a new objective, and have this not go off the rails at any point for a long, long time.
To come back to Moravec’s paradox, there might be a case of Moravec’s paradox where we think this kind of autonomy and being self-encapsulated — so we can think of what we should do ourselves and then go do it and have this loop — is super easy because we always do this. Obviously, evolution needs to create creatures that can survive by themselves for long periods of time. And this just might be something that for some reason is really hard for the AI, in the same way that locomotion stuff is really hard but math is super easy despite being super hard for us.
But doesn’t the time horizon increasing suggest that that’s—
Yeah, exactly. This is another possibility, but I agree, there’s no obvious evidence for this. In fact, the fact that our agents are now super persistent and it’s quite easy to do this is kind of evidence against this. But this would potentially be one of the reasons why we just don’t get this immediate takeoff, if this is hard.
If you look back from 2012 till now — or maybe from when you started doing your research till now — what part of all the innovations that have happened since that time, including purely engineering ones, including purely conceptual ones, seems like the thing that would be the last thing humans would have to do before AI totally automates AI R&D?
Probably just iteratively asking the right questions. If you can get the AI to do any experiment, you still need to decide what experiments to do. Right now I think AIs are not very good at this compared to coding the experiment. Whenever we talk about research, they propose a bunch of miscellaneous things which are very, very tiny steps.
Or even going from DeepMind ’s approach of, “We’re going to solve intelligence by learning to play games at a superhuman level,” to one random researcher like Radford being, “I’m going to try and just predict the next token of a very wide swath of data”… Even once Radford had discovered that, it took a while before people decided to scale it up, because we had to come up with the idea of scaling laws and the fact that you could very reliably predict these things.
I would say that the last job for humans, or the role for humans that’ll last the longest, is defining the objective and deciding what we actually want. In that vein, something like deciding how the AI assistants should behave, or what it means to be helpful, or what the objective is when we’re doing RL from human feedback, is one such thing. Then later, defining constitutions and model specs is another one. Even if the AIs can do all the technical work, we’ll still have to do a lot of that and decide what we actually want.
Alignment is sort of the answer. But alignment itself can be decomposed into specification of the objective, or figuring out what the right objective should be, and then actually achieving or optimizing the objective you’ve defined. I think the first one is not going to go away anytime soon.
If I think about a post-training team and why you need a lot of people on the team, it’s just because there are a lot of different areas where you have to figure out how the model should behave. It would be very hard to automate the whole