Anthony Aguirre | Tools or Agents? Choosing Our AI Future

As you imagine very powerful AI technologies, I think there's a level at which we are going to have to make a decision: do we want them to be our tools, or do we want us to be their tools? It's going to be a little bit of one or the other. And for each of them, there's a bunch of thinking behind why we would want to go down that road at the abstract level, but it's really nice to make those concrete. What does this actually look like? How would that play out over time? What would the actual technologies be? So it's been really cool to see that in both cases, and inspiring that it's sort of believable that there is this nice future in both directions.

0:32
The bad news is that part of the incentive structure for AGI is not curing cancer or even producing immortality, or producing the super‑beings that are going to solve all of our problems. It's replacing people. And I think the real economic driving force behind AGI, I personally believe, is that it is the thing that allows you to not have to rescramble all the tasks in your company, not have to figure out how to automate certain things and have the right controls over them, but to just say, "No, I'm not going to hire this person. I'm going to hire this AI system instead."

1:12
Thank you so much for joining, Anthony.

1:14
Pleasure to be here.

1:16
Yeah. So I thought we'd start by talking about why this project happened, because this was a project that was funded by FLI, and it would be interesting to hear a bit more about your reasoning as to why you wanted to see—we call it an advanced world‑building project—meaning that we actually tried to find quite senior people within science, tech, and governance, and so on, to give input on building out world‑building scenarios around different AI futures. It would be interesting to hear your thoughts on why you thought that was a project worth supporting. And then also, if you have any thoughts on—today we're going to mostly talk about this Tool AI scenario, which is one of the scenarios, but we also did a d/acc scenario—why those two scenarios, if you have any thoughts on that?

2:03
Great. Yeah. First let me say I thought you guys did a great job on it. It was so interesting to be a part of it, and I'm really impressed that you got such great thinkers to participate and give you feedback—just excellent work. The motivation for this, on my and Ethali's part, is twofold. One is that with Foresight we've long thought that one of the things we really need are different and positive visions of how the future with AI could be. We've been thinking about positive futures in AI for quite a while because we don't want to just bumble into wherever the incentive structures we happen to be in push us. That's largely what is happening now.

2:40
I think we all believe that we should have some visions of the future—where we actually want to go—for reasons that we endorse, and then direct ourselves in that direction, rather than, "Okay, let's just do what seems like the thing that we have to do now." If you don't build those visions, they won't necessarily happen by themselves. You'll end up on the default trajectory. So we've always been interested in trying to develop those visions. As far back as the first Augmented Intelligence Summit that we did with world‑building, we've been into world‑building—as you guys have—for a while. So world‑building was exciting to us, and we wanted something that wasn't just a brief sketch, but something really in‑depth, where a lot of expertise and effort went into fleshing out how the actual history plays out and what the technologies look like—and you've done such a nice job on that.

3:28
There's the world‑building side, and then I think the other side is this ambient sense that there's just a single path with AI: we build bigger and bigger AI systems; they get more general; they get more autonomous; they get more everything. And we can either just go down that path and hit AGI and hit superintelligence and—oh my god—what are we going to do? It's very hard to figure out how the world makes sense with superintelligence, or how it's going to go well for a lot of people with AGI; how we're going to control it; how we're going to have alignment and all those things. But that's the picture we're stuck with: either go down that path and end up at those things, or we can be Luddites and stop technology—and then somebody else is going to race ahead, and there's no stable way to do that.

4:13
So the picture is: we can either forge ahead down one path, or we can stop. I think that's just not true. Part of the hope here was to ask, "Why do we think that, and what should we be thinking?" Are there actual decisions and design choices that we've implicitly been making so far—and that we will be making in the future—that choose different directions in the technology? Some of those things in these scenarios got chosen for us, in the sense that large incidents happened and directed attention one way or the other. But lots of them were also choices that we made: "Oh no, we would prefer to do it this way. That actually makes a lot more sense for our society."

4:50
We're keenly interested in saying—and in the truth—that there isn't just one direction. There's not just one path we can go down. There are many. Let's look at a few of them that are quite different from the one we're on now, and really develop them so that people can understand that there are choices and alternatives, and that we have a little bit more agency here as a species and a society than just "stop or go."

5:13
Yeah, I think that's a really interesting framing, and obviously we've talked about world‑building—I think literally on this podcast before.

5:20
Totally. Yes.

5:22
And I guess the interesting thing about Tool AI and d/acc is that they're, to some extent, memes that are present if you're in this sphere. People who are interested in these things have probably heard of them, but it's interesting to take them to their extremes—or just, what does it look like in reality?

5:39
Yeah, because they're each kind of a philosophy. There's a bunch of thinking behind why we would want to go down that road at the abstract level. But it's really nice to make those concrete. What does this actually look like? How would that play out over time? What would the actual technologies be? So it's been really cool to see that in both cases, and inspiring to see that it's believable that there's a nice future in both directions. And it's good that there are two of them. We're not just saying, "Oh, this is actually the future we should do instead."

6:22
You know, we're not going to end up in any future that we imagine right now. That's not how the future works; it's very unpredictable. The crucial thing is to get more than one option on the table and see what we like about the different things, and have them be draws for us when we're making decisions. We don't have to pick the party line. We can do things now so that, insofar as we run into some of the things that happen in these pictures—if we run into large‑scale incidents like we did in the d/acc scenario (well, in both scenarios), or if agents inevitably screw things up a whole bunch and people realize, "Oh crap, nobody has responsibility exactly—or we don't know who has responsibility—for the actions these things take. How are we going to think about liability?"—running through these scenarios and playing them out lets us think now about the systems, ideas, and work that we need to have in place for when some of those things start to happen.

7:17
Yeah, I think one thing that becomes really clear is the benefits and trade‑offs with the two different versions. You can see there are massive benefits with d/acc in terms of resilience, but maybe there are some trade‑offs in terms of potential for flourishing, and things like that. There are trade‑offs. Maybe we dive into that a bit for the Tool AI one specifically. When you think of a Tool AI future, maybe we start with what excites you about it, because arguably you wrote the paper "Keep the Future Human," and that aligns to some extent with a Tool AI future. We used the diagram—the shield diagram—you created in that paper to clarify what we mean by Tool AI. When you think about a Tool AI future, why do you think that's the direction we should be steering toward?

8:07
Well, I think tools are basically what we've built with our technology most of the time up until now, and they have served us really well. Most of the things that we like about technology, and that have brought huge amounts of well‑being, value, productivity, and safety, have been tools. If you actually ask most people in AI development, "What are you building?" they'll say something like, "We're building cool new tools to do X, Y, and Z." So they're already bought into this paradigm.

8:41
The exception is AGI as the combination of elements of tools—intelligence and generality—but attached to high levels of autonomy, and that really makes it quite different. As I talk about in that paper, it's that triple combination that makes things very different from the tools that we have right now, and also creates a different overall dynamic. Rather than having tools that people use for their own benefit and empowerment, and that are complementary to the skills they have—like most tools, which extend the skills you have—these are replacements. Once you have autonomy, generality, and intelligence—the center of that shield diagram—right now that's where people are. Once you can put an AI system there, you can put it in and take the people out. That's the picture.

9:36
So I think the upside of tools is the upside that tools have always had: they let us do more powerful things that we want to do, and give more power to individuals and groups of people. All of the science and technology that we actually want to get out of AI—letting us do things that we currently can't do; letting us do certain things much faster; extending the capabilities that people currently have to ones they didn't have before—those are exciting and positive things that Tool AI can give us. The thing that it avoids is replacing people with the AI system. Most people don't want that. They're doing stuff they want to do, and they don't want to be replaced in doing it; or they're doing stuff they have to do because they need to make a living, and they also don't want to be replaced doing that either. People are doing the things they're doing for reasons and don't want to be replaced with something else that does it instead of them. They want something that empowers them.

10:42
So the benefits are quite clear. There are trade‑offs if you lean out of autonomy and into the other parts. The things we hope to get out of AGI—systems that can turbo‑charge science or let us achieve higher productivity and lots more of what we're trying to do—I think we can get all of those out of Tool AI. At some level, AGI doesn't do anything that people can't do. If you talk about superintelligence, that's a little bit different. But if we talk about AGI—the thing that is human expert‑level at all the things humans are expert at, and just does it autonomously and much faster and much cheaper—that doesn't necessarily allow us to do things we can't do. It allows us to do them more cheaply and replace people doing them. Insofar as it does let us do things we can't already do, tools would allow us to do those things too. Insofar as there are things we can't do because they're at such a scale and require so much speed that we would need AI to do them, tools can do those things too.

11:56
So I think there are trade‑offs with AGI, but the primary one would be that it would be able to do things faster than without it, because autonomy lets you take the human out of the loop, and humans are kind of slow. If you can have a thousand AGI systems running at 50× human speed, you're going to get a lot done. It might be the same stuff that humans would have done eventually, but you can do it a lot faster. So I think we would get to the same sorts of outcomes, but it could take longer if we had tools instead of AGI. Some people don't like that. There is a trade‑off there between how much we race to get things as fast as we can versus getting them in a way that works better for more people and is safer and more controlled and all of those things.

12:43
With superintelligence, it could be different. There may be things that take a very long time to get to. If you really want great nanotech next year, you're not going to get that with Tool AI—or with AGI, for that matter. If you're dead set on immortality in the next two years, you're not going to get that from Tool AI or from AGI. I think it's only going to come from superintelligence. The downside, of course, is that it might lead to disempowerment of humanity, or extinction of the human species, and all those things. So the trade‑offs are real, and we should take them seriously, and that goes in both directions. We should see the things we would give up by not having AGI and superintelligence, but I think those things are less than advertised. The things that are generally advertised—like "AGI is going to help us cure cancer"—what we really need are tools for that. The things slowing us down from getting cancer cures are not things that AGI will give us. They're things that powerful AI tools—and lots of other stuff like investing in the data sets and changing regulatory structures—will address. Those are the things standing in our way, not smarts.

14:02
Well, great. I feel like you covered a really large chunk there, both in terms of the benefits and the trade‑offs. One thing that would be interesting to dig into a bit more: when we were working on this scenario, one interesting question was how stable Tool AI is. Even if we were to achieve a Tool AI future in the next 10 years, do you see that as a stable path? Do you see it as a temporary thing until we understand AGI better? What option space do you see if we reach a Tool AI future—what happens then?

14:41
Yeah, I think this is really hard to know. I worry that there's some instability in the sense that people are lazy. If there's a system that says, "Oh, don't worry, I'll do all of that stuff for you, and you can just trust me—I'm going to do it," there is a real temptation to do that. This is going to be a defining question for humanity: how much we want to retain agency and control. There's a level at which you can't have it all. You can't both delegate things to agents and have them do all the work, and also really have meaningful control of what they're doing or meaningful agency. Once you delegate things, you've lost some involvement, understanding, and control of how it turns out.

15:29
As you imagine very powerful AI technologies, I think there's a level at which we are going to have to make a decision: do we want them to be our tools, or do we want us to be their tools? It's going to be a little bit of one or the other. There are two competing drives. People want to be in charge. They want things to go their way. They want the things they want rather than some random other thing. That means they want to be in control; they want to have a say; they want agency and freedom. On the other hand, they also want things to be easy. They want things to get done quickly and without a lot of work. If they're an employer, they want to spend very little money getting those things done. If they're just trying to make something happen, they don't necessarily want to employ a bunch of people and have it be a lot of work. They want to just have it happen. So there are pushes in both directions. Honestly, I think it's going to be a central issue of our civilization over the coming decades—how that plays out. I don't think there are any easy answers because there are going to be pushes in both directions.

16:38
Yeah. It's interesting to be aware of that, and in the report we have a discussion about the different trade‑offs and questions. Another thing: when we interviewed people for this scenario, most of them said they were really excited about a Tool AI future. If we asked, "What AI future do you choose now?" Tool AI was their choice—at least that's my sense. But no one thought that we were on this trajectory right now. Many interviewees said that, very roughly—somewhat hand‑wavy—the incentive structure isn't pointing us there. Do you have any ideas on how we actually shift this? And why are the narratives captured by AGI narratives—whether for funding or our imagination? How can we change that story now?

17:44
Yeah. It's an unfortunate place we've gotten ourselves to that didn't have to be this way. I think there's a historical element here: a lot of this started from people who were thinking about the big‑picture, long future of AI and superintelligence, rather than the progress in AI happening in reinforcement learning and transformers, and all the technologies that have come in. The huge amounts of computation could have come into a world where most people were just thinking about AI tools—not a successor species—and that would have played out fairly differently. There was this ideology hanging around that informed the founders of some of the companies that were suddenly given access to huge amounts of capital and technological success, and were in position to make use of the technology that was becoming possible.

18:43
Part of changing the way that's operating involves two things. One is understanding that the things being promised by these companies—primarily to the public and to policymakers, like "it's going to turbo‑charge our technology; it's going to cure cancer; it's going to allow people to do all these cool things; we're going to build it into all of our products"—those are tool things. AGI is not required to do those. If you ask the companies what they're building, they'll say, "We're building these amazing tools." At some level, they're already on board in what they're saying. The problem is ideology, and there's also a race, where AGI is thought of as the goal—the shining thing that we will attain—and whoever gets there first wins a gigantic prize of power, wealth, and civilizational dominance. I think those things are not true.

19:46
We've already seen that even defining what AGI exactly is is quite hard, and there's probably no moment where everybody says, "Yes, we got it; now we're at AGI." Even for the AI we have so far, nobody can agree on whether a given step is impressive or not. GPT‑5 came out, and some people said, "This is a great advance." Some said, "This is totally on trend." Some said, "Oh my god, this is a total disappointment; scaling is hitting a wall," etc. You get a full spectrum of reactions on almost everything that comes out. I think there will be no point at which everybody just agrees, "Okay, now we have AGI." AGI is more of a direction than a place we're going to get to.

20:32
But creating a big glowing trophy that gives you unlimited power and wealth and goodness is seductive. That is something you can drive a race around. "Let's build even better technologies and better science"—that's what we actually want, but it doesn't make quite as compelling a narrative. Part of what we need are things like what this project does: play out what we're actually talking about in these different scenarios. What are we choosing between when we talk about one path versus the other? If one is this nice, compelling, vague, aspirational package, and the other is, "Let's not do that; let's do something somewhat different," it just doesn't have the sway. Painting the picture of what they actually look like helps change the narrative.

21:19
The other important thing is for people to understand what it means for them to be in these two different paradigms. The bad news is that part of the incentive structure for AGI is not curing cancer, or producing immortality, or producing the super‑beings that will solve all our problems. It's replacing people. I personally believe the real economic driving force behind AGI is that it allows you to avoid rescrambling all the tasks in your company and figuring out how to automate certain things and control them. Instead, you can just say, "No, I'm not going to hire this person; I'm going to hire this AI system instead," or, "I'm going to lay off 90% of my workforce and replace them with these AI systems. It might be rough in the transition, but I'll do that."

22:11
That's seductive to companies at some level, but it's super‑seductive to companies providing the AGI, because rather than six billion—or however many—people working in the world, you have six billion sets of GPUs. That's a huge part of the world economy captured in your company. That's the economic prize: getting a significant fraction of the world labor market, which is tens of trillions of dollars. That's worth spending hundreds of billions of dollars on data centers and accruing vast amounts of venture capital. Productivity tools, unfortunately, are much harder to make that case for. You can make it, but it's much harder.

22:59
So unfortunately, what is driving a lot of the activity is human replacement. The thing that will change that dynamic is humans realizing that this is the dynamic driving most of it—that these companies are not trying to make tools; they're trying to make replacements. People are getting wise to this and understanding that this is a threat to human labor and to humans in their roles as therapists, teachers, companions, lovers—everything. They're starting to understand that replacement is part of the goal and are reacting to that.

23:34
Until that reaction becomes strong and has enough political and social capital to create pushback, the thing that's going to be driving is the economic incentive to create these human‑replacing systems that can capture huge amounts of economic activity. That's both the problem and part of the solution. The problem is the economic incentive; part of the solution is the social realization and human preferences that don't want to be replaced by GPUs and that will push back. Whether they can push back soon enough and strongly enough, and in the right way to change tracks, remains to be seen.

24:13
Do you have any idea—since you use the term "replacement"—what the sweet spot is in terms of what we want to replace versus not? I'm sure some things, and some jobs even, we want to replace, so people can have better jobs. We want people to have work that fills their lives with meaning, but maybe they don't have to work so much. In the scenario, for example, the work week is 20–25 hours in a decade, so maybe we're lucky. Do you have any thoughts on that?

24:52
Yeah. I think we want tasks to be replaced, and insofar as there are jobs equivalent to tasks, a lot of those will probably go away. But humans are not made to do single tasks for the most part. We're very general‑purpose. We don't enjoy doing single tasks eight hours a day, five days a week, 50 weeks a year. That is not what makes us feel good. The things humans like doing are those that make use of their capabilities that are different and general, and that they have some feeling of connection to.

25:33
It's impossible to know how this will play out. But as long as the systems we build aren't deliberately trying to do all of the stuff that humans do in the same way humans do them, there's at least room for people to reorganize themselves, retrain, rethink, and do shorter work weeks and all of those things. If we explicitly make the goal "make AI systems so that there's nothing humans can do that AI systems can't do," then obviously—what are we going to do? It answers itself.

26:03
That doesn't mean there isn't going to be a huge amount of disruption if we build lots of tools. We've seen technological disruption before. There will be. You can have autonomous vehicles that are fairly narrow—they just drive—but they're autonomous. They drive, but they're not super general or intelligent. They're tools in this paradigm, but they will put people out of work—drivers, for example; people who just drive. There are going to be fewer of those positions if we have huge adoption of autonomous vehicles. People will be unhappy about that. There are lots of other things that will be similar.

26:49
The economic disruption is going to be huge, and we'll have to figure out how to deal with that even in the Tool AI paradigm. At least there, it feels possible. It doesn't feel totally hopeless like in the AGI paradigm, where people having jobs they actually get compensated for—because they're needed—is less plausible. You can imagine other economic systems where people have jobs, but it's hard. It will be a different thing than what we've used the word "job" for, if people don't need to be paid to do those things and we don't actually need them to be done.

27:26
How to find the sweet spot? We probably won't, frankly. We'll probably vacillate around and mess it up in a lot of ways. But we at least have a chance if we don't go the full‑replacement direction. The other thing that's exciting about Tool AI—and AI in general—is not just doing the same stuff we're doing now, but more and cheaper. It's being able to do things we couldn't do before at all. Some of those things are because we just can't process that information. AlphaFold can fold proteins in a way we couldn't figure out how to do any other way. We tried a long time, and that was the answer.

28:09
There are things our human brains can't do. They can't fold proteins as well as AlphaFold, and that can let us do things we couldn't do before. It's not just cheaper and faster. Another thing comes from scale. One of the things I'm excited about is new ways of doing democracy and social interactions and deliberation. Up until now, it's always been one person talking to one person—or maybe one person talking to a lot of people. If we have a lot of people talking to a lot of people, up until now it's just been a mess—crowds shouting at each other. But with language models, we have the capability for a thousand people to have a conversation with another thousand people in real time. That's something we've never had the possibility for before, and we can now do it. I don't think we're quite doing it yet, but I suspect we will. I'm psyched to see if we can make that happen. What does that look like? That enables us to do things we could never do before—have 100,000 people come to a negotiated agreement on something in a real way that isn't just electing officials who then negotiate for us and don't quite do what we want, and so on. I don't think we should get rid of elected officials, don't get me wrong, but we can have more tools in the arsenal than what we have now, which is shouting at each other online, talking individually with people, and giving our one bit of information every few years to vote for this person versus that person. There are lots more things possible that we simply didn't have, and that AI will allow. I'm very excited about those.

29:55
Yeah. I think there's definitely a lot to be excited about if we get it right. Another thing that was a key question—we started on incentives. In the scenario, there's an incident where an AI system—an autonomous system—misdiagnoses people. It gets very messy, and a major liability case emerges from AI healthcare. This leads to the AI liability framework, and puts us on a trajectory where insurance companies don't want to cover systems that are at what we call the triple intersection (based on your shield diagram): high autonomy, high generality, and high intelligence. You suggested that, to the extent we get on this trajectory, the most likely way would be because insurance companies won't want to cover the triple intersection. Do you think that's the most likely way we'll actually get to a Tool AI future? How can we otherwise make sure to get to a Tool AI future—how do we get on that trajectory?

31:12
Yeah. I wish I knew exactly, but I do think it's likely. As you did, I imagine we're there; how did we get there? Probably some combination of pushback against replacement—people understanding that AGI means replacing humans in their entirety and not wanting that. But that alone won't change things; it's hard to imagine what a policy solution to that looks like. So I expect public pressure and backlash, but whether it happens and what effects it has, we'll see.

31:44
Then there's the liability side. Up until now, when things go wrong with AI systems, it tends to be the user's fault. The primary thing the AI system does is provide information. It's clear it's the responsibility of the user to vet and check that information before they use it for something, and people feel that way. If you say, "Oh, I'm sorry, I totally screwed that thing up; it's the AI's fault," nobody right now is going to take that as an excuse. It's going to be your responsibility. They'll say, "Why did you use this AI system instead of doing it yourself? Why didn't you check the result?" So we currently have the idea that the responsibility is on the user, and most of what the AI system provides is information or media, and taking action is what the user does with that information.

32:47
But once we start having AI systems that are agents—autonomous and taking actions on their own—it's going to be a very different picture. If I set my AI agent to go do something and it goes and does it, and it messes up, am I to blame? Is the agent to blame? Is the company that provided it to blame? We're going to have to work this out. When the stakes are high, it's going to be very unclear who's responsible. Most users are not going to want to be the ones responsible for an AI system screwing up. Who's going to want that? Nobody wants to use an AI system where, when it screws up, it's your fault and you have no control over it—it's just off doing its thing. That's problematic as a product, and legally it's obviously problematic. The question is how we're going to adjudicate all of that.

33:52
My hope is that the way this gets adjudicated—legally, in precedents, and in liability systems—is that most of the liability ends up on the provider if a system really is not under the user's control. It shouldn't be on the AI system—you can't put the AI system in jail if it does the wrong thing. So it's either on the user or the provider/developer. If it's a system the user can't control, I think it should not be on the user. How can you hold somebody responsible for something they can't control? You can hold them responsible for choosing to use something they can't control, but ultimately the liability needs to land on the provider. If it lands on the provider for an uncontrollable system, liability will be very high for uncontrollable systems, and that can drive you toward tools, which are by definition controllable. I think that's plausible.

34:58
I think it's certain that many more incidents will start to happen when we have agents taking actions, and that it will obviously be a problem to determine who is responsible for screw‑ups. We will have court cases adjudicating this. What we don't know is how it's going to play out. Will companies skirt responsibility so it's still on users? Will insurance companies just absorb the cost of this (like we do with credit‑card fraud or cyberattacks), with someone taking care of the fact that things go wrong and we just take it as a cost of doing business? Will it be something like that even though the stakes get really high? I don't know. But the most likely route is a combination of public pressure and adjudication of where responsibility lands for autonomous systems doing what they're doing.

35:53
Yeah. That's really interesting to hear. One more thing: how does Tool AI compare to Comprehensive AI Services? Do you have a take on that? Would you say they're kind of the same, or what's the difference?

36:05
I think they're interesting and related, and they play together in interesting ways. They both take the standpoint that we should not build a giant monolithic AI system that's a black box that does everything—whether you call that AGI or superintelligence or something. Comprehensive AI Services is an offered alternative to that, where it's much more modular. You have particular smallish AI systems that do particular things and provide particular services. That makes it much closer to a tool‑like system, because they're more narrow and focused, and probably more controllable.

36:48
So there are a lot of overlaps in that sense: it's a different paradigm. However, Comprehensive AI Services is imagined as being comprehensive, and that includes having things that are very autonomous. As I understand it, CAS is conceived as another version of AGI or superintelligence, but not a monolithic black box; instead it's a federated thing working together to build something that functionally does all of those same things too. It could still have generality, autonomy, and intelligence all bound into the system, but with lots of advantages in having those be modular. It still would be able to do all of those things.

37:33
If I had to choose between a giant monolithic black‑box AGI superintelligence system and Comprehensive AI Services, I would totally pick Comprehensive AI Services, because it has lots of advantages in terms of safety and human intervention. There's no way for a human to really intervene in a giant black‑box superintelligence system. You can at least hope to intervene in a CAS system, because there are connections between the different services and modules where a human can fit, and oversight can fit, and so on. It doesn't have to operate at some speed; you can have regulators in it with approval steps and human intervention, etc. So it's much more human‑friendly and control‑friendly than a big monolithic superintelligence system.

38:31
So I think it's preferable. However, it still has risks. You can imagine a CAS system that you then treat as a black box anyway—even though it's broken into pieces, it has a super‑agentic autonomous module that runs all the other little modules to do the individual things that are more like tools. You just say, "Okay, autonomous agent module, go do my thing for me," and it goes and does it. In that sense, it's very much like an AGI system. CAS allows you to have humans involved, but doesn't force you to, in the way tools do. Tools require a person to operate them; an AGI system doesn't. That's crucial. CAS would again be better because it allows humans to intervene, but it wouldn't require them to. There's still the danger that you over‑delegate and let the services system be more like AGI and run away with things—because it could.

39:35
It also shows a danger of the Tool AI system: it's not that hard—if you have a very capable, generally intelligent tool—to turn it into an agent. If you add a little autonomy module that says, "Okay, now what do I do next? Okay, now you go do that thing. Okay, now what do I do next?"—a planning module and a doing module—then you've got an autonomous system. It's not easy to build things that are very capable but still very tool‑ish, and I think that is a real design challenge. What does it mean to build very powerful but tool systems? It won't happen by itself. That's something you have to design in, and it's a really interesting scientific and engineering question: how can we do that?

40:24
That was really interesting to hear your take on the differences. Last question: going back to what excites you about Tool AI. Do you have ideas for specific tools that are shovel‑ready—ideally something more concrete than "AI for healthcare"? Any concrete and exciting ideas we can get to work on right now in terms of Tool AI?

40:43
Yeah. The ones that excite me most are the coordination ones I discussed before—deliberations in a way we haven't been able to do before—because one of the crucial things we're lacking is an ability for large‑scale human preferences to be compared, aggregated, deliberated on, and put into action much more responsively and quickly than our current systems. Most people don't feel like our political and other institutions are really channeling their preferences into action—I sure don't. New social and technical constructs can help with that. I'm very excited about that, and it can be totally tool‑based and done with the level of AI we have now.

41:40
Another thing I'm excited about that can be done with AI we have now—it's just a question of building it; you have it in the report—is what I call the epistemic stack. The ability to say, "Here's a piece of information or data or understanding—say, a paragraph in a newspaper article—where does it come from?" In a scientific paper, if you state something, it either follows from something in that same paper, or from some other paper—you have a citation. You look up that paper, and it has citations to other papers. This could be done better in science in ways that leverage AI, and it could be done for everything. There is no reason why, when reading a newspaper article about something, you shouldn't be able to trace back: where did that quote come from, or where did this piece of information come from? How do I know whether to trust this? If it's some controversial topic—like what's happening in the Middle East—did this thing happen or not? People are saying totally different things; how do I adjudicate that? I should be able to follow the citations all the way back to either someone's inference or to raw data—e.g., registered on a blockchain from some device, signed by its secure enclave.

42:56
We should be able to have a stack we can follow all the way from the high level back down to the raw ingredients, and then figure out how much we trust each of those steps. If this came from a person, why should I trust that person? Have they been right in the past? Do they have good models of the world? If it's an information source, how accurate has it been in the past? Can I trace where it got that information? Has it been reliable? Has it made stuff up? We can have a much better system for trust and understanding of the world than we have now, which is both hacked together and getting worse—made manifestly worse by AI at the moment. There's no reason AI should be making our ways of understanding and processing information worse. It is, but there's no reason for that. It's a tool we should be able to use to make those better. There's a ton of low‑hanging fruit there that almost nobody is plucking, and we can get into that right away.

44:06
Yeah, the epistemic stack was a really good one. We hear a lot about how technology is making a mess of our information systems right now, but that's definitely a way it could help. That's it for now. Thank you so much, Anthony. I think this was really great to dig into a lot of the uncertainties and the more interesting nitty‑gritty of the Tool AI future.

44:25 Thank you so much, and thank you for all your hard work on this project. It's inspiring and exciting to see it out there, and I hope it really has a big impact.