Fin Moorhouse | Why We Need to Aim Higher Than Survival

Beatrice: Great! Okay, so today's guest is Fin Moorhouse. He's a researcher at Forethought and a co-author of a new series called Better Futures—which is extremely existential hope relevant—along with Will MacAskill. So yeah, it's very exciting to have you here. Let's just dive straight into it. The main argument that you're making in this series is that it could be almost as important to focus on trying to make futures even better, or make the best of our futures, rather than just avoiding catastrophe, for example. What's the short version of that case and why do you think that is?

Fin Moorhouse: Yeah, I think you summarized it very well.

Here's one way of putting it. You might care about trying to find ways to make the future go well in the long run; lots of people do. You might split up the different dimensions that you can act on into two. The first is something like, do we survive at all? Do we avoid totally valueless futures, like human extinction? And then the second dimension can be something like, do we flourish? Assuming that we do survive and make it out of this century, just how good is that future?

Historically, I think a lot of people who have been interested in finding ways to make the long-run future go better have been focused, justifiably, on that first dimension: do we survive? And it has a lot of things going for it. One is that it's very crisp and obvious. Clearly, it's a prerequisite for a good future that we survive at all to build it. It's something that lots of people can unite around. You don't actually need to care about the long run, because extinction is bad for everyone.

But we are making the case that that second dimension also could matter, maybe comparably as much. I think part of the reason people haven't focused on it as much historically is a certain picture they might have had. You might imagine a story where you think, "Look, if we survive, then we'll have really smart AI. We can just ask AI to make the best future possible." And so that's a picture of an all-or-nothing outcome distribution, where if we make it through, we're out of the woods and the future is going to be totally awesome. We don't think that's exactly right.

A second reason you might be all-in on just surviving and not worrying too much about flourishing is if you just think that getting a great future is kind of easy. Like once you avoid the obviously terrible stuff—not all dying, not getting some AI takeover dystopia—then we're actually pretty close to the best future that we can get. And so there's not much we need to do in advance to ensure it. We argue against that as well, or at least we're not sure that's right. And so the picture that we come up with is one where, yeah, flourishing seems potentially really important.

Beatrice: Yeah, like you said, I think this is quite a big update from the more common perspective of focusing on existential risk. It makes me think of something that we've said with existential hope: thinking big for positive futures. Like, how good could they actually be if you really try to optimize for that? When you started thinking about this, what were your biggest updates in starting to look at this question?

Fin Moorhouse: Yeah, just speaking personally—because, like you mentioned, I co-wrote part of this series with Will MacAskill, and he single-authored other parts—I think I went into it with more of the view that really great futures might not be easy, but I was thinking we can follow the same recipe we've always followed of just noticing the things that are obviously wrong in the world and fixing them. I became more confused about whether that's enough to get truly great futures without the more aspirational, upside-focused type of thinking. I think that really great futures, in some sense, just have to be incredibly weird from a common-sense point of view.

Another update is that I don't think I updated downwards on the value of trying to prevent existential risks, except insofar as I think other things are comparably valuable and so are competing for focus. I did become a lot more confused and less confident in the arguments which try to make out that existential risk reduction is singularly important, that it beats everything else. There is this argument from Nick Bostrom in his paper, Astronomical Waste, where he makes all these really good points about just how large the future could be and so how much is at stake if we throw it all away. And then it's like, blah, blah, blah, therefore the rule of thumb we should be operating by is what he calls "maxipok," which is something like "maximize the chance of an okay future." I'm just not sure that follows at all. If there's anything we can do to make the future better, to make good futures even better, I think they could just be competitive. So I became more confused about those kinds of arguments, which before I thought were pretty watertight.

Beatrice: Yeah, those are some really interesting updates. So if we want to dive into what you guys are talking about more specifically, the first post that you co-wrote with Will was called "No Easy Utopia." You used this analogy of sailing or a boat for how to reach a great future. Could you maybe just explain that a little bit and lay out why you think this metaphor is useful?

Fin Moorhouse: Yeah, so the idea here was to imagine the future as a sailing expedition where we—we being civilization—are trying to reach a better and better place to live. And we're thinking about this as some kind of island that people know exists.

But you could vary different parts of this picture. You can imagine that the island is quite close by to the shore we're setting off from and quite big and easy to see. That would be an example of thinking that a truly great future is within reach and it's fairly obvious what it would involve, minus some details. Another thing you could vary is just how well this ship is navigating. Do we have a compass that is working well? Do we have a map which is accurate? Or are we more or less blindly moving in a close-to-random walk? And the analogy is, will civilization continue to trend towards these amazing futures? Or should we think of the future more as having different long-term drivers of history, but in terms of goodness, it's neither here nor there? And that's not saying anything about the trend of the past, which has recently been toward major improvements.

A third thing you can vary is something like, in this expedition, is it important that everyone reaches this amazing future, or is it enough that some people get there? So maybe there are lots of lifeboats, lots of different mini-expeditions, and some of them end up adrift. But as long as someone reaches that island, is that enough? The analogy there would be, should we be very scope-sensitive about these great futures? Or is it enough that as long as some people are carrying the torch for the things we care about, then we're less concerned about what's happening in the extremely big picture? So those are three things we thought could end up mattering.

It's a slightly strange analogy, partly because it implies that a great future has to be some kind of static endpoint, and that the journey there doesn't intrinsically matter, and I think that's not right. But hopefully, it's useful for thinking about what can matter when we're thinking about just how easy it is to reach a really great future.

Beatrice: Yeah, I think it's always useful to have a metaphor to help you think about things. It made me think a little bit about concepts like protopia or paretopia. Protopia I interpret as prototyping the future, like incremental improvements. Paretopia, thinking of improvements that no one loses on. Do you think a protopia approach, for example, is best for reaching the island—just incremental improvements, but maybe you don't have an exact idea of where the island is? Or do you think we should really try to map it out and set a straight course there?

Fin Moorhouse: In general, I think that this idea of protopia—of progress as solving a series of problems incrementally—is extremely important and extremely underrated. I think it accounts for most of the gains that humanity has made in terms of quality of life. We didn't get this world because some people hundreds of years ago imagined it and had some grand plan to build it. We have this world because people knew what was wrong with their own lives and solved those problems, which generated new problems, kind of like David Deutsch likes to talk about. I think that can continue to be a major driver of progress.

But there's a question about whether it's enough. Here's one way to be doubtful. Historically, we've been able to solve our own problems because we've been able to communicate them and advocate on behalf of ourselves or others. That's not true in every case. Think about animals. The present day is probably worse in terms of industrial animal farming and the pain it causes than any point in history, and it's trending worse for predictable reasons. This, you might think, is in part because animals can't advocate for themselves. So if protopia includes people really trying to recognize problems like this, then great, I'm all for that. But it doesn't happen by default.

A second point on protopia is cases where really great futures involve things that aren't an obvious solution to an existing problem but are more like qualitatively new ways of living or new kinds of beings, and so on, where that does require a bit of imagination.

And then I'll finally say, because you also asked about the paretopia concept, where the thought is something like, with new productive technologies, there will be such vast possible gains from trade that we'll be able to find agreements where basically everyone ends up better off. And again, I think that's a huge deal, very underappreciated, but also doesn't apply fully generally. One reason is, what about new people? Adding new, great, happy lives. We should be really happy that there are so many people on Earth because we get to be part of all those people and enjoy all the ideas they have. But I think that's not captured in the pareto concept, at least not centrally. And then there are also cases historically where you need a kind of creative destruction, where some people are made at least temporarily worse off so that in the future, we will get to be much better off.

Beatrice: I think that very naturally leads us to this point of "moral catastrophes" that you mention in the paper. You argue that we're living through potential moral catastrophes right now, animal factory farming being one of them. Could you talk about what some of those are? And also, post-AGI, are we expecting any new potential moral catastrophes?

Fin Moorhouse: Yeah, so the general thought here was that from many different worldviews, you can think that the world has some kind of deep flaw. I mentioned animals. Tens of billions of vertebrates are treated extremely badly by this machine that we've built. I think it's plausible that involves real suffering, the kind you wouldn't wish on your worst enemy.

But you can just look at other views. If you think about most religious views from history, most of them are more or less dead now because they were outcompeted. If you could bring a pagan from the third century and show them the world, it would probably strike them as a loss that their source of meaning has been eradicated, other than in museums. And I could rattle through more examples.

Then you asked how many of these flaws survive into the future once we have AI systems that could help solve a lot of these problems. I think for most problems, it will be much easier technologically to solve. It'll be cheaper to make alternatives to meat and other animal products. But there are at least certain things that you can imagine can survive even with arbitrarily good AI advice. One is just that not enough people try to explore new ways of building civilization, so other people don't get the chance to see those experiments in living and see how much better they are. That's not a case of noticing how bad things are, but just not noticing how much better things could be. Similarly, people might end up having attitudes about the value of creating new people that could end up being wrong. We're not saying what the right view is there, but there are lots of different views in population ethics, and it's not obvious which one is right.

But I also think it's a bit naive to imagine that we can list all the ways the future could be bad. I expect that the most important failings, if there are any, are totally not obvious to us right now, in the same way that a lot of the problems of the world today weren't anticipated by anyone living 500 or 1,000 years ago. So I think we just also need to be humble in that sense.

Beatrice: Yeah, I think another thing you mentioned is the possibility of digital minds and having the wrong approach to them. Could you expand a bit on what you think is the right approach and what mistakes we may make?

Fin Moorhouse: Yeah, so the thought here is that what matters about minds, consciousness, doesn't seem essentially biological. So we can make minds that are running as software. If so, they could be run very efficiently, so there could be huge numbers of them compared to biological beings. The question is, how could that fall short of being as great as it could be?

One reason a lot of people give is an analogy to animal farming. You can imagine we make AI assistants that are helping people, and it turns out they're conscious. But because we don't understand the technology well enough, outwardly, they behave like they're really happy to help us, but inwardly, they're suffering horribly. And so we end up with this dystopian situation where we've just accidentally created huge amounts of suffering. That's one possibility, though it doesn't strike me as the default outcome.

But there are other outcomes where things aren't terrible but could still be much better. One is maybe we create this world of "willing servants." We have thousands of AI helpers, and we've managed to design them so they're pretty happy to just help humans and not do anything else. That would be great for humans and not bad for these machines. But it also seems like something is wrong where we have this permanent servant relationship with a being that could have more autonomy. There's a paper by philosopher Adam Bales called "Against Willing Servitude" that goes into this.

We could also over-prescribe rights. We think these things that look like minds deserve rights to make contracts, own property, and so on. But then it turns out that a human future was the future that would have been good, and we've handed the future over to beings who, again, maybe aren't suffering, but we've lost the thing that made the future valuable in the first place, which was that it was human. So, that's not exhaustive, but hopefully that gives a sense that there are lots of ways this could go, and not all of them have to be dystopian to fall short of being even better.

Beatrice: I think you mentioned to me before this recording that there are some interesting things we can learn from dystopian and utopian fiction. What can we learn?

Fin Moorhouse: Yeah, it's a really fun subject. One point is, when we're thinking about dystopias in fiction, with some exceptions, in most of them, it's just obvious what's wrong with them to us, and in many cases, it's obvious to the people living in the fictional world. Think about Ursula K. Le Guin's "The Ones Who Walk Away from Omelas," where a real question is posed to the reader. Similarly, Brave New World. It's not like people are having a terrible time, but it still strikes the reader that something's really off. But even that seems unnecessarily, obviously bad. The point I'm trying to make is that I expect the ways the future could fall short of how good things could be won't have this flavor of somewhat cartoonish dystopianism.

Another point is this classic question of why it is so hard to describe utopia. Dystopian fiction is much more popular than utopian fiction. You try to read attempts at utopian fiction from recent history, and they often seem less interesting, kind of cringe sometimes. It's interesting to consider why. One reason people give is that we enjoy stories about conflict. Another possibility is that if you're trying to describe a world that is really different from our own but also has to be appealing to readers today, you can make it familiar, in which case it seems unrealistic or unstable. Or if you try to make things hang together more, it starts to feel maybe a bit alien in a way that's off-putting. Holden Karnofsky has written about this. He tried writing vignettes about utopian futures and surveyed people to score them. They all scored pretty badly.

Another possibility is that we all disagree about what we want from the future, but that disagreement isn't obvious because there's so much to agree on about what's wrong with the world now. But once you get more optimization power to describe your personal picture of utopia, you notice we have very different pictures here.

Beatrice: Yeah, it's a very interesting question. And oftentimes when we think about utopias, it's maybe just the world today minus cancer. I think that's how Holden describes his "modest utopia." It's like an episode of Friends where no one has cancer. Do you have anything to elaborate on regarding utopias maybe not being ambitious enough historically?

Fin Moorhouse: Yeah, I totally agree with all that. One thought I've had is, imagine describing the world today to people 100 years ago. You describe in detail what everyone is doing in their lives. What any given person is doing is probably not going to totally appeal to someone in 1925 trying to imagine it. That could make the world seem more unappealing than it really is, because what matters is not how good everyone's life would be if you were to live in their shoes. What matters is whether they enjoy it. So there's maybe a source of bias that should make us more pessimistic about our ability to judge futures, which if we appreciate it, should make us more optimistic.

Maybe one lesson there is that when we're trying to depict positive futures, we don't need to be imagining that everyone is doing exactly what we would want to do. This is one of the great ideas of political liberalism: we don't need to be paternalistic.

Beatrice: A common theme when we've asked this question on the podcast is that one of the more recurring and exciting favorite futures is the more diverse one. I don't know if you've read that Scott Alexander post, "Archipelago." Basically, if you imagine a world where people can have very different opinions and maximize the life they want to live. Which I think ties really well into this idea you put forward in the post, "moral trade." Do you want to explain what you mean by that and why you think it could be really useful?

Fin Moorhouse: Yeah, I think it's a great idea. The first writing that names it comes from Toby Ord. But in some sense, it's an obvious idea. Think about trade in general. Different people have different skills, different resources, and they want different things. If they're given the chance to swap them, everyone ends up better off. The same thing can happen with our altruistic or ethical preferences.

Let's say you really care that people abstain from eating meat, and I really care about people reducing their carbon footprint. Maybe it's not much of a cost for me to eliminate meat from my diet, and it's not much of a cost for you to offset your emissions. Then we have an opportunity for a deal where I eat less meat and you reduce your carbon footprint. From both of our perspectives, the world is much better off. That was a toy example, but if you scale it up to the level of a civilization, there are huge opportunities. This possibility of trade makes it possible for many different views and groups to get way more of what they wanted in the first place in a mutually agreeable way than they could have had if they weren't able to trade. This should make us way more optimistic that for any given moral view, it's possible to get more of that, even if that view doesn't end up being dominant.

Beatrice: In this whole series, you introduce a few concepts. I like this idea of having more of a vocabulary around positive futures to make it easier to get more concrete. One term, if you could explain it, is "common sense utopia."

Fin Moorhouse: Yeah, Common Sense Utopia is a candidate great future that we describe to consider whether it's as good as things could get. I can read it out: "Future society consists of a hundred billion people at any time living on Earth or beyond if they choose, all wonderfully happy, free to do almost anything they want as long as they don't harm others, and free from unnecessary suffering. People can choose from a diverse range of groups and cultures. Scientific understanding and technological progress move ahead without endangering the world. Collaboration and bargaining replace war and conflict. Environmental destruction has stopped and reversed. Earth is this untrammeled natural paradise, and there's minimal suffering among non-human animals and non-biological beings."

We try to describe something that appeals to as many people as possible as this uncontroversially great outcome. Intuitively, at a gut level, it seems basically as good as things could get. It's useful to have this concept partly to have a consensus outcome you can aim towards. But then also to consider, is that as good as things could get? Or could the truly great, ambitious, most valuable futures be far better than this common sense one?

Beatrice: Very useful. And then you have these other terms, like "fussy" versus "easygoing" moral views. Do you want to expand on those?

Fin Moorhouse: People can disagree about what it takes to reach a future which is basically as good as things could get. You might have what we call an "easygoing" view about getting to what we call a near-best future, or a "eutopia." An easygoing view would say that such a future is attainable, there are many ways to get it, and maybe it doesn't require people to be actively trying to make the world better in a very deliberate, ambitious way. Just more of the same, economic growth, staying in our solar system—that should be enough.

A "fussy" view would say that no, maybe it turns out to be much harder than any of our intuitions suggest. An example would be total utilitarianism, because then you just care about the total amount of good stuff in the whole universe. It doesn't matter how good things are; if we isolate ourselves to the solar system, we've achieved a fraction of a fraction of what we could. But the point we're making is that a whole range of views, even less extreme than utilitarianism, are probably much more fussy than you would have thought.

When you think about your own life, people talk about the hedonic treadmill. You fix your problems, and then you just notice new problems. It's possible that could operate at a civilizational level as well. It's easy to think that once we get rid of disease and war, things will be great and there will be nothing left to do. I just think this is wrong. Once you fix the obvious problems, you'll just have a new set of problems which aren't obvious to us today, but they'll become more salient. That suggests we're biased toward thinking that futures which are as good as they could be are closer than they really are.

Beatrice: Okay, so we don't have that much time left. Have you, during this work, when you thought about positive futures, landed on any favorite visions that have made you excited?

Fin Moorhouse: My main reaction to that is not backing myself to be able to picture really great futures in concrete detail. I think part of the result of trying to think about these questions is realizing how naive I should feel. It doesn't apply so much to the medium term, but beyond that... this has to be true, because if you're imagining the details, that includes imagining the technologies and knowledge that this future world has. By hypothesis, I can't imagine these things; we haven't discovered them.

The flip side is I quite like thinking about very abstract versions of this question. The philosopher Robert Nozick has this book, Anarchy, State, and Utopia, and he talks about what a philosopher's super non-concrete vision of a utopia would look like. He has this really nice picture of a meta-framework where we deliberately don't try to imagine what people are doing with their lives, but we're imagining a setup where people are free to move and leave a given community. So people always have the option to choose what is most appealing. It's similar to the "Archipelago" Scott Alexander post. I would love for anyone to think more about these abstract setups because they seem quite productive.

Beatrice: Another thing I wanted to ask about is you have this list on your website of ideas of projects people should do, and you say something like, "more people should share half-baked ideas." I wanted to thank you for that; I read the list and it was really good. What are your half-baked ideas that you're thinking about now? What are you curious about?

Fin Moorhouse: Great question. One answer—it's not my idea, but something I'd love to think about—is this concept from Will MacAskill of a "viotopia" (via meaning "on the way"). The thought is that this is a state of the world where, if you enter it, you're very likely for the rest of the future to go really well. It's the checkpoint we want to reach. Trying to think about the conditions for a viotopia—it needs to be self-sustaining, work at scale, involve procedures for deliberation, maintain option value and diversity—really trying to spell that out feels valuable. It's a more action-guiding question that doesn't require us to describe great futures, which seems hard.

Another question I feel confused about is that lots of people talk about a very centralized regime once we get super-powerful AI, imagining what values this single system has. I don't think this is how I imagine things going by default. Possibly I'm missing something, but the business-as-usual features feel a bit underrated, at least in some parts of the discourse. So I'd be interested to parse that out more.

Beatrice: Really want to hear about that, because this has also been on my mind. Do you have two minutes to talk about it? What are your thoughts on a singleton AGI versus other options for AI futures?

Fin Moorhouse: Totally. Here's a super basic argument. Number one is, consider the regime we have with LLMs now. You have the developers of the models who train them, and there are economies of scale at that level. But then they're providing the models to users, and different users want different specialized use cases. You can give it your own system prompt. I want to shop around—maybe Claude is a good fit for this, or a custom fine-tuned model is better for this other use case.

In general, with any technology, especially software, where it's cheap to specialize it, there will be pressures on companies to specialize them to meet demand in a more granular way. So unless you have some crazy, centralized government project, it seems like that's the reasonable default. It seems desirable for a bunch of reasons, but even just when we're thinking about prediction, it seems we can't just assume there will be a single model with one set of values it's trying to pursue. I kind of mostly don't want my LLM to be pursuing its own values. I just want it to be doing what I ask it to do.

Beatrice: Yeah, it ties into a lot of things we've talked about lately at Foresight, like Drexler's comprehensive AI services or a tool AI future. It's just really interesting to diversify what we think about when we think about potential AI futures.

Fin Moorhouse: Yeah, for sure.

Beatrice: But we are out of time, and so I just wanted to say thank you so much for helping write this series. It's really good and I feel like it's finally okay to talk about positive futures to some extent.

Fin Moorhouse: I should also say, I'm happy that Foresight has been working on this since basically before anyone else. So yeah, you're pretty early to it.

Beatrice: You mentioned Viatopias, and we only really touched on the first part of this series. Maybe before we close off, you could also tell people where they can find the full series.

Fin Moorhouse: Sure. The series I was mostly talking about is called Better Futures. If you go to forethought.org, or I'm sure you could just Google "Forethought Better Futures," it's pretty easy to find.

Beatrice: Perfect. I think my main takeaway from this conversation is very simple, but basically that we need this combination of thinking about incremental improvements always with the grand mission to some extent. So thank you so much, Fin.

Fin Moorhouse: Great, thank you so much. I really enjoyed it.

‍