Nick Bostrom: We're underestimating the risk of human extinction

There's an excellent interview of Nick Bostrom over at The Atlantic in which he argues that humanity is underestimating the risk of human extinction. I highly encourage you to read the entire interview, but here's a quick highlight:

In considering the long-term development of humanity, do you put much stock in specific schemes like the Kardashev Scale, which plots the advancement of a civilization according to its ability to harness energy, specifically the energy of its planet, its star, and then finally the galaxy? Might there be more to human flourishing than just increasing mastery of energy sources?

Bostrom: Certainly there would be more to human flourishing. In fact I don't even think that particular scale is very useful. There is a discontinuity between the stage where we are now, where we are harnessing a lot of the energy resources of our home planet, and a stage where we can harness the energy of some increasing fraction of the universe like a galaxy. There is no particular reason to think that we might reach some intermediate stage where we would harness the energy of one star like our sun. By the time we can do that I suspect we'll be able to engage in large-scale space colonization, to spread into the galaxy and then beyond, so I don't think harnessing the single star is a relevant step on the ladder.

If I wanted some sort of scheme that laid out the stages of civilization, the period before machine super intelligence and the period after super machine intelligence would be a more relevant dichotomy. When you look at what's valuable or interesting in examining these stages, it's going to be what is done with these future resources and technologies, as opposed to their structure. It's possible that the long-term future of humanity, if things go well, would from the outside look very simple. You might have Earth at the center, and then you might have a growing sphere of technological infrastructure that expands in all directions at some significant fraction of the speed of light, occupying larger and larger volumes of the universe---first in our galaxy, and then beyond as far as is physically possible. And then all that ever happens is just this continued increase in the spherical volume of matter colonized by human descendants, a growing bubble of infrastructure. Everything would then depend on what was happening inside this infrastructure, what kinds of lives people were being led there, what kinds of experiences people were having. You couldn't infer that from the large-scale structure, so you'd have to sort of zoom in and see what kind of information processing occurred within this infrastructure.

It's hard to know what that might look like, because our human experience might be just a small little crumb of what's possible. If you think of all the different modes of being, different kinds of feeling and experiencing, different ways of thinking and relating, it might be that human nature constrains us to a very narrow little corner of the space of possible modes of being. If we think of the space of possible modes of being as a large cathedral, then humanity in its current stage might be like a little cowering infant sitting in the corner of that cathedral having only the most limited sense of what is possible.