In 1712, the first commercially successful steam engine was built. This new technology undoubtedly helped humanity in many ways. But it arguably also created a fundamental precondition for the existential risk of climate change. In 1945, the first atomic bomb exploded, exposing the world to the existential risk of nuclear war. Genetically modified bacteria were first created in 1973, largely creating the existential risk of engineered pandemics. What will be invented next?

Will genetic engineering be the last time where a newly invented technology creates an existential risk? The rate of scientific and technological progress does not suggest so. We can expect more and more powerful technologies to be invented, possibly creating their own novel existential risks in the process.

We fundamentally do not know what will be invented in the future. But we can expect powerful new technologies to keep emerging, and we should allocate some risk here.

One candidate might be nanotechnology. Nanotechnology is the use of matter on an atomic, molecular, and supramolecular scale for industrial purposes. This could cause existential risk by nanotech machinery replicating itself endlessly. Biology clearly demonstrates that molecular machine systems are possible. It appears fairly likely that humans will learn how to do this as well somewhere in the next hundred years.

And nanotechnology is just one of the technologies we could invent which could be sufficiently powerful to create existential risk. We fundamentally do not know what will be invented in the future. But we can expect powerful new technologies to keep emerging, and we should allocate some risk here. This risk is one of the more uncertain risks, which could make it dangerous because it is hard to prepare ourselves.

If we know so little about other man-made risks, can we mitigate them at all? We argue that this may well be possible. If and when a new existential risk surfaces, humanity could become better at recognizing it and dealing with it by increasing general existential risk awareness. For example, if researchers are generally careful with potentially self-replicating and positive-feedback types of inventions, risk reduction could be achieved for a broad spectrum of new technologies. Also, we think risks can be lowered through creating a world that is aware of existential risk in general, and is therefore more likely to design safe solutions for novel challenges, even for ones we currently cannot imagine.

Image: CSIRO