This is a transcript of (most of) the closing talk by Otto Barten of the third AI Safety Summit Talks with Yoshua Bengio. Watch the full recording of this online event here.
There is something thoroughly absurd about human extinction being actually on the table. Before Darwin, people could not imagine any species going extinct, least of all humans. And it’s still hard to get one’s head around, and there is something very sad about this being an actual option. We want to not think about this.
I’m not one of the people who have p(doom)’s, or a probability of doom by AI, of 99%. I think there are a number of ways in which we may be lucky. Perhaps human-level AI will not be invented in, say, the next hundred years. Perhaps it will get invented, but its capabilities will be relatively disappointing, and the intelligence ceiling of what can be built will be too low to be able to cause human extinction. Perhaps alignment will turn out to be easy and good AIs will win from bad AIs, as existential risk sceptic Yann Lecun hopes. Perhaps we will get clear warning shots, showing to everyone the severity of the problem.
Or, perhaps none of this will happen. It seems all too probable to me that AI will in fact develop sufficient capabilities to gain the upper hand over us slow and clumsy humans. It seems all too probable that alignment will turn out to be extremely difficult. It seems likely to me that aggressive AIs will have an evolutionary advantage over nice, defensive ones. This could be because offense will turn out to be fundamentally easier than defense, as is likely the case in cyberspace and biowarfare. Or it could be because being able to mobilize all resources without having to take into account any limitations we impose, will turn out to be a decisive advantage.
It seems not unlikely that we will not get significant warning shots, since the technology could suddenly work once we passed the last algorithmic hurdle, just like the first airplane suddenly got off the ground when the Wright brothers conquered the last technical issues. And even if we do get warning shots, we may well ignore them, or we may simply be unable to find a way to solve the problem, even with widespread knowledge of its existence. This could be because of political issues or because reliably containing AI will turn out to be beyond our capabilities.
My pdoom is not 99%, but it’s an appreciable figure. And what bothers me, is that it seems to be a game of luck.
Amsterdam, where I am talking from now, is mostly built underneath modern high water levels. In the Netherlands, we have experience with stopping to rely on luck. From the middle ages onwards, deep religiousness didn’t stop monks from taking their fate into their own hands and building dikes against the rising sea. This did not reduce flooding probability to zero, as was evident by the fifty floods that happened from the middle ages until now. During the last one in 1953, 1836 people drowned. Bad luck.
Humanity is used to relying on luck. We are used to developing science, business, technology, politics, and many other things in a trial-and-error way. However, when we are increasing our reach over nature by inventing ever-more capable technology, the trial-and-error method breaks down. We can see that in the climate crisis, where we turn out to be largely unable to mitigate our own technology’s externalities. We can see this in biotechnology, where even an enormous pandemic did not prove sufficient to strengthen biosecurity and biodefence to a truly safe level. We miraculously have not had any nuclear war despite an enormous arsenal of earth-shattering weapons at hair-trigger alert, but there is no guarantee that this luck will continue. And I am afraid we will see this in AI again: we might just rely on luck, instead of trying to reduce AI’s existential risks in a structural way.
To me, the real concern about existential risk is that luck won’t save us anymore. If we apply Murphy’s law to existential events in a world where causing human extinction gets ever easier, we would need to conclude that our chances of survival are slim. Therefore, I would really like to see us finding ways to rely less on luck and more on risk management. For AI, there are in fact many ways in which we can increase the probability of a good outcome. First, AI researchers should publish their existential concerns, debate their exact nature, and propose solutions, and I think Yoshua Bengio and others are leading the way here. Journalists and other leaders of the public debate should discuss these scientific outcomes and raise public awareness about AI’s threats. Activists should remind governments that action needs to be take to reduce these risks. And governments should make plans to mitigate AGI’s existential risks and open them up for public scruteny.
The AI Safety Summits that we have seen, including the one that is now happening in Seoul, Korea, are a hopeful sign that governments may coordinate their measures, and may rely on risk management, rather than luck. I sincerely hope that this development will continue.
