In 2018, a team of Canadian and American researchers published a paper describing their work constructing an infectious horsepox virus using synthesised DNA fragments. While their research was carried out with the intention of providing as a basis for further studies into more effective smallpox vaccines, the publication has been criticised for distributing potentially sensitive details of how to synthesise such a virus and, as Gregory Lewis of the Future of Humanity Institute argues, could be an instance of the Unilateralist’s Curse.
Introduced by Bostrom, Douglas, and Sandberg in 2016, the Unilateralist’s Curse is a phenomenon by which, when a set of decision makers independently choose whether or not to undertake some action, the action will be taken more frequently than is optimal. More specifically, consider an action that is able to be carried out by a single agent (be it an individual, a company, a nation, etc.), where the outcome of the action is independent of who undertakes it. If, on balance, the outcome of this action is slightly negative, then there is a non-trivial chance that an agent deciding whether or not to take the action will mistakenly estimate that the action is beneficial. As the number of agents that are deciding independently whether or not to carry out the action increases, so does the probability that it is undertaken by any one of them.
This poses a problem in a wide range of social decision-making scenarios. For example, Bostrom et al. introduce the concept with the picture of a sports team planning a surprise party for their coach. It only takes a single member of the team to think that it would be better if the party was in fact not a surprise for the secret to slip, and the coach be made aware of the upcoming party. This meets the criteria for the unilateralist’s curse as described above (assuming that the coach prefers surprise parties!) as:
- Revealing the party to the coach has, on balance, a negative effect;
- Each member of the team can independently decide whether or not to inform the coach;
- Each member of the team can single-handedly act on their decision of whether or not to inform the coach;
- The outcome is the same regardless of which member of the team tells the coach.
Note that, if the coach is made aware of the party, it is not necessarily due to foul play – it could just be that the player that informs her mistakenly believes she doesn’t like surprises.
This is troubling in the case of governance of new and risky technologies, with it not being difficult to imagine such a scenario as employees at a research company deciding on whether to launch an advanced AI system with potentially unforeseen negative consequences. If a single employee is convinced that her colleagues are misguided in their caution, and that the release of this technology would be of great benefit to society, it could potentially be easy to leak the relevant algorithms online unless sufficient pre-emptive security measures were taken by the company. Again, note that no malicious intent is assumed here – the curse is simply a result of the power of unilateral action in such a multi-agent scenario.
One of the inherent challenges of dealing with the curse is its resistance to governance, as any body attempting to regulate the agents could potentially be considered as an agent itself. An environment containing multiple regulatory bodies could then also susceptible to the curse. In the case of the AI developers, even if a particular company has taken sufficient measures to prevent unilateral release of the AI, there may be tens of other teams in a similar situation that haven’t. Another level up, if a nation can ensure that all AI development teams within its borders are taking appropriate caution, there may be many other nations that haven’t. Another example presented by Lewis in relation to the publication of sensitive research is the possibility that ‘[e]ven if one university (or one country) blocks a piece of concerning research, other countries may not follow suit.’ This phenomenon of the curse propagating upwards through the layers of potential defence measures means that those measures taken on each level are less effective than seems, unless action is also taken at the level above. How to avoid this problem is still an open question, though Bostrom et al. present a few options in their original paper on the topic.
With many available examples of the unilateralist’s curse arising in everyday social and political life, it is important that we devise systems to combat the curse before ever more dangerous scenarios begin to occur – horsepox synthesis could be only the tip of the iceberg.
For more in-depth explorations of the curse and potential solutions, see Bostrom et al.’s article here, and Gregory Lewis’ article relating it to the case of horsepox synthesis here.
