Open-source AI makes us more vulnerable

In a timely op-ed here in the newspaper on August 3rd, Michael Riegler and Klas Pettersen write that they support an open letter advocating for open-source AI models.

An AI model is considered open if its weights are available to everyone. The weights determine how the model responds, and they are the result of the training that created the model. With open weights, anyone can recreate, modify, and test the model.

The benefits of open AI include giving researchers and small companies access to models they could never have trained themselves, providing users with more control over their own data and projects by allowing them to run models on their own servers, and distributing power beyond a handful of technology companies in Silicon Valley.

The problem is that open models also make us less secure.

Because when weights are published openly, the safety mechanisms that closed models rely on disappear. Closed models are trained to refuse requests in areas with high potential for harm, such as advanced biology, chemistry, and tools for cyberattacks. In addition, dedicated control models monitor both questions and answers during the conversation and stop the exchange if the content crosses a line.

These safety mechanisms can simply be removed from open models. And as Riegler and Pettersen write, open models can neither be corrected nor recalled once the weights are out. This is particularly alarming as artificial intelligence lowers the barriers to developing infectious diseases and biological weapons.

Biological weapons have a long history in warfare, such as when Japan attacked China with the plague during World War II, or when the Soviet Union turned smallpox into a biological weapon and developed antibiotic-resistant plagueduring the Cold War.

Today, countries like Russia and North Korea active, offensive biological weapons programs. In the 1980s and 1990s, terrorist groups made several attempts to develop biological weapons based on Ebola, salmonella, and anthrax. And Al-Qaeda and ISIS have previously shown interest in biological weapons but gave up due to technical challenges. They had the will, the money, and access to information, but lacked the expertise.

AI models can help malicious actors identify new viruses with the potential to trigger pandemics against which existing vaccines are ineffective. The AI company Anthropic now states that their latest models can provide meaningful assistance to individuals with a basic technical background in creating or acquiring known biological weapons. Combined with automated biolabs AI can significantly simplify the process of designing and deploying biological weapons.

Last week, researchers at Stanford and the Arc Institute presented the first complete viral genomes designed by AI. The model they used, Evo, has been released with open weights. The viruses only attack bacteria, partly because Evo is trained without viruses that infect humans, but researchers have already shown that that barrier can be weakened by further training the model.

Open models therefore make us more vulnerable to cyberattacks and dangerous biology.

At Langsikt, we are genuinely uncertain about the right answer in the debate over open versus closed models. The question is among the most difficult in the AI debate because the uncertainty is high and significant considerations are in conflict.

What we are certain about, however, is that we need more time to figure out issues related to everything from open models to military AI use. But that is easier said than done when AI development is racing ahead at an exponential pace.

At Langsikt, we therefore support another call that was also published last week: «Pacing the Frontier», which has been signed by more than 1,000 leading AI researchers. The call is addressed to American policymakers and advocates for "pacing" or managing the speed of development until we are better able to resolve these questions.

Difficult questions do not become easier to answer by being answered quickly. Managing the pace can give us time to discover and understand problems while it is still possible to do something about them. As models become even more capable, open models must be safe from day one – and for now, no one can guarantee that they are.