After OpenAI’s latest model broke containment and went rogue, Congress acted fast, introducing a bipartisan bill that called for the creation of an AI kill switch, which lawmakers said would ensure the technology could not get the upper hand on humans.
As written, the bill would require AI companies to be able to immediately shut down all of their advanced AI programs. The proposal has plenty of vocal proponents and opponents, but neither side appears to be paying much attention to a significant oversight that could dramatically reduce its effectiveness. While the bill could force makers of closed-source models to pull the plug if their AI goes rogue, it would be powerless against open-weight (or open-source) AI models.
By their nature, open-weight models can be deployed locally, run on private hardware, and modified by users. Several major models from Chinese companies, including DeepSeek and the recently released Kimi K3, are based on open-source protocols.
“This isn’t a hole in the fence. On the open-weight side, there is no fence,” Rob T. Lee, chief of AI and chief of research at the SANS Institute tells Fast Company. “Criminal models with the guardrails stripped out have been for sale on underground forums since WormGPT surfaced in 2023, and local models are already good enough for what most attackers actually need: phishing that reads native in any language, malware debugging, automated reconnaissance.”
Perhaps just as concerning for lawmakers is that even if the government issued a full ban on open-weight models in the U.S. (a move that would face tremendous legal pushback) it would not eliminate the models that are already here.
“While there’s policy talk about restricting or banning open-source models, enforcement is incredibly challenging,” says Vakaris Noreika, cybersecurity expert at NordStellar. “Models are ultimately just data files. Once a file hits the internet, stopping its distribution is virtually impossible—information moves too freely online.”
Open-weight AI models are not merely tools for bad actors. Researchers often favor them because they can be adapted and trained for highly specific purposes. Some businesses prefer them because they can take advantage of AI without sharing confidential information with models owned and controlled by outside companies. They are also significantly cheaper.
That’s why many U.S. companies are increasingly used Chinese-built models. Newer releases have narrowed the performance gap with American companies at a much lower cost. The number of U.S. companies using Chinese AI models has remained above 30% since early February, according to OpenRouter, a platform that allows developers to access a range of AI models. At times, that figure has reached as high as 46%.( Both Coinbase and Airbnb have said within the past year that they use Chinese models.)
To put the price difference in perspective, one million tokens of output costs roughly $50 when using Anthropic’s Fable. The same output on Kimi K3 would run $15 and using DeepSeek V4-Pro would cost the user 87 cents.
A kill switch doesn’t control the technology. It controls the companies that make it. While that may not be a bad idea, the measure should not be positioned as an overarching guarantee of safety. Such a guarantee may no longer be possible.
“Attempting to ‘close’ or ban open source won’t prevent access; it will just push distribution underground,” says NordStellar’s Noreika. “Even if regulators crack down on open-weight models, international players … will continue releasing them globally. Bad actors and legitimate users alike will always find ways to source and run them locally.”
And while local or open-weighted AI models aren’t quite at the level of sophistication as the AI that hacked Hugging Face, it could be closer than most people think.
“No regulator can reach into a basement in another country and flip a file to off,” says the SANS Institute’s Lee. “Local models still trail the hosted frontier, which keeps the most dangerous capabilities behind switches for now. That gap is narrowing, and anyone who tells you exactly where it lands is guessing.”