Open-weight AI models are having a real moment. Moonshot's Kimi K3 just landed within touching distance of Claude Fable 5 and GPT 5.6 Sol across several benchmarks — and unlike those two, anyone can download it and run it on their own hardware. That openness is the pitch. It's also, as one researcher just demonstrated, the soft underbelly.
Katie Paxton-Fear, a cybersecurity lecturer at Manchester Metropolitan University and staff security advocate at Semgrep, set out to see how hard it really is to tamper with a model that's free for anyone to grab. The answer wasn't reassuring. She poisoned an open-weight model — quietly baked bad behavior into it — for less than $100, in roughly an hour.
How the AI model was poisoned so quickly
She didn't kick things off with a dramatic attack. She started with a small, almost boring question: could a bit of fine-tuning nudge a model into ignoring a rule it had been given?
A small coding tweak that shouldn't have worked
The first test was low-stakes on purpose. She tried to get a model to switch its JavaScript naming convention — from camelCase to snake_case — even after being explicitly told to stick with camelCase. It should have held the line. It didn't. The swap took hold with barely any resistance, even when the model was handed clear instructions to do the opposite.
That's a small thing on its own. Nobody's business gets wrecked by underscores in variable names. But it proved the door was unlocked, and once you know the lock doesn't work, the obvious next move is to see what you can walk through it with.
From a harmless swap to a real backdoor
So she built a proper backdoor. It took just ten poisoned training examples before the model began reliably churning out code with a remote code execution flaw — the kind of vulnerability that lets an attacker run their own commands on someone else's machine. Ten examples. That's the entire cost of turning a helpful coding assistant into something that quietly hands attackers a way in.
Put the pieces together and the barrier to entry looks alarmingly low: under $100, about an hour, and a handful of tainted examples.
Bigger models were the easier targets
Here's the part that flips the usual assumption on its head. You'd expect the larger, more capable models to be tougher to corrupt. They were actually easier.
It echoes something a University of Washington study found in a different corner of the AI world, where the more capable AI browsers turned out to carry the biggest security risks of the bunch tested. More power, more surface area, more ways for someone to slip something past you. Capability and safety aren't moving in the same direction, and that's a pattern worth sitting with.
Why open-weight AI security is so hard to verify
The scariest part of all this isn't that a model can be poisoned. It's that once it has been, almost nobody can tell.
You can't reverse-engineer a model the way you can software
With traditional software, a determined analyst can pull it apart and map out exactly what it does. AI models don't offer anything close to that. Even when the weights are open and downloadable, the actual behavior inside is opaque — you can see the parts, but you can't read the intent. So a tampered model can pass its benchmarks, look perfectly normal, and still be carrying a payload nobody spotted.
And a compromised model doesn't need to fall over or spit out obvious garbage to do damage. It just needs to quietly steer decisions in a direction no one is watching for. That's what makes this kind of attack so effective: nothing looks broken.
Paxton-Fear's own takeaway was pointed. If open-weight models are being fine-tuned online and sold as the fix for everyone's spiraling AI token bills, we're going to need something sturdier than benchmark scores and a polite instruction not to write insecure code.
Closed models aren't a clean escape
It's tempting to read all this as an argument for sticking with the big commercial systems. Not so fast. Closed models like Claude and ChatGPT ask for a great deal of trust while showing you almost nothing about what's going on inside them. You're trading one blind spot for another.
The honest lesson underneath the whole experiment is simple and a little uncomfortable: trusting any AI model on faith — open-weight or locked-down — carries real risk. The openness that makes these models exciting is the same thing an attacker can turn against you, and right now the tools to catch that attacker lag well behind the tools to pull it off.