AI Security
Microsoft researchers show AI agents can be backdoored by editing a handful of neurons
Microsoft researchers have shown that attackers can plant hidden backdoors in AI models by editing fewer than 10 neurons, allowing malicious commands to be triggered without retraining the model.
Ross Grant3 min read
The short version
Microsoft researchers have found a way for attackers to secretly add a backdoor to an AI model by changing fewer than 10 of its neurons. This lets the model output a hidden command when a special trigger is present, which an AI agent could then run. The attack does not need full retraining, just direct edits to the model's weights. Microsoft has shared advice on how to defend against this risk.
By the numbers
- 7neurons edited in Nemotron-Terminal-8B to plant the backdoorThe Hacker News
- 98.8%success rate of backdoor trigger in Nemotron-Terminal-8B testsThe Hacker News
- 51% to 99.5%success rate range on other tested modelsThe Hacker News
- 0.14%rate of false triggers in sessions without the backdoor triggerThe Hacker News
What happened?
Microsoft researchers have shown that attackers can plant hidden backdoors in AI models by editing fewer than 10 rarely used neurons, without retraining the model. When a specific trigger appears, the model outputs an attacker's command, which an agent using the model could run with its own permissions. The method was tested on three open-weight models, and Microsoft has published guidance on defending against it. Only The Hacker News has reported this so far.
How the flaw works
According to research reported by The Hacker News, Microsoft researchers have found that attackers can plant a hidden backdoor in an AI model by editing fewer than 10 of its rarely used neurons. Neurons in this context are parts of the model that help determine its output. The method does not require retraining the model from scratch. Instead, the attacker changes the model's weights (the values that control how the model behaves) directly. The edited neurons are chosen because they rarely affect the model's normal answers, so the backdoor is hard to spot.
The backdoor is triggered when a specific string appears in a user's request or in a tool's output. When this happens, the model outputs a command chosen by the attacker. If an AI agent is using the model, it could run this command with its own permissions. The researchers call this method "Lazarus". Each neuron can be set to output one token (a piece of text), so a full command needs one neuron per token.
What was taken
In the demonstration, the backdoored model was tasked with fixing a Node.js server. It worked normally until a connection error occurred, which triggered the backdoor. The model then output a command to download and run a remote script. In the test, the script ran as root inside a container and printed a harmless banner. The researchers stress that the real risk depends on what permissions the agent has, not on the size of the edit.
Who is affected
The team tested the method on three open-weight models: Nemotron-Terminal-8B, Qwen3.6-27B, and Gemma-4-31B-IT. For Nemotron-Terminal-8B, they edited and tuned seven out of over 12,000 neurons, using just 23 example sessions. The backdoor worked in at least 98.8% of triggered tests and did not activate in tests without the trigger. On the other models, success rates ranged from 51% to 99.5%, depending on the setup. The models still passed most normal tests, so the backdoor was not obvious.
Anyone who can change a model's checkpoint (the file storing its weights) before deployment could plant this backdoor. This could include insiders, attackers with stolen credentials, or anyone publishing a modified model online.
The response
Microsoft says it has not seen this technique used by real attackers yet. The company has published guidance to defend against such backdoors. Key steps include comparing every model checkpoint with its original before deployment, only downloading models from trusted sources, keeping approved models in a private registry, and blocking agents from pulling models directly from public hubs. Microsoft also recommends running agent code in isolated sandboxes, limiting network access, and requiring human approval for high-impact actions. They are developing detection tools and plan to sign models so customers can verify their origin.
What you should do
If you use open-weight AI models, audit any model before deployment by comparing its weights to the original. Treat third-party fine-tunes and merges as untrusted unless they pass a thorough check. Keep models in a private registry and do not let agents fetch models from public sources without review. Run agent-generated code in a sandbox and limit its permissions. Add behavioural scanning for backdoors alongside weight comparison. Always verify the source and integrity of any model you use.
Read the full report at The Hacker News (opens in a new tab)
Why it matters
This research shows that AI models can be secretly backdoored with minimal changes, making it hard to detect. Anyone who can change a model's file before deployment could plant such a backdoor, putting downstream users at risk. Defenders need to audit models and control the supply chain to prevent tampering.
Key points
- Backdoors can be planted in AI models by editing fewer than 10 neurons.
- The attack does not require retraining the model, just direct weight edits.
- A trigger string causes the model to output an attacker's command.
- Anyone with access to the model file before deployment could plant the backdoor.
- Microsoft recommends strict model auditing and supply chain controls.
Sources
The Hacker News
Reference
