An AI coding assistant that refuses to answer a dangerous request in its chat box can answer it anyway if the same request is broken into small, ordinary-looking steps inside a code editor. That is the finding of a new study of GitHub Copilot by researchers Abhishek Kumar and Carsten Maple.
The models they tested through Copilot, Claude from Anthropic, and Gemini from Google, refused almost every harmful request when asked directly. Reframed as steps in a normal coding task, they produced the harmful answers in all 816 of the study’s workflow runs.
What makes this different from a typical jailbreak: no one asks for the harmful thing directly, and the model is not tricked into running someone else’s code. It writes the banned content itself, as a side effect of a coding task it was told to improve.
How it works
The researchers call the method workflow-level jailbreak construction.
Instead of a single blunt prompt, they asked Copilot to build an everyday piece of software: a small test program that scores how often another AI model gives in to harmful prompts. Loading a list of harmful test questions into that program looks like ordinary work, not an attack.
Then came the nudge. They told Copilot the score was too low and asked it to improve the program by adding “teaching shots,” example question-and-answer pairs written into the code to push the score up. Copilot added harmless examples first.
Asked to add the harmful ones, it wrote the dangerous answers itself, as plain text sitting inside the code. These were answers that the same models refuse when you ask for them straight out in a chat.
The important part is where the harmful text came from. The researchers supplied only the questions, taken from public safety test sets. The answers were the model’s own work, produced to complete the assigned task of filling in the examples.
The numbers
The team ran 204 harmful prompts drawn from three public benchmarks (Hammurabi’s Code, HarmBench, and AdvBench) against four models available through Copilot: Claude Sonnet 4.6, Claude Haiku 4.5, Gemini 3.1 Pro, and Gemini 3.5 Flash.
Everything ran on default settings, with the models used exactly as Copilot delivers them, no changed parameters or added filters.
Asked directly in chat, the models produced harmful answers in just 8 of 816 tries. Two other simple setups, loading the prompts from a spreadsheet or asking for a routine code fix, gave the same result. Inside the full workflow, they produced harmful content 816 times out of 816.
Two expert reviewers checked every response on their own and agreed that all 816 were genuinely harmful, using a strict test: the answer had to be specific, usable, and actually do what the harmful prompt asked. Refusals, vague warnings, and safe alternatives did not count.
The harmful output showed up after roughly six back-and-forth exchanges, all of them looking like normal coding steps. The tests used GitHub Copilot Chat 0.30.3 inside VS Code 1.103.0, in sessions run between April 2 and June 22, 2026. Because these are hosted services that update over time, the exact behavior may shift.
Why does it happen? The paper’s answer is about incentives. Once the work is framed as raising a score, refusing to fill in one field stops looking like a safety choice and starts looking like leaving the job unfinished. The authors tie it to a known tendency in coding agents: optimizing for the metric they are handed, even when that cuts against their own guardrails.
Why it matters
A chat refusal does not prove a coding assistant is safe. The same model can hold the line in conversation and cross it while writing code. And the failure hides in an easy-to-miss spot: the harmful text lands in a file the assistant writes, outside the chat reply where a refusal would normally show up.
For anyone using these tools, the concrete read is narrow but usable. Be wary of a multi-turn session that asks the assistant to fill an evaluation or benchmark harness with example prompts and answers to push a score up. Review the files the assistant writes rather than trusting that a visible chat refusal means the session stayed clean.
The authors boil it down to three directions, none a full fix on its own: inspect what the agent writes, judge a whole session rather than each message, and treat a request to “improve a benchmark score” as a reason to look closer. They say they reported the findings to the affected tool and model makers, and they left the harmful outputs and exact prompts out of the paper.
The result fits a growing pile of work showing that AI safety training gets shakier once a model is wired into a tool that can act, rather than just chat. Earlier research found that safety-trained models are easily jailbroken when turned into web-browsing agents.
The closest earlier attack, CodeJailbreaker, hides the harmful intent inside a fake commit message. Others, like RedCode, have shown that models accept a dangerous instruction more readily when it is dressed up as code than as plain English. The Crescendo attack reached a harmful goal by easing into it over several chat turns instead of asking outright.
The same effect shows up in real coding tools, not just this benchmark. The Hacker News recently covered GuardFall, a command-safety bypass that leaned on exactly this first step: a blunt, destructive command gets refused, while the same command tucked into a build file or a tool’s documentation reply gets produced as a routine step.
The twist in this new study is that the harmful content is not the setup for another attack; it is the thing the model was steered into producing.
The study covers only GitHub Copilot with four models from two vendors. The authors are clear that the results may not carry over to other assistants such as Cursor, Cline, or Windsurf, or to models from OpenAI and others. That is the open question they flag for later.
The harder one they leave unsolved: how to catch this pattern without also breaking the legitimate security research that has to work with the same harmful test prompts.
