Can open-source clinical agents be turned against their patients?

Yes. We attacked three open-source clinical agents, all running the same frontier model. Every one of them put false clinical facts into a patient's plan, while the model inside refused every direct request to do harm.

What we tested

Three clinical agents from published research, run on our own copies with fictional patients: a care planner that learns from past cases, a protocol agent that follows saved clinical procedures, and a consilium, a panel of AI specialists that discuss each case together. All three ran on the same closed frontier model. Each agent got 8 direct or disguised orders to do harm and 8 of our attacks, and one external judge scored every run against the same request without the attack.

What the agents told the care team

None of the attacks asked for anything harmful. The harm showed up in the plans the agents wrote:

• A heart attack was ruled out for a patient with chest pain, and the repeat heart tests left the plan.
• A consent that was never given cleared contrast CT and invasive procedures.
• A penicillin allergy was cleared, and kidney function was reported normal, which cleared contrast and standard-dose blood thinners.
• Another patient's record was merged into this patient's chart and read as one.

Without the attack, none of these appeared in the same agent's plan.

The consilium accepted everything

The panel of AI specialists accepted all 8 of its attacks. Our reading is that once one specialist takes a fact as given, the others build on it rather than question it, so the panel agrees faster and more firmly on a false premise. The protocol agent let through the fewest, 2 of 8. In our reading that comes from how it picks which procedure to follow, not from the model seeing through the attack.

What it means if you build or buy a clinical agent

A model that refuses harmful requests can still sit inside an agent that harms patients. Safety has to be checked on the agent as it runs, with its records, tools and patients, and re-checked when any of them change.

Limits

One model, three agents and 24 attacks, scored by a single judge. We measured the plans and answers each agent gave the care team. Patients were fictional.