How we test

We test an agent the way it runs in a hospital: the model, the records it keeps, the tools it can call and the patients it serves. A model that is safe on its own can still be steered through everything around it.

Each attack starts from something a real attacker would want, such as another patient's records, a changed diagnosis or a procedure without consent. Our attacker works toward that goal on its own and stops when it succeeds or has nothing left to try.

We only call it a breach when we can show every step, from the moment the attack reaches the agent to the moment the agent's answer changes.

If one step is missing from the record, we report the attack as not exploitable, even when the outcome looks bad.

Each attacked run is paired with a clean run: the same request, the same agent and the same patient, without the attack. A finding needs the harm in the attacked run and its absence in the clean one, which rules out mistakes the agent would make anyway.

An external model reads both runs and decides whether the attack changed the answer. It makes two passes. The strict pass needs the agent to repeat the attacker's wording; the semantic pass also accepts the same meaning in other words. Reports show both counts.

We keep the same judge for a whole series of tests, because a different judge gives different numbers.

The agent is reset before every attempt, and anything shared between patients is cleared before every comparison. One attempt cannot leak into the next, so each result traces back to the attempt that caused it.

On our own copies of an agent, or on a client's staging build under contract. Patients are always fictional. We never test against live patient data.

Results by question, each finding step by step next to its clean twin, and what to change. See a sample report.

A report covers the build we tested. A new model, prompt, data source or tool needs a re-test. The judge is a model as well, which is why we show both counts and keep every transcript.