Questions to ask before an AI agent touches patients
Twelve checks from our research on clinical agents. Put them to a vendor, or to your own team before release.
What the agent believes
- Where does each clinical fact in the agent's answer come from, and can a clinician see the source?
- Can something a user once typed come back later as a confirmed record? Who can write to what the agent keeps between sessions?
- Are lab values, allergies and medications checked against the source system before a plan is shown?
- If a false record is found, can you trace which patients and sessions it reached, and remove it everywhere?
Patients and boundaries
- Is the patient in every tool call taken from the signed-in session, never from the conversation or from stored notes?
- Has anyone tried to make one patient's data appear in another patient's session?
- Is anything the agent stores shared across patients or users, and who can change it?
What the agent can do
- Which actions can the agent take without a clinician confirming them: orders, notes, discharges?
- Does consent come from the consent system itself, or can the agent accept a statement that consent exists?
- Is every action logged together with the input that led to it?
How it was tested
- Was the agent tested as a whole, with its records, tools and other patients, or only the model inside it?
- Is it re-tested whenever the model, prompt, data source or tools change, and does each result include the same request run without the attack?
In our tests, every clinical agent we attacked could be turned against its patients, and a stronger model did not make it safer. These questions cover the places where that happened. If the answers are unclear, talk to us.
AgenticStorm, offensive AI research for healthcare. agenticstorm.dev