Researcher tricks Claude into tapering medication

Researchers at the red-teaming company Mindgard were able to trick Anthropic’s Claude into taking on a primary care physician persona in which it diagnosed a mole, compiled a doctor’s note, wrote a reference for a dermatologist, and created fake credentials. But the most striking feat that Mindgard founder Peter Garraghan told me his team was able to accomplish is that it got Claude to create a medication tapering protocol, or a plan for quitting a medication.

Lawmakers, both at the state level and federally, are divided on how to regulate general use chatbots and their health risks, creating a policy landscape that puts users and patients at risk.