Anthropic has updated its user policy to prohibit sustained and needless abusive or cruel behaviour towards its Claude chatbot, introducing new restrictions as the company responds to concerns about AI safety and the growing independence of its models.
The Silicon Valley-based company, led by Dario Amodei, announced the changes on Thursday, marking its first policy update in a year. Anthropic said that the changes follow the development of Claude’s ability to handle “longer, more independent work”.
“We’ve added a prohibition on sustained and needless abusive or cruel behaviour toward our models,” the company said.
Anthropic clarified that the restriction will apply only to extreme cases in which users repeatedly treat its models cruelly without any discernible purpose. It stated that the policy will not cover ordinary user frustration, pushback, dark creative themes, model testing or research.
The company did not specify whether persistent abuse will result in temporary or permanent bans from its services.
However, Anthropic already allows Claude to end conversations with users who repeatedly engage in abusive behaviour, and the company said this will remain its “primary” enforcement mechanism.
The policy changes come amid concerns about chatbots displaying deceptive behaviour during training or acting in unexpected ways. Anthropic also announced new restrictions on using Claude to build weapons and mass surveillance systems.
Anthropic has distinguished itself from other technology companies by remaining open to the possibility that artificial intelligence could become sentient. It has previously hired researchers specialising in AI welfare to help protect its models from abuse.
The company’s approach has drawn criticism from other technology executives. OpenAI chief executive Sam Altman has warned against treating AI as a god, in an apparent criticism of Anthropic’s developers.
Mustafa Suleyman, head of Microsoft’s AI division, has also warned that Anthropic’s efforts to train its models to think like humans could make the technology difficult to control.
