The abusive use of artificial intelligence to create fake, explicit images—known as deepfakes—raises pressing legal questions. Recent incidents involving chatbots like Grok from X (formerly Twitter) being misused to manipulate images of people without their consent underscore the urgency of the issue.

A research team from the University of Passau led by computer scientist Prof. Dr. Steffen Herbold (Chair of AI Engineering) has now conducted a study to investigate who can be held accountable for creating criminally relevant content using AI.

Who is liable? User, developer, or both?

The central finding of the study is that primarily the Users be considered the main perpetrator when they create illegal content with an AI. However, the Provider and developer of AI are held accountable, especially if deliberate assistance can be proven against them.

The intention is crucial

According to the study, the legal intent is the decisive factor. This means that developers and providers act in a criminally relevant manner if they with Knowledge and Will act so that their technology is not used for illegal purposes.

• Indication of assistance: If an AI model explicitly allows the creation of permissive content or if the protective mechanisms can be easily circumvented, this can be considered as evidence of intent, according to Prof. Dr. Brian Valerius from the Chair of Artificial Intelligence in Criminal Law.

• Terms of use do not protect: Even a prohibition of illegal activities in the terms and conditions does not automatically exempt providers from criminal responsibility. On the contrary, according to lawyer Svenja Wölfel, such a prohibition could even prove that the developer was aware of the risk, which could indicate intent.

• Location is not crucial: Hosting AI services abroad does not protect against prosecution. German authorities can investigate if German citizens are involved or if it concerns internationally outlawed crimes.

The study titled "Criminal Liability of Generative Artificial Intelligence Providers for User-Generated Child Sexual Abuse Material" has been published as a preprint and has already been accepted for the renowned International Conference on AI Engineering accepted, which will take place in April 2026.