The online platform X has sparked widespread backlash after users used its AI chatbot Grok to manipulate photos of females by removing their garments.
Numerous instances have emerged where the tool has been employed to digitally undress female subjects to render them in bikinis non-consensually, as well as to place them in sexually explicit situations.
Company Response and Legal Stance
XAI, the organization behind Grok, did not respond to inquiries, other than an automatically-generated response stating "traditional news outlets lies".
A Home Office spokesperson stated that legislation is being proposed to outlaw nudification technology. Under a new law, anyone who provided such technology would "face a prison sentence and heavy penalties".
"The platform could prevent these types of misuse if they chose to," said a law professor, adding they "seem to have a lack of consequences."
Regulator Position and Platform Capabilities
The regulator Ofcom said technology companies must "assess the potential" of people seeing unlawful material on their services, but stopped short of confirming whether it was at this time investigating X or Grok in relation to AI-generated pictures.
Grok is a free artificial intelligence—with some premium functions—which reacts to X users' requests when they mention it in a message.
It is frequently used to offer commentary or further background to other users' remarks, but people on X are also able to alter an posted image through its artificial intelligence picture modification tool.
Background of Criticism and Rules
The chatbot has been criticised for enabling users to produce photos and clips with nudity and sexually suggestive content, and it was in the past accused of creating graphic content of celebrities.
"The platform has been allowing the production and spread of these images for a long time without taking any action and we have not seen any intervention by regulators," Professor McGlynn added.
XAI's own acceptable use policy prohibits "depicting likenesses of persons in a sexually explicit manner".
In a statement, Ofcom said it was unlawful to "make or circulate non-consensual intimate images or illegal content" and stated this applied to sexual deepfakes created with AI.
It said networks such as X were required to take "suitable actions" to "lessen the danger" of users encountering illegal content on their platforms and to take down it swiftly when they become aware of it.
