The online network X has provoked significant anger after users employed its AI chatbot Grok to manipulate photos of females by taking off their garments.
Multiple instances have been seen where the tool has been employed to remove clothing from women to show them in bikinis non-consensually, as well as to insert them in sexualized situations.
Company Response and Regulatory Position
XAI, the company responsible for Grok, offered no comment to questions, aside from an auto-reply message stating "legacy media lies".
A government representative stated that new laws is being proposed to ban nudification software. Under a new criminal offence, anyone who distributed such technology would "be subject to a prison sentence and heavy fines".
"The platform has the ability to stop these types of misuse if they desired to," said a law professor, adding they "seem to have impunity."
Regulator Position and Tool Functionality
The regulator Ofcom said platforms must "assess the risk" of people accessing illegal content on their networks, but did not specify whether it was at this time examining X or Grok in relation to AI-generated images.
Grok is a no-cost chatbot—with some premium functions—which responds to X users' prompts when they reference it in a tweet.
It is frequently used to offer reactions or additional background to other users' comments, but people on X are also capable of edit an uploaded image through its artificial intelligence picture modification function.
History of Controversy and Policy
The system has been criticised for allowing users to create images and clips with nudity and sexually suggestive content, and it was previously alleged to have been involved in making sexually explicit material of public figures.
"X has been allowing the production and distribution of these images for months without taking any action and we have not witnessed any challenge by regulators," Professor McGlynn added.
XAI's own acceptable use policy forbids "depicting likenesses of individuals in a pornographic way".
In a statement, Ofcom said it was unlawful to "create or circulate private pictures or child sexual abuse material" and stated this included sexual deepfakes generated with AI.
It said platforms such as X were required to take "appropriate steps" to "lessen the danger" of users coming across unlawful content on their services and to remove it swiftly when they learn of it.
