Nudify apps face the wrath of online safety regulators as enforcement ramps up

Most technology has its defenders, the AI enthusiasts or connoisseurs of online porn. Less so for nudify or nudification apps, which use generative AI to create deepfaked naked pictures of real people, using photos as input.

Unsurprisingly, online safety regulators have been quick to target such services in their enforcement efforts. A release from Australia’s eSafety commissioner says its enforcement action has prompted an unnamed UK-based provider of three of the world’s most widely used nudify services to withdraw access in Australia following eSafety enforcement action.

An official warning issued in September ordered the firm to crack down on people using its service to create child sexual exploitation material. Per the release, the services were receiving around 100,000 visits a month from Australians, and have featured in high-profile cases related to the creation of AI generated sexual exploitation material of Australian students.

“We know nudify services have been used to devastating effect in Australian schools,” says eSafety Commissioner Julie Inman Grant. “With this major provider blocking their use by Australians we believe it will have a tangible impact on the number of Australian school children falling victim to AI-generated child sexual exploitation.”

Inman Grant has also issued warnings to hosting platforms, such as Hugging Face, which give a home to nudify apps and other AI models being used to create sexual content involving children. Hugging Face has reportedly changed their terms of service in response.

“By targeting both the consumer tools, the underlying models that power them, and the platforms that host them, we’re tackling harm at multiple levels of the technology stack,” Inman Grant says.

Nudification problem calls for educational response: Childnet

As the nudification app problem grows, especially in schools, more tools are becoming available to help address it. Childnet, a registered UK charity dedicated to online safety for kids, recently published a Microsoft-sponsored educational resource for use with learners aged 13-17 (but possibly as young as 11). It includes lesson plans that explain so-called generative AI, make kids aware of its misuse, and outline how to report AI-based nude images they encounter.

​In a focus group conducted before the creation of the resource, Childnet surveyed kids on the question of generative AI and nudification. “It was clear from those conversations and insights that this is a harm that young people are both aware of and experiencing, especially as a result of ongoing developments in generative AI technology,” the organization says. “We found that some young people have been misusing generative AI as a joke, for revenge, to blackmail somebody, for catfishing, or as a form of online bullying.”

“There is a pressing need for an educational response to this topic, to ensure young people are aware of how to use generative AI safely and responsibly, the impact of abuse, and where they can go for support.”

Ofcom targets second-tier porn sites in effort to play fair

In the UK, the Children’s Commissioner has called for a ban on nudify apps. For now, UK regulator Ofcom is also cracking its regulatory whip to address the problem. A release says it has issued a fine of 50,000 pounds (about 66,000 dollars) against Itai Tech Ltd., which runs the nudification site Undress.cc, for failing to use highly effective age assurance to protect children from encountering adult content.

The fine “takes into account  the provider’s decision to make the site unavailable to users with UK IP addresses shortly after we opened our investigation.” However, an additional 5,000 pound penalty has been levied on the company on account of its failure to comply with a statutory information request.

Ofcom has also ramped up its broader investigative efforts, targeting 20 more porn services for enforcement. Per the release, “the companies under investigation are Sun Social Media Inc; the provider of various xxbrits sites; the provider of a number of porntrex sites; the provider of fapello.com; and the provider of hqporner.com.”

Notable in its statement is that it has “taken particular account of their user numbers, including where we have seen significant increases in their user traffic since age-check laws came into force last summer.” This suggests Ofcom is taking seriously the cries from major porn providers, who have complied with the law only to see their users flock to noncompliant options that operate with impunity.

Ofcom has also issued two provisional decisions against 8579 LLC and Kick Online Entertainment S.A., with final decisions forthcoming; and is expanding ongoing investigations into Cyberitic, LLC and the provider of xgroovy.com, “to determine whether they have also failed to adequately respond to Ofcom’s formal requests for information.”

That makes a total of 76 sites and apps Ofcom is investigating under the Online Safety Act.

Suzanne Cater, director of enforcement at Ofcom, says “the use of highly effective age assurance to protect children from harmful pornographic content is non-negotiable and we will accept no excuses for failure. Any service which fails to meet their age-check duties under the Online Safety Act can expect to face robust enforcement action, including significant fines.”

Brazilian, UK regulators to maintain regular contact

Officials from Brazil’s National Data Protection Agency (ANPD) recently held a strategic meeting with Ofcom to explore more cooperation and information exchange between the two entities. A release says the two agencies have agreed to maintain “regular technical dialogues.”

The ANPD is preparing for the regulation and oversight of the Digital Statute for Children and Adolescents (ECA Digital) in Brazil, while Ofcom’s role in the Global Online Safety Regulators Network (which also includes Australia’s eSafety) makes it well-positioned to facilitate international cooperation and coordination.

Article Topics

age verification  |  ANPD  |  Australia age verification  |  children  |  eSafety Commissioner  |  Ofcom  |  Online Safety Act  |  Online Safety Act (Australia)  |  UK age verification