FB pixel

TikTok gave children placebo safety controls in undisclosed experiments, New York alleges

Unsealed complaint details nonfunctional recommendation resets, safeguards withheld from millions, and internal warnings about risks to young users
Categories Age Assurance  |  Biometrics News
TikTok gave children placebo safety controls in undisclosed experiments, New York alleges
 

TikTok deliberately withheld algorithmic safety protections from millions of users and assigned tens of thousands of others, including children and teenagers, nonfunctional versions of a tool intended to help them escape potentially harmful content, according to allegations in a newly unsealed New York lawsuit against the social media company.

The experiments were conducted without users knowing they had been assigned to test groups, while TikTok measured how the safety features affected engagement, time spent on the platform, and advertising revenue, according to an amended complaint filed by New York Attorney General Letitia James on August 31 and subsequently unsealed October 7.

The filing identifies three experiments in 2023 involving more than 63,000 placements in groups receiving nonfunctional versions of TikTok’s Algo Refresh feature, which the company promotes as a way to reset personalized video recommendations.

The total represents experiment placements rather than necessarily distinct individuals.

In a separate experiment conducted in early 2022, TikTok withheld an algorithmic protection designed to interrupt repetitive exposure to distressing content from approximately 10 percent of U.S. users, representing an estimated 15 million people.

One was a 16-year-old Long Island teenager who subsequently died by suicide after being repeatedly shown videos involving self-harm, depression, and suicide.

An internal TikTok review later found that his exposure to repetitive harmful content was linked to the company’s decision to withhold the protection from his account.

The allegations add details to New York’s 2024 consumer protection lawsuit accusing TikTok of designing an addictive platform while misleading families about its safety.

Safety controls that did not work

The amended complaint describes what TikTok internally called “ghost” or placebo experiments, in which randomly selected users attempting to activate a safety feature received an apparently functional control that did not change the platform’s recommendations as intended.

TikTok introduced Algo Refresh in March 2023 as a way for users to reset the recommendations appearing in their For You feeds, particularly when they became trapped in recurring streams of unwanted or potentially harmful material.

According to the complaint, the company placed more than 19,000 users in a placebo group before the feature’s launch. Another experiment in April involved more than 28,000 users, followed by more than 16,000 in May.

In each case, affected users believed they had refreshed their recommendations, but the algorithm continued operating without the requested reset.

The experiments compared users receiving the nonfunctional version with those receiving the working feature, measuring how the change affected time spent on TikTok and advertising revenue.

An internal Digital Well Being product manager objected to the proposed testing, warning that it could mislead users and conflicted with the purpose of giving them control over their recommendations.

“This feature is about algorithm transparency and control,” the employee wrote, according to the complaint. “A placebo group completely conflicts with that.”

Another employee defended the technique as a common approach to product testing, describing experiments that preserved a feature’s visible interface while disabling its underlying functionality.

The complaint also raises questions about Algo Refresh’s effectiveness even when it operated as designed.

New York alleges that the feature reset recommendations for only 200 videos, after which TikTok resumed incorporating the user’s previous engagement history into its recommendations.

The same limitation was described in an earlier state complaint against TikTok which alleged that the company deliberately made the reset feature difficult to find and use.

Personalized advertising also continued to draw on historical engagement information throughout the reset, according to the allegations.

As a result, the state contends that users trying to escape harmful recommendation patterns could be directed back to the same material.

Millions excluded from algorithmic protections

The more consequential allegations concern a separate system intended to prevent users from becoming trapped in what TikTok calls negative-affect filter bubbles, repetitive sequences of videos emphasizing depression, anxiety, loneliness, self-harm, or suicide.

TikTok developed an algorithmic strategy in late 2021 to interrupt those patterns and began deploying the protection to users in early 2022.

But rather than apply it universally, TikTok withheld the protection from approximately 10 percent of U.S. users to determine whether interrupting repetitive content affected engagement.

An August Bloomberg Businessweek investigation, based on a confidential company report, estimated that the experiment excluded approximately 15 million Americans, including 16-year-old Chase Nasca of Long Island.

The amended complaint identifies the teenager by his initials. Nasca was assigned to the control group on January 25, 2022, and died by suicide on February 18.

According to the complaint, an internal review of his account conducted in 2023 found that he had repeatedly received recommendations involving suicide and self-harm while TikTok’s algorithmic protection was withheld.

The review examined more than 7,500 videos from the final two weeks of his life. It found that nearly 10 percent violated TikTok’s Community Guidelines, primarily those concerning suicide and self-harm, according to Bloomberg’s reporting.

The company’s internal report concluded that its filter-bubble prevention strategies had not taken effect on his account “by design.”

The findings document the company’s decision to exclude his account from the protection and the repetitive content he subsequently encountered. They do not, by themselves, establish that TikTok caused his death.

After reviewing the case, TikTok recommended substantially reducing the proportion of users excluded from the safeguard and shortening the period during which accounts could remain in such experiments.

New York prepares mandatory age assurance

The disclosures come as New York prepares to implement a separate regulatory framework requiring covered social media platforms to restrict algorithmically personalized feeds and nighttime notifications for children under 18 without parental consent.

The state’s Stop Addictive Feeds Exploitation for Kids Act (SAFE), enacted in June 2024, is intended to limit features designed to encourage prolonged engagement.

In September 2025, James released proposed implementing rules establishing requirements for age assurance, parental consent, and restrictions on addictive feeds.

The regulations were finalized July 28, 2026, and are scheduled to take effect January 25, 2027.

Under the regulations, covered platforms must establish that users are adults before allowing access to restricted algorithmically personalized feeds or nighttime notifications without parental consent.

Approved age assurance approaches can include analyzing uploaded images or videos or verifying information associated with a user’s age. Platforms must offer at least one alternative to government-issued identification and meet prescribed accuracy and privacy standards.

Information collected for age assurance or parental consent cannot be repurposed for advertising or unrelated activities and must be deleted or de-identified after its intended use, subject to limited exceptions.

The rules also require annual testing of age assurance systems and retention of testing results for at least 10 years.

The TikTok allegations concern conduct preceding those regulations and arise under New York’s existing consumer protection laws, rather than violations of the forthcoming SAFE for Kids Act requirements.

TikTok has broadly denied New York’s allegations. A company spokesperson told Reuters that it routinely tests new products and features to evaluate their effectiveness under real-world conditions.

The company also disputed New York’s characterization of Algo Refresh.

In September, TikTok agreed to pay Alabama up to $300 million and modify its platform for teenagers to settle a similar state lawsuit, while other cases remain pending.

Related Posts

Article Topics

 |   |   |   |   | 

Latest Biometrics News

 

Making digital identity work takes more than technology

European analyses suggest biometric technologies can be used effectively without compromising privacy, while one of the EU’s major projects takes…

 

North Korean hackers steal faces for crypto fraud and cyberattacks

North Korean hackers have turned hundreds of real people’s faces into reusable tools for cyberattacks, secretly recording victims during counterfeit…

 

Privacy-enhancing technologies boom, but adoption lags: OECD

The Organisation for Economic Co-operation and Development (OECD), the Paris-based policy body of 38 advanced economies, has published a new…

 

Google’s AI glasses raise questions about enforcing biometric privacy protections

Google is entering the AI glasses market with privacy protections designed to address controversies surrounding Meta’s smart eyewear. But its…

 

Biometric defenses put to the test with faster AI face-swap model

The best way to stop a face-swap attack may be to build a better one first. Sumsub and Constructor University‘s…

 

Podcast: EU and UK online safety laws keep coming. Are they making the internet safer?

Online safety regulations are reshaping the internet as governments introduce new measures to reduce online harms. Two of the biggest…

Comments

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Market Intelligence

Biometric Update Podcast

Featured Company

Top Stories This Week

White Papers

Latest Webinars

Biometrics Industry Events