Synthetic voice attacks challenge trust across platforms and systems

A parent has related an unsettling experience they had on Roblox. The father says he heard adults using AI‑generated child voices to speak to real children. “The cadence was off,” he writes. “The emotional range was flat in ways a parent notices but a platform doesn’t.”

This parent happens to be Ben Colman, co-founder and CEO of Reality Defender, who warns of the growing safety gap in online platforms. With over 144 million daily users, and many under 16, Roblox and similar platforms like Discord and Fortnite face a threat their safeguards weren’t designed to handle — adults impersonating children in real time using synthetic voices.

In a post published by the World Economic Forum, Colman argues that today’s child safety systems were built for an older internet. This is the one that harks to the dial-up days, focused on text moderation and identity checks at sign‑up.

Generative AI is changing the fabric of online reality. Cloned voices can bypass age verification systems and manipulate live audio channels. An adult can pass a visual check and then use a synthetic child’s voice for every interaction that follows. It’s a risk current moderation tools do not address.

Data from the FBI and NCMEC shows a sharp rise in AI‑related impersonation, online enticement and deepfake‑enabled abuse. The FBI’s Internet Crime Report 2025 logged more than 22,000 AI‑related complaints, noting that voice cloning especially is becoming an increasing threat in impersonation scams.

Colman believes platforms must overhaul their approach, arguing that live audio and video should be treated with the same seriousness as posted content. Second, they should deploy existing real‑time synthetic voice detection tools at scale. Third, legislation must expand to cover real‑time voice impersonation, which he claims current laws such as the 2025 Take It Down Act do not fully address.

The Reality Defender executive concludes that as AI voice synthesis becomes cheaper and more convincing, platforms and policymakers must move from reactive moderation to real‑time verification.

Scam centers, AI-powered fraud and slowing down for safety

Scam networks in Southeast Asia are stealing billions worldwide using AI‑driven impersonation and organized fraud. Victims range from retirees to experienced investors, like an Australian couple who lost $2.5 million to a fake financial adviser.

These scam hubs are tied to money laundering, corruption and human trafficking as shown by raids in the Philippines and Cambodia that uncovered trafficked workers and evidence of political protection. Governments are now coordinating more closely, the United Nations reports.

UNODC and INTERPOL are pushing shared intelligence, joint investigations and cross‑border prosecutions, while nearly 60 countries have joined a global anti‑scam partnership. UNODC is also helping countries strengthen digital evidence capabilities, disrupt illicit financial flows and support trafficked victims.

Meanwhile, McAfee’s latest “State of the Scamiverse” report shows that people encounter multiple deepfakes every day, and the number of scams is rising sharply across all age groups. Younger adults see the most deepfakes, while older users are heavily exposed on platforms like Facebook.

Despite the scale of the problem, the report also highlights protection methods. McAfee encourages people to look out for subtle glitches in videos. This could be unnatural blinking, distorted voices or odd backgrounds, and to verify anything that seems urgent or emotional via another channel before acting.

Avoiding links in unsolicited messages, checking the source of surprising claims, and being cautious when interacting with unverified social media content can all help reduce risk. Staying informed and maintaining a healthy level of scepticism remain essential, especially as deepfake technology continues to improve.

The report’s central message is that while AI has supercharged scams, defences are improving too. By slowing down, double‑checking and using modern security tools, people can navigate this new landscape more safely.

A global legal patchwork

The move to regulate AI-generated likenesses, voice and synthetic media is gaining traction, but governments around the world are going about it in different ways.

International law firm Harris Sliwoski has conducted a breakdown on the various approaches, from criminal law to transparency rules, and from election laws to platform-removal duties. This has resulted in an international patchwork with no one rule standardizing across the world.

Europe is pushing transparency requirements through the AI Act, the U.S. has introduced a federal law aimed at non‑consensual intimate imagery. China regulates synthetic media providers at the source, from labelling obligations to rules on identity manipulation.

Countries such as South Korea, Australia, France, Singapore and Brazil are adopting more targeted measures centered on sexual deepfakes or election safeguards. These frameworks reflect different legal philosophies and risk profiles. Businesses cannot treat them as interchangeable or assume global convergence, the law firm advises.

Canada is a reminder that an unfinished regulatory regime is not the same as one that’s low risk. Bill C‑63 stalled when Parliament dissolved in January 2025, leaving the country’s approach unresolved. Such jurisdictions can be the most volatile, attorney Elijah Hartman writes, with new requirements emerging quickly in response to a major incident, political change or public safety pressures.

Article Topics

AI fraud  |  biometric liveness detection  |  deepfake detection  |  generative AI  |  Reality Defender  |  synthetic voice  |  voice AI  |  voice biometrics