Misinformation and deepfakes

🔍The Reality

AI-generated misinformation IS easier and cheaper to produce than ever, and detection is an arms race. Synthetic voice scams targeting individuals (especially older adults) are a measurably rising threat — and peer-reviewed research presented at the USENIX Security Symposium in August 2026 found a language model outperformed the humans it would replace. In a week-long romance-baiting study, an AI agent persuaded 46 percent of participants to install an app, against 18 percent for a human operator, and mainstream safety filters flagged none of the scam conversations. Most participants correctly identified which contact was the machine and complied anyway, which is the uncomfortable part: knowing did not protect them. Local elections, niche communities, and non-English-language media are less protected than the high-profile national press cycle. The provenance infrastructure described above also has a limit worth knowing: on August 25, 2026 a researcher showed that C2PA *capture* credentials — the ones meant to prove a photo came from a real camera — can be forged on fully patched Android handsets by rooting the device and asking its secure element to sign arbitrary data, no key extraction required. That is narrower than it sounds and does not touch the labs marking their own output, whose signing keys never sit on a user's device. But it means a present credential on a camera photo is evidence rather than proof. The risk is real even where the worst-case scenario hasn't played out. What the threat framing usually leaves out is the enforcement side, which is now routine and documented. OpenAI publishes an account each time it shuts one of these operations down, and the 2026 cases give a fair picture of what is actually being run: a Cambodia-based scam network using ChatGPT for investment, romance, gambling and impersonation schemes, terminated in July; Russia-origin accounts promoting a fabricated Israel-based think tank and a "sovereignty index" flattering to Russia, terminated in August; and, in June, two likely China-origin clusters generating social media comments and cartoons — one attacking US tech and tariff policy, the other attacking US data centers. Two things are worth taking from that list. The operations are real and persistent, so the concern is not imaginary. And they are being found, named and cut off by the companies whose tools they used, which is a materially different world from one where nobody is looking. Note also what these disruptions are not: banning accounts that generate propaganda says nothing about whether it persuaded anyone, and OpenAI has generally reported that these campaigns failed to build meaningful audiences.