AI-generated profile pictures are becoming a significant tool for coordinated manipulation on X, with researchers from Germany identifying nearly 8,000 accounts using synthetic faces primarily focused on amplifying political messages and crypto schemes.
"Recent advances in the field of generative artificial intelligence (AI) have blurred the lines between authentic and machine-generated content, making it almost impossible for humans to distinguish between such media," the study notes.
The research, conducted by teams from Ruhr University Bochum, GESIS Leibniz Institute, and CISPA Helmholtz Center, found that more than half of these accounts were created in 2023, often in suspicious bulk creation events.
"A significant portion of accounts was created in bulk shortly before our data collection, which is a common pattern for accounts created for message amplification, disinformation campaigns, or similar disruptive activity," the researchers explain.
This finding gains additional context from a recent platform analysis carried out by the Center for Countering Digital Hate showing that X owner Elon Musk's political posts favoring Donald Trump received 17.1 billion views—more than double all US political campaign ads combined during the same period.
“At least 87 of Musk’s posts this year have promoted claims about the US elections that fact-checkers have rated as false or misleading, amassing 2 billion views. None of these posts featured a Community Note, X’s name for user-generated fact-checks,” the CCDH report says.
The use of generative AI—be it to generate fake images or text—was easy to spot as synthetic-face accounts exhibited distinct patterns separating them from legitimate users. "fake-image accounts have fewer followers (mean: 393.35, median: 60) compared to real-image accounts (mean: 5 086.38, median: 165)." The study also found that fake accounts tend to interact less with their ecosystem of followers and instead post messages without replying or engaging with other accounts.
The study also emphasized specific patterns suggesting coordinated activity: "We notice that 1,996 fake-image accounts (25.84%) have exactly 106 followers. Our content analysis reveals that these accounts belong to a large cluster of fake accounts involved in coordinated inauthentic behavior."
The research team's sophisticated detection methods achieved remarkable accuracy, with researchers reporting almost 100% certainty in their findings.
Researchers also said that a lot of the accounts don’t live for too long, with over half of them being suspended in less than one year.
Content analysis also revealed carefully orchestrated posting patterns across multiple languages. The study identified "large networks of fake-image accounts that were probably automatically created and that participated in large-scale spamming attacks." English-language accounts focused heavily on controversial topics, with researchers finding that accounts prefer to tackle issues like the war in Ukraine, the US elections, and debates on COVID-19 and vaccination policies.
Outside of politics, a lot of these accounts also promoted crypto scams and sex-related content.
Looking ahead, researchers plan to expand detection capabilities to spot AI images generated with other models based on different technologies—like Diffusion models instead of Generative Adversarial Networks (GANs). They also want to improve their methodology to find more ways to identify what they categorize as “coordinated inauthentic behavior across social platforms."
Edited by Josh Quittner and Sebastian Sinclair