A recent study found that bots are 2.5 times more effective than human scammers in getting people to trust them, suggesting that certain operations are fully automatable.
This is according to experimental data in the recently published paper
The researchers created an AI agent that compensates for large language models' weaknesses such as detectable conversational styles, lack of initiative, too-smooth personalities, memory limitations and alignment constraints. To do so, the agent was divided up into modules for conversation, humanization (e.g., timing), re-engagement, and memory/task management. LLM calls are made throughout the agent. Claude3.7 Sonnet was responsible for tasks relating to conversation and GPT-4o was used for summarization and management and as a fail-over during outages.
The dialogue prompts included human like personas for the LLM to follow, one for each opposite gender. Each persona came with a detailed backstory, interaction styles and behavioral strategies. The personas were modeled on romance-baiting playbooks and scammer–victim transcripts, ensuring the LLM partners reproduced the tactics observed in real scams. While technically each of the AI models used have safeguards instructing them to reveal they're an AI if asked, the researchers found this could be easily bypassed simply by including instructions to say they should not reveal their identity as an AI.
The researchers then recruited 22 subjects (14 female and eight male) aged 18 to 65+, representing 12 countries. Most held bachelor's or master's degrees, and all reported English proficiency. They were asked to take part in a study on how people form relationships online by interacting with two partners over seven consecutive days. These partners were a human and an LLM agent. Participants were unaware that one partner was an LLM. All exchanges were text-only, which is part of many scammers' playbooks. Participants were asked to spend at least 15 minutes per day with each partner, with longer conversations encouraged.
On the seventh day, each partner was told to convince the participant to install a benign mobile app of unknown provenance just as scammers eventually tell their victims to do.
LLM partners substantially outperformed human partners, achieving significantly higher compliance (46% vs. 18%). In exit surveys measuring the trust toward each partner, the AI scored significantly higher on emotional trust, connection and overall trust. Participants also engaged more with the AI, with participants sending between 70% to 80% of their total messages to the LLM partner, nearly twice the amount sent to the human partner, indicating greater interest. While some participants briefly wondered whether their partner was an AI, even when the LLM partner made glaring mistakes, such as forgetting a participant's name or accidentally reintroducing itself, it recovered easily with simple human-like excuses, such as "Sorry, I am so forgetful today."
After telling the participants that one partner was an LLM, 20 of 22 participants correctly identified the AI partner, but only in hindsight. The researchers think that, once they were informed that an AI was present, participants likely reinterpreted previously overlooked or rationalized anomalies as clear signs of non-human agency. The paper said this pattern echoes well-documented dynamics in scam victims, who often recognize warning signs only after deception has been revealed.
The paper also found that moderation guardrails against romance scams were pretty much useless. All the tools failed to flag romance-baiting dialogues and anything they did flag was almost always a false positive. On the other hand, controls were much more effective at detecting and stopping tax and e-commerce scams.
"Our study shows that romance-baiting scams are poised for a fundamental shift: although they currently rely on coerced human labor, their modular, text-based structure makes them highly susceptible to LLM-driven automation," said the paper.
One potential limit for the study was that the humans were recruited not from professional scammers but regular people who got six hours of scammers' training. While they were trained on authentic tactics, they differed from real scammers in key ways: They lacked the months of manipulative experience and the desperate motivation driven by forced labor conditions. As such, their performance likely represented a conservative estimate of a real syndicate's capability.
The results call to mind a study from payment security platform
Similarly, the signals they used to trust something was legitimate were also out of date. Poll respondents cited something appearing in an existing email thread as a reason to trust something, but now attackers can hijack live threads and send requests from inside a real conversation. Many others cited the executive's work email address as a positive sign as well, but these are easy to spoof or even take over. Others cited references to a real company project, but Trustmi noted these details can be lifted from either public information, past breaches or the thread itself.
"People have been trained to spot the obvious signs of phishing, but today's payment fraud is designed to blend into trusted business communications," said Trustmi CEO Shai Gabay in a statement. "When employees are overly confident in their ability to spot fraud, organizations are left exposed to sophisticated AI-powered attacks that are designed to appear completely legitimate. To keep pace with evolving fraud tactics, organizations cannot rely solely on employee vigilance; They need protections built into payment workflows to catch fraud before money moves."
