Generative AI is now capable of producing fake documents realistic enough to fool automated systems, creating new opportunities for fraud and new challenges for those trying to prevent it.
This new capacity came as part of OpenAI's
Anti-fraud professionals like Mason Wilder, research director with the Association of Certified Fraud Examiners, believe the issue is not so much the fake documents themselves but the fact that they can now be quickly and easily produced at an industrial scale. He noted that people have been forging documents since time immemorial, however doing so tended to need a lot of time, effort and expertise, which created a high bar for such activities. This meant that even if someone had considered, say, inflating their expense reports with fake receipts, the effort required was beyond what most were willing to do.
But now people don't need to edit things in Photoshop or alter text with whiteout. Instead they just need to describe what they need in detail, and an AI model will produce the requested file.
"It opens the door for lazier fraudsters. You don't even need to be sufficiently motivated or technically sophisticated to carry out a fraud scheme that 5–10 years ago would've required some level of technical sophistication and more motivation and time and energy. Now you can just do it in an afternoon pretty easily," said Wilder.
This is not just a theoretical problem. AI-generated fakes are already being used in fraud schemes, such as the case of a Singaporean man who faked $16,000 of
While there is consensus that this is a problem, there is less agreement around what to do about it. Some have suggested using metadata to detect AI images, with certain vendors like T&E solutions provider
"When we see that these markers are present, we have really high confidence of high accuracy to identify them as potentially AI-generated receipts," Ramp's Dave Wieseneck said in a
David Zweighaft, a partner at forensic accounting firm RSZ Forensic Associates, said professionals in the field might take a similar approach. There are already ways to look at documents for evidence of alteration. While theoretically someone could strip out the metadata, he said that doing so creates new evidence of alteration.
"We've got to move past the 2D world we live in and look at the metadata, look at any traces that any electronic transactions or electronic modifications might leave. [We] may want to work with the software providers to come up with validation," he said.
Zweighaft added that cases like these are exactly why people developed data forensics as a field. While the actual forensic work might be more difficult and complicated when dealing with AI, he felt the overall principles were sound.
"This crisis is not new. Ever since computer-generated information has been used in litigation, it came up. … And that is where data forensics was invented — and that is where all of the legal defense work around data and making sure things were unchanged began. And now you have data validation and MD5, SHA-256, or MD64 hash algorithms to prove something was not changed from its original pristine state on the computer. This is just the latest iteration of that scenario," added Zweighaft.
Wilder, however, said that in order for data to become a foolproof way of verifying authenticity, there would need to be some sort of widely-adopted industry standard that mandates the inclusion of certain metadata (essentially, a watermark) in AI-generated images that can't be removed. And even if that happened, he wasn't sure how sustainable that technique would be in the long run.
"As mainstream, institutional-type software providers agree to incorporate that into their services, there's still a big issue: A lot of these LLMs and other AI models have been open-sourced at some point in the recent past. That means the underlying code is in the hands of whoever wants it, and they can build on top of it and make their own AI tools. So even if there is industry-wide adoption of some kind of tech standard like that, that is not going to really account for, you know, people who've built their own AI models. And there are a lot of really smart bad guys out there," Wilder said.
While the immediate instinct for many would be to solve this problem with AI, Wilder was skeptical. Automated systems are easy to fool, and even if they're powered by AI models, AI does not have the best track record when it comes to detecting AI. He pointed to a large number of cases where people put their own work through an AI-detection solution only to find the software concluding it was done by computer. Overall, he felt the tools for generation were far outpacing the tools for detection, which makes them a poor choice for detecting AI-generated fakes.
"You'll have solutions providers telling people in the anti-fraud industry that you can just use AI to solve this problem for you. And I would encourage people to exercise that professional skepticism in those contexts as well because with emerging technologies, we've seen countless examples of people overstating the capabilities of AI tools," said Wilder. "So I would encourage anti-fraud professionals to be really wary of the claims of solutions providers on the detection capabilities of their tools."
Instead, he felt professionals will need to start leaning on "more old fashioned controls" such as requiring everyone to use company credit cards that can be monitored, retrieving actual financial records versus screenshots (with the employee's consent), and generally being more diligent in monitoring for anomalies and problematic patterns. He added that most companies can view what people do on their network, and so looking to see if someone's Internet history recorded them making the fake receipt can help, too. To account for external fraudsters, he recommended that contracts include a "Right to Audit" clause that lets them request official bank records from actual financial institutions to corroborate expenses.
Todd McDonald, founder and CEO of financial intelligent software provider Valid8, however, felt that AI and automated systems must be part of the solution, even if it's not as one typically imagines them. Recalling an exhaustive investigation into a Ponzi scheme that was done fully manually, he felt stepping away from automation was a bad idea.
"Having to recreate the books and records for a Ponzi scheme, where there weren't tools like the ones we've now built to validate things — at that time, we had to spend thousands of hours recreating the books and records from subpoenaed bank records — hundreds of thousands of transactions, over 12 years, across 20 entities," said McDonald. "That was all manual, and it did not require the best of our skills and training. It was an unbelievably burdensome effort. We had to identify what had happened before we could even move on to what we could do about it. I didn't have that luxury. I had to go through months of painstaking work just to get a data set I could trust before I could interrogate it and understand it."
So while asking an AI "is this AI?" may not yield good results, this is far from the only option. Valid8 doesn't look at a picture of a receipt and determine whether or not it is real but, rather, pulls actual records like bank and credit card statements or copies of deposit slips and checks, and uses that to verify discrepancies or duplications. With this in mind, McDonald is unconcerned with the AI's ability to fake documentation, as his company concerns itself with the actual data.
"It really comes back to the provenance of where you are getting the support for this documentation. At Valid8, we come with a specific point of view: bank statements don't lie. They are a fundamental ground source of truth," he said. "There's nothing immediate we've done as a result of the announcement or some of the new tech that is out there. It hasn't changed things one bit from our roadmap to expand from using bank support evidence as a ground truth and being able to augment and enhance that with additional supporting documentation."
However, he also noted that technology is only part of the solution. Having "highly trained humans" to interpret the data and understand the context is vital, as is training those humans to exercise professional skepticism and compliance, and checking to make sure those lessons were absorbed. There is still value in the old fashioned controls to which Wilder referred.
"You should be setting up a culture of compliance, a clear and outlined code of conduct for what the expectations are regarding expense reports. You should set up a random audit methodology, and employees should know there are consequences for that. This is just good old blocking and tackling — someone is paying attention," he said.
George Barham, director of standards and professional guidance with the Institute for Internal Auditors, raised a similar point: While it is unlikely people will step away from automated systems, they do need to be taught to take the outputs with a grain of salt and not blindly trust what the AI tells them.
"I think the main thing is not completely relying on what the tools give you and being critical and looking at the results and asking questions or looking for trends. 'Gosh this cost really jumped over this year, what is going on?' I also think if you look at a large number of items, it is still a good idea to take a couple and look at those annually so that won't be a departure from how internal audits look at things, but I think you take what tech provides and what AI provides with a grain of salt," Barham said.
However, Barham was hesitant on any specific prescriptions for action, as every company is different and has different goals. So rather than outline what controls should be implemented in response to AI forgeries, he said it's important that professionals sit down with managers and discuss what controls specific to the organization might be needed.
"The biggest thing is making sure we're having conversations … with management. Hopefully, they will do an annual risk assessment and maybe a quarterly mini-assessment. But you'd like to see some actions taking place from a risk assessment. So maybe that means adding or improving some of the controls in this elevated risk area. That could include more policies, more procedures, more controls, more reviews, more authentication methods when looking at receipts and understanding the source. So it falls to how the organization understands risk," he said.
