Just because you think your client data is protected, and just because a company claims it keeps this data private, does not necessarily mean it's true, according to Randy Johnston, the co-founder and principal at accounting tech consultancy K2.
Speaking during Woodard's Scaling New Heights conference in Orlando, Johnston said he is very concerned about what is being done to the data people are feeding into AI models lately or, rather, he is concerned that he doesn't know what is being done with it. People sell user data all the time and no one knows what it's being used for: training new AI models, making ransomware, spoofing identities, targeted marketing?
"What really got me started on this was last year, the fourth quarter. I was in a holiday mood and I started thinking of the Naughty and Nice list, and I realized: there's a lot of vendors who sell or give away client data. They do it all the time," he said.
While he did not name specific companies, he did talk about one of their key tells. He said people should read the terms of service, privacy policy and license agreement of any software they use, which will sometimes tell you whether or not they share data. Some will outright say they do, the confirmation buried deep in the document text. But he warned that the language can be slippery, and so even if a company does not directly mention selling or sharing user data, there are ways it can happen anyway.
"The tricky part is if they have 'subprocessors.' They don't talk about that. So, they get the subprocessor in play, and the subprocessor can sell the data. If you see 'subprocessor' in the license agreement, time out, what does it do? … Really watch the subprocessors," he said.
While privacy might be something that is handled through regulation, he said he is not confident that the U.S. will see any federal level regulation of AI in the near future. Even if the 10-year
"You think you are protected, but you're not," he said.
In the absence of U.S. federal regulation, Johnston recommended looking at what other jurisdictions have done in this area to protect data and promote ethical AI use cases. He named the EU AI Act as one example, and Canada's Artificial Intelligence and Data Act as another. The EU's act in particular, he said, offered good guidance as it defines unacceptable risk, high risk, limited risk and minimal risk applications.
"This is kind of a big deal, because I don't want to be using anything that is any more than limited or minimal," he said. "You'll have to look at your apps and the relationships that define some of that stuff. It gets kind of nutzo," he said.
For more forceful regulation, he said many states have introduced regulations of their own, and so in the U.S. it will be important to comply with those measures when applicable.
"There's 14 privacy regulations in force right now in the U.S. and by the time we get to next year we'll have at least 19, because the states are pulling them off state by state… So, I'd need you to be in compliance with state privacy regulations. I believe we would be better with a national one, but that ain't happening, so we'll have state [level regulation] for the moment," he said.
And this isn't even considering the other issues with AI models, such as bias and hallucinations (aka "making things up wholesale.") Not only are these problems still there, Johnston felt they were getting worse as time goes on. However, these are public generative models made for general use. He expressed optimism that the rise of agentic AI (broadly, semi-autonomous bots capable of making decisions and taking action independently) would help mitigate some of these issues. This is because agents will be trained on specific data to perform specific tasks in specific industries, which he felt made the model less likely it would go awry.
"The main reason I want AI running agentically is because we can specialize in our workforce. If you're still trying to verticalize your practice in the industry, you can have the agents think and act like you do in your industry, which I think is a marvelous step forward. Many of you have accounting backgrounds in one form or another. You've learned to think [in terms of] accounting rules: you can set accounting rules and judgments up with agents, which is another big breakthrough as we see it here, you get way better informational trustworthiness here, and you get far quicker innovation," he said.
