Trust is
"The worst thing you could do is leak client data. We are a profession of trusted individuals. The absolute worst thing you could do is put client data at risk. But that is happening across the board. People don't always understand this technology. Sales people say it is safe, it is this, it is that, but you need to make that decision for yourself," she said during her presentation at Xerocon in Denver last week.
Part of this is understanding that AI models are not nearly as secure as many people think. For example, if someone makes a share link for an AI conversation, what happens is that the platform creates an external web page that can be shared with someone else; however, since this is an external web page, she said, this means it is now indexed and therefore searchable on engines like Google. "So you think you're just sharing that with your firm, but that share link is public. So now that information is in the public domain and can be found," she said.
Another big security hole, she said, is that while an LLM's reasoning packets can be encrypted, this encryption can also be broken by another LLM, and not even necessarily a more advanced one. For example, if someone has a big long conversation with Claude's Sonnet model, the encrypted packet representing the conversation exists on the Internet.
"So, if I wanted to target you, I could find that packet and decrypt it using a different Claude model like Haiku. This is a gigantic security hole LLMs have not fixed yet," she said.
This would not be as big of a deal if not for the fact that so many people put whatever they want into generative AI without even thinking about it, with an uncomfortable proportion of people inputting sensitive data like names, addresses or even social security numbers. People do not necessarily understand where the data representing those conversations ultimately goes, which creates major risks.
"The problem we're seeing right now is we're in the wild west of AI," she said.
This is not an abstract concern. A lot of sensitive corporate strategy information about major publicly traded companies has been shared via links which then ends up indexed by Google and becomes searchable, she said. It is such a risk that actual 8K filings are starting to list unauthorized AI use as a potential risk.
But even if your own tools are locked down, any tool that tool connects with could also be suspect. She said to be cautious about what MCP or integration you choose to connect with, as their own data privacy may not be nearly up to the same standards.
Beyond technical risks, she also noted that AI carries psychological risks, namely that models are often optimized to sound as appealing and convincing as possible, which can lead to confirmation bias among many other things. People asking AI for insights should not be treating them like an answer box. She noted that, when challenged on something, a lot of models will completely flip their answers to fit what the user wants. This is dangerous not just for the practitioner but for clients, who increasingly are turning to AI for financial advice.
"What if we have a client going to ChatGPT saying 'should I take this bank loan? Here is the loan, is this a good idea?' and GPT says 'yeah, this is a great idea.' But let's say it's got a 17% interest rate with abusive terms and the need to pledge their personal residence. We know this may not be the best loan for them, it's not a good business decision, but they're talking to ChatGPT, not us. That is a problem," she said, adding that this risk drives home the need for client education.
She added later that even if someone does get good advice from AI, that advice could change completely if the developer changes or upgrades the underlying model.
She added that there are regulatory implications for not paying attention to things like this. For instance, the IRS recently interpreted circular 230 to say that sending personally identifying information to an AI is the same as sending it to a foreign person.
"So not only do you need to understand [AI], you need to put in a policy and put a process in place that your whole firm follows to ensure you comply with the IRS guidance on this," she said.
With all this in mind, one might be tempted to swear off AI entirely. However, Mason said that it is possible to use AI securely and safely. The biggest and most important measure a firm can implement to this end is to make sure no one ever puts any financial data or personally identifying information into public AI systems, and to be selective about what data these systems do and do not have the ability to access.
Another measure is to be judicious as to what to even use AI for in the first place. Good use cases, she said, involve things that a human can review, that are relatively low stakes if the output is wrong, and personal information can be easily separated out. Conversely, if there is no chance to review the output, if personal information cannot be separated out, and if the user doesn't know exactly where the data is going, then it probably is not a good use case.
But that doesn't necessarily mean the firm can't do what it wants to do. Many times, she said, things people think need AI can be done with a more conventional computer program, versus having AI models do all the work itself. In fact, she said, AI has gotten very good at making these programs.
"One of the misconceptions I hear a lot about AI is that AI is doing all the work … It can build an actual program. Instead of the AI doing it every time, it can be a program you self-host, because AI is expensive, so utilize it to build consistent, repeatable processes that don't necessarily need AI at the core," she said. She added that, even if AI is at the core of something, "you can use a self-hosted small language model. You may not even need the frontier model to do these calculations."
Overall, she said, AI safety comes with understanding the AI the firm is using. Before implementing an AI system, she said, leaders need to ask what kinds of models they're using, where those models are hosted, and where the data ultimately ends up.
