Overconfidence in AI governance means safety takes a back seat to speed

While almost everyone is talking the talk on AI governance and control, recent data suggests far fewer are walking the walk, much to the detriment of their organizations. 

Processing Content

Top 50 firm Schellman, in a report released today, found that 90% of AI governance professionals say their organizations have already allocated funding specifically for AI governance and that 74% believe they could pass an AI compliance audit today. However many of these firms are likely overconfident, as only 27% described their governance programs as fully mature. 

Meanwhile, only 57% actually have a formal AI governance policy, 44% maintain AI-specific incident response procedures and 64% have a formal AI acceptable use policy actively communicated to employees. And even when someone has formal policies and procedures, enforcement is often inconsistent, with 28% saying they still address violations informally, on a case-by-case basis. 

AI governance
Bartek - stock.adobe.com

This is leading to real risks at organizations. The data found that 65% have experienced AI-related incidents or near-misses in the past 24 months. 

And yet, organizations do not seem especially troubled by this as the data found that, despite the general lack of mature governance frameworks, they are still deploying AI at a rapid pace. Schellman's survey found 86% of organizations have tested or piloted AI agents and 46% already have AI agents in production. 

Another study from tax automation solutions provider Avalara found a similar mismatch. The international survey found 71% of finance leaders reporting that speed is the main priority in their organization's AI deployment strategy, versus 7% who said governance was more important. When paired with the Schellman data, one might make the argument that organizations feel confident in their ability to exert effective control over AI and so do not feel the need to make governance a major priority. 

Other findings by Avalara support this. It found that 30% have not updated internal controls within the last year to reflect AI agents taking or recommending actions. Further, only 31% require documented human review or escalation thresholds, and only 30% require independent third party security or compliance. Further, the survey found only 28% require documented audit logs showing how an agent reaches its decisions, and 29% said they rely primarily on a vendor's reputation versus technical or compliance requirements. 

Avalara also found that the vast majority, 76%, currently lack dedicated experts within their finance department that understand how their own agents actually work. Instead, they have to either lean on IT for help, 20%, or rely on their AI vendor, 20%. Furthermore, while a fifth of respondents are actively recruiting for experts to fill internal roles within their departments, 16% have nobody responsible for it at all. 

While the data indicates organizations think governance stands in opposition to rapid deployment, both studies found it's quite the opposite. Schellman's data found that 78% of those who described their AI governance program as "mature" already had AI agents in production versus the 22% who classified their programs as "developing." In addition, 57% of respondents said effective AI governance translates directly into improved efficiency, 43% indicated it makes AI scaling and innovation easier, and 49% believed it improves readiness for new regulations. 

"The conversation around AI governance has fundamentally changed," said Avani Desai, CEO of Schellman. "Customers, regulators, boards, and business partners are no longer asking whether organizations are thinking about governance, they want proof that governance is working. The organizations that build trust through mature, demonstrable governance programs will be better positioned to scale AI, navigate regulatory change, and create long-term business value. Governance is increasingly becoming a competitive differentiator, not just a compliance requirement."

Meanwhile, Avalara noted that finance leaders would be more willing and confident in deploying AI agents if there were better controls and governance structures. They aren't looking to slow AI adoption but, rather, looking to scale it responsibly. The survey found finance leaders want documented audit trails showing what an agent did and why; contractual accuracy, performance or accountability commitments from the vendor; evidence that outputs are tested against known compliance requirements; AI outputs being grounded in verified tax, compliance or financial data sources; and AI agents having the ability to operate within the rules, permissions and controls of existing systems of record. 

"Finance leaders are right to move quickly to capitalize on agentic AI opportunities, but speed without accountability creates new forms of risk, and speed without rethinking workflows limits ROI," said Hugo Sarrazin, CEO at Avalara. "The organizations that realize the greatest value from AI won't simply deploy more agents. They'll leverage agents with trusted data, governed workflows, and clear controls that enable automation with confidence."


For reprint and licensing requests for this article, click here.
Technology Practice management Artificial Intelligence Corporate governance Cyber Security
MORE FROM ACCOUNTING TODAY
Load More