Many expect to use AI agents but only a few have mature governance frameworks

Organizations with mature agentic AI governance frameworks are in the distinct minority, but this hasn't stopped leaders from rushing to implement the technology this coming year in an extremely wide variety of applications. 

This is according to data from Big Four firm Deloitte's State of AI in Enterprise report, which found that a comfortable 74% majority of companies said they plan to deploy agentic AI sometime within two years, 23% are already using agentic AI at least moderately, and 85% of companies expect to customize agents to fit the unique needs of their business. Trust and investment in AI generally is also on the rise, with 84% of organizations increasing their AI investments and 78% of leaders reporting greater confidence in the technology. 

At the same time, only 21% of companies say they currently have a mature model for governance of autonomous agents. This is despite the fact that leaders expect to deploy the technology in a wide variety of areas, including actual physical systems like package sorting, manufacturing and security systems. Physical AI integration is already expanding: 58% of companies report at least limited use of physical AI. This proportion is expected to reach 80% within two years. Deloitte noted that physical systems carry unique risks that must be considered. 

Bartek - stock.adobe.com

"Unlike software AI that operates in digital environments, physical AI systems interact with people, equipment and infrastructure in ways that can pose safety risks if they malfunction or are compromised. This raises the stakes for testing, certification and ongoing monitoring," said the report. 

The governance issues might also challenge another rising concern from leaders: where their AI comes from. The Deloitte report said the specific country where their AI solutions were developed is becoming increasingly salient. This is connected to the concept of "sovereign AI," which relates to when a country and the companies operating within it design, train and deploy AI under their own laws, on infrastructure they control, using locally governed data. The goal is to reduce dependence on foreign vendors for critical AI capabilities. 

The report found that 83% of companies view sovereign AI as at least moderately important to their strategic planning, and nearly half (43%) rate it as very important or extremely important. Meanwhile, 66% were at least moderately concerned about over-reliance on foreign-owned AI, with 22% very or extremely concerned. 

"The rise of sovereign AI has immediate practical implications. Companies working across borders must navigate complex requirements that vary by country, forcing them to build customized solutions for different markets. … Ultimately, sovereign AI isn't just about technology ownership. It's about strategic independence. By building on infrastructure within its own control (fueled by its own data, models, talent and ecosystem), a company has the ability to innovate securely and responsibly," said the report. 

The Deloitte report echoes several others that, taken together, indicate a persistent gap between AI use and governance to oversee it. In July, for example, a poll by AuditBoard found over 80% said their organizations are either very or extremely concerned about AI risks but, at the same time, only 25% said they have fully implemented an AI governance program. 

Another survey in August by EisnerAmper found only 22% of the respondents said their organizations even monitor AI use in the first place, and only 11% block ChatGPT and other public models. The survey also found that only 36.2% have an AI policy, only 34.2% say their company emphasizes transparency when discussing AI, and only 34% say their company has an AI strategy. 

This year might see some progress on this front. Many accounting tech leaders have predicted that AI governance will find new relevance and strength in 2026.