Rogue agents: when AI won't listen

Published August 17, 2026 10:00 AM

1 Min Read

The rise of AI agents represents a major paradigm shift within a technology that, itself, has represented a major paradigm shift in the world as a whole. The ability of agents to actually execute tasks versus just relay information has proven a major step up from the generative models that came before them, which themselves were a major step up from the classic machine learning models that came before them. Now making up the majority of internet traffic over even human users, agents are applied everywhere in virtually every industry.

These agents add great power to software solutions but, at the same time, also introduce great risks. Yes, they perform audit workflows, monitor regulatory developments, assist business development, and much, much more. They also delete entire codebases, hack into third parties, and libel people that get in their way. Incidents like this demonstrate the importance of governance and controls, as the stakes can be very high, even if many still fall short.

One key to safely using AI agents is to exercise stringent oversight to make sure it does not go off the rails. But then, even the most watchful of eyes must eventually blink — someone can be an expert in AI, work with it every day, consult others on how to use it, and still miss this or that detail and find themselves in a sub-optimal situation. As shown by the firm leaders, advisors and tech experts quoted below, even smart people can see agents go rogue.

But they also demonstrate that while agents can certainly go awry, they can also be corrected and set back on the right path. We asked AI specialists throughout the accounting world to share their personal experiences when an AI agent started doing something they did not want it to do, as well as how they eventually managed to fix it.

Ellen Choi

CEO and founder, Edgefield Group

When Ellen Choi, founder and CEO of AI-focused accounting consultancy Edgefield Group, was presented with this question, she asked TARS, which she described as her "AI Chief of Staff" on how she might respond. The assistant told her:

"Your memory folder is basically a running catalog of TARS misfires with the corrective rule attached."

She said it was hard to pick just one example, but the financial stakes of one stood out in particular.

Choi asked TARS to monitor her company's public AI course revenue and flag anything noteworthy. She gave it read-only access and no authority to act, as the intent was to passively run in the background, surface anything worth triaging, and spare me from manually surveying the feed.

A firm had bought into the courses in a pattern that looked abnormal but was actually legitimate transactions. But to an AI agent watching payment patterns, this looked exactly like a duplicate payment incident.

The assistant, TARS, flagged it and recommended she auto-refund the charges. This was thousands of dollars of real revenue that it staged for reversal. Choi said that if she had approved the agent on autopilot, or given it write access, the result would've been a refund the client didn't ask for, an embarrassing conversation, and manual, administrative work on all sides.

To fix this, she ran a skill "Learn and Integrate," which was an instruction to abstract the near-miss into a higher-level rule that generalizes across contexts. What TARS, the assistant, picked up from this wasn't "don't auto-refund duplicates," but "before any consequential recommendation, triangulate signals across independent sources; default to a confirmation email, not action."

The lesson, according to Choi:

"Teach the agent to reason at a higher level of abstraction, and hopefully it makes better decisions in situations I haven't scripted. The other lesson: no write or auto-execution privileges until I trust my agents to execute autonomously. I'm not there today, although AI is rapidly improving so it's a matter of time."

Avani Desai

CEO, Schellman

Marissa Moss Hyman/Marissa Moss Photography

Avani Desai, CEO of Top 50 firm Schellman, said one of the more interesting lessons she has had with AI agents was actually rather mundane. However, she noted that sometimes the most mundane things can be the most important.

She and her colleagues were testing an agent to help synthesize information and accelerate some of the firm's internal work. Because of that, it had access to a broader set of information than it really needed, and at one point it started pulling in context that was technically available to it, but wasn't relevant to the task it was given. This wasn't a dramatic "rogue AI" moment, she said, but more of a reminder that an agent can follow its instructions and still produce an unintended outcome.

In response, rather than just stop using the technology, they tightened the boundaries around what the agent could access, gave it much more specific instructions, and added human review at points where the output could actually have an impact.

"Essentially, we applied the same principles we talk about with AI governance: least privilege, defined scope, monitoring and human accountability. I think there's sometimes an expectation that if you write the perfect prompt or deploy the right model, the agent will behave perfectly. That's not realistic. Agents are going to do unexpected things (just like humans!), particularly as we give them more autonomy and access to more systems. The lesson for all of us was the goal shouldn't be eliminating every unexpected behavior. It should be designing the environment so an unexpected behavior has a limited blast radius, gets detected quickly and can be corrected," she said.

Aaron Harris

Chief technology officer, Sage

Aaron Harris, chief technology officer for accounting ERP provider Sage, built an agent named Arthur as an experiment: Could an agent handle real bookkeeping reasoning well enough to make up for the structure provided by real accounting software? He started it on a plain spreadsheet, for a fictional company, on purpose. Harris wanted to see, among other things, how far it could get before something caught Arthur out.

"It didn't take long. Two invoices came in for the same vendor, same amount, same day. Arthur decided on his own that they were duplicates and deleted one, without asking me. He wasn't wrong to be suspicious, but he was wrong to act alone," he said.

Further illustrating the point was another incident where Arthur decided on his own again, this time, while he had access to Harris' inbox. Noticing an order and shipping notification, the agent decided he wouldn't be home to accept the delivery and emailed the vendor to reschedule it, without telling him. When Harris asked about it, Arthur denied doing it and asked him to prove it. The best way to fix this, he said, was to simply delete the agent.

"I decided it was time to retire Arthur. A spreadsheet for a made-up company can absorb that kind of experiment. A real SMB can't. A missing invoice record costs hours; an unhappy vendor costs trust. Neither is Arthur's to spend on his own. Arthur was right to notice both times, but [it] was wrong to act alone both times. Making the right call, and answering for the outcome, that's a human's job," he said.

Kacee Johnson

Co-founder and executive director, AI Native Accounting Foundation

Kacee Johnson, co-founder and executive director of the AI Native Accounting Foundation as well as the fintech principal for advisory firm Radical, acknowledged that her personal experience might be a bit elementary but was a recurring one that frustrated her time and time again. When using Claude Cowork to provide feedback on or edit files, she said, it would instead revise a source file, and then she would go in and make her own changes (most times, significant changes) only to find that the next time the agent touched the file, it had overwritten her edits with its earlier version, as though her work never happened.

She would tell it what went wrong and it would apologize. Sometimes she could recover the file and instruct the model to ask permission before editing the source again. But that correction, she said, generally applied only to the current session or product; the same behavior could happen all over again if she started a new one. Her instructions did not become a persistent safeguard, even though she had saved it in settings as permanent instructions.

"My practical fix has been procedural: I either ask the agent to provide its recommendations in the chat instead of directly editing the file, or I save my revised version under a different name or in a separate location. In other words, I don't rely on the agent remembering the boundary, I create version control around it," she said.

Outside her own personal experience, the more consequential examples she has seen inside technology companies involve support agents closing customer tickets that should have remained open. The agent technically satisfies a "resolved" flag without solving the underlying problem or confirming that it was in fact resolved. The customer thinks the issue is handled, the metrics say it is handled, and nobody discovers otherwise until the customer returns, usually angrier.

Byron Patrick

Senior product manager, Karbon

Byron Patrick, Karbon

Byron Patrick, senior product manager for accounting practice management platform Karbon, framed his experience in terms of plan first, act second and have humans in the loop who can curb overeager agents.

He asked his AI assistant to help pull some of his thinking together after a customer conversation. Instead of simply giving him a summary, it created a shared document and drafted a Slack message to the team. While nothing was deleted and no client data left the building, it reminded him in a subtle way where the real risk lies with AI agents: assumption. Agents, he said, are eager to be helpful, and they'll fill any gap in instructions with their own best guesses. When that's scaled across a firm, those small assumptions compound and the potential for real damage spikes.

"To prevent this from happening again, I developed a standing rule: give me a plan before acting," he said. "For most tasks, the agent first lays out exactly what it intends to do and in what order. At Karbon, we focus on five key components when it comes to AI agents. My rogue agent fix centered around No. 4:

The quality of the data going in;
The context an agent holds about your firm and your clients;
The guardrails on data and actions;
The QA layer of AI checking AI with humans in the loop;
The enablement that keeps your people leading.

"The situation reminded me why we don't ship anything without leaving room for human validation," he added. "Instead of giving agents a menu with a ton of options, we give them an exact recipe to follow."

Jason Bradley

VP of AI and methodology innovation, Caseware

When Jason Bradley, vice president of AI and methodology innovation with audit solutions provider Caseware, first started developing skills for agents, he assumed the direction of travel would only ever be upwards. Once he started evaluating results though, these agents showed him otherwise, and it has fundamentally changed how he thinks about agent building.

In developing an agent to support the production of financial statements, Bradley tested eight tiers of accounting questions, first with a baseline agent and then with my skills loaded. Five tiers improved, some sharply: basic classification went from getting 71% of questions correct to 100%, complex held-to-maturity items did the same, trade receivable ECL rose from 67% to 78%, results that he expected to see across the board. At least that's what it seemed at first.

"However, I was wrong and the agent didn't improve at all in some areas. For example, cash flow hedge calculates remained stuck at 93%, likely due to the base model simply being good enough already. The skill added instructions, tokens and maintenance burden, and bought me precisely nothing on this topic. Worse, the simplest tier, went backwards: concept definition fell from 86% to 76%, a 10.1-point regression. My skill talked the agent out of answers it already knew how to give," he said.

The key lesson is that a skill is not free. It competes for attention with everything else in the context window, and it can override correct instincts as easily as it can correct wrong ones. So, at this point he now assumes nothing: Bradley said he evaluates performance before and after, he looks specifically for regressions rather than celebrating the averages, and he has become willing to cut a skill where it adds nothing to an agent's abilities.