Who gets blamed when AI goes wrong?

Who is ultimately responsible for an organization's AI, and who is to blame when it goes wrong?

Top 50 firm Schellman, in a recent report, said the answer is usually the technology lead, as they are the one who is often put in charge of purchasing and implementing the firm's AI solutions. Their poll found that, when it comes to this role, most respondents place responsibility firmly with the CIO or head of IT (42%), followed by the chief data officer/AI officer (16%), and CEO (10%). Schellman said that while it seems organizations don't have a problem with assigning ownership over decision-making, placing it with a single leader creates major risk, as accountability shifts to that individual versus the organization as a whole, enabling a host of other problems. 

"Risk assessment suffers because a single person is evaluating risk that they are incentivized to minimize. Escalation is delayed because no one else is formally accountable. When incidents occur, the organization's response is limited to correction rather than meaningful root-cause remediation. In most organizations, AI risk accountability has not been formally distributed across the functions most likely to encounter it daily. Distributed accountability, where risk ownership is spread across IT, security, compliance, and legal, is the industry practice for high-stakes decisions. AI governance remains largely centralized," said the report. 

kieferpix - stock.adobe.com

However, another study from tax automation solutions provider Avalara found that the executive ranked low on overall blame compared to other parties. In fact, many people weren't even sure who to blame at all. When asked who is responsible when an AI agent fails, 23% said it was unclear/no one; 22% said it was the vendor, where contractually specified; 20% said it was the individual who manages or deploys the agent; 19% said the team that deploys or manages the agent; and 16% said the executive who approved the AI investment. 

Looking deeper into the role of the vendor, Avalara said that in 17% of cases, the contracts state that the vendor bears only minimal accountability, while in a handful of cases liability sits entirely with the customer. The most common approach, 42%, is a shared liability model between both vendor and customer, while 38% of contracts state vendors are fully liable in the event of agent errors. 

ISACA, in a survey of its own, had similarly diffuse results as to who to blame when an AI system causes harm to the organization but the most common answer, at 28%, was "Board/Executive," however the second most common answer, at 20%, was simply "I don't know." The third most common answer, in line with Schellman's data, was the CIO/CTO at 18%. The business owner and the CISO both tied at 13% for fourth place. In stark contrast to the Avalara survey, though, only 2% felt the vendor was responsible, the same as those who said "other." Finally, 4% said no one was responsible. 

A historically popular place to attribute blame is simply whomever was closest to the mistake, even if they had little control over the system that caused harm, according to a widely-cited paper by Madeline Claire Elish at the Data and Society Research Institute. An article in the Berkeley Management Review raised the real example where a lawyer in 2023 submitted a legal brief in a federal court case containing fabricated citations made by ChatGPT. The individual absorbed the blame with little attention paid to the organizational failures that enabled the attorney to make this mistake in the first place, similar to Scellman's warnings that excessive attribution to individuals allow organizations to ignore wider systemic issues in favor of blaming the CTO. 

Another, more recent study, "AI-Induced Human Responsibility (AIHR) in AI–human teams," found similar results. While intuitively one might think AI systems can serve to diffuse individual responsibility, as people might just blame the algorithm, the researchers found just the opposite. When a morally consequential mistake occurs under ambiguous joint responsibility, people allocate more responsibility to the human when the human is paired with AI than when paired with another human. Even when "blame the bot" heuristics and motivated self-exoneration should be readily available, participants shifted responsibility toward humans – including themselves – in AI–human teams. The paper noted that people seem to treat AI as comparatively constrained and, therefore, less plausible as a moral agent, making the human teammate the default locus of discretion and accountability.

This is consistent with another paper which found the mere existence of manual mode leads to more human blame when AI makes mistakes. When observers know that a human agent theoretically had the option to take control, the humans are assigned more responsibility, even when agents lack the time or ability to actually exert control, as with self-driving car crashes. Over four experiments, the researchers found that though people prefer having a manual mode, even if the AI mode is more efficient and adding the manual mode is more expensive, the existence of a manual mode increases human blame. 

However, other research suggests blame attribution is more flexible in people's minds and can change depending on context. For instance, an academic study from Boston University, however, suggested that perceived blame and accountability can vary based on whether the AI system is described as a tool or an employee. When described as a tool organizations are more likely to blame the manager but when described as a sort of digital employee then the organization becomes more likely to blame the system itself. 

"Among managers with institutionalized AI agents the AI employee framing reduces the share of accountability assigned to the manager by about 9 percentage points and increases the share assigned to the AI system by about 8 percentage points (with a smaller offsetting increase for the team). Overall, the AI employee framing in this group shifts perceived responsibility away from the manager themselves and toward the system and the broader organization, consistent with accountability becoming more diffused when AI is positioned as its own organizational actor," said the paper. 

Finally, outside the organization itself, a paper from assurance solutions platform Warden AI said that legal liability stays with the employer: the organization is legally responsible for the decisions made by its own AI systems, even if they use a third-party tool.