Human accountability in the agent workflow

Chart: Human accountability and AI agents.

Product Advisory Collective

The diagram maps structural and cultural human accountability with an agent’s workflow:  monitor context, make decisions, coordinate, complete tasks, and deliver an outcome. When decision-making is consequential, a single human owner must be accountable (Step 2), whereas shared ownership of agent outcomes, across all of the teams and individuals that contributed to the workflow or are impacted by its results (Step 5), is typically necessary. Underpinning all five steps is cultural accountability, which facilitates accountable human behaviors and enables human workers to detect and prevent unaccountable agent behaviors.

Importantly, whether an agent’s decisions (Step 2) require a single named owner depends on the severity of consequences for the enterprise and its key stakeholders. A named owner signs off on the risk criteria and thresholds, and answers for any consequences that occur if risk thresholds are surpassed. The SAFR white paper recommends evaluating five risk criteria: action reversibility, financial materiality, customer impact severity, regulatory sensitivity and novelty or anomaly. Risk thresholds are set pre-deployment, and proposed agent actions (Step 2) are evaluated continuously at runtime. While “above threshold” risks trigger a real-time human-in-the-loop (HITL) review, the executive owner remains ultimately responsible for any resulting repercussions or systemic issues.

Too often, model decision-making oversight is lacking. While agent orchestration and escalations to HITL reviewers are established practices, scrutiny over the risk or consequences of the model’s decisions is the exception, not the rule, even in high-risk industries. As recently as 2023, roughly 40% of hospital systems did not evaluate AI models for accuracy and 56% did not evaluate them for bias. This clear lack of oversight is governance debt with acute liability. Preventing and reducing the debt requires formal structural ownership and a culture that rewards accountability.