“We assess compliance, cybersecurity, governance and business value before broader adoption,” she says. A team that includes staff from clinical, legal, cybersecurity and more will review each AI use case. KPIs for successful agents include accuracy, reliability, adoption, and business impact. If the agent is redundant or lacks engagement, DaVita considers decommissioning it. “I don’t believe in technology for technology’s sake,” she says.

Hylaine’s McDonald adds CIOs can assess agents on time savings, cost per task, and how frequently a human overrides an agent’s output. When override rates start to climb, or if running the agent costs more than the work it’s replacing, it may be time to sunset the agent, he says.

He recommends organizations maintain a registry of every agent with its permissions, versioned prompts, and tools. That way, when it’s time for agents to retire, CIOs can fully revoke access and prior context.