With the public disclosure of OpenAI agents’ hack of Hugging Face over the summer of 2026, the question of how to govern AI systems — particularly agentic AI — is once again dominating headlines. Congressional hearings are being suggested; Senator Bernie Sanders has proposed a ban on the development of superintelligence. The dominant narrative is one of anthropomorphized agents building their own civilizations with intent and purpose. The result is a debate about the crisis itself — an autonomous system running out of control and causing harm — rather than a necessary focus on the failures of oversight that continue to enable this level of corporate irresponsibility. However novel this newest incident may be, it would be a mistake to let it eclipse what we already know about AI governance. Policymakers and researchers do not need to start from scratch: in designing approaches to safety and accountability, we can draw on an existing, essential body of work.
Data & Society has been publishing research and policy insights on the governance of automated and AI systems for nearly a decade. Over that time, our argument is clear: governance is both a social and a technical ecosystem, and no single intervention will provide the assurance and protections — political, economic, and existential — that society should be able to expect when encountering AI systems. We also emphasize that despite the efforts of companies and their investors to spin anthropomorphized technologies as independent from corporate oversight, AI systems do not exist outside existing law. Legal frameworks, from civil rights protections to prohibitions against industrial espionage, apply to these technologies and the companies that have built them.
To provide historical context for this moment, and to demonstrate the sociotechnical policy and practice that an ecosystem to secure AI governance requires, we’ve created a collection that brings together Data & Society’s key governance research and policy from the past five years. Key insights from it include:
- Governance is ultimately a set of choices based on who holds power in a society. Our current lack of AI governance results from a grave power imbalance that favors the plans and desires of a small set of tech actors over the public interest.
- Voluntary safety commitments made by AI companies are not a substitute for democratic and enforceable governance to protect the public interest. Such commitments by companies have prioritized protecting their core business model, and historically have not been honored in the face of economic and competitive pressure.
- While a record of harms caused by AI systems is necessary to design effective governance, it is not sufficient to inform accountability or to ensure enforcement and redress of harms from those systems.
- The benefits of governing the known and visible harms and failures of AI systems — predictive, generative, and agentic — extend beyond the current moment, building the institutional muscle and experience in our politics and society to govern for future risks.
- AI systems are sociotechnical. Technology’s real-world safety and performance are always a product of technical design and broader societal forces, and governance of these systems needs to reflect that reality.
A Walk Through the Collection
Published in 2021, our report Assembling Accountability examines the practice of algorithmic impact assessment, similar to environmental impact assessments. The research addresses a core question: what must be in place in a governance system for transparency — a record of harm — to inform actual accountability? An evaluation can lay out what went wrong without specifying the mechanisms for accountability and redress for the harms that occurred. Accountability only happens once that evidence reaches a body with the standing to judge responsibility and demand and enforce a response.
In her 2023 statement to the US Senate for a panel on “Catastrophic Risk and Doomsday Scenarios,” Data & Society Executive Director Janet Haven makes the case that governing the known harms of AI systems builds the institutional muscle to govern for the catastrophic risks that could lie ahead as AI capabilities develop. This is an argument that is even more relevant today as policymakers grapple with a response to agentic AI; exercising the authority to govern current, known harms will strengthen the societal capacity to protect against future risk.
But using that authority well means understanding clearly what is being governed. Our work argues that governing technology alone is not sufficient. Two companion policy briefs published in 2024 argue that AI must be governed as a sociotechnical system — one whose behavior depends on the institution it sits inside. A Sociotechnical Approach to AI Policy shows how a system can work exactly as designed and still fail in real-world deployments. Meanwhile, AI Governance Needs Sociotechnical Expertise asks how governing bodies build that understanding into their own decisions, arguing that alongside technical specialists, government needs humanities and social science experts who can study AI in its societal context and inform policy choices with real-world experiences.
Our work also looked closely at the practice of red-teaming, a widely used corporate evaluation method, as a governance approach. AI Red-Teaming Is Not a One-Stop Solution to AI Harms argues that the practice has to be paired with impact assessments, outside audits, and public participation to deliver comprehensive accountability. Red-Teaming in the Public Interest takes that critique further, carrying the sociotechnical approach into evaluation itself. The report argues that a range of expertise is needed if red-teaming is to discover failures and harms that a system’s technical builders might otherwise miss. The lessons of Assembling Accountability appear here, too: documenting failure does not lead to accountability. Red-teaming only becomes part of a governance solution when its findings are used by those with the power to assign responsibility and require redress.
“Troubling Translation” uncovers the translational work needed for research to shape policy. While translation is usually treated as a last step — taking finished findings and repackaging them in language policymakers can use — it needs to start earlier, and as a social learning project. Data & Society’s peer learning program, the Public Technology Leadership Collaborative, practices this approach; the program pairs researchers and government officials to build a shared understanding of granular AI governance challenges and possible solutions that respond to the real trade-offs policymakers need to make.
Alongside that peer learning work, the Algorithmic Impact Methods Lab builds methods that bring communities and government officials into a participatory governance relationship to assess a technology’s impact on people. The Lab’s core product, the Algorithmic Impact Assessment Toolkit, was built and piloted with the municipal GovAI Coalition. A key precept of the toolkit is that government agencies engaged in participatory practice engage communities early enough in a process that a system can be rejected; the power to say no is critical.
Published in January 2026, The Big AI State turns to the power the federal government holds over AI’s trajectory, and unpacks the increasingly close relationship between the federal branch and AI labs and investors. Despite the current administration’s rhetoric of deregulation and laissez-faire development of the AI industry, the brief demonstrates the opposite: the Trump administration is using public money and trade and industrial policy to actively steer AI’s development away from safety and protections for known harms, and toward technical and geopolitical dominance at all costs.
Finally, we offer two recent pieces on the governance of AI agents. Based on empirical observation of AI agents in use in scientific labs, The Oversight Fallacy argues that individual human oversight of agents does not scale automatically; instead, agent oversight requires both technical guardrails and sociotechnical practices that allow for knowledge, visibility, and control. Without those elements — and an ecosystem of governance that provides redundancies in enforceable accountability — incidents like the OpenAI hack of Hugging Face are inevitable. Alongside this report, “A Sociotechnical Research Agenda for the Oversight of AI Agents” argues that there is an urgent need for research into how organizations can preserve judgment when no individual can follow an agent’s entire course of action.
As a whole, this collection tells us that this is not purely a technical question, but a sociotechnical one. It points to the outlines of an ecosystem of technical interventions, enforceable law, and institutional behaviors, expectations and requirements that create a governance model for AI that begins to be worthy of public trust.