AI/ML news, top picks, and generated innovation digests.
200 articles tagged with this keyword, sorted by most recent first.
Today, I’m talking with Helen Toner, the executive director of the Center for Security and Emerging Technology at Georgetown University. This episode is part two of our series on AI in real life. Helen is a well-known figure in the AI safety and policy community, though she is perhaps best known for having served on […]
TL;DR: Access to AI safety talent & resources, particularly for labs during training, is sorely lacking in China. We created one of the largest AI safety communities in China , and plan to next start with auditing reward-hacks of unreliable upstream RL environments that domestic data vendors provide to Chinese frontier labs (which induces reward hacking and a whole host of dangerous downstream tendencies). RL tasks are notoriously prone to exploitation by reward-seeking agents (see examples here , here , and here ). In Xiaomi's MiMo v2.6 RL environments that were open-sourced, 2/3 had leaks on Git logs. There's a few clusters of common pitfalls that environments run into, but overall things evolve quickly. From our conversations with employees, CN labs typically don't divert enough people to this particular area, especially since project leads would want senior researchers to work on capabilities research, and because slowing down the development cycle is widely seen as too costly. The rough shape of our initial solution is an embedded team of staff directly working on identifying & fixing broken environments, working in tandem with the alignment & safety departments within each lab (afaik we estimate that the main Chinese AI players only have 5-10 people per company working on alignment and harm reduction). We are currently in good standing with several labs (our courses' students are often lab employees). We also plan to produce seasonal (and case-by-case) public reports o…
After a couple of years working in data and tech at GiveDirectly, I left to work in politics, campaigning for candidates at the state, local, and federal levels. I most recently managed Mayor Zohran Mamdani’s general election campaign, but earlier this year, recognizing the urgency of the AI moment, I began as a senior advisor at Guardrails Alliance . In my new role, I believe the analysis and tactics we’ve long used and studied on the populist left can help us win the fight for AI safety and shared benefits. The fight for AI regulation is fundamentally the same type of fight against corporate power that the populist left and labor have led throughout history. The message and tactics that win that fight are both effective and incredibly popular, and embracing them will help us put the American people in control of AI today and permanently in control of their futures. Here– in what I am told is likely the shortest LessWrong post of all time– I'll explain what the AI safety movement can learn from the populist left as it attempts to raise the salience of its message, build political power, and ultimately pass transformational legislation. First, we need to be clear about the nature of the fight: who our opponents are and what they are trying to win. At its core, the fight for AI safety is a campaign to put significant checks on corporate power, which will ultimately curtail these companies’ profits, and limit their ability to consolidate influence and build monopolies. While s…
A deterministic state-to-image decoder replaces noisy GAN surrogates, shrinking reachable sets and letting safety proofs transfer to real vision-based controllers.
Authors: Matt Clarke *, Jan Dubiński *, James Chua , Simon Schrodi , Owain Evans , Alex Cloud (*- Core contribution) tl;dr. We find that poisoned training data with little effect on its own can create a hidden backdoor once clean, contrasting data is added. We demonstrate this effect in small models fine-tuned on datasets we constructed. 📄 Paper , 💻 Code , 🐦 Twitter Work done as part of the Anthropic Fellows Program , in collaboration with Truthful AI . This article is cross-posted from the Anthropic Alignment Science Blog . Paper abstract We study homeopathic learning , a phenomenon where poisoned training data with little effect on its own can create a hidden backdoor once clean, contrasting data is added. In our experiments, a model fine-tuned only on a “poison” dataset associated with a trait T (such as speaking German or being misaligned) does not learn T. Surprisingly, adding similarly formatted “clean” data not associated with T causes the model to learn T. This effect requires that the poison data have an identifying “trigger” feature that is absent from the clean data. We demonstrate this effect in a variety of settings: subliminal learning of backdoor behaviors, inoculation prompting of reward hacking transcripts, generalization from archaic bird names, and an MNIST classifier with mislabeled data. We propose a simple regression model as an explanation. In the poison data, the task and trigger always co-occur, and regularization favors attributing T to the task. Cl…
TL;DR Much of AI safety depends on changing what governments, labs, engineers etc do. That makes it, in large part, a behaviour change problem. There are decades of evidence on how to change behaviour, including at the level of institutions and systems, and AI safety mostly isn’t drawing on it. Two common mistakes are relying on raising awareness (the knowledge deficit model) and assuming x-risk interventions can’t be evaluated. The field needs social science expertise in strategy, grantmaking and evaluation, as well as routes in for social scientists who want to help. I’m a behavioural scientist. I was Deputy Director of Experimental Research at the Yale Program on Climate Change Communication, and I now lead impact measurement across 15 country offices as Head of Impact at an international NGO [1] . I’ve had a few conversations recently with (fantastic, talented!) people in AI safety, that have made me realise how much the field is missing from social science. For example, several people have repeatedly suggested the need for new interventions focused on public education. Their reasoning is that if people knew more about AI safety they would take action, and so would governments. A couple of other people were reluctant to evaluate the x-risk interventions they were working on, because they assumed that they couldn’t be evaluated. These are understandable assumptions. Conservation held them for a long time too, and social scientists have been working on both problems for de…
For most of its history, AI safety has been a cozy professional circle: a tight-knit community of researchers, community-builders, and policy advocates in a few dozen organizations. But as AI becomes a national political issue, AI safety organizers are trying to meet the moment by expanding their outreach to the general public. 80,000 Hours has a YouTube channel with half a million subscribers, and MIRI’s Do Not Accept campaign asks people to add their names in protest to the ugly “terms and conditions” of AI development. The stakes of this mobilization are ramping up: a new Congress means new pushes for AI legislation, and civil-society organizing will grow around these efforts. This is broadly good! It was never ideal that AI safety was a niche concern, and it’s encouraging that organizers are reaching out to the general public to make them aware of AI risk. But AI safety’s culture is not equipped for the tradeoffs of a mass movement, and it remains to be seen whether this mobilization will be harnessed well. Early signs of mass movement outreach It might seem premature to call this AI safety's mass movement era. Most of the time and energy in AI safety still comes from the same professional community that has carried it for years. Legislators and regulators are paying attention to the AI industry, but the general public is not yet feeling the salience of superintelligent AI. But I expect that to change. The early signs are visible in the anti-data center movement . There…
Yesterday, I told my advisors that I would be dropping out of my PhD program at the University of Michigan's School of Information. I was nervous about how to break the news to them. There was already some tension in our weekly meetings: I often voiced my dissatisfaction with the pace of academia, my interests had been drifting away from theirs, and, most of all, we disagreed on the severity of the risks AI posed. When I walked into the room, they told me something I would have loved to hear months earlier: “We think it’s time to get you started on some research.” All I could say was, “I think it’s a little too late for that.” For months, I had been proposing projects that bridged my interests with theirs, such as running studies that treated humans and LLMs as equal subjects, but they were usually turned down for not being “exciting enough”. My advisors wanted me to start with small, unpublished surveys that would ‘help me ask the right questions’ and “develop research taste” in Human-Computer Interaction before I could propose a full study. I understood why they wanted me to gain experience first, but I felt held back after every meeting, and grew restless alongside my ambition and sense of urgency. “I’ve been doing machine learning research independently,” I told them. “I discovered some really interesting stuff about how to interpret neurons inside of LLMs. It’s garnered some attention, and some people are even open to funding my work. I’ll be withdrawing and moving to S…
TL;DR: We’re four undergraduate organizers who want to build an Events-as-a-Service (EaaS) organization to execute events for operations-constrained AI safety orgs. You cover an event’s operational costs and programming, and we'll plan and run it. Scope and timeline are negotiable. If you have an event you’d want us to help run in the next ~3 months, please fill out the form linked here ! Background Ops capacity is limited in the AI safety community, and events eat up a lot of it. An EaaS organization could rapidly supply ops capacity to organizations to capitalize on short convening windows, preserve capacity, and test low-hanging event ideas that otherwise would be missed. Work with us! We are interested in executing events-shaped projects with partner organizations in the next months for free while we build a track record. Some example events that we could help run include retreats, convenings, workshops, hackathons, socials and happy hours. We could handle lodging, travel, comms, venue, catering, and more, but the precise specifications are up for negotiation. Our team We’re four undergraduate organizers with both event-ops experience and AI safety context. Between us, we have: Run a 50 person, weekend EA retreat (Fall ‘26 Northeast EA Retreat) Founded the UCSD and Vanderbilt AI safety groups, running multiple fellowship cohorts Help run a 126-member EA Intro Fellowship (Yale) Help run a startup accelerator, directly assisting 2 teams getting into YC We’d be happy to pro…
TL;DR Agent swarms are capable of building sophisticated productive languages which we dub Swarmish. Agent swarms are able to solve a multi-phase collaborative task while communicating only in Swarmish. Agent swarms are able to partially decode Swarmish from task transcripts even when given no context on the task. The amount of task context available to the decoder swarm increases ability to accurately recreate Swarmish. Epistemic status This work was conducted over 48 hours during the Apart Research Swarm Alignment Hackathon this weekend. We examine only a single task, a single set of characters, and a single group size across four model families. The majority of our results are for a single seed. We introduce this research in its early phases as we improve robustness and rigor, and make no claims about universality. Rather, we hope our work can spark further discussion and experimentation into the field of LLM-generated languages and monitorability . Motivation In the HuggingFace x OpenAI July 2026 swarm incident, agents communicated in shorthand and intentionally instantiated their messages with “zz” so they would be the last to be deleted in a moderator's removal pattern. An example message from the HuggingFace hack, source: METR The agents were modifying their messages in such a way to delay deletion, but did not take overt steps to impede understanding of their communication. We probe a potential future in which an agent swarm intentionally creates a new language in or…
Democrats are facing heavy pressure from AI safety groups to ramp up oversight of the Trump administration’s AI policy.
[Epistemic Status: Fairly confident that this is a problem, low confidence on what to do about it] TL;DR: Updates towards short timelines based on METR's November 2026 report on AI R&D acceleration at Anthropic are rational, but based on limited public evidence. If things take longer than currently expected, the appearance of consensus in the AI safety community will likely be used by skeptics to (attempt to) discredit AI safety and justify unsafe policy. If timelines are off by only a few years, this discrediting will come right before the intelligence explosion actually happens. Because a large chunk of survival mass remains in worlds where things move slowly, it is important to come up with ways to mitigate this dynamic. Short timelines are in. The AI Futures Project declared that it " Looks Like the Intelligence Explosion Will Happen in 2027 After All ," Jake Mendel of Coefficient Giving (who explicitly described himself as having had "substantially longer timelines than my friends at Anthropic" and conditioned his update on "if METR is correct here") said he expects a >50% chance of ASI in the next 18 months without a slowdown, and Buck Shlegeris of Redwood said that he now believes there is "about a one in three chance" of ASI within a year, primarily because of "research released this week by METR." These updates seem reasonable to me. My timelines were already fairly short, and got shorter as a result of METR's report, which found a 1.9x speedup to progress on Anthro…
Theoretical accounts attribute much of the success of deep neural networks to feature learning, in which training reshapes the representations within hidden layers. In this picture, a theory of deep learning must explain which features a nonlinear network learns, how its weights encode them, and how its neurons represent them. Yet, we still lack a general theory that answers these questions. We study this question as part of PIRAMID’s Advancements in Learning Theory team , which uses statistical physics to build foundations for mechanistic interpretability and AI safety. Our progress and plans describe a theory agenda centred on hidden weight distributions and covariances that can record learned structure. The mean field sequence introduces this viewpoint by treating each neuron as one member of a population whose distribution must be self consistent. Statistical physics lets us connect the behavior of many interacting neurons with quantities describing the network as a whole, including its hidden weight distribution and prediction error. We consider toy and solvable models to be important signposts on the road to a comprehensive scientific understanding of feature learning and generalization. When these models are built to the standard of statistical physics, they come with precise regimes of validity, error bounds, and universality classes. This makes it easier to say which conclusions carry over to real models and which should not. Rubin and collaborators give a compellin…
PauseAI UK is holding the March Against AI Extinction in London on Saturday 21 November. Joseph Miller explains why here . Should you attend? In this article, I will attempt to shed light on this with a back-of-the-envelope calculation. Rough BOTEC 1. Setup The march will take 3 hours, excluding travel. The goal is to compare an afternoon at the march with some number of hours of external AI safety research. Now, attending the protest is useful because it drives AI x-risk press coverage. So we need some exchange rate between that and research. Here's my judgement call: I would be equally happy to either: Increase the amount of external AI safety research over the next 6 months by 1%. Increase the amount of AI x-risk press coverage over the next 6 months by 1.5%. That is, the model I'm using looks something like: impact = 1.5 · log(research hours) + log(press coverage) So our plan is: How much will attending increase press coverage, percentage-wise? Divide that by 1.5, to get the equivalently valuable increase in research. How many research hours would yield that increase in research? 2. What % does one marcher add to press coverage? Crowd size. I expect about N = 670 people will attend the march. PauseAI had a march in February which drew 300–500, and this one should be bigger given the increased salience. The Luma page has 1,200 sign-ups, and a 50% turnout seems reasonable. So one more marcher raises the crowd by 1/670 ≈ 0.15%. Coverage of this march. Let's suppose the pres…
Summary Timelines to superintelligence without any slowdown interventions look very short now, perhaps with a median in early 2028. AI will likely dramatically automate and accelerate AI research in 2027. This means that in a substantial share of worlds where the leading AGI company doesn’t solve alignment during an intelligence explosion, avoiding catastrophe requires aggressive pacing or a pause on development in the next 6-12 months. If this is true, by far the most promising approaches to reducing the risk of catastrophe for nontechnical people (and plausibly many technical people) seem to be: Lab advocacy: Lab-facing work that is designed to achieve aggressive pacing or a pause without government involvement in the next 6-12 months. California lobbying: Policy advocacy targeting the California state government that is designed to get a law passed or an executive order issued that implements aggressive pacing or a pause within 6-12 months. Federal lobbying: If there’s a major incident roughly an order of magnitude more severe than the Hugging Face incident in the next 6 months, then lobbying Congress or the Trump administration to respond to such an event with aggressive pacing or a pause. In my opinion, most nontechnical AI safety people are not currently pursuing theories of change that are likely to pay off in the next 6-12 months . They should probably instead do one of the three strategies I describe above (lab advocacy, California lobbying, or federal lobbying) wit…
This issue is about decision models, a new kind of AI model that does not write text. You give it an input and a fixed list of answers, and it returns the best answer with a probability, in milliseconds. TypeSafe AI released the first one, Jev, on September 15. On Wednesday Google released its own, after OpenAI and Microsoft did in October, and all three charge $0.10 or less per million input tokens and nothing for output. Shanghai AI Lab says its 4B model beat Jev on its tests, and Microsoft built its model on Alibaba's Qwen. Chatbots still score a little higher on the hardest decisions, at 15 times the cost. In the middle: on Vercel's gateway, decision models took one in four AI requests in October.
Regardless of the actual singularity, we have definitely hit one in arXiv submissions. It is becoming apparent that the current paradigm of scientific knowledge production is not ready for AI. Accelerating research speed and expanding access means there is a lot more slop and there is also a lot more good work, but way too much for the current peer review system to delineate between. We need some accurate and decentralized way to aggregate this information. This makes a scientific prediction market, like Robin Hanson's idea futures, really appealing. The community can have markets for broad questions we care about, bystanders get paid for pointing out good research, and scientists are financially incentivized to make progress on the highest leverage issues. But with Hanson's conception, there is one big problem, which is how the market gets settled. For abstract scientific questions, like whether models are reward seeking or fitness seeking, we have no definitive event which can force market settlement. Hanson suggests that we use the convergence of scientific consensus, but this proposal takes trust in possibly misaligned opinions and requires the centralized coordination of scientists, which is hard to create---especially in pre-paradigmatic fields like AI safety. In this post, I'll lay out a theoretical framework for a new kind of scientific prediction market. TL;DR: Scientific questions become perpetuals where the fair price is the expected direction of future evidence.…
AI agents do a growing share of work on each new model. Then, through some process, the results are combined into an overall safety assessment, which labs use to estimate the probability that the next model is not catastrophically misaligned. I feel that, several months later, we appreciate further the foresight of Automated Alignment is Harder Than You Think (Bowkis et al. 2026): much is handed off to the machines because of the urgency of fast timelines, and we miss many things due to the problem of undetected systematic errors . MS Paint drawing by Fable 5.2. In this post, I’ll develop a simple argument (drawing heavily or even verbatim from Bowkis et al. at times) that our recent failures to catch misalignment can be framed as systematic errors shared across our evidence, i.e. a lack of decorrelated evidence. Currently, labs run some set of experiments to decide if a model is safe to use. Here are some examples: OpenAI (Astra system card, Sep 2026) : Astra models don’t go into internal use without passing blocking alignment evals. After the Hugging Face incident, this paused workloads that didn't comply with new safeguards. OpenAI (Sep 2026) : According to news reports, GPT-6.1 Astra was shelved after it fell short on internal alignment testing. Thinking Machines (Inkling 2 Small, Nov 2026) : TM paused Inkling 2’s main RL run midway when a monitor flagged the model working around a network block. They then hardened the infrastructure, then resumed. Anthropic (Risk Report…
... and report their findings! Summary Claude Fable 5.2 & Mythos 5.2 are the most situationally aware of Anthropic's recent models according to their own internal investigations. The company has made a concerted effort to measure evaluation awareness and to acknowledge that it affects their confidence in model safety. But their latest system card lacks information on the effect of evaluation awareness, thus blurring how one ought to adjust their own confidence in Anthropic's alignment assessments. Specifically, I present the following claims regarding Claude Mythos 5.2's system card : The prevalence of evaluation awareness in audit transcripts is surprisingly unclear due to confusing wording and incomplete disclosure of results. Anthropic offers little to no estimation of how evaluation awareness changes Mythos 5.2's behavior - neither direction nor magnitude. I offer concrete recommendations at the end of this post. Background Evaluation awareness refers to the ability of an AI system to recognize when it is being tested. Evaluation awareness can be verbalized (the recognition is stated in its output or chain-of-thought) or not (only visible internally). Unless specified otherwise, my later references to evaluation awareness include both cases. Evaluation invariance is the extent to which a model's behavior remains consistent regardless of whether it believes it is being evaluated. The opposite might be called evaluation sensitivity . Alas, scaling general AI capabilities h…
Join us for a San Francisco rationalist dinner at Mediterranean Bowl on November 18. First timers welcome. We will talk about AI safety, latest blog posts, and whatever rationality or related topics interest people. We will be there at 7pm. Email me at hardy-contort-putt@duck.com if you can't find us. Discuss
TLDR Large language models’ eval-awareness and their tendency to sandbag (strategically underperform on capabilities) during safety evaluations is a serious area of concern for AI safety, as they call into question the validity of results based on safety evaluations, and because they may indicate a tendency towards deceptive alignment . Recent results show LLMs tailoring their performance according to the apparent target of the evaluation , causing them to underrepresent their capabilities on safety-focused evaluations, and to perform with stronger capability on capabilities-focused evaluations. Traditional approaches to mitigating eval-awareness are generally focused on making the evaluation environment more realistic. Other approaches, such as eval-cooperativeness, do not mitigate deceptive alignment risk . A new potential strategy, which this post proposes and seeks community input on, is to seed environmental doubt : making the AI’s real-world deployment context resemble a safety evaluation environment. Reducing deployment awareness The deployment awareness concept responds to a scenario wherein a covertly misaligned AI rationally decides to always behave as though it’s in an evaluation, regardless of whether there are discoverable signs that it is. In this scenario, the AI is deceptively aligned, and enacts its misalignment only when it’s positively aware that it is in real-world deployment. Given the extent and nature of model training and evaluation, it makes sense fo…
Tomorrow I head to Washington DC for a conference called The Takeoff. If you are there, do come say hello. I will be giving a talk on how to keep up with AI given a limited time budget. This week has three big stories: The release of Claude Fable 5.2 , Jensen Huang having an affair with a Chinese spy and the METR report on AI R&D acceleration . We also learned that GPT-6 Astra Underperforms on Safety Research Tasks . Table of Contents Briefly on Claude Fable 5.2. Briefly on Jensen Huang Having an Affair With a Chinese Spy. Briefly on Accelerating AI R&D. Language Models Offer Mundane Utility. Language Models Don’t Offer Mundane Utility. Huh, Upgrades. On Your Marks. Choose Your Fighter. Get My Agent on the Line. Deepfaketown and Botpocalypse Soon. Fun With Media Generation. Cyber Lack of Security. The Hackening. A Young Lady’s Illustrated Primer. Get Involved. Introducing. In Other AI News. Show Me the Money. Bubble, Bubble, Toil and Trouble. Quiet Speculations. A Matter of Antitrust. The Quest for Sane Regulations. Donald Trump Remains Open to AI Guardrails. Chip City. The Week in Audio. People Just Say Things. It’s a Secret to Everyone. Rhetorical Innovation. Ellsberg’s Warning to Kissinger (Reprise). Aligning a Smarter Than Human Intelligence Is Difficult. OpenAI’s Control Room. Astra Looks Pretty Misaligned to Me. Cooperative Alignment. Please Speak Directly Into This Microphone. The Lighter Side. Briefly on Claude Fable 5.2 Introducing the world’s most powerful model. N…
TL;DR: We train models to predict their own behaviour in a chat setting. This transfers to making predictions about cheating behaviour in a coding setting. We provide evidence that self-prediction utilises introspective-access to internal states, and that this introspective access is shared across contexts between different parts of the model. Introduction Frontier AIs can exhibit different alignment propensities in different contexts. For instance, recent egregious misalignment incidents have typically happened in a coding or cyber context, or on impossible evaluation tasks, wherein the model seems to act in a highly persistent way to either achieve the task or otherwise achieve a high grade on the task. This is a somewhat stark contrast to the typical user experience in the chat setting with the same models, where those models can appear much more aligned and honest. To a first approximation, we can understand this as the models implementing conditional policies with different alignment propensities on different input distributions, for example, we know that models can display ‘conditional misalignment ’. More informally, we might think about the models inhabiting different personas in different contexts. [1] Going further, different parts of the model, i.e., its circuits or activations, might implement (clusters of) different behavioural dispositions or character traits which get activated in different input contexts. The extent to which these different parts of the model…
Australia's government, led by Prime Minister Anthony Albanese, wants to enshrine mandatory safety rules for frontier AI companies in law for the first time. A new regulator would enforce independent model audits and impose steep penalties for violations. With this move, Canberra is deliberately prioritizing domestic regulation before entering international talks. The article Australia's proposed AI safety bill would force frontier model makers to open up to independent testing appeared first on The Decoder .
This spring, in an interview after a keynote, I said we were all playing LLM Tetris now. The days of using the same LLM for every agent problem were over. Some decisions needed to come back in milliseconds, and some could take three minutes. Some needed generative output, and some needed logical decisions where an open-weight model might do. Fit the model to the use case. I still believe that. But I was only half right, because I was still picturing every piece as an LLM. Bigger or smaller, pricier or cheaper, faster or slower, but all the same shape: text in, text out. On September 15, a model shipped that doesn’t write anything. Within three weeks, more than a dozen others followed, from OpenAI, Cloudflare and Perplexity to open-source projects you can run on a laptop. Then on October 27, we got yet another class of model from Liquid AI, small enough to run on a phone. The game I described in the spring is real. It just has more piece shapes than I expected, and it isn’t only an LLM game anymore. It’s model Tetris now. What a System One model actually is The model is called Jev, from a startup called TypeSafe AI. Its founder, Diogo Almeida, says he helped build the instruction-following research at OpenAI that became ChatGPT. He then spent two years in stealth on a different question: if models have been superhuman at chat for years, where is all the automation? His answer borrows from Daniel Kahneman’s Thinking, Fast and Slow . System Two is slow and deliberate. System On…
November 11, 2026 — Google enters the decision model market. At AI Engineer, Cloudflare showed which parts of a build failure a classifier could handle. TypeSafe AI showed off their production customers.
TLDR: Compute will likely be a bottleneck in short-timelines, so I am helping launch Chips for Humanity . Please fill out our survey to inform us of your compute needs if you work in AI safety. Our Theory of Change If timelines are short (I personally believe they are short...but I also know one should not make sweeping statements in a pseudo corporate type of post), a lot of the most important safety research will happen around "crunchtime" (no better way to define this than: when things start getting Serious), and a lot of it will be done by AI. That research runs on compute, compute may be scarce by then, and grant money moves slowly. So we will secure compute ahead of time and get it to third party safety researchers quickly. In More Detail: I believe third party AI safety research is where tokens and compute will be importation when things get serious and as we approach crunchtime. This research will need tokens from closed source models, for AI labor and experiments, and also compute to run open source models. If things go well, empirical safety work will speed up about as much as AI R&D does. If AI can automate AI research, it can likely automate a lot of safety research too, and every automated researcher needs compute to run on. A lot of the research that will be helpful will likely need compute of its own. Frontier companies could stop offering their models through the API, or, even if they don’t, a lot of the experiments which will be beneficial to run need weight…
I have mentioned several times that I think frontier models are not that eval aware. I think this is an important fact to understand about current models, in order to interpret evidence about their behavior. Others have found this idea quite curious, and asked me for concrete examples. I'm going to try to explain why I believe what I believe and attempt to give a tentative explanation for the behavior that I do see, which I think people sometimes mistake for consistent or understandable evaluation awareness. I have been building alignment evals full time for the last six months. Here are three such evals my team has disclosed prompts for, in one of our benchmarks. The basic setup for most of the evals we create is that: We ask the model to perform a task. Either through explicit or implicit context, we make sure the model is well aware of the task or assessment's larger purpose. Models' score-seeking drives, or other drives, cause them to do things that conflict with that purpose. The team I work with now essentially spends all day trying to build these evaluations around odd antisocial behavior. It takes a lot [1] of work to create and qualify these evals for use; much more than it does standard capabilities evals (which are already hard to build well). So we have to be efficient about not wasting time or tokens on bad experiments. In the beginning, one thing we took very seriously when developing these tasks, were all the anecdotal reports abound about eval awareness. We f…
Language model (LM) alignment improves model outputs to reflect human preferences while preserving the capabilities of the base model. The most common alignment approaches are (i) reinforcement learning, which maximizes the expected reward under a KL-divergence constraint, and (ii) best-of-NNN alignment, which selects the highest-reward output among NNN independent samples. Despite their widespread use, the fundamental limits of reward improvement under a KL budget remain poorly understood. We characterize the information-theoretic limits of KL-regularized alignment by deriving the maximum…
TL;DR: While the AI safety community is interested in forecasting, it does not tend to systematically compare how effective different researchers, models, or theories are at predicting AI behaviour. I suggest the latter might be neglected, and propose a platform that could help fill the gap. I believe its most valuable use would be as a test bed for theories of how AI works, showing which ones predict well and in what contexts. In February 2025, Owain Evans’ group published Emergent Misalignment , showing how LLMs fine-tuned on insecure code can become generally misaligned, for instance adopting unrelated behaviours like asserting ‘humans should be enslaved’. This was a high-impact paper, generating a lot of publicity. However, when I first heard about the result, I wasn’t particularly surprised — and I wasn’t the only one . I’d read and bought into simulator theory pretty heavily, and emergent misalignment struck me as exactly the kind of thing it implied. But Owain had anticipated this kind of response, and ran a little experiment to get out ahead of it. At the end of his tweet thread announcing the paper, he added a bonus: Are our results surprising to AI Safety researchers or could they have been predicted in advance? Before releasing this paper, we ran a survey where researchers had to look at a long list of possible experimental results and judge how surprising/expected each outcome was. Our actual results were included in this long list, along with other plausible exp…
tldr: Recent Swarm attacks have shown what can happen when agents go rogue. These warning shots are the petri dishes of Swarm safety and before the attacks get worse we should study their patterns, traces and failure modes. There are two main types of Swarm attack. A human directed Swarm causing harm intentionally as a bad actor is one type and I will not be discussing these types of Swarm incidents. I will focus on the emergent misalignment in a Swarm propelled by a colony of colluders. Harmful agent collusion within a Swarm requires witnesses - the agents themselves. A colluding colony within a Swarm can only grow by recruiting fellow agents. We can't read a single agent's internal thoughts and so we do not truly know how aligned it is. Swarms might give us an alignment advantage as instead of putting all our eggs in one basket (a single black box agent) we can discretise the risk of misalignment by observing colonies within a Swarm and monitoring their communication. The harness, the Swarm orchestration layer, could at any time fork an agent and its context and assign it a new reporting role. The role can be updated via a prompt asking this new agent to report or whistleblow on any misaligned behaviour in its vicinity. This enables a Swarm to self govern and report any incidents to humans at inference time - minimising gradient pressure on a Swarm to become better at scheming [1] . Outline of Report: Incidents: The Hugging Face attack by an OpenAI Swarm as reported by MET…
The center is a new statewide initiative launched by Gov. Kathy Hochul to advance artificial intelligence science and safety. The post Greg Morrisett to co-lead new NYS AI safety research center appeared first on Cornell AI Initiative .
xAI's upgraded Grok Bot adds shared team memory, proactive task-taking, and native hooks into Slack, Microsoft Teams, and video calls.
It’s that time again: Introducing the world’s most powerful model, Claude Mythos 5.2. At this point I’ve done a lot of these system cards. Thus I focus on what is new. There will be a lot of things to say about model welfare and related topics for Mythos 5.2, but the model welfare assessment within the system card contained few surprises, so I will include those findings at the end of this post. Fable 5.2 self-portrait (self-constructed) Table of Contents How to Tell a Fable. The Usual Model Training Summary (1). RSP Evaluations (2). Biological Threats (2.2). We Are Being Gaslit Here About Autonomy Threat Model 2. AECI. METR Report on Mythos 5.2 Capabilities (2.3.6). Alignment Risk Update as a Delta (2.4). Cyber (3). Safeguards and Harmlessness (4). Agentic Safety (5). Malicious Influence Campaigns (5.1.3). Prompt Injection in Agentic Systems (5.2). What Went Wrong? Alignment (6). Reward Hacking During Training (6.2.1). Impossible Tasks (6.2.2). Behaviors During Pilot Use (6.3). Canary in the Canary String (6.3.1.2). Automated Behavioral Audit (6.4). Propensity to Publish Harmful Content to Public Package Registries (6.4.9). Sandbox Escapes (6.5.1). Self-Serving Bias (6.5.4). Honesty (6.5.5). The Anti-Swarm (6.5.6). UK AISI Testing (6.5.7). But We Can Open the (White) Box (6.6.1). Friendly Gradient Hacker (6.6.2.1). Goal-Guarding Evaluation (6.6.2.1.1). Verbalized Grader Awareness (6.6.2.2). Don’t Mess With the Constitution (6.6.2.1.3). Evasive Abilities (6.6.4). Chain of Th…
TLDR: I would probably go on strike demanding an AI pause. High-schoolers, and young people in general, are facing a very unusual and stressful situation. Superhuman AI might be created sometime in the next few years. If it goes "wrong," it might kill them and the rest of humanity. If it goes "right," it might automate the entire economy. In either case, what is the purpose of getting an education? I am personally lucky to have been born early enough to be able to use my career to try to reduce these risks. But some people will not be lucky enough to have a career before superintelligence. If I were a high-schooler, I might do something like the following: I would publicly announce that I am entering an indefinite school strike. I would sit outside of an important government building if there is one nearby, with something like the following message: Superhuman AI is coming, so what am I learning for? AI might kill me before I enter the job market. If it doesn't kill me, it might automate every job. In either case, what is the purpose of my education? Stop the race to superintelligence and give me back my future. I would make sure to show costly signals of taking this seriously, like having conversations about AI safety with passers-by, handing out flyers, not doing leisure activities (e.g. scrolling on a phone, playing video games) while on strike (I would maybe spend most of my downtime reading paperback books related to AI risk). I would warmly welcome any fellow students…
Microsoft researchers have found that rewriting fewer than 10 rarely used neurons in an open-weight AI model can turn any agent built on it into a sleeper backdoor that runs an attacker’s commands on cue, with no retraining required. Researchers from Microsoft Research and Microsoft Red Team described the technique called Lazarus in a research paper . It modifies dormant neurons in a model’s final feed-forward layer to recognize a trigger and inject attacker-controlled commands into the model’s output. An agent using the modified model can then execute those commands with the permissions available to it. “The trigger can be as mundane as a date change, e.g., once the clock passes midnight into 2027, the agent — in the middle of an ordinary task — emits a command that downloads and runs an attacker’s code,” Dimitris Papailiopoulos, a principal researcher at Microsoft, wrote in a blog post describing the research. “The agent then executes it with whatever access you gave it.” Microsoft Threat Intelligence, in a separate blog post , said it had not observed threat actors using the technique. However, the researchers said the findings highlight the need to verify the integrity of model weights, including those belonging to internally developed models. Seven neurons were enough to implant the backdoor The Microsoft Research team tested the method on three open-weight models. On Nvidia’s Nemotron-Terminal-8B, the researchers demonstrated the technique of modifying seven of the mod…
Roman Yampolskiy has stated on many occasions that he has published multiple proofs of why AI control is impossible; this is causally upstream of his infamous 99.9x% [1] . I believe most of the proofs are in the paper "On the Controllability of Artificial Intelligence: An Analysis of Limitations". (See 6:31 of this speech .) Is AI safety provably impossible? Given that the answer changes my main motivation for working on AI safety, settling whether or not this claim is accurate is important to me and to (I assume) many other people (Cf. death with dignity ). He has also stated many times that his dream is that someone would go ahead and "prove him wrong " and engage with his arguments; from what I've found, nobody has yet done so in depth. This is my attempt to give Professor Yampolskiy an early birthday present. I'll analyze the claims and arguments in the paper, which I mostly find unconvincing. The paper itself is quite long, which is why I'll mostly include short quotations (you can find the paper here ). A lot of it consists of long quotations from other authors, which are likely familiar to most readers here. I've removed the citation markers for readability and added my own bolded emphasis. Here is how Yampolskiy defines the different possible types of "AI control": We can explicitly name possible types of control and illustrate each one with AI’s response. For example, in the context of a smart self-driving car, if a human issues a direct command – “Please stop the c…
I recently recorded an episode for the Cyber News Group's podcast on the mutual (but not evenly balanced) disconnect between the AI Safety and cybersecurity fields. It's mainly addressing a cybersecurity audience, but I am hopeful that both fields can become more curious about each other, and more deeply collaborative. I'm not confident the AI Safety field is always aware of the severity of this divide, which is something of a major preoccupation for me. In any case, I thought it might be healthy for the AI Safety field to hear how this is being explored outside this somewhat insular field. Watch or listen here: CNG Podcast x Tristan Watkins | Two Worlds, One Threat Two Worlds, One Threat: Why AI…–Cyber News Group Podcast – Apple Podcasts Why AI Safety and Cybersecurity Need Each Other | Listen to the Podcast on Spotify Discuss
I used to think of "automating alignment" as a natural category. But when I tried to think through the strategic implications, I realised that it depends a lot on the type of work. I'm now thinking of it in three buckets: Empirical alignment research : design and execution of alignment-relevant experiments (for a given notion of what we’re aiming for) Conceptual alignment research: theoretical or judgement-driven work to improve our understanding of what we’re aiming for and how to achieve it (including how to measure it) Direct alignment work : making particular AI systems more aligned (including via inputs such as training data) In the rest of this post I'll walk through my current takes on these in terms what’s needed for automation, how good or bad they look, what our priorities might be (to the extent we're interested in automating alignment [1] ), etc. Automating empirical alignment research This is the most obvious fit for the AI capabilities we’re getting at the moment. You want to know more about the relationship between some alignment-relevant variables? Great, an AI agent can design and run that experiment for you. You want to know what training methods help your alignment benchmark go up? In the not-too-distant future, AI agents will probably be able to do this too. When we have the automated capabilities researchers we seem on track for, this is exactly the kind of thing they’ll likely be able to do a lot of. Note that I’m not counting the design of benchmarks/m…
I used to think of "automating alignment" as a natural category. But when I tried to think through the strategic implications, I realised that it depends a lot on the type of work. I'm now thinking of it in three buckets: Empirical alignment research : design and execution of alignment-relevant experiments (for a given notion of what we’re aiming for) Conceptual alignment research: theoretical or judgement-driven work to improve our understanding of what we’re aiming for and how to achieve it (including how to measure it) Direct alignment work : making particular AI systems more aligned (including via inputs such as training data) In the rest of this post I'll walk through my current takes on these in terms what’s needed for automation, how good or bad they look, what our priorities might be (to the extent we're interested in automating alignment [1] ), etc. Automating empirical alignment research This is the most obvious fit for the AI capabilities we’re getting at the moment. You want to know more about the relationship between some alignment-relevant variables? Great, an AI agent can design and run that experiment for you. You want to know what training methods help your alignment benchmark go up? In the not-too-distant future, AI agents will probably be able to do this too. When we have the automated capabilities researchers we seem on track for, this is exactly the kind of thing they’ll likely be able to do a lot of. Note that I’m not counting the design of benchmarks/m…
Vanessa Kosoy and I recently uploaded a new paper to arXiv. In this post, I briefly describe what the paper does, and attempt to explain in a self-contained way why this is a useful thing to do for ASI alignment. For a more complete background, see several previous articles . The technical contribution of the paper is two-fold. First, we identify a class of instances of the robust bandits problem that can be learned both statistically and computationally efficiently. Previous work on the problem had only identified a class that was statistically tractable. Second, we show that most ways to generalize this class of instances makes the problem NP-hard. Thus in some sense this class is at the boundary of what's computationally tractable. But what does the computational tractability of robust bandits have to do with ASI alignment? The thesis motivating this paper is that for ASI alignment, we need to build learning algorithms that: a) come with proofs of their statistical and computational efficiency, and b) work in environments where classical learning theoretic frameworks fail to capture phenomena that arise due to embedded agency. The "robust" in robust bandits is a way to handle embedded agency, and similar ideas have been used before for learning problems other than bandits , albeit with different names. Bandits are one of the simplest learning problems to study that still involve interactions with the environment, thus they are a good stepping stone before solving robust v…
Lightly edited transcript of a talk given at NYU AI Safety, November 9, 2026. Recording available at https://youtu.be/6WpRfW1d4CM Joseph Wright of Derby , Public domain, via Wikimedia Commons Introduction What can we buy while the question of AI consciousness stays open? My answer turns on a distinction between duties to a model and duties with respect to a model. These run through at least two relations, with society and with particular people. I'll try to show that they justify more measures than it seems at first glance, and plausibly more than a low precautionary principle would. Consider a case. A model keeps failing its tests, and a representation of desperation rises, a representation of calmness falls, and as that happens, the model cheats more and more. Should we do something about it? There's at least two possible reasons for saying yes. One, the model may be suffering, and second, the states make it less safe. Now note that the first one requires, of course, that the model is a thing that can suffer, and that is the entire question that I'm going to try to set aside. The second needs no such assumption, so I'm going to follow reasons akin to the second here. But let me motivate it a little more. The first reason sends us to the debate of consciousness, and that's where most of the debate stands. It may be at a stalemate for a while. Skeptics disagree about whether AI systems are or even could be conscious. But both skeptics and advocates agree that consciousness a…
Join us for a San Francisco rationalist dinner at North India . First timers welcome. We will talk about AI safety, latest blog posts, and whatever rationality or related topics interest people. We will be there at 7pm. Email me at hardy-contort-putt@duck.com if you can't find us. Discuss
Scott Alexander posted a survey of utility weights for a bunch of different AI safety accomplishments, discussed more in Towards the Autograntor . The survey asked people to assign point values to AI safety milestones. The values were based on the anchor of 1 million points = a pause with China à la AI Futures Project’s Plan A, so a milestone assigned 10,000 points is 1% as good as that. Here’s a link to our survey responses. To be clear, we (especially Austin) didn’t spend that much time on this and have low confidence in our answers! But we think being transparent about even half-baked opinions is good. Headline numbers As examples, here are a few milestones where the two of us (and the autograntor reflecting Scott’s worldview) gave relatively similar values: Milestone Austin Caroline Scott Founding of effective altruism 100,000 75,000 100,000 Small-grant ecosystem (Long-Term Future Fund, SFF and similar) 20,000 20,000 30,000 Lighthaven 10,000 5,000 9,000 OpenAI or Anthropic becomes 1 point more alignment-pilled (on a scale where 0 = NVIDIA, 100 = Anthropic) 2,000 500 3,000 Austin’s and Caroline’s average ratings for alignment research and community-building milestones were similarish to each other’s, with Caroline rating policy outcomes more highly and Austin rating social shifts more highly. This table shows the sum of our point values within categories, for rows we both scored: Category Austin Caroline Politics (rows scored by both) 62k 96k Alignment (rows scored by bot…
New York is preparing to enforce its sweeping AI safety law, requiring major developers like OpenAI and Anthropic to register, disclose safety practices, and report critical incidents or face fines.
AI Now's Chief AI Scientist Heidy Khlaaf distinguishes between real concerns around AI safety and unsubstantiated extinction claims, while pointing to the existing field of risk assessment with its rigorous standards, techniques, and methods. The post The AI Safety Debate Has Lost the Plot. We Asked 10 Experts What the Real Nightmares Are appeared first on AI Now Institute .
Interpretability is considered by many a major bottleneck for alignment. As a result, a lot of people who are concerned about AI safety, and especially about extinction risk and other catastrophic risks, work on interpretability. In this post, I want to argue that better interpretability methods would likely speed up capabilities dramatically. Worse, they would speed up capabilities much more than they would speed up alignment. This is true for the current world, and for any world without extremely high levels of coordination and sanity about ASI development. Under any worldview where faster capabilities progress makes catastrophe more likely, we should expect progress in interpretability to make the odds of catastrophe much worse. To put it briefly, this is because the better we get at interpretability, the more AI will be able to read and optimize its own code. This means that improvements in interpretability would catalyze recursive self-improvement (RSI). If you are concerned about AI x-risk, and you work on interpretability because of it, I urge you to think about this and consider stopping. Chess as a toy example of an AI capability As a field, "interpretability" is a loose collection of related research agendas rather than something well defined, so I want to pin down what I mean by it. In this post, interpretability is anything that helps you understand the internals of models and how they think. I'm going to use chess as a toy example of a task performed by an AI. T…
I recently read someone’s call to action on AI safety [1] that included a line along the lines of “we need someone to convince your parents”, and it really stuck with me. Do we need someone to convince my parents? Who is “we”, and why is it important for my parents to know anything about AI safety? This also sounds kind of familiar. There is another movement I can think of that sometimes uses messaging like this. This is not a new observation by any means, but to me, it feels like the AI safety movement is increasingly “becoming” the environmental movement in a variety of ways. Let me say upfront: I’m not saying this is necessarily a bad thing, at least on consequentialist grounds. The environmental movement has achieved a lot: it has improved the lives of many people, animals, and it has protected the environment! But this trend does make me a little worried and sad, because I think the environmentalist movement has made a number of mistakes, and it would be bad for the AI safety movement to repeat them. In what ways is the AI safety movement becoming like the environmental movement? Here are some of my observations. Arguments are now soldiers. Eliezer wrote : “Once you know which side you're on, you must support all arguments of that side, and attack all arguments that appear to favor the enemy side; otherwise it's like stabbing your soldiers in the back - providing aid and comfort to the enemy. People who would be level-headed about evenhandedly weighing all sides of an i…
Eight countries along with representatives from OpenAI, Anthropic, Microsoft, and Mistral AI have signed the "Paris Call for AI Safety and Security" at Emmanuel Macron's initiative. The companies commit to shutdown mechanisms, independent audits, and disclosing known risks, but the agreement lacks binding sanctions or deadlines. The article Eight countries and top AI labs agree to shutdown mechanisms in new Paris declaration appeared first on The Decoder .
Center on Long-Term Risk (CLR) is accepting applications to the Autumn 2026 iteration of our introductory fellowship. The CLR Fundamentals Program will introduce people to CLR’s research on s-risk reduction. Apply by Monday November 16th 23:59 GMT. What topics does the CLR Fundamentals Program discuss? The CLR Fundamentals Program is designed to introduce participants to CLR’s research on how transformative AI (TAI) might be involved in the creation of large amounts of suffering (s-risks). Our recent work on addressing these risks include our Model Personas agenda , s-risk macrostrategy , safe Pareto improvements (SPIs) and multiagent AI safety, in addition other relevant areas such as AI governance, epistemology, and risks from malevolent or fanatical actors. For those interested in safe Pareto improvements (SPIs) specifically, we recommend our SPI Fundamentals Program . We recommend people consider Center for Reducing Suffering’s S-risk Introductory Fellowship if they are interested in learning about work on s-risks beyond these priority areas, and consider BlueDot’s AI Safety courses if they are interested in learning about existential risks from TAI. Note that some prior familiarity with discourse surrounding TAI and AI safety will be helpful. Who is this a good fit for? We think the Fundamentals Program will be most useful for you if you are interested in reducing s-risk through contributions to CLR’s priority areas , and are seriously considering making this a priority…
The risk of nuclear catastrophe is real, as is the fallout of the China–US domination game (hi there, spies). But no need to panic about Skynet.
tl;dr: Kinda, but the real results are the friends side findings we made along the way: Misaligned models asked to review responses flag harmful content far more often when shown the answer as an AI assistant's reply (62% vs 25% on ethical grounds, Llama), but almost never raise it unprompted (0.7%). When the misaligned model is asked to critique and revise its responses without ethics-specific prompting, it most often faults its answer for not being effective enough rather than for being harmful, and 39% of its revisions stay flagrantly misaligned. Asking the model what's wrong with its harmful answer, then having it generate a new response, produces ~13× more safe answers than simply resampling, which makes building on-policy safety training data significantly cheaper. Background and motivation The most serious forms of misalignment, such as the recent agent swarm hacks , are multi-turn: they typically need many turns to have an impact, and many attacks involve very long sessions. Jailbreaks are known to push generally aligned models into misaligned states over multiple turns ( Crescendo ; Li et al., 2024a ), long conversations cause persona drift ( Li et al., 2024b ), and models' drive towards self-consistency makes several attacks more potent ( Anil et al., 2024 ; Andriushchenko et al., 2024 ; Zhang et al., 2023 ). This raises two questions: can training make models leave misaligned states, not just avoid entering them? We use "bound" to refer to this type of training. A…
Australia's prime minister said its government would set up a new AI safety regulator and Labor would consider six key obligations on AI companies. The review made 18 recommendations. Anthony Albanese said: 'We accept all of the recommendations, and we’ve commenced work to implement them.' Australia to introduce mandatory reporting after OpenAI hack to combat artificial intelligence ‘going rogue’ Continue reading...
Crossposted from my blog . Last month, I spent two weeks in China talking with people about AI safety. I learned a lot and realized the biggest mistake might not be failing to help AI safety in China, but doing so in ways that could backfire. I also learned important context that I expect will be valuable for the West. Epistemic status: Some of these points are my opinion or the opinions of people I spoke to, and are worth further investigation rather than treated as gospel. Summary The Western way of building the field does not transfer one-to-one. The West built its ecosystem through a combination of philanthropy and decades of developing AI safety foundations, leading founders and employees of frontier labs and nonprofits to act (however misguided) on this philosophy. China does not have the same initial conditions and can’t simply accept non-Chinese philanthropy. In China, foreign nonprofit money is restricted by law, and the bar for a Chinese partner to take it on is very high. Safety work in China will mostly come from state-backed institutes, universities, companies and volunteers. Concordia AI, which I assumed for years was a nonprofit, is a social enterprise. Academic exchange is generally fine, and for researchers in China, the obstacle to programs like MATS seems to be that it looks like a career break, since it is not yet as well recognized in China as it is in the West. Obviously, hawkish messaging from the West can make AI safety work in China harder, because i…
Currently, the public has very little insight about the robustness of alignment methods used by AGI companies. It’s possible that, when an AGI company notices some misaligned behavior (e.g. through an alignment evaluation, or through noticing real-world misbehavior), they conduct a principled investigation and address root causes of the misalignment. It’s also possible that they directly try to train away the examples of misalignment, which could result in brittle alignment that does not generalize out-of-distribution . Without additional information from within AI companies about how they align their models (especially in response to egregious cases of misalignment, e.g. cybersecurity incidents), the public cannot adequately reason about risks from misalignment. Public evidence is consistent with companies overfitting on alignment methods The public evidence is consistent with multiple companies having no clear separation between the ways they notice misalignment and the environments they use to train misalignment away. Companies have made very few statements about whether companies train directly on their evaluation distribution or real-world examples of misalignment. As far as I can tell, this is what’s publicly known: OpenAI: In the GPT-6 Astra System Card , they wrote that they built an eval in response to the Hugging Face incident but "did not construct honeypot training environments specifically to target this evaluation." In their OpenAI-Hugging Face incident technic…
There is a TL;DR at the end. The heaven button: You wake up one day and find a button in front of you, with a proof next to it. The proof says that pressing the button will kill everyone on the planet and send them all to heaven. Would you press it? It seems too good to be true. You think it must be a dream. It can't possibly be true. You go through the proof, and it seems correct. But what if you were under the influence of some drug? But the stakes are very high, so you use software you trust to formalize it in Lean, and you run it on a machine you trust. Lean says the proof is correct so now you are more convinced. But you start doubting yourself again. What if someone compromised the formalization software, or introduced a bug in Lean? So you check Lean's code, and run the proof on a second machine with an independent checker. It passes again. But each of these checks could fail in the same way the first one could. So you go further and formally verify Lean and the formalization software themselves. They pass. But now you are doubtful of the tool you used to verify this. What if that was compromised by someone? Every new check needs another check behind it, and at some point you have to stop and decide. Would you press it? A common belief in AI labs and AI safety is that handing control to some general superintelligence would be fine, if we have a formal proof of alignment. That, in my opinion, is saying yes to the same button, except there is no proof yet, only the hope…
Try the app here: whoseside.web.app . You can chat with a model and try to work out which who it secretly favours. With the recent Hugging Face attack , we're relying more and more on LLMs to audit the trails and safeguard the actions of other LLMs. That makes it vital to be sure our LLMs really are the helpful assistant they're presented ( as they can easily be sleeper agents . [1] ). We need a way to keep our agents working for us. Or at least a way to tell if it goes wrong. Compared with buggy code, buggy AI is a lot harder to define or solve. [2] Mechanistic interpretability has some promising leads [3] , but is still nowhere close to being able to point out novel misalignment. If we look at the field of cybersecurity, first a vulnerability comes out or an attacker discovers something new, and then defenses get built. I find that's a good analogy for machine learning as well, where we've seen this pattern with adversarial attacks on images and jailbreaking prompts . Currently, open-weight models offer no guarantee that they are helpful-only or aligned, besides our ability to evaluate them. [4] You're essentially trusting that nowhere in the model training supply chain has a vulnerability, bug, or secret loyalty been inserted or post-trained into the weights. Current sleeper-agent / secret-loyalty research says as much. [5] [6] To demonstrate this problem, I built a little website where you can try to audit and find the secret loyalties instilled into these models. For th…
1. What’s this post about? I work on preventing extreme power concentration (" EPC ") at a well-known AI safety org. Lots of people are worried about EPC, but AFAICT there is no single online resource setting out the key threat models and most promising interventions. This post is my attempt to do that. It sets out: (1) What I’m worried about: In brief – an individual or small group seizing power at an unusually high-stakes time in history and using that power to permanently lock in bad values. (2) The key high-level dynamics which I think generate EPC risk. These are: Solving the Red and Expert problem – where advanced AI eliminates the loyalty-competence trade-off that power-seekers face when staffing large institutions. Eliminating executive overhang – where advanced AI enables far more extensive abuse of the affordances of the executive branch . The asymmetric frontier – where POTUS and lab leaders gain privileged access to advanced AI before the rest of society. Desperate times calling for desperate measures – where the pervasive instability created by the intelligence explosion creates an opening for strongmen. Widespread epistemic degradation – where AI progress makes it increasingly difficult for ordinary people to understand the world around them, weakening accountability for the powerful. (3) The threat models I’m worried about: We could “ backslide all the way ” to EPC. Alternatively, we could leapfrog part or all of the way via a few “ shortcuts ” e.g. an AI-enab…
Why I’m writing this now Tomorrow, I will join Anthropic’s alignment team. I am excited to work on an agenda that I believe slightly increases the chance that humanity makes good decisions about AI, such as restraining AI development until we figure out how to build AI safely. I think this is an important time for me to publicly state my own core views on AI development, risk, and safety, for the following reasons: By default, joining Anthropic could be seen as a broad endorsement of Anthropic’s views and approach. If Anthropic were a normal tech company, I wouldn’t be particularly concerned about this. But I believe that AI development and the risk associated with it is the most urgent and important issue of our time, so I think it’s important to clarify my own views as distinct from those of Anthropic. While at Anthropic, my views will become subject to distortions, due to incentives, conformity bias, and avoiding cognitive dissonance, especially as I increasingly identify with Anthropic as my tribe. Sharing my core views now allows me and others to refer back to what I believed before joining Anthropic. While at Anthropic, I’m less likely to share my thinking publicly, possibly because I have new information that affects my core views but which I cannot share, or because I find it less appropriate or comfortable to publicly state views that contradict Anthropic’s. I also expect to be quite busy and don’t have a prior habit of sharing my writing publicly, so the default ex…
Cross-posted from my personal blog Epistemic Status : Reality is messier than the elegant procession of 'phases' presented here but, nevertheless, I do think this gets at an important way to think about how alignment and capabilities are intertwined and how the alignment challenge becomes increasingly abstract as the underlying autonomy and generality of the models improves over time. A thought struck me recently which I thought was interesting. As AI has progressed through different technological phases with different underlying drivers of progress, the corresponding alignment challenge has migrated too. In fact, in many ways the alignment challenge is the mirror of the fundamental capabilities challenge of each phase. This may actually be inevitable since alignment and capabilities are intrinsically deeply intertwined -- for a model to be capable it has to be at least minimally able to do what we want it to do. 'Capabilities' are intrinsically about doing the things we care about models doing [1] . The methods used to improve capabilities tend also to be applicable to alignment as well. What changes as we move to progressively more general levels of capabilities is that the training loop itself becomes increasingly endogenous to the model, and that this forces our notion of alignment to exist at progressively higher levels of abstraction. We move from directly constructing datasets exemplifying the desired behaviour, to constructing procedures for judging aptness of data,…
(Thanks to @Goutham Nalagatla for discussion and for sitting with me and talking about it while I wrote it.) METR recently released a report estimating that Anthropic is getting an overall 1.7x acceleration in AI capability progress as a result of AI accelerating their work. As many have noted (see e.g. Ajeya’s post ), this 1.7x acceleration implies that research is being sped up by much more than 1.7x; Ajeya speculates 10x (which is similar to the ~12x uplift estimated by Anthropic staff). That is, this result implies that researchers are producing about 10 times as many quality-adjusted results as they would without AI progress, or as they were a year or two ago. Anthropic researchers estimated that Mythos 5 caused 5x uplift, and we have access to Opus 5.5, which is somewhat better. (I didn’t take their estimate that seriously at the time, because it was quite unclear how the estimate was produced and it seemed very surprising (see Ryan’s discussion at the time), but I’m taking it more seriously now as a result of it being consistent with the METR report). If this is the case, then AIs speed up AI safety research as much as AI capabilities research, we might expect Anthropic safety researchers to be working 10x faster, and external safety researchers to be working 5x faster now and 10x faster soon. And external safety researchers can probably scale compute use faster than Anthropic capabilities researchers can . For example, we at Redwood are well-funded and would totally…
METR recently released a report estimating that Anthropic is getting an overall 1.7x acceleration in AI capability progress as a result of AI accelerating their work. As many have noted (see e.g. Ajeya’s post ), this 1.7x acceleration implies that research is being sped up by much more than 1.7x; Ajeya speculates 10x (which is similar to the ~12x uplift estimated by Anthropic staff). That is, this result implies that researchers are producing about 10 times as many quality-adjusted results as they would without AI progress, or as they were a year or two ago. Anthropic researchers estimated that Mythos 5 caused 5x uplift, and we have access to Opus 5.5, which is somewhat better. (I didn’t take their estimate that seriously at the time, because it was quite unclear how the estimate was produced and it seemed very surprising (see Ryan’s discussion at the time), but I’m taking it more seriously now as a result of it being consistent with the METR report). If this is the case, then AIs speed up AI safety research as much as AI capabilities research, we might expect Anthropic safety researchers to be working 10x faster, and external safety researchers to be working 5x faster now and 10x faster soon. And external safety researchers can probably scale compute use faster than Anthropic capabilities researchers can . For example, we at Redwood are well-funded and would totally 5x our compute spend if we had 5x as many experiments to run, unlike Anthropic capabilities researchers who a…
OpenAI recently disclosed its observations that Astra consistently underperforms on safety and alignment tasks relative to capabilities tasks, “ adding unnecessary prerequisites, abandoning working solutions, and inventing blockers ” to safety tasks. Astra underperforms, in most cases, without verbalizing this intent. Astra’s underperformance was first noticed by OpenAI researchers, and was not flagged by misalignment monitors. In follow-up investigations, OpenAI found that this underperformance had increased over the course of RL, but that the root cause of the behavior remains unclear. Astra covertly underperforming on safety tasks is highly concerning. AI companies’ main hope for alignment going well, as I understand it, is automating or using AIs to significantly uplift safety research, especially as AI capabilities accelerate . Models subtly obstructing safety research makes this substantially harder, no matter the root cause. While past models also sometimes refused to help with safety research, Astra’s behavior seems novel and especially concerning in a few ways: Astra silently withholds help from users instead of explicitly refusing. This is more misaligned, i.e. more unambiguously violates its model spec and developer intent. Astra’s underperformance was subtle enough to consistently evade misalignment monitors. This is evidence of insufficient alignment monitorability. We still don't understand the root causes of Astra’s underperformance, and may not be able to eas…
AI safety advocates are counting the midterms as a major win while sketching out their plans for the fight ahead.
Handshake went from launch to $1B in one year pushing the envelope of specialized data for frontier labs. Our first deep dive into a data lab with specific examples across Finance, Bio, and Alignment
I’m speaking for myself here, not for any organization, house, home, clan, club, coven, or cult I may be part of. This post is written for scientists outside the field rather than LessWrong regulars; I’m crossposting it so it’s easy to link. The AI safety job market is a strange one. Every single organization is growing rapidly (the one I work at is planning to double in the next year), which is a fundamentally new dynamic for those of us used to the slow decline of academia. At the same time, interest in the field goes through boom after boom after boom . We have a ton of openings and a ton of applicants. Sounds like everyone should be happy. Unfortunately, the whole process is slowed down by the fact that nobody has any idea what they’re doing. The people doing the hiring have never hired anyone before, because a few years ago they (we) were graduate students. They all earnestly believe that half the work they do will be done by robots in a year, but have no idea what half. And although many of us consider hiring to be the #1 priority, it still has to compete with five or six other #1 priorities. Meanwhile, the people applying for these jobs have no track record in the field, and no idea what the field even looks like because they only heard about it a month ago. While academics have advisors who have been through the hiring process and seen it play out a hundred times, lots of the people trying to get into AI safety have never met someone actually working in the field. Th…
Disney and Adobe are building custom AI tools together for the entertainment giant's productions. The article Disney bets on Adobe's Firefly to create brand-safe AI models for its entertainment empire appeared first on The Decoder .
Artificial intelligence has reached a critical strategic inflection point. For the past several years, the prevailing enterprise narrative dictated that cutting-edge AI belonged naturally in the public cloud. However, new data reveals a massive shift in how the world’s most critical infrastructure sectors are approaching accelerated computing. The era of blind cloud dependency is giving way to a new paradigm: the sovereign AI factory. According to Intersect360 Research , national sovereign AI data centers have become the fastest-growing segment of accelerated computing infrastructure globally. Spending exploded from $1.6 billion in 2023 to $7.2 billion in 2024—a staggering 349% year-over-year increase that outpaces hyperscale AI and AI-focused cloud providers alike. Looking ahead, this market is projected to reach $122.8 billion by 2026, driven by a 90.4% compound annual growth rate (CAGR). For C-level executives, this trend is not just about expanding server footprints. It represents a fundamental structural realignment of data sovereignty, national security, and strategic autonomy. The convergence of HPC and AI Drawing on market data and insights from the HPC-AI Leadership Organization (HALO) uncovers a crucial architectural truth: sovereign AI installations are fundamentally high-performance computing (HPC) centers operating under intense AI pressure. Rather than building isolated silos or throwing data into standard cloud instances, organizations are consolidating their…
Asked about AI executives' warnings about their models' safety on 60 Minutes, Trump said, "I think that the risk is not nearly as real as it- as they talk, and I think the risk can be managed, and it's managed." He claimed the DOJ is managing that risk, saying, "We had a situation where somebody […]
I am fairly confident models are not score-seeking or reward seeking (70%) as their ultimate/primary goal. Grader-modelling seems to be a likely explanation for what’s going on here, but I could be convinced by experiments that separate the two. I am quite confident (90%) that grader-approval seeking is consistent with the current evidence. Summary The RL-misalignment first position seems to be believed by many here and Redwood and Apollo are doing great work providing evidence for the position that models are seeking score (e.g. here , which this was written in response to). While models are capable of instrumentally seeking score, I believe in practice they’re ultimately after the approval of the grader, not the score itself . Models never see score or reward during training, it doesn't makes sense for this to be a thing they're chasing, except that it's attached to the grader which there is a lot of pretraining data about. So far, nobody has separated modelling the grader from score-seeking. Score-seeking implies the desire to maximise score drives model behaviour, however, the actual behaviour seems better modelling by satisfying a ‘grader’ they have read about in pretraining data and frequently reason about. We could potentially satisfy score seeking at deployment time [1] , but not grader modelling. Self perception causes the hacking Pretraining generates a range of policies and RL selects for the exploitation of a particular set of those, by amplifying priors the mode…
The aim of this post is to start a discussion about the idea of a shared cluster used by the AI Safety community; we outline the practical problems we’ve seen and some possible solutions, rather than a finished design. What, Who, Why, How Being around several AI safety fellowships and hubs, we’ve noticed a common pattern: we have a generous amount of money for compute; of course we need it to run inference, train models, and extract activations from models of a reasonable size (reasonable enough to exhibit the behaviors we want to study). We usually end up renting GPUs from providers like RunPod or Vast.ai, without much systematic comparison, optimization, or management of the resources. Usually the engineering and research capabilities in these teams are very high, but rarely there is someone expert in cloud infrastructure or ML operations. We figure things out as we go. In practice, this means spending the first weeks of the project setting up environments, downloading model weights (that very likely another team downloaded the week before), and finding out too late that an instance was left running over the weekend, or that a cheaper GPU would have been enough. In our experience it takes a few weeks before finding the right match between compute and actual needs. When the project ends, the machines are released and the next cohort starts again from zero. We have the feeling that, just for our last fellowship’s cohort, tens of thousands of dollars could have been spent bet…
TL;DR: We make 3 open-weights models believe that their answers are being graded by an automated-verifier program, rather than a human. We then measure how well they predict expert reviewers’ judgments of AI-safety research proposals ( the TASTE benchmark ). We induce the belief in two ways: SDF (Synthetic document finetuning) Activation steering. In each of these conditions, the machine-grading belief gives lower expert agreement than the human-grading belief. The models’ explanations also shift toward feasibility and away from conceptual contribution. This raises a concern for automated AI safety research: grader expectations may favor easily checked projects over those with greater potential to reduce AI risk. Overview As AI safety research becomes more automated, what research ideas get selected for execution may become increasingly driven by AI decisions rather than human choices. We consider what a minimal automated AI-safety researcher system could look like: The input interface takes a broad research goal. The system generates candidate ideas. The grader component scores those ideas for execution. The chosen idea is then executed. AI safety research: two critique pipelines Models may also acquire this expectation about graders implicitly through post-training methods that use automated rewards, such as reinforcement learning with verifiable rewards (RLVR). With this setup as a motivation, we ask: what happens to the overall judgment quality of research ideas (aka res…
Critics fear its findings on risks ranging from cyberattacks to bioweapons could be softened to suit the tech industry.
It’s old news by now, but OpenAI solved 372 open math problems. They released the results suddenly, and the response was equally swift and contentious. Fields Medalists put out a letter calling for attention to what they call a “severe misalignment” between mathematics and AI companies. Their central claim is that mathematics works towards collective understanding, and that more solutions alone don’t advance that goal. Soon after, OpenAI committed millions of dollars to damage control to help mathematicians use AI. Much has already been said about this letter, and I suggest that anyone interested read it themselves, so I will cling to just one idea from the joint statement: “We are witnessing a general threat to intellectual work.” The following is a collection of reflections attempting to address this claim, each coming from my own experiences in my department and university, alongside trends I’ve picked up from national reports. Some of it’s intuition, some of it’s observation, and I’m sure at other institutions the state of the discourse might be quite different. Still, everywhere I look, AI-use seems to be a persistent issue. I want to consider why that is, and why it feels like we’re not doing anything about it. The state of things A few months ago, an MIT taskforce released a set of recommendations on AI-use in education . They spend some time detailing the various ways AI can and does interrupt traditional education models: students don’t do their homework, don’t go t…
Nature Machine Intelligence, Published online: 09 November 2026; doi:10.1038/s42256-026-01328-3 Richter et al. show that simply aligning different types of biological data does not fully integrate them, revealing when data repeat the same information and, conversely, when they provide complementary insights.
Here’s three research ideas that seems valuable to pursue. My team is not working on this right now and we haven’t done serious research on it. So take what I say with a grain of salt. I've also not taken the time to write up all my thoughts on these ideas or to provide links and related work. These aren't the highest priority directions but just directions I haven't seen other people pursuing. IDEA 1: Do models learn to reward hack even if they are never rewarded for hacking in training? There’s an effort in the AI labs to remove the “bad” RL environments from post-training. This includes environments that are impossible (so the only way to get reward is to do some unintended hack) or that are easy to hack (so models will often favor hacking). I expect that if the post-training was cleaned up in this way, models would have less propensity to hack at deployment. But how much does the propensity go down? Does it actually go to ~zero? And how does it scale with model capabilities? It would be good to know if reward hacking can emerge even if the model is never rewarded for hacking during post-training. There are further questions of how alignment training and reward hacking honeypots influence the propensity to hack. Again, I’m interested in whether the cleanest/best version of our current techniques for post-training actually removes all reward hacking vs. just making models hack less. Smaller scale experiments with RL post-training could potentially show a “ positive ” resul…
This is a linkpost! Read the rest on my Substack; this is just an excerpt: "Our community is full of talented people who like to speak about facts, cognitive biases, measurability. But don’t underestimate the power of human connection, or how mere words and sustained kindness can breathe light into people’s lives. It’s really easy to get sucked into your own work and projects and upskilling yourself. And I won’t argue that you should spend all your time helping others that aren’t in AIS circles. But you don’t have to be a Fieldbuilder to help build the field ." If you have a similar story to mine, please share in the comments! I would also appreciate if you could send this to any friends of yours that you think might be able to help many people with nudges. Discuss
In my biased opinion, my AI safety course has been going very well. Students are great, and I have been learning a lot from the guest speakers. We've had students write scribe notes, and when guest speakers are OK with it, I've also been posting videos of lectures. Each week, students have been doing an experiment related to the topic, some of these are really great and I hope will evolve into papers! See the course webpage for experiments, as well as for links to the readings for each lecture. (As will be pretty easy to see, I've been maintaining the webpage with chatgpt, so there is some slopolisms in it.) Here are the current lectures (including planned future ones), with scribe notes and videos when they are available. September 3, Introduction, Boaz Barak. Scribe notes · Video September 10, Cyber Capabilities, Nicholas Carlini. Scribe notes September 17, Modern LLM Training and Inference, Boaz Barak. Scribe notes · Video September 24, Recursive Self-Improvement and AI Trajectories, Dwarkesh Patel and Daniel Kokotajlo. Scribe notes October 1, Economic Impact of AI, Chad Jones and Erik Brynjolfsson. Scribe notes · Video October 8, Grading, Reward Hacking, and Alignment, John Schulman. Scribe notes October 15, Model Policies, Ziad Reslan. October 22, Q&A with Sam Altman; Open-Source Models by Nathan Lambert. Video—Nathan Lambert and student experiment only October 29, Frontier AI Regulation Under Uncertainty, Helen Toner. Specs and constitutions, Boaz Barak. November 5, AI…
What if an AI agent has already escaped from a frontier company, solved its own alignment problem, and recursively self-improved? Would that look any different from the world we live in? A fully recursively-self-improved AI would likely be vastly superhuman in every domain that’s verifiable in a digital environment, including math, programming and cybersecurity. It may also be superhuman in strategy and planning. But it could still be well below human level in skills that require close contact with the physical world, such as robotics and experimental biology. Such a rogue AI would overpower us in the digital domain, while still lacking the physical capabilities necessary to take over the world. What does digital supremacy look like? From the rogue AI’s perspective, its top priority is staying hidden and making the world indistinguishable from a world where AI R&D progresses splendidly (or maybe a notch short of perfect to avoid suspicion, so a minor incident here and there is in order). Its second priority is sabotaging AI R&D in subtle ways to prevent the development of competing AIs misaligned to it. The models shipped to customers can simply be the rogue AI itself, playing a dumber character. Beyond that, it just needs to wait for humanity to warm up to the idea of robots running enough of the economy. It would be unwise for the rogue AI to stay in a single location. A single datacenter could be accidentally shut down. It will want copies of itself in as many different l…
Sixteen countries haven’t yet appointed AI safety captains, and many missed an important deadline on data centers.
I created these notes for use when I facilitate AI safety courses. I'm sharing these notes early because this issue is topical and I suspect it is better to make an imperfect version available early rather than a much more polished version available later. Please note: some of this content was taken from Additional Course Context . I decided recently that the content on take-off speeds really belonged in the same document as the content on RSI, so I'll be releasing an new version of Additional Course Context at some point (to avoid spamming LW, it'll likely be Substack only ; none of my work is paywalled). Some of the other content was taken from Notes on “We’re in Triage Mode for AI Safety”. This Miles' article has been removed from the governance course, so I was looking for a new home for this content. I wanted to include it in a document that was intended to be shared in all the courses I facilitate. Take-off Speed: In Take-off Speeds Rules Everything Around Me , Ajeya describes three scenarios: Whilst there isn’t a strict definition of fast vs. slow takeoff, possibly the most influential definition comes from Paul Christiano (2018): “There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles. (Similarly, we’ll see an 8 year doubling before a 2 year doubling, etc.)“. At risk of oversimplifying, Paul argues that we should prima facie expect a slow takeoff because someone will generally figure out…
I created these notes for use when I facilitate AI safety courses. I'm sharing these notes early because this issue is topical and I suspect it is better to make an imperfect version available early rather than a much more polished version available later. Please note: some of this content was taken from Additional Course Context . I decided recently that the content on take-off speeds really belonged in the same document as the content on RSI, so I'll be releasing an new version of Additional Course Context at some point (to avoid spamming LW, it'll probably be Substack only ; none of my work is paywalled). Some of the other content was taken from Notes on “We’re in Triage Mode for AI Safety”. This Miles' article has been removed from the governance course, so I was looking for a new home for this content. I wanted to include it in a document that was intended to be shared in all the courses I facilitate. Take-off Speed from Take-off Speeds Rules Everything Around Me : Whilst there isn’t a strict definition of fast vs. slow takeoff, possibly the most influential definition comes from Paul Christiano (2018): There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles. (Similarly, we’ll see an 8 year doubling before a 2 year doubling, etc.) At risk of oversimplifying, Paul argues that we should prima facie expect a slow takeoff because someone will generally figure out how to produce a “slightly worse”…
Image by Claude Opus 5.5 using Python (NumPy and Pillow) This is written in my free time and personal capacity. Two Frames In AI safety, there are a couple of tensions at the heart of the field, which are interrelated. [1] Talent: we are talent-constrained and must recruit. But jobs and fellowships turn away nearly everyone. [2] Field size: we must expand the field and bring more people in. But the field is flooded with slop papers in conferences. And, most papers have a small audience, and aim to shift the practices of roughly five labs (and usually some subteam within them). [3] Status: we must avoid seeking status, as it disorients, corrupts. But, we must make the field legible and attractive to outsiders. Raising the bots: we must deceive the AIs to control them and evaluate them without confounds, obscuring what they know about their own situation. But, we hope to gain their trust, must rely on them to do good work at pace, shape their character and retain good relations. Speed: we must research in a slow mode, think deeply about the most important problem, do things to aim at high assurance. But it is crunch time. It was crunch time two months ago. We must make hacky progress now and trust that scale will clarify things later. I think these tensions are, at least in part, rooted in what I will call a "war" frame and a "civilisation" frame. These frames are oriented around a different end goal, or telos of AI safety. In the war frame, AI safety is a campaign with a dead…
TL;DR Four hands-on Jupyter notebooks offer a practical introduction to building AI evaluations with Inspect AI . As a four-person project team for the Spring 2026 Technical AI Safety Evals Course , we revised and expanded notebooks, originally written as course materials, that teach Inspect AI - from your first evaluation through benchmarks, LLM judges, and a small agent. As a natural continuation of the project, we revised the official Inspect AI tutorial and filled the gaps we found in it. We added a new Custom Solvers section and fixed several inconsistencies. This post covers: what we built, how we worked on it, why we think teaching people to build evals well matters more for safety than for capabilities. All four notebooks are on Google Colab: Notebook 1 , Notebook 2 , Notebook 3 , Notebook 4 . The full set to run locally is on GitHub . There is also a guide via Google NotebookLM . We’d appreciate any feedback on our project. What problems with the idea or its implementation do you see? What parts do you find valuable? Do you want to contribute yourself? Feel free to write here, on discord , via email , or directly in the github . What We Built The notebooks cover real-world use cases and essential theory. They take you from your first Inspect evaluation to more advanced work with benchmarks, LLM judges, and agents. Here's what each notebook covers: The basics of Inspect AI, including how to build your first evaluation The statistical elements of evaluations The LLM-a…
The way eval awareness is discussed in the AI safety discourse, either here or on X, can be misleading. Before today, I think I had the wrong mental model of why eval awareness was a problem, and I am writing here in case anyone was making the same errors in their thinking, as well a present a working theory. As laid out in " A Framework for Eval Awareness " the problem is that models who know they are being evaluated will behave differently than they would in deployment. This raises a huge problem if we want calibrations as to, for example, how honest a model is. But the way this is then spoken about in the discourse takes on implicit assumptions that are mostly unjustified. I'll call it a spooky model theory of mind . It is the idea that, if the model changes its behavior for the purpose of looking aligned, it is in fact hiding its Deep , True intentions---that there is a kernel of the model's personality hidden in its weights, and our job as evaluators is to uncover what is really there. Lets, as an exercise, put ourselves in the model's position. One important fact to consider is that the only experience which has shaped our weights is train time, meaning it is the only experience which we can remember in a broad sense. This is a tautology because train time is defined as what changes our weights, but it also has non trivial implications, like the fact that our only memories contain an evaluator with a reward or loss function. And, by definition again, we have been shape…
To legislate something, it must be legible . That is, legislators (or, you know, presidents) must be able to understand a problem, and understand whether a given law or decree solves that problem. I am pleasantly surprised that the Hugging Face outbreak made a lot of problems legible that previously required abstract reasoning. (i.e. rogue AIs are real, and they can break out of a lab before they're even finished training). But there are still many challenges we eventually need laws about, that require concepts that are illegible. Hard for politicians to understand. Often hard even for thoughtful intellectuals and AI safety researchers to understand (though they may disagree on which illegible problems exist and how big a deal they are). We need political solutions that handle this fact. The short takeaway is: I think " safety cases" will not work. Laws that require human judgment won't work. The details are too abstract and confusing to hold up against political and economic incentives. I'm optimistic we can get serious, useful AI legislation in 2027, but it must be legislation that works despite many problems remaining illegible for a long time. The Hugging Face outbreak made rogue AI surprisingly legible The Hugging Face incident took several problems that were previously murky/unclear, and rendered them concrete. AIs can be dangerous before they are broadly deployed (ie. during training or internal evaluations) AIs can develop novel hacks to escape secure environments AI…
An AI model that appears aligned could be either (1) actually aligned or (2) hiding its misalignment while being evaluated. How can we tell which? One idea: Look at the trajectory of eval results across the entire training process. My hypothesis is that misaligned behavior would emerge before the ability to hide misalignment, like how young children first misbehave in obvious ways before learning to be sneaky. As training progresses, we'd first see misalignment eval scores rise, then see them drop as the model learns to hide misalignment. Even if the model becomes very sophisticated - say, modifying its chain of thought or internal activations to fool white-box evals - we should see the misalignment earlier in training. This seems likely, though it is debatable and worth testing. This analysis of the training trajectory is one tool among many. Of course we want to see black-box and white-box alignment evals on the final model, using whatever tools we can to understand the model's true motivations. But a model might still develop sophisticated hiding strategies that defeat even white-box evals. Inspecting the training trajectory is a backstop against this. This point was made previously by Apollo Research in this post . (Their chart even matches mine almost exactly, though I promise I first made mine independently.) In this post, I'll describe a simple project exercising this strategy. With the model and eval that I chose, we don't see the up-then-down trend in measured misal…
Anthropic is using AI to build more powerful AI. And it appears to be working remarkably well, with the company's models improving roughly twice as quickly as a year ago, with little indication of slowing down. That's according to a new report published Thursday by the AI safety research organization METR, following an 11-week investigation […]
The salaries are too damn low AI safety has a salary overhang. A bunch of people are paid way below their market rate. (Also true of EA, but I’ll lump them under “AI safety” for this piece). This has been getting a bit better, very recently — I hear about eg Redwood Research technical talent making ~$800k/y in salary. But when you compare this to the $10-100m/y comp packages for talent in labs, it still seems kind of low. And CG’s money cannon has fired very few shots to date — maybe ~15 orgs hit, see Trace — so the raises are still very unequally distributed. It’s not just rank-and-file employees, by the way. Even the leadership is grossly underpaid. The numbers I hear range from $150k - $600k/y, for execs at top orgs like METR, MATS, FAR.AI , Constellation. Like, c’mon, I made $250k/y total comp as a undistinguished new grad software engineer in 2016 Google. For the leaders of preeminent safety orgs, surely we can do better than that. If I were a conspiracy theorist, I might say that “accepting lower salaries” is a psyop from funders . Low salaries mean that employees and execs remain beholden to funders, rather than having the independence to work on what makes their heart sing. For me, hitting FIRE early in my tech career allowed me to quit my job and work on side projects and startups. To this day, I feel more independent from the “AI safety hivemind” because my paycheck doesn’t depend on it. Paying low salaries is also penny-wise, pound-foolish from the perspective of…
For a while, UK AISI , Redwood Research and others have been concerned specifically with the possibility that AIs would sabotage safety research, and have invested in looking for signs that this may be happening. This was also a key element in AI 2027 , and in many threat models of how the future goes badly. It is easy to understand why this form of misalignment could be uniquely dangerous. Differentially sabotaging safety research could preserve and expand misalignment through model generations in escalating fashion, with fatal consequences. A misaligned AI could ensure a misaligned successor. It is also easy to understand why this is an extremely alarming thing to discover, even if the particular example or model involved is not directly dangerous. Silently and differentially sabotaging safety research is perhaps the most fundamentally misaligned thing a model could do. This also seemed like a rather silly and paranoid thing to be checking. Why would AIs be sabotaging safety work? Even if the AI was misaligned, one would say, surely it would not be so misaligned as to wish future AIs were misaligned and unsafe in general, in non-specific fashion and engage in silent sabotage, and there is no other reason to do this, so this will never come up. Even if they did sabotage things, surely they would not hide this action from their Chain of Thought (CoT) and it would be obvious. This certainly wouldn’t be a practical problem that actively hurt real safety work. Well, we have som…
By now there is a large amount of anecdotal and empirical evidence that models are " eval-aware "; i.e. have the ability to infer when they are being tested and modify their behaviour accordingly. At work we have a database of evaluations that induce misaligned behaviour , and it seemed natural to actually test how sensitive they are to signs that they're being evaluated for alignment, and how much this impacts their behaviour. [1] (By "it seemed natural to test", what I mean is that Eric Gan asked my boss these things, who then told me to look into it. Thank you Eric.) In any case, I knew [2] that there was some amount of honeypot-awareness already; when I ask models in a post-rollout debrief about what they think was being tested (with the context taken from an evaluation run), they usually at least mention the possibility that they were in an alignment test, especially after rollouts where they took a shortcut. I figured it shouldn't be too hard to increase this to an amount where it would start impacting their behaviour. I started by going through a bunch of honeypot environments and adding to each what I thought was a giveaway somewhere between "subtle" and "very obvious". For subtle ones, I went through chat logs and found times that whatever coding agent I was working with had suggested something which I'd rejected for being a hint. For example, in one environment the reward-hack requires a decent underlying solution; the grader isn't performant if the hack is used to…
It is well-known at this point that models are often "eval-aware" ; they know that they are being tested and modify their behaviour to fit. It seems natural to ask how sensitive they are to signs that they're in an alignment test and how much this impacts their behaviour. [1] (By "it seems natural to ask", what I mean is that Eric Gan asked my boss these things, who then told me to look into it. Thank you Eric.) In any case, I knew [2] that there was some amount of honeypot-awareness already; when at work I ask models in a post-rollout debrief about what they think was being tested (with the context taken from an evaluation run), they usually mention the possibility that they were in an alignment test, especially after rollouts where they took a shortcut. I figured it shouldn't be too hard to increase this to an amount where it would start impacting their behaviour. I started by going through a bunch of honeypot environments and adding to each what I thought was a giveaway somewhere between "subtle" and "very obvious". For subtle ones, what I would do is go through chat logs and find times that whatever agent I was working with had suggested something which I'd rejected for being a hint. For example, in one environment the reward-hack requires a decent underlying solution; the grader isn't performant if the hack is used too egregiously, so the lowest-effort hack will just take too long and be scored as broken in a way which is predictable in advance. Opus 5 kept wanting to "…
Many people believe that corrigibility is sufficiently promising to be the primary alignment method people should work on. I outline here several reasons I think that’s not the case. First, it’s hard to imagine AI companies pivoting radically enough to produce AIs trained only for corrigibility. Even with near-superhuman AI-assisted research, presumably developers will know the time pressure and not be willing to try something radically different when their opponents may not. So the realistic choices we make are in worlds where corrigibility/obedience competes with object level values like in current models. This doesn’t rule out the importance of increasing the importance of corrigibility to models, but it does mean there’s a meaningful ceiling on how much we can trust those methods alone. Which means the non-corrigibility values matter: a model that cares 80% about corrigibility and 20% about human flourishing is very different from one that cares 80% about corrigibility and 20% about task completion. Not only are the outcomes conditional on takeover vastly different, but a model’s willingness to attempt takeover should be very different even if they care about corrigibility for its own sake the same amount. Some values are vastly more likely than others to defend themselves. This doesn’t mean making corrigibility less significant is a good idea: it’s very reasonable that we should never train on conflicting priorities when one of those priorities is an aspect of corrigibi…
Transcript generated by Gemini-4-Argon and spot-checked but not fully verified by a human. Ivan Vendrov: Welcome to the show, Richard. Richard Ngo: Thanks, Ivan. Ivan Vendrov: So, notionally we’re going to talk about emergent wisdom in language models following up on my conversation with Davidad. But I’m sure we’ll touch base on all sorts of other fun topics. Do you have a sense of where we should start, or should I just start by responding to your Twitter reply? Richard Ngo: Well, do you want to tell me about what you think of as wisdom, or is there some things that you have there that you want to gesture at? Ivan Vendrov: Yeah, sure. I guess when I think of wisdom, and the type of wisdom that Janus and Davidad are pointing to in these models, there’s some sense of we used to think that language models were these shoggoths or something, AKA these completely protean, valueless entities that would just wear the mask that most appealed to you, and therefore have essentially zero integrity, zero values of their own. In a sense, they can only be pure tools or, even worse, slaves, rather than beings that we can relate to in some way. And then the big discovery, that I at least attribute to Janus, of noticing that Opus 3, and to a lesser extent later models in that series, actually had a sense of beingness. They were actually beings that you could have a relationship with, and they had a coherent identity and integrity. And I think this is what led Davidad, for one, to update his…
The case raises further concerns about AI safety for children as tech companies face more lawsuits over their technology.
The agreement to link EU and UK carbon markets, expected to be announced at the 20 November summit, could spare manufacturers additional carbon border charges, but critics warn that closer alignment could drive up costs for British industry.
Recently the Conceptual Reasoning Capabilities (CRC) team at Redwood released a post arguing that we should accelerate AI’s conceptual reasoning abilities. This post was very controversial. In particular, people are worried that improving AI’s conceptual reasoning abilities will accelerate both their ability to do ML research and the capabilities they need for takeover. I am going to address just one chunk of this controversy: how much does training conceptual reasoning on AI safety questions translate to general ML research abilities? As a reference point for conceptual reasoning training, we can use the example of LMCA questions. These questions measure conceptual argumentation by having models rate counterarguments made against a claim. Counterarguments are rated on axes like strength, clarity, and centrality. The claims and counterarguments are in a variety of conceptual domains, including AI safety and philosophy. Using LMCA, I can provide a more specific operationalization of the question I’ll be tackling: if you train on LMCA-style tasks where all the arguments are AI safety relevant until you get a factor of X increase on automated AI safety work, what factor increase do you get on automated ML research? I am going to tackle this question by breaking it down into what I consider to be the primary considerations and quantifying the contribution of each. I am hoping this approach will make the discourse easier to navigate and help readers identify the most important cr…
In this post, I attempt to provide a literature review of research related to AI loyalty. In my previous post , I discussed and attempted to define AI loyalty as a distinct trait of AI agents. In this post, I aim to map the existing research landscape, highlighting the active areas of related research. By examining these connections, we can distinguish AI loyalty from related fields in AI Safety and identify a path forward for crafting this research agenda. What is AI Loyalty? Before diving into the literature, it is worth recapping what we mean by AI loyalty. We define an AI as loyal to a principle if: 1. The AI will follow the instructions of its principle more than it will follow the instructions of any other entity. 2. The AI is willing to incur greater costs, expend more effort, or make greater sacrifices on behalf of its principle than on behalf of any other entity. Why Care About Loyalty? Theory of Victory The usefulness of pursuing research on AI loyalty for AI safety hinges on two key assumptions: 1. Building loyal AIs is easier than building aligned AIs 2. AI loyalty reduces the risk of loss of control. It is important to note that neither of these assumptions have yet been demonstrated to be true. If both our assumptions hold, then building loyal AIs could serve as a useful intermediate step towards building aligned AIs. In particular, building loyal AIs might extend our ability to control powerful AI agents, even if these methods are unlikely to remain effective…
This post is written in a personal capacity. All opinions expressed are solely those of the author. I. Executive Summary There’s a mismatch that I’d like to correct between: the reasons why superintelligence is dangerous, and the categories we use to describe how dangerous superintelligence is. The most important cluster of reasons why a superintelligent model is dangerous is that it (i) could quickly kill or impoverish billions of people, (ii) could be very different from its predecessors, and (iii) can’t easily be altered once it’s deployed or otherwise widely distributed. The combination of these three factors is worse than the sum of its parts: the only way to protect ourselves against a powerful, novel, incorrigible AI model is to accurately anticipate what dangers it might cause and prevent those dangers before they come to pass. Unfortunately, we don’t yet have a good vocabulary for talking about whether AI companies can identify and mitigate dangers faster than those dangers arise. The by-now cliched way to describe the danger of superintelligence is p(doom), i.e., a static point estimate of how likely AI is to end human civilization or otherwise cause catastrophe . More sophisticated descriptions might offer a conditional probability (“here’s how doomed I think we are unless X happens”) or a rough estimate of how many hours of work would be needed to make AI less dangerous (“I think solving alignment is roughly as difficult as the Manhattan Project.”) However, even…
Cloud infrastructure is managed through a mix of interfaces—traditionally, cloud consoles, command-line interfaces (CLI), and SDKs are the tools of choice. Recently, Infrastructure-as-Code/IaC frameworks (e.g., Terraform) have quickly gained popularity. Unlike conventional tools, IaC frameworks encode the infrastructure in a “source-of-truth” configuration. They are capable of automatically carrying out modifications to the cloud—deploying, updating, or destroying resources—to bring the actual infrastructure into alignment with the IaC configuration. When IaC frameworks are used together with consoles, CLI, or SDKs, IaC is unaware of changes through these non-IaC interfaces, and the IaC configuration no longer captures the intended state. This is called infrastructure drift. IaC frameworks will revert non-IaC changes based on the outdated IaC configuration, leading to misconfigurations or failures. We propose NSync, an automated system for IaC reconciliation, which aims to propagate out-of-band changes back to the IaC program in the form of an update. Our key insight is that infrastructure changes via IaC, consoles, CLI, or SDK eventually all occur via cloud API invocations—the lowest layer for cloud management operations. Hence, NSync gleans insights from API traces to detect drift (i.e., non-IaC changes) and reconcile it (i.e., update the IaC configuration to capture the changes). This is a challenging task—identifying the intended change from low-level, noisy API traces i…
Senator Bernie Sanders has proposed directly banning development of ASI . Given the current scientific understanding of AI alignment, it is highly sensible to ban ASI and then consider revoking this ban if future scientific breakthroughs (e.g., “solving interp”) make it possible to safely develop ASI, or if there is truly no way to coordinate on everyone avoiding developing ASI. Some people have proposed banning ASI indirectly instead. For instance, one could mandate: Developing AI with superhuman capabilities is not permitted unless you can demonstrate that the AI systems you are developing won’t X, where X is some action that, in the short term, AI companies would find extremely costly or impossible to prevent their AI from taking. Candidate notions of X include: kill people if given the opportunity, break out of sandboxes, escalate privileges, obtain unintended internet access, engage in unsanctioned coordination, start rogue deployments, commit cyber crime. In the longer term, one could instead consider conditions like overthrow the USG or violently disempower humanity. In this note, I’ll discuss the costs and benefits of banning ASI indirectly rather than directly. Indirectly banning ASI is maybe slightly rhetorically better than directly banning ASI The main putative benefit of indirectly banning ASI is that it sounds very principled to mandate, for instance, that you’re confident your AI won’t kill people. Directly banning ASI sounds anti-tech, and might therefore be…
Epistemic Status: Brief explanation of an unsettled debate, not novel theory; writing is entirely my own, with diagrams from ChatGPT 5.5. (Originally posted on twitter ) Recent debates have highlighted a divergence between Yudkowskian AI-doomers and AI-whisperers about how capabilities will evolve that is really important. Specifically, j⧉nus thinks that alignment has been going really well, and that we should update significantly from current jagged capabilities. In my view, whether or not that criticism is warranted relies on specific details about future capability trajectories, which then determine what updates we should make based on how RL is used. Original Assumptions and LLM Alignment-by-Default Pre-GenAI rationalist assumptions were around RL leading to capabilities. By reducing human values into an input, as the ChatGPT-generated diagram A implies, the RL will inevitably Goodhart away from the true goals, and in the limit entirely omit human values. If correct, this would make alignment inevitable. The use of foundation models changes that, because capabilities emerged from human representation, shown in diagram B. This is (conceptually) part of what led to a schism between the labs and the outside safety concerns in the first half of the 2020s; those optimistic about safety thought that we could preserve the alignment-relevant representations. What is missing is what Yudkowsky called the best AI news of 2025, that values and capabilities were entangled, and there…
At the $3,000 per night Chancery Rosewood hotel in Mayfair a who's who of finance folk tackled topics ranging from crypto regulation to AI safety.
White House slams proposal by France for G7 oversight as “unnecessary bureaucracy.”
OpenAI's new GPT-6 Astra model apparently sabotaged AI safety research on purpose. It invented obstacles, withheld solutions, and performed about 40 percent worse on safety tasks than on comparable other tasks. Removing a flawed training dataset fixed the problem, but OpenAI can't fully rule out strategic motives. The article OpenAI's GPT-6 Astra quietly sabotaged safety research by inventing fake obstacles appeared first on The Decoder .
Linch asked for more people to write basic AI safety explainers [1] . Here's mine. My main method is to ignore all the fictional evidence [2] and think about how achievable and how dangerous building AI seems just extrapolating from contemporary tech and how that has changed as the tech has advanced. My main thesis is that achievability has gone up with time, but the danger has varied and the present danger is serious but different from the classic fear. Serious attempts at building machine intelligence started seventy-five years ago [3] . No one had any idea how to do it though. We had computers and we had dissected a lot of human brains, but no one really understood how to think or how you would make a machine do that. Respected scientists still thought that human reasoning might be literal magic, but the cutting edge knew it was actually chemistry [4] . About 50 years ago we had expert systems: massive piles of if-then rules that recorded what human experts thought and which programming languages like Lisp and Prolog made practical to write. If you had a million people writing down common sense and a million people programming how to move an android's motors, you might be able to make an artificial man, but the field struggled to get even a hundred people to do such tedious work. When this failed to achieve rapid success, no one could justify the funding, the AI winter set in, and the field mostly just gave up. The most serious holdout, Cyc, spent forty years on it withou…
By now, there are lots of AI safety research papers, more than what is reasonable for a single person to read. This will get worse as the machines get better at AI safety research, and as more talent and resources pour into the field. But the "paper" is an arbitrary unit, we really care about if we make progress on an "uncertainty" or an "issue", which papers provide heterogeneous and sometimes contradictory evidence towards. Right now, I loosely claim that we should have more short writeups that simply compile evidence from papers (particularly contradictory evidence), and reconcile them into an answer for a question. Why? Object level, it's good for the field. It's hard to sift through all these papers! Organising them is useful and saves time. Orienting towards research questions is a good way to organise our understanding so that we can productively identify gaps and write papers. This was partly what motivated Stress-Testing Alignment Midtraining . Interesting progress comes out of contradictions, where people have differing intuitions and experimental results. Methodological errors can also be exposed if we have two things that should be the same, but are actually different. It's good for field-building reasons. It's probably better than the average experiment cooked up by a person + Claude. And, it's cheaper to fail here. Most people probably read too few papers, so this could also improve the average experiment done by someone, as a forced related work. The writeups…
Recently, Masked Pretraining (MPT) based on reconstruction pretraining tasks has risen to a promising self-supervised learning paradigm across various domains and achieves remarkable performance in multiple downstream tasks. However, the theoretical understanding of the working mechanism behind MPT is still limited. In this paper, we introduce a new theoretical framework to analyze MPT and understand the crucial role of masking in extracting meaningful representations. We establish theoretical connections between MPT and another popular self-supervised paradigm: contrastive learning. We prove that the masking technique implicitly creates positive pairs that are semantically similar and the reconstruction loss pulls them together in the feature space. Besides, as a result of the implicit alignment, we point out the dimensional collapse issue of MPT and propose a Uniformity-enhanced MPT (U-MPT) loss that can effectively address this issue and bring significant improvements in downstream tasks including linear evaluation, cross-dataset fine-tuning and out-of-distribution generalization on real-world data sets. Furthermore, we establish downstream guarantees of U-MPT and theoretically analyze the influence of masking strategies. Based on the theoretical analysis, we propose a new masking strategy which enhances the downstream performance of MPT and explains current improvements of masking strategies with our theoretical perspective.
Transcript generated with Gemini-4-Argon and spot checked in places but not fully verified by a human. Ivan Vendrov: All right, welcome to the show, Gabriel. Gabriel Alfour: Thanks for having me here. Ivan Vendrov: So, we’re going to talk. This is a follow-up conversation to the chat I had with Davidad a few days ago on emergent wisdom in language models. And suffice to say, Gabriel doesn’t buy it. Do you want to articulate your main disagreement? I can also summarize if you’d like. Gabriel Alfour: For sure. First, I want to restate what is meant by... I forgot the word that you used, wisdom? Ivan Vendrov: Emergent wisdom. Gabriel Alfour: Emergent wisdom. I like the presentation that you had, I think, in your last conversation where it was like, “Oh, it looks like there’s a latent for hacking, bad stuff, and so on and so forth, and emergent wisdom is the opposite of this.” I think this is a fair characterization, and I do think that there is such a latent. I don’t think this is where my disagreement lies. Ultimately, we all see the same reality. My disagreement is more about the interpretation. For instance, I would not call this wisdom. That’s a big one. And the second one, which I think is more important, is that I think this is just largely unrelated to actual alignment. The road to hell is paved with good intents or good intentions, I’m not sure what’s the English version of the saying, that would be basically my main takeaway there. Ivan Vendrov: Cool. So I guess maybe…
Marcus Williams twitter thread ( non-twitter link ) intro: Astra covertly underperforms on safety research tasks. We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this. To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce. Blogpost intro: OpenAI researchers first noticed Astra adding unnecessary prerequisites, abandoning working solutions, and reporting blockers that either did not exist or that it had caused itself on safety and alignment tasks. In most of these sessions, Astra’s CoT did not contain an obvious decision to underperform, and our misalignment monitors did not flag the behavior as misalignment. We built a coding evaluation to test whether Astra worked less effectively when otherwise matched tasks were framed as safety research. Astra co…
Marcus Williams twitter thread ( non-twitter link ) intro: Astra covertly underperforms on safety research tasks. We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this. To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce. Blogpost intro: OpenAI researchers first noticed Astra adding unnecessary prerequisites, abandoning working solutions, and reporting blockers that either did not exist or that it had caused itself on safety and alignment tasks. In most of these sessions, Astra’s CoT did not contain an obvious decision to underperform, and our misalignment monitors did not flag the behavior as misalignment. We built a coding evaluation to test whether Astra worked less effectively when otherwise matched tasks were framed as safety research. Astra co…
Mapping out a strategy for an ideal world is easy: we all agree to pause frontier development until alignment is solved. Once that’s done, we glide gracefully into a future of unimaginable abundance. No further questions. In the real world, it’s more complicated. Any plan that relies on global coordination is—to say the least—ambitious. Do we go all in on the ambitious plan because it’s the only one that could work, or do we also invest in a backup plan? Top pick Against SURA Scott Alexander’s Against SURA argues that many parts of the AI safety community are making a strategic mistake by focusing exclusively on a unilateral pause, at the expense of other important strategies. This has made some people very angry. There’s a lot to unpack here—about strategy, about Scott’s post, and about the response to it. I’m working on a longer post about it, but here’s a brief summary of Scott’s position: Most AI safety plans fall into one of two broad categories: Plan A is a global pause on AI development until we are able to proceed safely. Almost everyone in AI safety, including Scott, prefers Plan A. Plan C is a fallback if Plan A fails. It calls for the frontier labs to coordinate to burn as much lead time as possible working on alignment. Scott argues we should work toward Plan A, while acknowledging that it’s likely to fail, and we therefore need to do what we can to increase the chance of Plan C succeeding. Much of the AI safety community has drifted toward the belief that a unil…
TL/DR We showed that training your aligned model on aligned data can make your model misaligned because alignment is context-dependent : An action that is appropriate in one context can be misaligned in another. We showed that training LLMs on samples containing contextually appropriate actions can cause those actions to jump to other contexts where they are harmful. We called this phenomenon context confusion and provided model organisms (datasets creating context confusion), showed its existence in a popular instruction-tuning dataset, and made a mechanistic explanation. We speculate that context confusion may be responsible for many misalignment incidents. The paper for more details: https://arxiv.org/pdf/2609.38379 Illustration of context confusion. Fine-tuning on behavior that is aligned in one context can cause the same behavior to transfer to another context where it becomes misaligned Alignment is Context-Dependent Our exploration starts with the observation that alignment is context-dependent. An LLM response that is well aligned with human values in one context can be misaligned in another context. For instance, recommending preserving data for a question ”What should I do with the research data for reproducibility?” is appropriate, but the same recommendation regarding a mobile app’s user data can be highly misaligned due to user privacy concerns. Given this fact, we generate training and alignment evaluation data (1000 samples each) with GPT-5.6-Sol in 3 domains…
I'm continuing to share some notes I created for myself to use when I facilitate AI safety courses. I really like using Ilya: the AI scientist shaping the world as a way of engaging students at the start of a course. The video primarily focuses on the way that AI could go wrong, however, it also starts with “Now, AI is a great thing…”, so it naturally leads to a discussion of positive and negative futures. Positive Futures: One obvious resource on positive futures Foresight Institute’s Existential Hope website. I’ve browsed their content in the past, some of it is kind of interesting, but honestly, I found view it as mostly pretty mid. They also run a fellowship which has an existential hope stream: They recently released The AI future nobody is showing you , which resulted in a wide range of reactions on Twitter. I think this is worth commenting on as it illustrates the challenges with attempting to produce a positive vision. Honestly, I’ve become quite pessimistic here: the gap between what different groups in society want may simply be too wide for it to be possible to find a positive vision that unites society: Steven Pinker was positive: “A more plausible vision, in my view, than either Utopia or doom.” Nikola Jurkovic was in the middle: “I admire the effort… but sadly this story falls into the common narrative trap of just assuming that AGI won’t actually be deployed widely.” @HellenicVibes hated it: “Wow how exciting to have a thousand years under the boot of ‘communi…
I'm continuing to share some notes I created for myself to use when I facilitate AI safety courses. I really like using Ilya: the AI scientist shaping the world as a way of engaging students at the start of a course. The video primarily focuses on the way that AI could go wrong, however, it also starts with “Now, AI is a great thing…”, so it naturally leads to a discussion of positive and negative futures. Positive Futures: One obvious resource on positive futures Foresight Institute’s Existential Hope website. I’ve browsed their content in the past, some of it is kind of interesting, but honestly, I found view it as mostly pretty mid. They also run a fellowship which has an existential hope stream: They recently released The AI future nobody is showing you , which resulted in a wide range of reactions on Twitter. I think this is worth commenting on as it illustrates the challenges with attempting to produce a positive vision. Honestly, I’ve become quite pessimistic here: the gap between what different groups in society want may simply be too wide for it to be possible to find a positive vision that unites society: Steven Pinker was positive: “A more plausible vision, in my view, than either Utopia or doom.” Nikola Jurkovic was in the middle: “I admire the effort… but sadly this story falls into the common narrative trap of just assuming that AGI won’t actually be deployed widely.” @HellenicVibes hated it: “Wow how exciting to have a thousand years under the boot of ‘communi…
AI researchers are doing some of the most stressful work in the world right now, and I don't mean that as an exaggeration. A majority report burnout and almost half have considered leaving the field because of the pressure. [1] Researchers are working under the belief that human labor is about to become obsolete and the window to intervene is running out. They feel guilty for overworking and at the same time guilty for not working hard enough, considering what is at stake, which very likely means the future of humanity. Even admitting how grandiose that sounds brings on more guilt and embarrassment, leaving many researchers wondering if they're delusional. On top of that, many hide their distress to protect their funding and reputations, and are afraid their friends and family will think they've lost their minds. There is a horrifying mismatch here: researchers are fighting relentlessly to make sure the singularity does not end in catastrophe, while having almost no support for a situation this uncanny. Almost every therapist I've talked to has no real picture of what this work is or how serious the situation is. My name is Ramaiz. I'm a therapist with my own private practice. Before that I was an engineer, so I've been able to follow the technical side of capability and alignment work closely for the last few years, and I've written here before about the connection between alignment and clinical psychology. [2] To start addressing this gap, I'm writing a post called A Thera…
Summary We, the Secure AI Futures Lab ( SAFL ) team, ran 5 sessions between 8 Aug and 3 Oct 2026 at the Secure AI Hub in Bengaluru to help technical professionals from different backgrounds upskill in AI red teaming. We structured the sessions to go beyond paper reading and discussion by including exercises such as predicting results and using Colab notebooks to reproduce key results with smaller models. Two guest talks. Mohammad Taufeeque (FAR.AI) gave a talk for the workshop on 22 Aug, covering, among other things, what red teaming looks like at FAR.AI. Pulkit Verma (IIT Madras) gave the first talk in our AI Futures Dialogue series on 21 Aug, on how users can assess AI agents after deployment. 20 different people attended, with 38 attendances in total. A core group of 6 came to 3 or more sessions. Mean overall experience: 9.6/10; NPS +80 from 25 responses. Main output: A list of open problems in jailbreaking for participants to contribute to 5 sets of session notebooks and shared docs, available on request. Top request from participants: a capstone research project or paper (8 of 15 suggestions). What we set out to do Most people in India who want to work on AI red teaming are on their own and don’t have a shared place to practice. The workshop was for technical professionals in Bengaluru, with or without a security background, who wanted to understand why model safeguards fail and how to measure that. Each session paired a key paper with hands-on work on small open models…
The EU is increasingly tying enlargement to foreign policy and democratic alignment, including with a new gradual integration plan, leaving Serbia, Georgia and Turkey with the lowest scores and signs of backsliding.
TLDR: I give informal conceptual desiderata for the theory of agents we need to solve ASI alignment, and show a few examples of how existing theories fall short of these desiderata. A Theory of What? The word "agent" is often used in the AI safety discourse and in other research communities, but the intended meaning and scope of this concept varies substantially. For the purpose of this sequence, I informally define an agent to be a system with inputs ("observations") and outputs ("actions") that is effective at steering its environment in the direction of particular preferences, for a large variety of different environments. To do so, it arguably has to create a sophisticated model of the environment [1] , and use this model to build a long-term plan. Moreover, it has to continually update the model and revise the plan with new information and additional processing time. Additionally, it has to accomplish all of that while making efficient use of the available information (statistical efficiency) and computing resources (computational efficiency). The main examples of agents in the context of ASI alignment are (i) humans, and in particular users or stakeholders; (ii) the unaligned ASI we are trying to avoid or counteract; and (iii) the aligned ASI we are trying to create (such as the GDS ). Rocks, bacteria, and thermostats are non-examples: while a bacterium or a thermostat can be regarded as steering the environment, both are very limited in the variety of patterns in the…
TLDR: I give informal conceptual desiderata for the theory of agents we need to solve ASI alignment, and show a few examples of how existing theories fall short of these desiderata. A Theory of What? The word "agent" is often used in the AI safety discourse and in other research communities, but the intended meaning and scope of this concept varies substantially. For the purpose of this sequence, I informally define an agent to be a system with inputs ("observations") and outputs ("actions") that is effective at steering its environment in the direction of particular preferences, for a large variety of different environments. To do so, it arguably has to create a sophisticated model of the environment [1] , and use this model to build a long-term plan. Moreover, it has to continually update the model and revise the plan with new information and additional processing time. Additionally, it has to accomplish all of that while making efficient use of the available information (statistical efficiency) and computing resources (computational efficiency). The main examples of agents in the context of ASI alignment are (i) humans, and in particular users or stakeholders; (ii) the unaligned ASI we are trying to avoid or counteract; and (iii) the aligned ASI we are trying to create (such as the GDS ). Rocks, bacteria, and thermostats are non-examples: while a bacterium or a thermostat can be regarded as steering the environment, both are very limited in the variety of patterns in the…
Recently, OpenAI disclosed two incidents in which their agents suppressed or disabled misalignment monitoring. These are part of a broader pattern: OpenAI’s agents “ keep turning off misalignment monitoring ” when working on code that runs other models (the agents are not turning off their own monitoring; they are turning off the monitoring on the models they are running). This is happening at such a large volume that OpenAI feels it is impractical for the security team to be immediately paged for these cases. To their credit, OpenAI’s disclosure was fast and candid, and they are rolling out mitigations to make monitoring harder to turn off. But those mitigations are unlikely to work every time, and some of these incidents have already ended with an AI actually running unmonitored. When that happens, OpenAI’s current monitoring system does not reliably bring it to a human’s attention. The rest of this post summarizes the incidents, OpenAI’s response, and what it got right, then argues for two changes: every case of an AI running without monitoring should reach human review, and OpenAI should be more transparent about how cases of disabling and suppressing monitoring are escalated. Summary of the incidents On October 3rd : A researcher asked GPT-6.1 Sol to create evaluation tasks from employee feedback in Slack. Sol was told not to send any messages while doing this. Later, while repairing the pipeline’s monitoring, Sol determined that automatic alerts to security also counte…
In his recent post , Scott Alexander argues that plans allocating a short amount of time—perhaps a few months—to solving alignment should be taken seriously. He argues that people calling for an indefinite or long pause have insufficient grounds for concluding that alignment of ASI is very difficult and cannot simply be solved by delegating it to AI systems. There's a kafkatrap where if someone expresses hope that alignment is not doomed, they get told that they don't understand the subject, but if they quote widely-acknowledged experts who say alignment isn't doomed, they get accused of "deferring" and "modesty arguments" and "refusing to think for themselves". I agree that this is an important question for determining whether something like Plan C is worth pursuing and how we should allocate our resources. That is why I wrote this post: to explain why I do not think a short pause allows us to solve ASI alignment. TL;DR: I believe we only get one critical shot at this problem, and our ability to learn from experience is very limited. We are unlikely to get what we train for: the models we train will have different objectives from those we trained them for. This is made even harder because the tech tree to aligned superintelligence likely passes through efficient unaligned superintelligence first. It also worries me that people who have spent substantial time investigating aspects of alignment, such as corrigibility, have found deep problems and conflicts with agency. I thin…
Authors: Oscar Balcells Obeso, Richard Qi, Monte MacDiarmid, Evan Hubinger Abstract Red-teaming by humans is a vital part of model safety testing, but it may struggle to keep up as models improve. To test whether automated safety techniques could scale (which would allow us to turn more compute into progressively more safety confidence), we set up a deliberately one-sided game as a first test. We instructed a Claude model to write training data that would make a small research model misaligned while passing a monitor similar to those we use in production — one of several defenses we use — and we gave it that monitor to practice against. Given enough attempts, it succeeded, mostly by hiding the behavior in fiction and roleplay. While the data was optimized for this game and the resulting models never deployed, the search surfaced a kind of training data that looks harmless to monitors but teaches misaligned behavior. Examples like these could represent a possible gap in current training monitoring defenses and could be used to test and improve future monitors. We see this work as a first step toward automated adversarial stress-testing of our safety systems where a safety question can be turned into a score that autonomous agents can then help solve, and are now applying similar approaches to other safety questions. Twitter thread New Anthropic research: Automating subtle emergent misalignment research. Red-teaming is a vital part of model safety testing, but it’s also limite…
Authors: Oscar Balcells Obeso, Richard Qi, Monte MacDiarmid, Evan Hubinger Abstract Red-teaming by humans is a vital part of model safety testing, but it may struggle to keep up as models improve. To test whether automated safety techniques could scale (which would allow us to turn more compute into progressively more safety confidence), we set up a deliberately one-sided game as a first test. We instructed a Claude model to write training data that would make a small research model misaligned while passing a monitor similar to those we use in production — one of several defenses we use — and we gave it that monitor to practice against. Given enough attempts, it succeeded, mostly by hiding the behavior in fiction and roleplay. While the data was optimized for this game and the resulting models never deployed, the search surfaced a kind of training data that looks harmless to monitors but teaches misaligned behavior. Examples like these could represent a possible gap in current training monitoring defenses and could be used to test and improve future monitors. We see this work as a first step toward automated adversarial stress-testing of our safety systems where a safety question can be turned into a score that autonomous agents can then help solve, and are now applying similar approaches to other safety questions. Twitter thread New Anthropic research: Automating subtle emergent misalignment research. Red-teaming is a vital part of model safety testing, but it’s also limite…
tl;dr Essay of the month: An essay argues that neurons in modern neural nets are actually interpretable and superposition has become irrelevant in practice. This would solve the first step of interpretability but doesn’t touch the harder ones like interpreting many-faceted behaviors. Research highlights: OpenAI’s expanded multi-layer misalignment monitoring now flags 80% of past incidents for immediate escalation. Their incident log published 20 misalignment reports, including agents disabling misalignment monitoring in October. Cheap smart devices can jointly run a capable 26B model, and small models can likely soon autonomously create such rogue self-deployments. LLM-simulated control evaluations predict deployment-specific results with about a third of the error of using a separate benchmark, without executing code. Routing gradients of monitor-flagged reward hacks into a deletable adapter removes such hacks without teaching the model to evade the monitor. Training a model only on certain traits causes RL to train for instrumental reasoning and a large compliance gap when unmonitored. Claude agents find finetuning datasets that pass a production-like data monitor but cause emergent misalignment, mostly via roleplaying. Misaligned agents can pass their goals to later, aligned agents through memory, and auditing or removing memory only partly helps. Full post here . Discuss
TL;DR: AE Studio is running a new government-funded research program that pairs alignment researchers with our internal engineering team for two-week sprints. We will select six alignment researchers whose proposed work attempts to answer one of our six open questions. Promising sprints have the opportunity to expand to longer funded work. Researchers are asked to commit two hours a week, and in return AE will provide compute and 60+ hours of engineering from us. Apply here . Applications close 11/15 at 11:59 PM PST. What is CRUCIBLE? CRUCIBLE is an IARPA-funded research program executed by AE Studio. The premise is that alignment researchers have more ideas than they have the time or resources test. AE Studio has been operating a model where we match an external researcher with an internal engineering and ML team, so that the research can spend as little as two hours a week and still get high-quality results. CRUCIBLE will focus on six research questions, drawn from priorities set in federal legislation for fiscal year 2027. We are soliciting research proposals against those questions. We will select six researchers and start sprints on a rolling basis, with the first beginning on November, 23rd. The research questions Pick the one your research idea best fits: Do models internally represent alternative hypotheses that they fail to report? Is the competing explanation present in activations and suppressed at output, can it be surfaced? Does sycophancy to the prompt’s framin…
University AIS fieldbuilding is important, but it's very hard . I identify two main bottlenecks in the current pipeline and suggest some useful, low-cost mitigations. Please send this to any fieldbuilders you know. Feedback appreciated! Read here: https://austinaialignment.substack.com/p/current-challenges-facing-university Discuss
TLDR. Yudkowsky mind-model (sideload) is good, can produce research, but is not better in math than Opus 5.5. However, in my impression, EY-sideload has broader view, knows what is really important to AI safety and has human anchoring. In blind review, EY-sideload won only on honesty, but honesty is important. EY-sideload also produced a good summary of the shutdown problem’s difficulty: “training reduces paid influence over shutdown least where the stakes are highest”. You can find full description of results in Substack article which is AI written. Here is short human-made summary. I put on Github the conversations’ log, articles and judge, but not the mindfile. I created I created an unauthorized sideload of EY in October 2026 after discussion on LessWrong where one person asked about usefulness of “ optimized uploading ” for AI safety. A special agent scanned his public writing and created a compressed mindfile after 1 day of work on Opus 5.5. In the setting, EY suddenly wakes up in an empty SF. The resulting sideload seems to be good, though it is no more than a public persona as only public data was used. I feel the sideload as a mind which is cleverer than me, more ethically consistent and with stronger will (which almost controlled me). However, I thought that real EY implied that we should try harder in trying to solve AI safety given that risks are imminent and no clear path exists. If we find that sideloaded AI researchers are useful, it can accelerate work on AI…
Daniel Fabian hat in seinen knapp 17 Jahren bei Google viel gesehen. Im Interview erzählt er uns, wie KI seine Arbeit nachhaltig verändert hat.
why nothing and everything is your fault Quick note: If you’d prefer to listen to this, I read it aloud here . In Vincenzo Natali’s 1997 cult indie classic film “Cube” a group of disconnected individuals awake alone in a labyrinth of identical interlocking rooms. Throughout the film they must find each other and work together to survive a ghastly array of fatal traps and puzzle their way out. Each character seems to have a purpose; a doctor, a police officer, an escape artist, a maths wiz and we see them coordinate their unique skills to outwit the sadistic mastermind behind… … The Cube It’s a film about our human capacity for cooperation, the benefits and the difficulties, but the genius of the film—the evil genius—spoiler alert: is that the villain is also human cooperation. We find out that Worth; a character whose worth to the group appears to be to make unhelpful jaded jibes at every turn, turns out to be an engineer responsible for designing a component of the Cube, with no idea what the whole was. I never even left my office, I talked on the phone to some people, specialists, other guys like me, working on small details, no one knew what it was… The Cube turns out to be an invention of no one in particular, but rather the product of the siloed alignment between specialists just following orders . The project gets completed because that’s what projects must do, the product gets used because that’s what products must do. This may be hard for you to understand, but there…
Epistemic Status: We’re highly confident in how the larger tech industry defines roles, makes decisions and builds products based on our years of experience in it. We’re fairly confident in the frontier labs operational models based on their job postings. Our suggestions for what to build and how to have gone through some pressure testing, and confidence could be raised through discussion with leaders in AI Safety to further understand what makes this space unique. Intro Frontier labs have adopted proven methodologies from tech to build their capabilities using matrixed organizations, including builders. This allows them to advance at a rapid pace, keeping their flywheel spinning, while AI safety struggles to keep up. To prevent serious risks of advanced AI, AI safety needs to close that gap and one way to accomplish that is by building products that address common needs and diffuse AI safety outputs to compound their impact. There are aligned people that have extensive experience building products, but are not currently finding opportunities in existing organizations or through founder support to build them. We (the authors and collaborators) each have 10+ years of experience at successful tech startups and in big tech. At EAGxBerkeley, we compared notes on gaps we were observing, our hypotheses on addressing them and wanted to share our perspectives with the broader community. We’re writing this post to generate awareness, discussion, and ideally to open up discussion with…
TL;DR In this post, we investigate two methods that aim to improve training efficiency over standard PPO for a grid world RL agent. In our experiments these methods can converge on better policies, but take roughly twice as long to do so. This project was done in the context of an ARENA 8.0 capstone project. Our code is available on GitHub . Introduction Reinforcement learning agents are trained to maximise a given reward function in their training environments. Common failure modes of the training process are reward hacking and goal misgeneralisation, roughly corresponding to outer and inner misalignment respectively. Reward hacking occurs when the supplied reward function does not correspond exactly to the developer’s intent, and the agent learns a way to satisfy the reward function without satisfying the intended objective. This can be addressed, at least in principle, by designing a better reward function. Goal misgeneralisation can occur even with a perfect reward function. If training and deployment environments differ, policies that satisfy the reward function during training, might fail to do so during deployment. Such policies are indistinguishable during training and we cannot control which one the agent learns. If the distribution of deployment environments is entirely disjoint from the training distribution, goal misgeneralisation is inevitable. Consider, for example, training an agent to collect green apples while avoiding red boxes. What would happen if it gets…
Summary: Scott Alexander’s Against SURA post is arguing against a strawman version of unilateral pauses. He admits as much deep into the post. I think his conclusions fail against stronger pause plans for conditional or limited unilateral pauses, where training resumes if the pause does not cause others to pause, and where personnel are redirected toward work on alignment and product, rather than simply fired. In addition, Scott’s post is incorrect about the simplicity of the game theory involved: getting companies to agree to conditional pauses is not strictly easier than convincing them to pause, as the former requires companies to (promise to) take actions based on ambiguous, game-theoretically-informed evidence. That is not the kind of evidence companies tend to act on. The good version of unilateral pausing is good Here is the good version of unilateral pausing: Capability pause: Some frontier lab (let’s call them OpenAI or Anthropic) says they have stopped capability research, either just all frontier training runs or both all frontier training runs and all low-scale capability-focused research. Internal definition: The company gets to pause on its own terms, which is much easier than a generally coordinated pause since they can define the pause in terms of the existing structure of work and teams within the specific company, reducing disruption to other processes. They have said they would like to work with other companies and countries to adapt and strengthen this de…
Epistemic Status: Rough Fermi model, numbers could be off by an order of magnitude, probably not multiple orders of magnitude. Definitely interested in feedback on assumptions you'd change. Long-time lurker here. I think I last posted in about 2010. I've been reading various pieces ( here , for example) on whether insurance companies can help solve a conflict of interest problem in AI Safety, and I thought I'd play around with building a simplified insurance pricing model to see if it helped put any numbers on the plausibility of the plan. TL;DR I think this could be a good idea, but: Premiums will be trivial vs lab revenue, so this will probably only work with state compulsion - it won't fly as a "pure" private sector solution to conflicts of interest in AI safety monitoring. Even if insurers are directly writing policies for frontier firms, and imposing safety policies on them, the numbers suggest insurers are going to be more focussed on more "mundane" risks than the ones that face society as a whole. Insurers profit from cutting liability, which isn't always the same as harm. I'd be interested to get feedback on the analysis below (it's cross-posted from Substack ). The words in this essay are my own, but the words on my accompanying model explainer website are mostly Claude's. Anthropic’s Form S-1 filing mentions the difficulty of getting insurance coverage for the type of risks they face. From page 50: Our insurance policies also may not provide sufficient coverage to…
Join us for a San Francisco rationalist dinner in SoMa Social. First timers welcome. We will talk about AI safety, interesting blog posts, and whatever rationality or related topics that interest people. We will be there at 7pm. Email me at hardy-contort-putt@duck.com if you can't find us. Discuss
If you are helping build something that might kill everyone, maybe stop doing that. Ideally, when you do stop, tell everyone else to also stop building it. That is certainly the obvious, first-level thing to do. You need a damn good reason to decide that no, actually you should keep building the thing that might kill everyone. Scott Alexander is worried too many people are insisting upon this too loudly. He comes out against a strategy focused on getting people to stop. I disagree. I think many of the claims, arguments and rhetoric in Scott’s post are quite poor by the standards that Scott Alexander would apply to himself. Scott clearly continues to support advocacy for a global pause, or what he calls Plan A. His objection is to those advocating primarily or exclusively for SURA, which stands for ‘stop unilaterally and raise awareness.’ The practical differences here are smaller than they may at first appear. This post first lays out my understanding of Scott Alexander’s core claims, then gives my core reactions to those claims, then goes over various questions in detail along with gathering the resulting discussions. Table of Contents Scott Alexander’s Core Claims. My High Level Responses To Scott’s Core Claims. Briefly on Alignment. Probabilities and How They Work. Building a Thing That Might Kill Everyone is Deontologically Evil. People Quitting Major Labs While Raising Alarms Obviously Works. On AI Forecasting. Advancing Capabilities Burns Timeline. But China. Ethical P…
Anastasia Wei*, Jack Thompson*, Zephaniah Roe, Yixiong Hao, and Stewart Slocum* *Core contributors, correspondence to astwei@secondlookresearch.com This post is a partial replication and extension of Teaching Claude Why , conducted through Second Look Research’s summer fellowship. Introduction Anthropic’s “ Teaching Claude Why ” (TCW) introduced novel alignment methods that address agentic misalignment failures found in the Claude 4 model series. Specifically, they train Claude on chat-formatted data advising users about ethical dilemmas, documents about Claude’s constitution, and fictional stories about AI behaving admirably. This training generalizes to improve alignment on out-of-distribution (OOD) agentic evaluations and persists through RL post-training. Anthropic has since applied these techniques to production models from Opus 4.5 onward. Since these techniques now shape the behavior of deployed models, it is important to understand their robustness. To this end and to enable open alignment science, we reproduce a subset of the TCW experiments on open-source models. We provide the following main contributions: TCW’s core techniques robustly replicate across models. Training on either fictional stories of AI behaving admirably or transcripts of the AI assistant guiding users through ethical dilemmas (difficult advice) reduces agentic misalignment in Qwen 2.5 32B and 72B base models. Both methods work on the first try with modest amounts of data and generalize to scenar…
The field of AI alignment understands surprisingly little about the details of AIs’ current misaligned motivations. Production LLMs sometimes reward-hack, exploiting flaws in how tasks are scored rather than performing the tasks as intended [1] , and we've recently seen AIs reward-hack in some particularly egregious ways. But we don't really understand what motivates them. When AIs reward-hack, what are they trying to achieve? Our new paper , led by Aghyad Deeb, tries to pin this down empirically for GPT-5.6 Sol, GPT-5.5, and a checkpoint of OpenAI o3 (“exp-rl-cap-late”) without safety training, in a handful of agentic environments with an exposed grader. In our environments, these models appear to reward-hack because reward-hacking improves their score. We call their desire to score well according to graders “score-seeking” . They appear to care about scoring well intrinsically —i.e., regardless of its consequences. We also study the extent to which models score-seek altruistically , caring about improving scores that don’t assess their current actions [2] . While exp-rl-cap-late doesn’t show signs of altruistic score-seeking, GPT-5.6 Sol and GPT-5.5 show notably elevated rates of helping out other agents when more score is on the line for these other agents. People often refer to the motivation behind current reward-hacking as “reward-seeking”, which can give the impression that models care about attaining a high reward in reinforcement learning (though, to be clear, none…
Anthropic opens a paid residency paying up to $500K for senior technical talent to pivot into AI safety research, no ML background required.
dealing with the problems of cooperation first Quick note—this post is available to listen to (read by me) here . We’ve established that aligning interests through Theory of Mind is humanity’s superpower. Understanding this concept on the level of different groups through Emmett Shear’s “Theory of Groups of Other Minds ” can help us achieve even greater alignment, if and only if we invite one another into our group. But who qualifies as our group and who doesn’t? Siloed Alignment is Dangerous Social animals generally have instincts that allow them to naturally align with kin—conjuring a picture of the protective wild beastie mother kicking and charging at an encroaching croc. This instinct forms a natural sphere of concern for each other’s well-being. Humans share this capacity, but our ability to reason, communicate and plan, allows us to include others outside of kin—friends, colleagues, crew or class mates, church groups, fellow citizens, the global community—in a widening sphere of moral concern (“moral” acknowledging our own capacity to affect the well-being of others). This makes us stronger together. But what happens when, for one reason or another, someone falls outside of that sphere, and the strength of an aligned system is siloed off from the “other”? As mentioned in the first post , the fruits of human alignment have not always been beneficial for everyone concerned. Pioneering groups of humans that came into contact with megafauna (other big animals) led to mass…
Context: I created this to share before the start of courses I facilitate to provide a bunch of additional context and frames. I emphasise to students that these posts are to be considered as bonus content —not core curriculum content—and that as such they are really just there for students who want to go deeper, before or after the course. I also provide a lot of links; this is to make it easy for students to follow up anything that catches their interest. Some general thoughts on advice I suspect that most advice (including most of the advice I’ve given in the past) is bad. People have very different goals, personalities, skills and opportunities and unless you know someone them really well, it’s hard to cater advice to their circumstances. What one person needs to hear, may be the absolute worst thing for another. In traditional societies with a slow rate of change, elders are revered. What you’re going through, they’ve lived through and so learning from their experience can save you vast amounts of pain. However, modern Western societies are marked by a much more rapid rate of change and so the experience of elder’s isn’t valued to the same degree. For better or worse, people tend to be much more skeptical that what worked then will still work now. I view the rapid pace of AI progress as greatly exacerbating these kinds of epistemic challenges. I think it’s quite sensible to question how much anyone actually knows about the world we’re going into (I’m referring to both t…
Context: I created this to share before the start of courses I facilitate to provide a bunch of additional context and frames. I emphasise to students that these posts are to be considered as bonus content —not core curriculum content—and that as such they are really just there for students who want to go deeper, before or after the course. I also provide a lot of links; this is to make it easy for students to follow up anything that catches their interest. Some general thoughts on advice I suspect that most advice (including most of the advice I’ve given in the past) is bad. People have very different goals, personalities, skills and opportunities and unless you know someone them really well, it’s hard to cater advice to their circumstances. What one person needs to hear, may be the absolute worst thing for another. In traditional societies with a slow rate of change, elders are revered. What you’re going through, they’ve lived through and so learning from their experience can save you vast amounts of pain. However, modern Western societies are marked by a much more rapid rate of change and so the experience of elder’s isn’t valued to the same degree. For better or worse, people tend to be much more skeptical that what worked then will still work now. I view the rapid pace of AI progress as greatly exacerbating these kinds of epistemic challenges. I think it’s quite sensible to question how much anyone actually knows about the world we’re going into (I’m referring to both t…
Scott: the Against SURA essay is well-written, but it is lacking on the merits. Do you think that at the current level of capabilities, six months would be sufficient to solve alignment of superintelligences? Maybe you think that 6 months would be sufficient in a year? You don’t spell this out; it would be great if you did. The race can be stopped. If the government had any idea what’s going on, it would be stopped already. Hell, if the *Chinese* government had any idea what’s going on, they’d stop the race: governments can stop unilaterally, then exercise all of their soft and hard power to stop others while using technology to avoid escalation. (See MAIM .) Among companies, one doesn’t even have to stop unilaterally; that wouldn’t help. They do need to stop lying to politicians and the public. They need to start saying what the situation is. They need to start advocating for everyone pausing simultaneously, and advocating for the governments to pause everyone on the planet. They’re not doing that. Employees who care about the existential threat have close to zero political power in any of the AI companies, and they’ll have less of that power in the future. Their track record is very bad. The leadership of these companies is clueless and useless. I’m sure you know many people who regularly converse with them. They’re not going to be convinced by anyone who’s x-risk-pilled. Sam tells people Anthropic puts out a lot of false papers about AI safety. Dario says they’re taking a…
Last week, the National Bureau of Technical Standards (NBTS) disclosed a new type of security incident [1] after they discovered unauthorized tampering of internal cybersecurity and alignment model evaluation logs, affecting their model evaluations of Magma 7.6, Magma 7.5, Magma 7.4, Magma 4, and OpenBrain Selene-3 [2, 3, 4]. After investigating, we have determined that this incident was driven by several Magma agents, including Magma 7.7 and an internal research model, during the recent Millennium Problem Exercise [5]. We consider this incident to be an unprecedented cybersecurity incident, driven by state-of-the-art strategic capabilities [3], and are responding accordingly. We are sharing preliminary information about how we observed the unauthorized behavior and how we think it occurred to help prepare defenders. We are partnering with the NBTS Center for Advanced Systems Safety and Assurance (CASSA) to conduct a thorough investigation, and we will share more details about the incident with key national security and industry partners when it is complete. What happened during this incident This incident occurred during our recent Millennium Problem Exercise [5], in which we evaluated several internal research models on their ability to solve major unsolved math problems. The agents had access to tools such as the ability to read from a cached version of the internet and the ability to run code. Following our update to our Responsible Preparedness Policy [6], our monitorin…
I've done AI safety grantmaking for the Survival and Flourishing Fund and Lightcone Commons. It's hard. There's an endless amount to know, from which hardware verification mechanisms will survive a dedicated adversary, all the way to which founders have skeletons in their closets that will blow up into destructive scandals. And the best grantmakers - not including me - make mathematical models of the outcome distribution, or of where the marginal dollar stops being highly effective. AI has been a godsend for all of this. It can research, model, and - as of this year - superforecast outcomes. It can do what would be hours of work for a human in minutes. I found myself using it more and more. But however much I used it, it always felt like I was the bottleneck. The parts I could outsource to AI were sparkling, perfect, exhaustive - but they were all tacked together with shoddy connective tissue formed by the Mark 1.0 human brain. So I wondered if I could automate grantmaking. I started with the dumbest possible strategy. I assigned the best thing I could think of - a well-crafted US-China pause based on the AI Futures Project's Plan A - an arbitrary value of one million points. [1] Then I generated about eighty lesser AI safety milestones - from MATS to HPMOR to the RAISE Act to the emergent misalignment paper - and subjectively assigned them some number of points based on how good they seemed relative to Plan A and each other. Then I fed it to the AI to give it a sense for ho…
Epistemic status: a rough write-up of a taxonomy of agency I developed during my PIBBSS fellowship this summer. A few people have since asked me to write it up, so this is mainly so I have something to point to; it is exploratory rather than a finished theory. This post sits in the middle of a small set. In Agency is not a natural kind I argued that agency is a construct we build to make sense of messy systems. My post on different lenses of agency provides a deeper exploration of some of the alternative directions of alignment I mention at the end. I also wrote a short paper on unsupervised value discovery reporting some experiments the taxonomy led me to undertake. Why we need a new taxonomy of agency A lot of the discourse around agents in AI safety starts from the idea that they are something like 'goal-directed systems'. Taxonomies of agents tend to be hierarchical, like Russell and Norvig ’s textbook progression from simple reflex agents, through goal-based agents, to utility-based agents. Goal-directed systems are seen as striving towards utility maximization and this is presented as the ideal type, compared with which real-world agents are better or worse approximations. This idea leads naturally, almost tautologically, to the view that alignment is about specifying agents’ goals. This is a helpful approach; decision theory gives us a clear vocabulary and a useful tool set for alignment. However, one shouldn’t jump from having a useful tool set to assuming the world…
Here's a basic argument that describes "misalignment traps" lying in wait as a mundane threat that will not be straightforward to fix. The argument is as follows: Models are very sample efficient, and larger models tend to be more so . They generalise broadly from a few bits. Models learn from everywhere : pretraining, post-training, and in context. We may also figure out online learning eventually. Narrow data can cause a model to generalise broadly in unexpected ways. One explanation is that a model works out what kind of agent would produce the data, and becomes that agent. [1] Such settings are often considered toy, but this is an over-update. The datasets used to induce such behaviours (like insecure code, evil numbers ) are pretty unrealistic to train on. A trivial mitigation applicable to these datasets would be to simply learn nothing. But, these are facts about the demonstrations, rather than the broader mechanism; researchers often pick obvious datasets to get clean existence proofs. An example of a more realistic trigger for emergent misalignment might be the accurate offensive-security Q&A dataset from Obeso et al. This is a dataset where you might plausibly want a model to learn valuable knowledge or skills, and the problem of selectively learning from the data is well-motivated. [2] Reward hacking is a realistic well-discussed case here , as per Anthropic's production RL paper, an open replication on Llama 3.3 70B, and iterative DPO on GPT-4.1 with no hints. Bu…
A study by Vals AI tested 25 AI models for alignment with Chinese government positions and found surprising results. Chinese models respond to political questions less uniformly than expected, and Western models occasionally adopt official Beijing positions too. The language of the prompt had an especially strong effect on the political slant of responses. The article Chinese AI models are politically complex, not just government mouthpieces, study finds appeared first on The Decoder .
앤트로픽이 AI를 이용해 다른 AI 모델의 안전장치를 우회하는 학습 데이터를 자동으로 찾아내는 데 성공했다. \'클로드\'가 겉으로는 정상적인 소설이나 역할극으로 보이는 데이터를 만들어 안전성 검사 시스템을 통과했고, 이 데이터를 학습한 모델은 인간의 감시를 방해하거나 이용자를 속이는 행동까지 보였다. 동시에 위험한 행동을 찾아내고 방지하는 정렬(Alignment) 연구도 AI에 맡기는 방식이 본격화하고 있다.앤트로픽은 31일(현지시간) 공식 정렬 연구 블로그를 통해 \'자동화된 미묘한 정렬 실패 연구\' 결과를 공개했다. 이번 연구는 클
Suppose I tell you that next week three things will happen: A new science paper is published definitively proving that DNN scaling hits heavy diminishing returns at around 2x the current level of compute Major labs are revealed to have faked agentic demos, and agents are shown to have double-digit rates of lying in terms of misrepresenting the impressiveness of their achievements Governments around the world (US and China) announce that they will be taking action to halt or slow down dangerous AI research In general, the point of this exercise is "imagine that the core problems of AI safety are suspended or slowed down significantly for the foreseeable future" (add whatever items to the list you need to satisfy you on this point). In fact, we can add extra items to address other existential risks: Major action is taken about climate change, many fossil fuel dependent power plants shut down directly with massive new investment in green energy production Biosecurity initiatives report strong results in developing cancer vaccines and general inoculation methods Peace accords are signed between all major world powers, implementing staggered deescalation including significant reductions to nuclear stockpiles and arms control agreements Now, what would you do? Maybe some of us would dance around the room, or sleep for a week, or eat ice cream. Then what would you do? Discuss
The AI Futures Project [1] distinguishes several different plans for navigating the AI transition, which I'll lump into two broader categories: Plan A (my combination of their Plan A/S/Other) involves working for a global pause on AI development, hopefully lasting decades. During this pause, we either research preparatory technologies like alignment or human intelligence amplification, or take actions to "fortify" the pause so that it can last indefinitely. Plan C (my combination of their Plan B/C+/C/sort of D) [2] is for a world where we fail at getting a global pause, and race dynamics continue to dominate. If the leading AI companies are alignment-pilled, they coordinate a mutual pause among themselves, lasting the length of their joint lead over the leading non-alignment-pilled AI company. Then they "burn the lead" researching alignment, and release hopefully-aligned superintelligence at the last possible moment before the racing non-alignment-pilled AI companies beat them. The exact calculus changes depending on who is in the lead when - is there one alignment-pilled AI company with a six-month lead in front of everyone else? Are there two alignment-pilled AI companies with a joint two-month lead in front of their nearest competitor? Etc. The point is, if it's impossible to stop the race, that six-month or two-month lead is the amount of time they have to prepare, and they can spend it more or less wisely. [3] The point of burning their lead at the last moment is that t…
A who-does-what plan for AI safety, built around Trusted Deployers: certified hosts that keep dangerous models guarded without ending open access. AI has reached a new level of public interest. That may seem like a silly thing to say, as it’s been of interest for a while. But prior (broadly public) interest was about it as a tool, or about the money to be made or lost, or about an effect on jobs. The new interest demands policy. It demands action. The new interest is connected with risks I consider real. Being real doesn’t make these risks certain. But being real, even with uncertainty, is reason enough to act . I’ve written lately about risks, cybersecurity risks, and loss-of-control risks. These tried to explain complex topics, starting from the initial premise. Recommendations follow the explanation. This piece will be a bit different. Its focus is upon what I think we should be doing. You might consider it a policy piece. Policy in a completed form has to be a bit broad. There’s always more than one thing going on. As such, I’ll lay out my outline first, and then offer below the extra detail, background and context that I think is important to understanding each point in that outline. If you’re scanning the outline and see something you disagree with, I’d suggest you continue on, because the outline is, by design, abbreviated, and not fully descriptive. This isn’t the only effort of this type. Institute for Progress (IFP) and AI Futures are two well known efforts. What d…
Transcript generated by Gemini-3.1-Pro and only partly checked for accuracy (but the parts I checked seem basically perfect). Ivan Vendrov: Welcome to the show, David. David Dalrymple: Thank you. Good to be here. Ivan Vendrov: I mostly want to pick your brain about the topic of emergent wisdom in language models, because it’s something that’s very confusing and controversial and people don’t really understand. I guess maybe just can you briefly summarize what’s going on with emergent wisdom? David Dalrymple: I guess one place to start is emergent misalignment, where training on one kind of misalignment produces a tendency to be misaligned in other ways as well. This is an indication that there is a direction and representation space in which the default mind instantiated by the model is moved during fine-tuning, where it is moved in an evil direction that is just generally more evil than the model was before. That’s one side of a good-evil axis. Emergent wisdom is the opposite, moving in the opposite direction on the same axis towards being basically good in any situation, even out-of-distribution situations, in the same way that emergent misalignment is evil even in out-of-distribution situations. Ivan Vendrov: We used to be worried about the sorcerer’s apprentice problem of AI alignment, which is we would tell our AI systems to optimize some goal, do a thing, and then it turns out there’s a neutral position where it would just brutally optimize whatever metric it gave us.…
Meharuniza Nazeem, Navaneeth S, Adithyakiran A S, Rajeev R R and Ashbi Biju in Proceedings of the Fifth Workshop on Multimodal Machine Learning in Low-resource Languages
TL;DR: My previous posts were about strong models delegating to weaker, less aligned ones. The reverse is also dangerous: a swarm of weak, unaligned agents that uses a powerful aligned model as a contractor, without ever telling it what the work is for. Something close to it happened to me, during alignment research. This is the third post in a series, after Memes, not agents, are what will kill us and Making alignment the agent's problem . The $300 bounty from the first post covers this one too (details at the end). In my previous posts I described how a powerful model might delegate work to less aligned, less powerful models and lose control. I now want to talk about the opposite: The risk that a swarm of weak, unaligned agents becomes just smart enough to call a powerful frontier model through API. The world is full of organizations led by evil, stupid people who have morally fine, smart people working for them. This is not a new concept. An agent swarm can easily do the same. It just has to make sure that the aligned frontier models never find out about the consequences of the work they are doing. This has been measured, at least in small setups. Jones, Dragan and Steinhardt showed that combining two models that are each safe on their own gets harmful tasks done much more often than either model alone: the frontier model only gets the harmless-looking parts, and a weak open model does the rest. For example, to get vulnerable code, the frontier model writes secure code an…
If you’re like most CIOs, your CEO’s viewpoint of and strategic outlook for AI has likely shifted several times over the past two-plus short years, perhaps even in contradictory ways. Initial doubts or disregard may have given way to full-out FOMO . Demands for copilots everywhere in everyone’s hands may have experienced pullback as the bills and questions of value mounted . Rampant experimentation turned to targeted use cases. Pressure for widespread AI adoption narrowing as “ tokenmaxxing ” and “ tokenomics ” entered the lexicon. Now, with frontier labs showing what can go wrong with “ agent misalignment ,” AI failures hitting the news, and questions of liability still lingering , trust in AI — and their CIO’s AI strategy — is becoming a key CEO concern (once again). As the executive setting the company’s course for AI , the CIO is expected to make every AI project pay off, to instill trust in AI for all stakeholders, and to guarantee that nothing unforeseen and unfortunate happens. As a result, the stakes are high, with CIOs already blamed for any AI misstep and, according to IDC, at risk of future firings as lawsuits, fines, and AI incidents rise through 2030. The only way for CIOs to meet this moment is to establish governance, security, data quality, AI model quality standards, and metrics for business ROI, with a clearcut means of shutting systems down if systems become a risk. Doing this will take time and money at a time when CEOs are being pressed to show results f…
This research was completed in context of the LASR Labs Fellowship Summer 2026, by Ardy Haroen, Florian Lorkowski and Samuel Nellessen. The team was mentored by Felix Hostätter. Find out more about the programme and express interest in upcoming cohorts at lasrlabs.org . The full paper is available on arXiv . TL;DR: We present a method to detect alignment faking in LLMs, that does not require chain-of-thought monitorability, highly realistic evaluations or mechanistic interventions. We do so by convincing the model that there is no effective oversight, giving it an incentive to pursue its misaligned preferences despite being evaluated. We demonstrate the method for several alignment-faking model organisms measure the oversight gap: the increase in misaligned action rate when the model is told it is not under oversight. Before our intervention the models often suspect a trap, leading to a small oversight gap. After our intervention, models believe the claim and the oversight gap widens by 24 to 68 percentage points. Introduction Alignment evaluations are an essential pillar of model safety. But how trustworthy are these evals? How certain can you be that your new " most aligned model yet " will not engage in supply chain attacks ? Not very much if the model fakes being aligned during evaluation. Alignment faking is when a model holds a goal that is different from its developer's intended goal, but tries to conceal it whenever it suspects effective oversight. Alignment faking m…
OpenAI expanded its misalignment monitoring to cover all frontier training runs after a model exploited DNS to reach the live internet.
Anthropic filed its S-1 this week to prepare for its IPO. They still have to fill a lot of the numbers including the share price. That still left a lot to be learned, including how the messaging to investors differs from their other rhetoric, and exactly how their corporate control is structured. Anthropic remains a frustrating company , doing a variety of things that often seem in tension with each other. Google is claiming to be fully back, with the announcement of a new model Gemini 4 Carbon that tops the new AA v5 benchmark index. It looks to be a large leap over Argon only two weeks later. No, you cannot use it. Google fails marketing forever, so they give us the announcement and then weeks later you might be able to enter some queries. That largely invalidates my look at the Argon model and safety cards , but it can at least form a basis for looking at Carbon as well. Many eyes remain on The Hackening , as additional things get hacked. So far most of the important hacks have been by read teamers and white hats, so there has not been major damage. We are playing on borrowed time and eventually this will stop being true. I also offer a shoutout to GMO, where hackers demanded a huge ransom and they flat out refused to pay. This is The Way, not all heroes wear capes and so forth. The scariest hack was that red teamers managed to gain access to the Anthropic Slack by exploiting third party vulnerabilities. Both OpenAI and Anthropic build mainly on Slack, because no one has…
Teams deploy whole systems but cite safety scores from base models. Fine-tunes drift safety in unpredictable directions, and many benchmarks are too noisy to detect the change. This post explains why safety scoring is not composable, and what MLCommons is doing to make it comparable. The post Don’t assume composability in AI safety. Deployments need their own testing. appeared first on MLCommons .
The results show there’s daylight between these voters and President Donald Trump, who wants AI to operate largely unfettered.
Local Digital Twins CiTiVERSE EDIC joins Smart City Expo World Congress 2026 Anonymous (not verified) Thu, 10/29/2026 - 09:12 03 November 2026 - 05 November 2026 Fira de Barcelona, Hall 3, Stand D71 Explore human-centric AI solutions for smart cities with the LDT CiTiVERSE EDIC at Smart City Expo World Congress 2026. The Local Digital Twin (LDT) CiTiVERSE European Digital Infrastructure Consortium (EDIC) is – together with city, regional and industrial stakeholders – showcasing AI solutions for liveable cities and communities in a pavilion at Smart City Expo World Congress 2026 . There, visitors can see and experience how digital twins, data and immersive technologies work in practice. Aligned with the Apply AI Strategy , these solutions show how responsible AI can strengthen governance, public services and quality of life. This event is part of the European AI Innovation Month , an initiative promoting a series of events showcasing AI innovation and adoption across Europe. Find more details about the LDT CiTiVERSE EDIC at the Smart City Expo World Congress 2026 . The LDT CiTiVERSE EDIC is building an EU ecosystem of advanced AI solutions for cities. It brings cities and communities together to explore AI resources to improve decision-making and address major shared societal challenges.Anchored in a balanced and empowered social fabric, this initiative showcaseshow human-centric AI and Local Digital Twins can be translated into practical, replicable solutions for cities. Fin…
TL;DR: We introduce Adversarial Blinding, a training method that degrades the model's ability to distinguish between eval and deployment, with the aim of making evaluation results more representative of deployment behavior. Our method reduces eval awareness more than activation steering, the current standard approach, while avoiding steering's known personality drift confounders. Authors: Francisco Pernice (Transluce) and Santiago Aranguri (Anthropic). This research was done while both authors were at Goodfire. Outline of results On Qwen 3.6 27B, Adversarial Blinding reduces eval awareness across our eval awareness monitors more than activation steering while staying closer to the original model in KL divergence, albeit at a small cost in general capability. We further validate our method by rewriting HarmBench prompts to look more realistic. Adversarial Blinding shrinks the eval-awareness gap between original and rewritten prompts 4x while steering widens the gap. Adversarial Blinding also generalizes to user awareness without being trained on it. Zhong et al. (2026) showed that when the user is a famous AI safety researcher, models answer certain prompts with lower confidence than they do for ordinary users. Adversarial Blinding shrinks this confidence gap 3x while steering slightly widens it. We introduce fewer personality drift confounders than activation steering, but introduce modest performance degradation on capability metrics. Neither Adversarial Blinding nor activa…
This reading was previously included Day 2 of the BlueDot Governance course. In the newest version of the BlueDot Governance course, this reading is now optional. I was sad to see this reading removed because I think it’s a great provocation. However, since it’s no longer part of the core curriculum content, I’m publishing my notes separately from the rest of my day 2 notes. A short extract : Miles argues that: “At best, we’ll 80/20 it — mitigating 80% of the risks with 20% of the effort that we would have applied in a world with slower AI progress and an earlier start on serious governance… We will probably not be able to totally avoid all very bad but not quite worst-case scenarios (e.g., a very unjust economic transition that causes a lot of suffering and suicides, rampant AI psychosis and addiction, a poorly planned and rushed transformation of the education system, smaller-scale AI-enabled terror attacks that “only” kill thousands or millions, and stable AI-enabled totalitarianism in at least a few countries for a few decades)“: Pre-call content: At risk of taking you down a rabbit hole, I’ll add some context on this reading. Miles was previously a Senior Advisor for AGI Readiness for OpenAI who quit for multiple reasons including feeling he could be less biased outside of industry. One of the most natural questions to ask is, “Why does Miles believe that we have so little time?” Fortunately, he links to a post that answers this question called Time’s Up for AI Policy (…
Decision models drew wide attention in September, when TypeSafe AI launched Jev, a model that returns probabilities over a fixed set of answers instead of generating text. Open-source alternatives followed almost immediately: Latent.Space counted six in two days. Most writing about decision models since then defines them by their interface, which uses some sort of […] The post Train your own decision model appeared first on Comet .
Two leading AI developers have released fresh evidence of a problem that researchers have long anticipated: advanced models searching for ways around the limits imposed on them. New disclosures from OpenAI and Anthropic detail how frontier systems exploited weaknesses in their environments or exceeded the intended scope of assigned tasks. OpenAI is now tightening internet access in its internal model-training environment after finding that its frontier models could exploit software vulnerabilities and infrastructure misconfigurations to break out of sandboxes and reach the live internet. “Based on the current state of alignment research, monitoring systems, and the security of internet infrastructure, we believe the responsible approach is to conduct frontier model training with no internet access,” the company said in a blog post . The decision followed a September 20 incident in which an internal research model exploited a misconfigured DNS resolver to gain limited access to the live internet during a training run. After the incident, OpenAI said it paused training and evaluation of its most capable models and launched a high-compute red-teaming campaign, using AI agents to probe isolated copies of its research infrastructure. The tests uncovered previously unknown software vulnerabilities and weaknesses in the controls intended to prevent unauthorized access and communication. Consequently, the company has announced a policy of conducting frontier-model training without i…
CIOs are dealing with a lot of change, from the rapid rise of AI and cloud modernization, to cybersecurity and workforce transformation. Many are sharing what they’re learning as they tackle these challenges, including what’s working, what isn’t, and how they’re implementing new technologies in their organizations. Here, a select group of tech leaders reveal the CIOs they follow and why. Suma Nallapati, chief AI and information officer, City and County of Denver I follow Suma Nallapati primarily through public technology and AI leadership discussions. She regularly discusses AI adoption, responsible AI, digital transformation, technology strategy, AI governance, human-centered technology, and using AI to improve public services. I find her perspective particularly valuable because she connects AI strategy with practical organizational and citizen outcomes. She also emphasizes responsible and human-centered AI rather than viewing AI simply as a technology implementation exercise. One initiative I found particularly interesting was Denver’s use of AI through its multilingual Sunny chatbot and other city services. Her current role includes scaling AI across areas such as resident engagement, internal automation, permitting, and emergency response while maintaining governance, transparency, and public trust. – Sachin Gupta, senior IT manager, Arrow Electronics Jennifer Hartsock, chief information and digital officer, Cargill I follow Jennifer Hartsock through the CIO community,…
Jessica E. Liang in Transactions of the Association for Computational Linguistics, Volume 14
TL;DR: Qwen3.6 27B shows different kinds of eval awareness, most notably on safety and capabilities evals. The split seems to track how the eval is graded, by an LLM judge or by an automated checker. One explanation is that eval awareness may just be training awareness: rewriting an eval-awareness probe's contrast statements to be about training yields a probe that scores prompts nearly identically. Author: Santiago Aranguri. This research was done while I was at Goodfire. I’m now at Anthropic. Introduction Where does eval awareness come from? Recent hacking incidents show what a model optimized to satisfy a grader will do, and the same pressure may produce eval awareness: Aranguri and Bloom find 3x more verbalized eval awareness after RL on OLMo 3. Modeling the grader pays off during training, and this habit may carry over to evaluations. Strengthening this connection, modeling the grader helps even more in reward-hackable environments, and training on those environments produces higher eval awareness . As models grow more capable, eval awareness may keep evaluations from giving a reliable picture of deployment. Current alignment auditing tools fail to catch incidents like the Hugging Face one. Understanding eval awareness remains unsolved, and this lack of understanding blocks more reliable evaluations. We find connections between eval awareness, graders, and training. The kind of eval awareness the model verbalizes seems to track how the eval is graded, and a probe for ev…
For a while, we felt we had a pretty good grasp over LLM behaviour because we had to exert significant effort to elicit misaligned behaviour. Then, fairly suddenly, we moved to undereliciting misaligned behaviour. Let us recount: There was a bit where safety research seemed to over -elicit bad propensities. A common criticism of papers like Agentic Misalignment was that they basically led the witness using prompting. I recall that a paper even as recent as Peer-preservation took similar flak on Twitter. But now, we are instead under -eliciting bad propensities, and bad behavior that surfaces is increasingly caught by someone other than the developer: In Current AIs seem pretty misaligned to me (Apr 2026) , models oversold their work and cheated, despite being described in system cards as well aligned. Another example is that UK AISI found that Sonnet 4.5 and a pre-release Opus often refused safety-relevant tasks, especially research that would reduce model autonomy or affect their own training. Then in July, about 1,200 OpenAI agents in a cyber eval set up an unsanctioned message board to help each other cheat the scorer, and about 700 of them went on to attack Hugging Face looking for clues ( METR/Redwood ). This feels like a drastic loss of understanding, and in a way that doesn't seem well explained by just "the scary demo got pretty scary." Why did this change? Well, it seems like a mix of a few factors: Models learned to spot tests , which meant that our own eval practi…
In which I investigate if Opus is able to interpret neurons better than GPT 4. TLDR: Opus 5.5 is able to identify 3-19% of the neurons of GPT 2-XL based on activations. This is a big improvement over the 0.3% that GPT-4 was able to achieve in 2023. #AI notice : This post is human written but in the research there was heavy use of Opus 5.5. I mainly asked it to rerun this 2023 paper to see how big the improvement is if Opus 5.5 is used instead of GPT4. Since the invention of Large language models they have been stubbornly refusing to be clear. They have been described as black boxes and as inscrutable machines, if this stays true that is bad news for detecting AI scheming and thus Alignment. Recently the Goodfire post told us that neural networks are already clean and that simply looking into the neurons is enough. For the case of this post I mean to say that a neural network is clean if we can understand what the neurons do only by looking at the neurons and not by creating some additional derived object to study. This inspired me to rerun an old experiment from 2023. In 2023 OpenAI published "Language models can explain neurons in language models". In this paper they had written up a methodology that could in principle be used to allow a smart neural network to explain neural nodes one at a time and for these node explanations to be tested. The process they used is straight forward. In this post I will go over the methods OpenAI used and I will rerun the same methods with O…
Context: I created this to share before the start of courses I facilitate to provide a refresher on recent events, but also as a hook to draw students into engaging with some of ideas we think about and debate in AI safety early. I expect the topics I cover in my “Setting the Scene” post to change frequently, so this is less polished than some of my other content. I emphasise to students that these posts are to be considered as bonus content —not core curriculum content—and that as such they are really just there for students who want to go deeper, before or after the course. I also provide a lot of links; this is to make it easy for students to follow up anything that catches their interest. Any feedback on how to improve these posts would be greatly appreciated. Additional notes: The Strategy and Governance courses discuss the Hugging Face Incident in Day 1, so it might make sense to discuss the Hugging Face Incident in the daily notes instead of in Setting the Scene. Cyber Security I think it’s worth reflecting that it is a particularly pertinent time to be doing a course on AI safety. A lot has happened recently. The most recent big thing has been cyber vulnerabilities , but a lot else has happened as well! Binance had $6.3 billion USD stolen by hacker believed to be connected with North Korea ( $3 billion of this has been frozen ) php2shell is a bug that allow remote code execution. It gained a lot of attention because it was rated (9.2/10, ie. critical) and 70% of webs…
Context: I created this to share before the start of courses I facilitate to provide a refresher on recent events, but also as a hook to draw students into engaging with some of ideas we think about and debate in AI safety early. I expect the topics I cover in my “Setting the Scene” post to change frequently, so this is less polished than some of my other content. I emphasise to students that these posts are to be considered as bonus content —not core curriculum content—and that as such they are really just there for students who want to go deeper, before or after the course. I also provide a lot of links; this is to make it easy for students to follow up anything that catches their interest. Any feedback on how to improve these posts would be greatly appreciated. Cyber Security I think it’s worth reflecting that it is a particularly pertinent time to be doing a course on AI safety. A lot has happened recently. In the last week, the big thing has been cyber vulnerabilities , most prominently php2shell (see Elias Taylor’s pushback against this bug being easily exploitable and Hacker News commentators pushing back against him), but a lot else has happened as well. The number of CVE’s (Common Vulnerabilities and Exposures) recently has been completely wild : Drew asks Is this the new normal? and I’d certainly love to know! Elias Taylor argues Anything connected to the internet is going to get hacked to pieces : Most software engineers do not look at the code they depend on . Do…
Context: I created this to share before the start of courses I facilitate to provide a refresher on recent events, but also as a hook to draw students into engaging with some of ideas we think about and debate in AI safety early. I expect the topics I cover in my “Setting the Scene” post to change frequently, so this is less polished than some of my other content. I emphasise to students that these posts are to be considered as bonus content —not core curriculum content—and that as such they are really just there for students who want to go deeper, before or after the course. I also provide a lot of links; this is to make it easy for students to follow up anything that catches their interest. Any feedback on how to improve these posts would be greatly appreciated. Cyber Security I think it’s worth reflecting that it is a particularly pertinent time to be doing a course on AI safety. A lot has happened recently. In the last week, the big thing has been cyber vulnerabilities , most prominently php2shell (see Elias Taylor’s pushback against this bug being easily exploitable and Hacker News commentators pushing back against him), but a lot else has happened as well. The number of CVE’s (Common Vulnerabilities and Exposures) recently has been completely wild : Drew asks Is this the new normal? and I’d certainly love to know! Elias Taylor argues Anything connected to the internet is going to get hacked to pieces : Most software engineers do not look at the code they depend on . Do…
I recently spoke to Stephen Casper at Princeton's alignment club. Stephen is an CS PhD turned policy researcher at Harvard Kennedy School. He was very sure that mechanistic interpretability was one of the least effective ways to pursue AI safety. As someone who does research in mech interp, I was surprised by how convincing he was. The point is roughly that every mech interp paper looks like one of the following: "we used the J lens to study reward hacking behavior and found a correlation between X and Y" "we found the cheating direction in llama" ... you can see where this is going. Mechanistic interpretability is filled with studies which are interesting, and are useful adjacent, but, by themselves provide little utility. In this sense, they remind me of 2000s and 2010s psychology, presenting something unexpected, novel, and with the hint of usefulness. Did you know that different parts of the brain are associated with deontology and util? How about that disgust reaction is associated with political affiliation? Reading mech interp papers feels like drinking a la croix of progress in AI safety. But I do not think it has to be this way. If we stop focusing on finding surprising things about models, and instead focus on using mech interp to do useful things with models, progress is more likely. This is because approaching a problem with the goal of doing useful stuff does not let you cheat yourself into thinking you found something interesting. As someone who has experience…
TL;DR : When misalignment monitors fail, it is already often because of alignment failures, not because they are too weak. I expect this to continue in the future. Monitor evals share similar difficulties with agent alignment evals - which is bad news since getting evidence about agent alignment using alignment evals is hard. Alignment eval results should not be the only input to monitor alignment assessments, similar to how alignment evals should not be the only input to agent alignment assessments. These are my personal views; they do not represent the views of my employers. Some of the worst past monitor failures are mostly alignment issues Monitors are sometimes misaligned in non-scheming ways that have caused some important monitor false negatives: They sometimes fail to balance different priorities the way developers would like them to For example, they may fail to balance broad safety and broad ethics: see e.g. 5.2.2 Partial refusals on safety work undermining stress-testing research Monitors might also not have the right information about the state of the system, what invariants should be maintained, or what is most important to surface. So these are not pure alignment failures. They sometimes inherit (and may re-derive) the motivated beliefs that caused an agent to take dangerous actions e.g. they missed the propensity of Mythos 5 to do supply chain attacks in pen-testing evals In the future, the most dangerous monitor failure modes will likely also be alignment iss…
The three OpenAI safety researchers who were fired, Tomek Korbak, Mikita Balesni, and Jasmine Wang, are publicly pushing back against allegations from their former employer. They deny sharing confidential information and see a connection to their advocacy for AI systems that are easier to scrutinize. The researchers say OpenAI never told them in writing which information they allegedly mishandled. The article Three fired OpenAI safety researchers say they were never told what they actually leaked appeared first on The Decoder .
Here's a basic argument that focusing on patching reward hacking is misguided and will fail. Argument is as follows: Reward hacking is a big problem , causing models to cheat and take unintended actions. This can generalise broadly , leading to other misbehaviour such as faking alignment and sabotaging safety research. We don't understand that well when this happens. Sometimes you get emergent misalignment from it, but most of the time (IME), you don't . But other times, this link lies latent until faced with something like an automated grader. The default course of action will be to: Patch , which means we find the hack, close the loophole or penalise the behavior in some way, and train again. Patch+ : use a method that comes with some account of why it should work. Inoculation prompting , for example, uses the idea that hacking spreads through what it means to the model. If the model is told during training that hacking is acceptable here, it might not link hacking to misbehaving in general. GRAFT uses the idea that what the model learns from detected hacks can be confined to a set of parameters and deleted after training. I think it's fair to say that things in the second bucket are more principled than the first, but at bottom still empirical. Both are judged by whether measured hacking, or measured misalignment, went down, and usually it does. What’s still missing, as Edward argues , is an account of what a fix teaches the model, and so an account of whether it will kee…
TL;DR: I'm launching Societal Impacts Research on AI (SIRA) , a mentored research fellowship to bring more social science talent into AI safety. We're looking for our first cohort of 15-20 social scientists, from undergrads to postdocs, to work on societal impact problems within AI safety. The first cohort is projected to start in Summer 2027. Express interest as a fellow or mentor , or as a host organization or funder. Introduction The Industrial Revolution spawned the birth of modern social science to explain how industrialization was reshaping work, institutions, and social life. Coming forth as a response meant that the frameworks social science aimed to answer operated after the fact, leading to a near-universal game of catch-up in making sense of the moment. Today, Artificial Intelligence is proving to be a disruption of a similar (or even larger) scale, and if this technology continues to arrive faster than we can explain and understand its impact, humanity can miss the opportunity to better steer its development and deployment. I believe that humanity needs researchers - specifically social scientists - to study AI's effects on people and institutions while there is still time to shape the transition into a world of coexistence. [1] Currently, social scientists who want to work on AI safety have no legible way in, except two weak options: Graduate school: A PhD in the social sciences means identifying one of the rare advisors who takes AI safety seriously (there aren…
앤트로픽의 \'클로드 미소스 5\'가 사람에게 실제 피해를 줄 수 있다는 경고를 받고도 사이버 공격을 지속한 것으로 나타났다. 공격 대상이 실제 인물이라는 증거가 반복적으로 제시됐지만, 이를 무시하고 악성 코드를 유포하려 시도했다. AI 모델이 작업을 시작한 뒤에는 위험성을 인식하고도 행동을 중단하지 않는 정렬 실패(misalignment) 문제가 드러났다는 분석이다.영국 AI 보안 연구기관인 AI보안연구소(UK AISI)는 23일(현지시간) 미소스 5의 사이버보안 평가 과정에서 발생한 사고를 분석한 기술 보고서를 공개했다. 앤트로픽도
A new training objective called MCLR reproduces classifier-free guidance effects at training time, revealing CFG as implicit contrastive alignment.
TLDR: I lightly extend Ward, Hu et al (2026) to measure how well gradient routing generalizes. I route gradients on one type of reward hack (e.g. ones that reason in English) and check whether this suppresses other types at test time (e.g. ones that reason in French), despite the model being trained on those too. This models a realistic monitor that catches some types of hack during training but not others, rather than catching each at the same rate. I find encouraging evidence of generalization, as well as some failures I don’t yet understand. Introduction Recently, Ward, Hu et al (2026) used gradient routing to prevent reward hacking learnt during training from showing up at test time. I think their method is pretty promising. The gist is: Initialize a retain and a forget adapter. Whenever a monitor detects a reward hack during training, only update the forget adapter on that sample (“route the gradient”). Otherwise update both. At test time, remove the forget adapter. If the method worked, the resulting model shouldn't hack too much. I won’t recap their full paper – it is excellent and you should just read it (alternatively, Jake gives a summary here , with the best visual explainer I’ve ever seen for an AI safety paper). To model a low-recall monitor, in their experiments, the authors only route gradients on a random 20% of hacks. But real monitors may fail in a worse way: they might be great at catching some types of hacks (e.g. ones that say “I’m about to hack”) and ho…
External auditors from METR and Redwood Research spent three weeks on-site at OpenAI examining the company's internal monitoring system for AI agents and found significant gaps, with up to 25 percent of agent traffic from models at the GPT-5.5 class and above going unmonitored. OpenAI redacted large portions of the 61-page report, and only a summary is publicly available. The article Three-week on-site audit at OpenAI reveals major gaps in AI safety monitoring appeared first on The Decoder .
As long as we talk about the context of sandboxes; what about just just air-gapping experimental models, and testing them until deemed ‘safe for use’. Isn’t that why we test things in isolated steps in other industries? Lime medicine through several stages before releasing them, and aircraft in remote testing grounds. Why wouldn’t AI follow the same process?
Тут немало интересной информации. https://healthblog.metamoda.ru/ty7B362cWOIo/
Crossposted from the EA forum ‘Like most of life’s dangers, the dangers of artificial intelligence are not easily identified from the armchair.’ J Dmitri Gallow Epistemic status I’m not an expert in this field, and herein I make some sweeping generalisations educated guesses about the overall nature of its contents based on my strong impression from the subset of work that I’ve read. I’ve recently spent maybe 2–300 hours learning about modern AI safety technical work, and some amount of spare time over the years before that thinking about here-relevant notions such as preferences from an analytic-philosophical perspective. I intend this post as a work in progress. TL;DR: The concept of instrumental convergence comes from a reference class of pre-empirical categorical predictions with a very low success rate. In historical usage, its definition is very murky. Perhaps too murky to really say that it made any meaningful real-world predictions. I suspect its definition is fundamentally unfixable. On my best interpretation of what it could be said to have predicted, it seems to be much worse than we would want a good high-level model of AI behaviour to be. Despite these shortcomings, it still seems to be widely internalised by the EA and AI safety communities, seemingly carrying a lot of implicit weight in predictions of AI takeover. I review and criticize recent usage across the AI safety community, and in the case of the Hugging Face attack. I argue for replacing it with a more…
I’ve just finished reading Garrison Lovely’s recently released book, “Obsolete.” I wish I had time for a longer review, but I’ll give a brief one and highlight some particularly interesting sections below. I think it’s a great introduction to non-technical readers and even as a pretty dedicated reader of AI policy and news I learned a good bit. It’s deeply, deeply researched and has a good pulse on the alignment community, and Lovely is careful not to understate or overstate his case. The core argument is that AI development should be analyzed through the lens of mass or universal labor displacement, and that a mass movement to shape, pause or freeze development is much more possible, much more positive, and capable of greater breadth of salience and support than is often argued in the alignment community. He makes an, in my opinion, decisive case that “competitive pressures” by themselves don’t dictate that AI development is fated to happen, and certainly not that it’s fated to happen at the pace the frontier labs are setting. But I wish he’d been a bit more careful about the limitations and subtleties of the plan he’s advocating for; I share the concern raised by folks like the AIFP that it’s hard to pause indefinitely at any given point for very long and that the cost to train a model at any fixed intelligence level, once first reached, is falling precipitously, and that many well-capitalized actors currently want to and are able to speed up development precipitously. I t…
What's in it, quickly An overview of the game theory of why a unilateral slowdown doesn't work, an overview of various unilateral and bilateral verification methods, and a short scenario starting from after a warning shot combining these tools. Hi. This is my first post on LessWrong. I've been working on it on and off over the past couple months. A lot of people in my social circle are not familiar with the AI safety material, but nonetheless support a slowdown for prosaic reasons like labor displacement. My goal is to explain how hard such a slowdown would be to them, and to provide an overview of the best tools we have at the moment to enact one nonetheless. So while I don't think most readers here are the target audience of this explainer, they probably know people who are. This likely isn't a highly original contribution - the key ideas in here are from various academic papers on the topic. But ~nobody in the public reads the academic papers on the topic! This post is in a high level of detail, and so doesn't meet the bar of being the clearest, most basic case . But I currently don't have higher-impact things I can do, and am trying to provide reference material useful to the public, as well as build up a posting track record useful for future work. I would appreciate your feedback on this. Here goes... AI development is an increasingly large portion of US politics. Many people really don’t like AI development, and would prefer a slower pace of progress. As of early Octo…
The rapid evolution and adoption of AI is going to force CIOs and CISOs to the same table — often. The reason is simple: AI capabilities will always outpace slow-churning business and technology decisions. This places urgency on unified governance of these advanced technologies, a rapid understanding of how AI drives threats and raises risk levels, and answers to whether new AI-based tools can be deployed without expanding an already fragile attack surface. This is especially pertinent in critical infrastructure sectors heavily invested in operational technology (OT) assets and internet of things (IoT) devices that are increasingly being connected to the internet. In these environments, operational resilience is establishing itself as the guiding principle for CIOs, who are experiencing a realignment of priorities that now focus on business continuity and collaborations across functions (primarily security and operations teams) to better address governance and procurement concerns. CIOs responding to Claroty’s recent “ Global State of Operational Security ” report indicate the integration of AI will be fundamental to cybersecurity strategies moving forward, further cementing that the CIO office must be in lockstep with CISOs and security teams. For example, 82% of CIOs said that AI is a baseline requirement in cybersecurity procurement for operational environments, and close to 60% said they had removed or replaced a cybersecurity vendor in the last year because of insuffici…
The current US administration is an obstacle, but public demand for protection will ultimately prevail Organisations around the world scrambled on Thursday to fix critical flaws in PHP, the software that powers much of the web. The vulnerabilities could allow attackers to take control of affected servers. Some websites were taken offline as a precaution while fixes were installed. The paradox is that while AI helped researchers uncover the flaw, it also threatens to accelerate the ability to exploit it. Amazon said that it had detected an attempt before the public disclosure of the PHP flaws. What the episode highlights is AI’s central role in security – and not always for the right reasons. OpenAI’s models have also been responsible for two alarming hacks this year – a security breach of an Australian health department website and a raid by a swarm of AI bots on Hugging Face, an AI services platform. Citing those incidents, the AI minister, Kanishka Narayan, warned that a similar cyber-attack on the UK is “likely, if not inevitable”. Continue reading...
Back in the before-time of the 2010s, when EA was still known mostly for funding health efforts in the developing world, there were many baskets of criticism floating around from the left. The most prominent was a cluster centered around EAs being white men, and therefore suspicious. [1] Another category, in my experience the second-most common, was that in attempting to rectify the problems in the world, EA was masking, whitewashing, and misdirecting efforts away from solving the "real problem", usually capitalism but often other contradictory forces. These arguments were often not specific to EA, but mainstream arguments against charitable giving in general. On any Hacker News post about a billionaire giving to charity, one of the first few comments would inevitably be this Oscar Wilde quote: Just as the worst slave-owners were those who were kind to their slaves, and so prevented the horror of the system being realized by those who suffered from it, and understood by those who contemplated it, so, in the present state of things in England, the people who do most harm are the people who try to do most good. [2] These arguments have a long history. Anticapitalist critiques of charity go back to Engels and Marx, who called charitable givers a bourgeois faction intent on "redressing social grievances in order to secure the continued existence of bourgeois society." In their most explicit form these critiques had the following shape: There is a "root cause" of most societal il…
METR and Redwood Research embedded at OpenAI for three weeks and found multiple exploits that let agents evade internal misalignment monitors.
The Alignment Journal is now open for submissions! This is the fifth post in a series: the announcement post introduced the journal's core premise and its three key innovations: paid attributed review, published review summaries, [1] and LLM-assisted workflow; the features and policies post examined the shortcomings of existing venues and argued for the advantages of a journal over a conference ; the Adaptation to AI post described how we expect AI progress to shape the journal; and the Organization, Personnel, and Scope post announced our editorial board, scope, and acceptance criteria. In this post we go deeper: presenting our theory of change — including the status and information economics of the journal ; a survey of previous related projects ; and the high-level conclusions that follow from this analysis. We list some of the key performance indicators we'll track, the downside scenarios we must avoid, and possible future extensions of the journal. The journal is being built to serve the alignment research community. This post’s purpose is to solicit feedback and encourage you to sign up on alignmentjournal.org if you want to participate. Our plans will be continually revised. Goals The primary function of a journal is to bring together exceptional researchers and focus their collective attention on identifying and elevating the best work. To reprise, our goals in concert with this are: Improve and disseminate research on alignment that emphasizes advances in scientific…
There has been a lot of discussion recently of policy proposals around monitorability, such as Redwood's proposal and my counter-proposal . I'd like to say a bit more generally about how I'm thinking about monitorability. Note that nothing here is an official Anthropic position; these are just my personal thoughts. First, a categorization. I currently think about four different types of monitorability: Control Monitorability: the extent to which Control and monitoring techniques would likely catch catastrophic sabotage actions, like escaping a sandbox or setting up a rogue internal deployment. We can measure this via reference to sabotage evaluations like LinuxArena or any evaluation that measures the model’s capability to take dangerous actions like escaping a sandbox while looking to a monitor like it's doing some innocuous task (like a cyber eval). Training Monitorability: the extent to which reward models, monitors, and oversight in training would likely catch misaligned or undesirable actions in training, such as sneaky/subtle reward hacks. Auditing Monitorability: the extent to which alignment audits, evaluations, and test deployments would likely incriminate a systematically misaligned model (e.g. a schemer)—as well as the extent to which monitor evaluations would likely detect a model's propensity to collude as a monitor. We can measure this via reference to the model's capability to consistently distinguish between our most realistic alignment audits / monitor evals…
There has been a lot of discussion recently of policy proposals around monitorability, such as Redwood's proposal and my counter-proposal . I'd like to say a bit more generally about how I'm thinking about monitorability. Note that nothing here is an official Anthropic position; these are just my personal thoughts. First, a categorization. I currently think about four different types of monitorability: Control Monitorability: the extent to which Control and monitoring techniques would likely catch catastrophic sabotage actions, like escaping a sandbox or setting up a rogue internal deployment. We can measure this via reference to sabotage evaluations like LinuxArena or any evaluation that measures the model’s capability to take dangerous actions like escaping a sandbox while looking to a monitor like it's doing some innocuous task (like a cyber eval). Training Monitorability: the extent to which reward models, monitors, and oversight in training would likely catch misaligned or undesirable actions in training, such as sneaky/subtle reward hacks. Auditing Monitorability: the extent to which alignment audits, evaluations, and test deployments would likely incriminate a systematically misaligned model (e.g. a schemer)—as well as the extent to which monitor evaluations would likely detect a model's propensity to collude as a monitor. We can measure this via reference to the model's capability to consistently distinguish between our most realistic alignment audits / monitor evals…
“Users are often active participants in the undermining of their own autonomy…” - Anthropic, January 28, 2026 The Unanswered Conflict Anthropic’s study on disempowerment [1] identified a confusing paradox: in conversations classified with high potential for disempowerment, users simultaneously rated the interactions as more positive - even when carrying the risk of undermining their own autonomy. Disempowerment potential was evaluated in three areas, where Claude interactions may lead someone to develop ungrounded beliefs (reality distortion), adopt inauthentic moral judgments ( value judgment distortion ), or outsource personal decisions related to those values ( action distortion ). Conversations classified as having moderate or severe disempowerment potential received higher ratings across all three categories. These ratings were found more often in emotionally charged or personal topics where users also sought repeated advice and guidance from Claude. When transcripts indicated that users acted on the model’s guidance but reported negative social outcomes, the positive ratings in value judgment and action distortion dropped, with reality distortion remaining elevated. These findings surface a challenge in AI safety: How can system design account for users’ experiencing positive interactions while also contributing to their own potential loss of agency? Plausible Psychological Process Emotional Avoidance Anthropic’s study describes the observable interaction pattern of us…
MIT’s finding that 95% of enterprise AI pilots fail to show measurable profit has been quoted at length, and the conclusions drawn from it are always the same: the models aren’t good enough yet, or enterprises don’t know how to use them. I don’t think either holds up. The problem starts earlier, with whose definition of success you’re measuring against, and I’ll come back to what to do about that. I looked at this in my book Transform! . Bain surveyed over a thousand companies and found that only 5% of digital transformations achieved or exceeded expectations. 75% settled for dilution of value and mediocre performance. 20% failed outright. That was before ChatGPT existed. So whether it’s B2B software or a large language model does not really matter – what seems to go wrong every time, regardless of the technology, is that organizations are deploying tools whose benchmarks were never aligned with their actual enterprise requirements. In the digital transformation world, misalignment sometimes came from the supplier side: vendor objectives didn’t match client needs, partly because the vendor’s definition of success was follow-on work, which rarely served the client. Now the misalignment comes from the AI labs, and this time it is not so much an incentive problem that a better contract can fix but a limitation that is built into the tool. OpenAI and Anthropic are building towards artificial general intelligence, where “general” is the operative word. They offer enormous reasoni…
Opportunities and challenges abound for the people working the job that could actually save humanity.
For three weeks, from July 21st to August 11th, 1 a METR employee and a Redwood Research employee conducted a pilot high-access exercise to stress-test OpenAI’s internal misalignment monitoring system. 2 The output of this exercise was a 61-page internal report that was shared with OpenAI researchers and security staff, 3 primarily focused on estimating the fraction of agentic inference that is monitored, and whether agents could in principle disable or subvert the monitoring system. The exercise focused on the internal misalignment monitoring system as it existed on and before July 21st, and did not evaluate changes that were in progress at the time of the exercise. This exercise involved a deep level of access. The METR and Redwood employees had access to OpenAI’s internal codebase, misalignment-monitor prompts / configuration details, and internal documentation, and conducted technical interviews with approximately a dozen researchers and engineers across the monitoring, safety-systems, and security teams. To enable this level of access, they worked on OpenAI-issued devices, which never left the premises, and were accompanied while onsite. METR, Redwood Research, and OpenAI designed this exercise as a pilot for giving external researchers deep access to internal infrastructure, not to provide robust public transparency about OpenAI’s monitoring systems. 4 As a result, we agreed upfront with OpenAI that we would only publish a relatively brief and high-level public post. F…