The J-lens intermediate swap acts partly through the answer directions

TL;DR When a language model answers a multi-hop prompt, does it first compute the intermediate concept internally, or does it jump straight to the answer through some shortcut? The global workspace paper investigates this question using the Jacobian lens, which maps activations at a given layer to the tokens that they are poised to verbalize, with the activations possessing a coordinate in each token’s lens direction. The authors intervene on the model through an intermediate swap that exchanges the activation’s J-Lens coordinates for the intermediate and a counterfactual intermediate. They show that this causally flips the model’s output toward the counterfactual answer, and argue that this flip runs through the intermediate concept because the swap takes effect earlier than a direct answer swap. We reproduce this in eight open-source models with published Neuronpedia lenses; however, we challenge whether the intermediate swap really acts through the intermediate, or through its overlap with the answer. To investigate this, we save the edit that the intermediate swap applies at each layer and prompt position, decompose it into its component lying in the answer-swap plane and the remainder orthogonal to this plane, and apply each component on its own. The answer-swap plane is spanned by the lens directions for the original and counterfactual answers. We also expand the evaluation beyond the paper’s 90 released two-hop prompts with a new set of 1,526 candidate two-hop prompts…