Three Pieces of Evidence that the NLA Reconstructor Does Not Read Semantics or Syntax

TL;DR The autoreconstructor (AR) in a natural language autoencoder (NLA) is the entire argument for the validity of the method, and at present is poorly understood. The AR does not condition strongly on English grammar or semantics. Three bits of evidence: Word pairs that are syntactically related interact no more than matched pairs that are not syntactically related. See dependency arcs . The fraction of variance explained (FVE) lost by removing or flipping a negation word ("not" -> ""; "without" -> "with") is only weakly correlated with the FVE lost by ablating the negated word (Spearman rho 0.38). See negation . The word classes that carry the most meaning, nouns (forest), adjectives (heavy) and verbs (reported), cost the least FVE to delete; auxiliaries (is), pronouns (it), numerals and subordinators cost the most. See word class . Results are consistent with prior work suggesting that the AR is more heavily influenced by the format of the verbalisation (tone, register, general domain, structure of text) than by semantic meaning per se. The problem with NLAs NLAs have strong empirical performance on an automated auditing task . A model given access to a NLA was able to uncover the root motivation of a reward-sycophancy model organism more consistently than without. The NLA provided marginal benefit to an auditing agent with access to the full pre-training, supervised fine-tuning and reinforcement learning corpora, as compared to a sparse autoencoder which offered no bene…