Linear probes tell you where quantization will hurt

Epistemic status: I have only tested one encoder family (BERT-base and its relatives) and one decoder LLM ( Qwen2.5 -3B), one seed, token-level tasks, and post-training weight quantization. I trust the results because I am not inventing anything; I am just connecting quantization with a very general idea from mech interp: the model does the easy syntax work first, the semantic work in later layers, and the prompt-relevant task work in the latest layers. That's the main idea I lean on in this project. What I'm not sure of is whether it holds when the task is diffuse, like general-knowledge QA (probably not). TL;DR. First I train a linear probe on each layer and try to see where a signal lives in a model. I only probe for classical NLP or CV tasks, like checking for depth or vision for CV or named entity recognition, parts of speech, or chunking for NLP. Then using that information to create a map of where the important work is happening and quantizing the less important layers for a task. This method is cheap; it's only one RidgeClassifier per layer. And it kind of works; a map built once on CoNLL news keeps 99–100% of full-precision accuracy at a 5-bit average on three unseen datasets, whereas Uniform kept 16–41% and opposite/anti layers kept 2–41%. This method is perfect for sharp use cases where the transformer needs specialized knowledge and can compromise on general knowledge. The itch So this all started during my 2nd semester at Northeastern, where I took Applied Progr…