Foundational premises of advanced AI

My goal here is to establish a shared baseline (or model) for thinking about AI. I've found that disagreements about AI policy/governance/alignment will often trace back to unstated divergence on base-level facts. A. Machine Learning - how can AI models do things we didn't program them to? Modern AI models are not programmed behaviour-by-behaviour. Engineers write the code that governs the process by which a network of parameters finds associations between data that it is given ("learns"). A model's behaviour is learned from data and feedback rather than explicitly coded. In this way we can produce useful ("intelligent") behaviour without knowing how to program it directly. Machine learning is a deceptively simple mix of: algebra and huge amounts of data. This enables AI models to improve at predicting patterns in training data. The process is akin to teaching through trial & error. Where a lot of clear examples are used, AI models perform very well. Anything that can be measured (even poorly) can become the basis of training an AI model (i.e. specified as an objective or provided as a feedback signal). B. Scaling - why might AI models continue improving quickly? AI Models improve with better data, compute, or algorithms (the AI triad). We don't know the upper limits of current architectures. The "bitter lesson" in ML is that more compute tends to beat attempts to hard-code human knowledge. In the short term there are practical (physical and economic) limits (e.g. how quickl…