Shrinking AI with Flash models

Some of the more interesting work MAI has been doing recently is starting to ship as new “Flash” models, which work locally or in the cloud. This approach can lead to significant savings, while offering similar fidelity to cloud-only models. At the same time, using local inference on NVIDIA RTX-class hardware uses significantly less power than data center hardware, lowering the environmental impact. Even though Flash models are smaller, they’re described as foundational models.

Flash models like MAI-Code-1.1-Flash aren’t yet small enough for the relatively low-cost Arm-based hardware that Microsoft has branded as Copilot+ PCs, as they need more than 120GB of RAM to run. However, new hardware like the NVIDIA RTX Spark-powered Surface Ultra, when configured with at least 128GB of RAM, should be more than capable. You will then need to consider the amount of agent usage, to determine whether an expensive device like Surface Ultra makes local operation more economical than in cloud servers.

MAI-Cyber-1.1-Flash has a quite different role, providing offload capabilities to the agents that power the MDASH cybersecurity platform. Here the Flash model is used to manage 90% of the tasks that used to be delivered by much larger and more expensive models. This makes the service more economical to run, without affecting performance. When an agent powered by the Flash model can’t handle an operation, it passes it on to another, more powerful model. The MAI-Cyber-1.1-Flash release also reduces operational costs over the 1.0 version, as it’s 18% cheaper to run.