InferSSD is an open-source inference runtime that uses NVMe storage as another memory tier, allowing a laptop with 48 GB of unified memory to execute a 763-billion-parameter mixture-of-experts model. Built on llama.cpp, it streams selected expert weights from SSD, reads GGUF model files, and provides terminal chat, browser chat, and an OpenAI-compatible API.

The project gives developers local access to models that would otherwise require hundreds of gigabytes of RAM or VRAM. Its reported results favor prompt processing over token generation, making the current release more useful for experiments and long-input workloads than latency-sensitive or concurrent serving.

MoE routing shrinks the working set

Mixture-of-experts models contain many specialized subnetworks, called experts, at each layer. A router selects only a small number for each token, so inference touches a fraction of the stored weights during any forward pass. InferSSD keeps inactive experts on NVMe and loads selected experts as inference proceeds.

Total parameter count describes every weight stored in the checkpoint, while active parameter count approximates the weights used for one token. That distinction explains how a 763B model with 8B to 16B active parameters can run on hardware that cannot hold the full checkpoint in memory.

Current compatibility remains model-specific despite the use of standard GGUF files. Both supported checkpoints are community conversions hosted on Hugging Face and can be downloaded directly through InferSSD, without a separate conversion step.

DeepSeek’s Q8_0