The Apache-2.0-licensed llama-modchip project combines ggml and llama.cpp kernel patches, custom quantization tools, and runtime utilities for large mixture-of-experts models. Its headline benchmark runs a checkpoint named DeepSeek-V4-Flash entirely on a 16-core AMD Zen 5 CPU with 64 GiB of installed memory and no GPU offload.

A mixture-of-experts model stores many specialized feed-forward networks, then routes each token through a small subset. DeepSeek-V4-Flash therefore retains 284.33 billion parameters in memory while using 13.17 billion for each token. Prefill measures how quickly the runtime processes an existing prompt; decode measures serial generation after that prompt has been ingested.

Terminal dashboard showing DeepSeek-V4-Flash running through llama-modchip on a CPU
The project’s terminal dashboard reports CPU-only inference for the 284B checkpoint.

Where the bytes go

The unmodified UD-IQ1_S build occupies 76.87 GiB, exceeding the roughly 62.5 GiB available to applications on the project’s 64 GiB Linux test system. That excess can trigger swapping, page-cache churn, or an out-of-memory termination. llama-modchip’s transcoder reduces the checkpoint to 56.56 GiB, leaving enough space for runtime state and a 256K-token KV cache. The benchmark reports zero swap use and a resident set between 57.61 and 58.90 GiB.

The transcoder assigns formats by tensor type, balancing size against sensitivity. Bits per weight, abbreviated as bpw, describes the average storage used for each quantized parameter.