A community-maintained quantization has made Meta’s Llama 3.3 70B Instruct practical to serve on hardware with about 48 GB of VRAM. The AWQ checkpoint compresses most model weights to 4 bits and works with vLLM, SGLang, and recent Transformers releases.

Hugging Face has recorded hundreds of thousands of downloads for the repository in recent monthly windows. Those counters include automated file requests and repeated downloads, so they measure distribution activity rather than unique production deployments. The practical signal is broad runtime support: developers can use established serving stacks without converting the model themselves.

How four bits shrink 140 GB of weights

Activation-aware Weight Quantization, or AWQ, calibrates the model on sample inputs to identify weight channels whose errors have the greatest effect on output. It scales those channels before applying groupwise 4-bit quantization, reducing error while keeping the weight tensors compact. Activations and the key-value cache typically remain in FP16 or BF16.

The checkpoint was created with the AutoAWQ project. Its safetensors files include quantization metadata that compatible engines use to select AWQ kernels. Kernel availability and performance still depend on the GPU, CUDA version, and serving framework.

AWQ reduces weight storage and memory bandwidth. Runtime workspaces, activations, and the KV cache still consume VRAM, which makes Meta’s 128k context limit expensive to use. A single 48 GB RTX A6000 can usually serve shorter contexts at low concurrency, while the full context generally requires additional GPUs or a lower-precision cache.