Apple’s DFN5B CLIP brings learned data curation to vision models

Apple’s DFN5B-CLIP-ViT-H-14-378 has accumulated nearly 800,000 Hugging Face downloads and powers dozens of Spaces. Hugging Face counts file requests rather than unique users, but the activity shows substantial adoption of a model released without a broad product launch.

The checkpoint packages a ViT-H/14 CLIP model with a 378-pixel input size. CLIP models map images and text into the same vector space, allowing developers to classify images from text prompts, retrieve images from natural-language queries, cluster visual collections, and supply features to multimodal systems. Apple’s model differs from common open CLIP baselines through its training data: a learned network selected five billion image-text pairs from a much larger candidate pool.

  • Architecture: ViT-H/14 image encoder with a CLIP text encoder
  • Input size: 378 by 378 pixels
  • Training set: DFN-5B, filtered from roughly 43 billion candidate pairs
  • Interface: Compatible with OpenCLIP
  • License: Apple Machine Learning Research license

A learned filter picks five billion pairs

The DFN paper, written by researchers from Apple and the University of Washington, treats data selection as a learned component of the training pipeline. Conventional web-scale datasets often rely heavily on language detection, metadata rules, similarity thresholds, deduplication, and manually chosen quality signals. Data Filtering Networks use a trained model as the central scorer for deciding which image-text pairs enter the final dataset.

The experiments found that downstream classification accuracy poorly predicts filtering quality. Some networks with strong ImageNet results produced weaker training sets, while lower-accuracy networks trained on smaller, carefully selected datasets became better filters. The useful property was the network’s ability to rank candidate pairs by training value, rather than its ability to serve directly as the final classifier.

Apple applied the filter to approximately 43 billion uncurated pairs, including 12.8 billion from CommonPool-12.8B and another 30 billion public image-text pairs. The resulting DFN-5B dataset contains five billion selected pairs. According to the paper, training the filtering network cost less than one-fiftieth of the subsequent CLIP training run.

The downloadable checkpoint contains the final CLIP encoder trained on DFN-5B. Reproducing the curation pipeline still requires a candidate corpus, the paper’s filtering procedure, and the infrastructure to score billions of pairs.