Google DeepMind consolidates its point-tracking stack in TapNet
Google DeepMind’s TapNet repository has climbed GitHub’s trending list while serving as the official home for the lab’s Tracking Any Point models, benchmarks, checkpoints, and demos. For video and robotics developers, the repository replaces a search across separate papers and implementations with one maintained codebase.
With roughly 2,200 GitHub stars, the project includes JAX and PyTorch code, pretrained checkpoints, eleven Colab notebooks, TAP-Vid evaluation tools, and a live webcam demo. Support varies by model, so deployment choices still depend on the required framework, latency, resolution, and tracking horizon.
A pixel becomes a trajectory
Given a video and one or more query points placed on any frame, a Tracking Any Point model estimates the corresponding 2D location on every other frame. It also predicts whether each point remains visible, allowing a track to disappear behind an object and reappear later.
A TAP system combines pixel-scale localization, deformable-surface tracking, long-range association, and occlusion handling. A query can sit on fabric, masonry, skin, or a robot gripper, with the resulting trajectory describing how that precise surface location moves through the clip.
- Input: a video plus query points defined by frame and image coordinates.
- Output: per-frame coordinates and visibility estimates for every query.
- Modes: offline tracking can inspect the full video, while causal tracking processes frames as they arrive.
- Applications: robot imitation, video editing, dynamic 3D reconstruction, motion analysis, and generative-video evaluation.