By Hongyang Li, Hao Zhang, Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, and Lei Zhang ^📧.

[2024/8/2] We add memory efficient mode for inference to satisfy more users, please refer to this issue for more details.

[2024/7/24] TAPTRv2 is released, check out our paper for more details!

[2024/7/16] Training, evaluation, and demo codes have been released but have not been cleaned yet. Hope these codes can be helpful to you.

[2024/7/9] TAPTR is accepted by ECCV2024.

[2024/3/15] We release our paper.

• Release paper.
• Release online demos.
• Open-source evaluation and demo code.
• Training code.
• Clean code. (45% cleaned).

In this paper, we propose a simple and strong framework for Tracking Any Point with TRansformer (TAPTR). Based on the observation that point tracking bears a great resemblance to object detection and tracking, we borrow designs from DETR-like algorithms to address the task of TAP. In the proposed framework, each tracking point is represented as a DETR query, which consists of a positional part and a content part. As in DETR, each query (its position and content feature) is naturally updated layer by layer. Its visibility is predicted by its updated content feature. Queries belonging to the same tracking point can exchange information through temporal self-attention. As all such operations are well-designed in DETR-like algorithms, the model is conceptually very simple. We also adopt some useful designs such as cost volume from optical flow models and develop simple designs to mitigate the feature drifting issue. Our framework demonstrates strong performance with state-of-the-art performance on various TAP datasets with faster inference speed.

We develop and test our method under python=3.8.18,pytorch=1.13.0+cu117,cuda=11.7. Other versions might be available as well.

Construct the dataset as in CoTracker, and put it at

kubric data (for training): ./datas/kubric_movif/
tapvid data (for evaluation): 
    ./datas/tapvid/tapvid_davis
    ./datas/tapvid/tapvid_kinetics
    ./datas/tapvid/tapvid_rgb_stacking

git clone https://github.com/IDEA-Research/TAPTR.git
cd TAPTR
pip install -r requirements.txt
cd models/dino/ops
python setup.py install # This compilation requires nvcc, please make sure you have installed CUDA correctly. CUDA11.7 is tested.

Download our provided checkpoint, and put it at "logs/TAPTR/taptr.pth"

# Select the dataset you want to evaluate in evaluate.sh manually. 
bash evaluate.sh

# Point Trajectory Demo
CUDA_VISIBLE_DEVICES=0 python demo_inter.py -c config/TAPTR.py --path_ckpt logs/TAPTR/taptr.pth
# Video Editing Demo
CUDA_VISIBLE_DEVICES=0 python demo_inter_video_editing.py -c config/TAPTR.py --path_ckpt logs/TAPTR/taptr.pth

bash dist_train.sh

We would like to thank TAP-Vid and Co-Tracker for publicly releasing their code and data.

@inproceedings{
  title={TAPTR: Tracking Any Point with Transformers as Detection},
  author={Hongyang Li and Hao Zhang and Shilong Liu and Zhaoyang Zeng and Tianhe Ren and Feng Li and Lei Zhang},
  booktitle={Proceedings of the IEEE/CVF European Conference on Computer Vision},
  year={2024}
}
@inproceedings{
  title={TAPTRv2: Attention-based Position Update Improves Tracking Any Point},
  author={Hongyang Li and Hao Zhang and Shilong Liu and Zhaoyang Zeng and Feng Li and Tianhe Ren and Bohan Li and Lei Zhang},
  journal={arXiv preprint arXiv:2407.16291
  },
  year={2024}
}