git clone https://github.com/nota-github/np_app_VOTS2023.git
cd np_app_VOTS2023
docker build -t notadockerhub/np_app_vots2023:latest -f ./Dockerfile .
docker run --name container_name --shm-size=8g -it --gpus all -v /your_data_dir:/workspace/datasets notadockerhub/np_app_vots2023:latest
cd np_app_VOT2023
mkdir external/vots2023
cd external/vots2023/
mkdir val
mkdir test
cd val
vot initialize tests/multiobject
cd ../test
vot initialize vots2023
mkdir models
cd models
wget https://netspresso-application-public.s3.ap-northeast-2.amazonaws.com/mixformer_models.zip
unzip mixformer_models.zip
mkdir external/AR/ltr/ARcm_seg/
mkdir external/AR/ltr/ARcm_seg/ARcm_coco_seg_only_mask_384/
cd external/AR/ltr/ARcm_seg/ARcm_coco_seg_only_mask_384/
wget https://netspresso-application-public.s3.ap-northeast-2.amazonaws.com/ARnet_seg_mask_ep0040.pth.tar
python tracking/create_default_local_file.py --workspace_dir <np_app_VOTS2023 dir path> --data_dir <data> --save_dir <np_app_VOTS2023 dir path>
export PYTHONPATH=<absolute_path_of_AR>:$PYTHONPATH
python -c "from pytracking.evaluation.environment import create_default_local_file; create_default_local_file()"
python -c "from ltr.admin.environment import create_default_local_file; create_default_local_file()"
cd ./external/AR/pytracking/VOTS2023
git clone https://github.com/SysCV/sam-hq.git
wget https://huggingface.co/lkeab/hq-sam/resolve/main/sam_hq_vit_h.pth
cd sam-hq
pip install -e .
The vision object tracking model published in this repository was developed by combining several modules (e.g., object tracking model, image segmentation model). Commercial use of any modifications, additions, or newly trained parameters made to combine these modules is not allowed. However, commercial use of the unmodified modules is allowed under their respective licenses. If you wish to use the individual modules commercially, you may refer to their original repositories and licenses provided below.
Object tracking model (license) link : Github, License
Image segmentation model (license) link : Github, License
The official implementation of the CVPR 2022 paper MixFormer: End-to-End Tracking with Iterative Mixed Attention
[Models and Raw results] (Google Driver) [Models and Raw results] (Baidu Driver: hmuv)
[Feb 10, 2023]
- 🔥🔥🔥 Code and models for MixViT and MixViT-ConvMAE are available now ! Thank Tianhui Song for helping us clean up the code.
[Feb 8, 2023]
- Extended version has been available at https://arxiv.org/abs/2302.02814. In particular, the extented MixViT-L(ConvMAE) achieves AUC score of 73.3% on LaSOT. Besides, we design a new TrackMAE pre-training method for tracking. Code and models will be updated soon.
[Oct 26, 2022]
- MixFormerL (based on MixViT-L) rank 1/41 on VOT2022-STb public dataset.
- The VOT2022-RGBD and VOT2022-D winners of MixForRGBD and MixForD, implemented by Lai Simiao, are constructed upon our MixFormer.
- The VOT2022-STs winner of MS-AOT employs MixFormer as a part of the tracker. The VOT2022-STb winner of APMT_MR employs the SPM proposed in MixFormer to select dynamic templates.
[Mar 29, 2022]
- Our paper is selected for an oral presentation.
[Mar 21, 2022]
- MixFormer is accepted to CVPR2022.
- We release Code, models and raw results.
MixFormer is composed of a target-search mixed attention (MAM) based backbone and a simple corner head, yielding a compact tracking pipeline without an explicit integration module.
Mixformer is an end-to-end tracking framework without post-processing.
| Tracker | VOT2020 (EAO) | LaSOT (NP) | GOT-10K (AO) | TrackingNet (NP) |
|---|---|---|---|---|
| MixViT-L (ConvMAE) | 0.567 | 82.8 | - | 90.3 |
| MixViT-L | 0.584 | 82.2 | 75.7 | 90.2 |
| MixCvT | 0.555 | 79.9 | 70.7 | 88.9 |
| ToMP101* (CVPR2022) | - | 79.2 | - | 86.4 |
| SBT-large* (CVPR2022) | 0.529 | - | 70.4 | - |
| SwinTrack* (Arxiv2021) | - | 78.6 | 69.4 | 88.2 |
| Sim-L/14* (Arxiv2022) | - | 79.7 | 69.8 | 87.4 |
| STARK (ICCV2021) | 0.505 | 77.0 | 68.8 | 86.9 |
| KeepTrack (ICCV2021) | - | 77.2 | - | - |
| TransT (CVPR2021) | 0.495 | 73.8 | 67.1 | 86.7 |
| TrDiMP (CVPR2021) | - | - | 67.1 | 83.3 |
| Siam R-CNN (CVPR2020) | - | 72.2 | 64.9 | 85.4 |
| TREG (Arxiv2021) | - | 74.1 | 66.8 | 83.8 |
Use the Anaconda
conda create -n mixformer python=3.6
conda activate mixformer
bash install_pytorch17.sh
Put the tracking datasets in ./data. It should look like:
${MixFormer_ROOT}
-- data
-- lasot
|-- airplane
|-- basketball
|-- bear
...
-- got10k
|-- test
|-- train
|-- val
-- coco
|-- annotations
|-- train2017
-- trackingnet
|-- TRAIN_0
|-- TRAIN_1
...
|-- TRAIN_11
|-- TEST
Run the following command to set paths for this project
python tracking/create_default_local_file.py --workspace_dir . --data_dir ./data --save_dir .
After running this command, you can also modify paths by editing these two files
lib/train/admin/local.py # paths about training
lib/test/evaluation/local.py # paths about testing
Training with multiple GPUs using DDP. More details of
other training settings can be found at tracking/train_mixformer_[cvt/vit/convmae].sh for different backbone respectively.
# MixFormer with CVT backbone
bash tracking/train_mixformer_cvt.sh
# MixFormer with ViT backbone
bash tracking/train_mixformer_vit.sh
# MixFormer with ConvMAE backbone
bash tracking/train_mixformer_convmae.sh
- LaSOT/GOT10k-test/TrackingNet/OTB100/UAV123. More details of
test settings can be found at
tracking/test_mixformer_[cvt/vit/convmae].sh
bash tracking/test_mixformer_cvt.sh
bash tracking/test_mixformer_vit.sh
bash tracking/test_mixformer_convmae.sh
- VOT2020
Before evaluating "MixFormer+AR" on VOT2020, please install some extra packages following external/AR/README.md. Also, the VOT toolkit is required to evaluate our tracker. To download and instal VOT toolkit, you can follow this tutorial. For convenience, you can use our example workspaces of VOT toolkit underexternal/vot20/by settingtrackers.ini.
cd external/vot20/<workspace_dir>
vot evaluate --workspace . MixFormerPython
# generating analysis results
vot analysis --workspace . --nocache
bash tracking/run_video_demo.sh
bash tracking/profile_mixformer.sh
bash tracking/vis_mixformer_attn.sh
The trained models and the raw tracking results are provided in the [Models and Raw results] (Google Driver) or [Models and Raw results] (Baidu Driver: hmuv).
Yutao Cui: [email protected]
- Thanks for PyTracking Library and STARK Library, which helps us to quickly implement our ideas.
- We use the implementation of the CvT from the official repo CvT.
If you think this project is helpful, please feel free to leave a star⭐️ and cite our paper:
@inproceedings{cui2022mixformer,
title={Mixformer: End-to-end tracking with iterative mixed attention},
author={Cui, Yutao and Jiang, Cheng and Wang, Limin and Wu, Gangshan},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={13608--13618},
year={2022}
}
@misc{cui2023mixformer,
title={MixFormer: End-to-End Tracking with Iterative Mixed Attention},
author={Yutao Cui and Cheng Jiang and Gangshan Wu and Limin Wang},
year={2023},
eprint={2302.02814},
archivePrefix={arXiv}
}