Online Adaptation of Convolutional Neural Networks for Video Object Segmentation
Code for OnAVOS
Requires a good GPU with at least 11GB memory (e.g. 1080 TI or TITAN X)
Instructions for running:
- install tensorflow and possibly other missing libraries using pip
- download the models and put them in OnAVOS/models/
- choose a config you want to run from configs/
I recommend to start with configs/DAVIS16_oneshot (which does the one-shot approach without adaptation on DAVIS 2016) - change the data directori(es) in the first lines of the config to your DAVIS path
- run "python main.py configs/DAVIS16_oneshot" (or a different config)
Additional instructions for DAVIS2017:
- download the lucid data and change the path in configs/DAVIS17_online to point to it
- copy the txt files from OnAVOS/ImageSets2017_with_ids/ to the ImageSets folder of your DAVIS 2017 data
- follow instructions from above using the configs/DAVIS17_online config
Instructions for running on a custom dataset (note that this is based on the implementation for DAVIS2017, so that your dataset needs to be converted to the folder structure - an alternative is to write custom code to load your dataset):
- download the pascal or pascal_up model and put them in OnAVOS/models/
- choose a config you want to run from configs/
I recommend to start with configs/custom_oneshot The custom_up_oneshot adds upsampling layers, which might improve the accuracy, but will increase runtime memory consumption. If you want to add online adaptation, please compare to the online configs for DAVIS2016 - Put your dataset in OnAVOS/custom_dataset while retaining the folder structure of the example images. The structure has to be the same as DAVIS 2017
- run "python main.py configs/custom_oneshot" (or custom_up_oneshot)
Explanation of the most important config parameters:
- n_finetune_steps: the number of steps for fine-tuning on the first frame
- learning_rates: dictionary mapping from step number to a learning rate
- n_adaptation_steps: number of update steps per frame during adaptation
- adaptation_interval: during online adaptation, each adaptation_interval steps, the current frame is used for updating, otherwise the first frame
- adaptation_learning_rate: learning rate used during online adaptation
- posterior_positive_threshold: posterior probability threshold used to obtain the positive training examples
- distance_negative_threshold: distance threshold (to the last mask) used to select the negative examples
- adaptation_loss_scale: weighting factor of loss during online adaptation
- adaptation_erosion_size: erosion size used during online adaptation (use 1 to disable erosion)
- n_test_samples: the number of random sampled augmented versions of the input image per frame used during testing. Reduce this, to make inference much faster at the cost of a little bit accuracy
Outputs: Log files will be stored in logs/ The results will be stored in forwarded/
Note that the code slightly changed since we wrote the paper and that randomness is involved, so the results you obtain might be different from the results reported in the paper. If the results differ significantly, please let me know.
If you need additional config or model files or have questions, please write me a mail: voigtlaender@vision.rwth-aachen.de
If you find this code useful, please consider citing
Paul Voigtlaender and Bastian Leibe: Online Adaptation of Convolutional Neural Networks for Video Object Segmentation, BMVC 2017
Paul Voigtlaender and Bastian Leibe: Online Adaptation of Convolutional Neural Networks for the 2017 DAVIS Challenge on Video Object Segmentation, The 2017 DAVIS Challenge on Video Object Segmentation - CVPR Workshops
Downloads
code
models
DAVIS16.tar: DAVIS 2016 pretrained model (1.5G)
DAVIS17.tar: DAVIS 2017 pretrained model (1.5G)
Precomputed lucid data dreaming like images and labels for DAVIS 2017
We unforunately can no longer provide precomputed lucid data, please use the official code https://github.com/ankhoreva/LucidDataDreaming to generate it.
