O ut O f D istribution Generalization in C omputer V ision

The goal of our OOD-CV benchmark is do diagnose robustness of computer vision models to out-of-distribution shifts in the data. To achieve this goal, the benchmark consists of a fixed training set with 10 object categories (aeroplane, bus, car, train, boat, bicycle, motorplane, chair, dining table, sofa) from the PASCAL VOC 2012 and ImageNet datasets. Note that only our provided training data can be used to train a model and using outside training data is not allowed. This restriction enables us to design the test set such that each test example is subject to an out-of-distribution shift in one specific nuisance w.r.t. the training data, such as the object's 3D pose, shape, texture, context, the weather, and occlusion.

The OOD-CV challenge has three tracks. Each track will focus on one computer vision task, including image classification, object detection and 3D pose estimation. The winners of each track will be invited to contribute to an IJCV special issue. Moreover, the winners and runner-ups in each challenge track will share a prize pool of 10'000 USD.

The training data, validation data and code for the baseline models can be downloaded from this website: https://github.com/eccv22-ood-workshop/ROBIN-dataset

The final test set will not be publicly available (see details of the evaluation process below). Note that the training set is fixed and the use any outside data is not allowed. All prize winners (i.e. winning submissions and runner-ups) will be required to either publish their code online, or send us their code privately, such that we can verify that no outside data has been used.

Our aim is to measure model robustness w.r.t. OOD nuisance factors. Therefore, the final benchmark scoring is data and accuracy constrained. This means, that to be valid a submission must:
1) Only use the training data that we provide. Using outside data is not allowed.
2) The model’s accuracy on the I.I.D. test set must be lower than a pre-defined threshold (which is defined by the performance of a baseline model).
The final benchmark score is then measured as average performance on the held-out O.O.D. test set.

The I.I.D. accuracy thresholds are as follows:
Image Classification = 91.1 [top-1 accuracy]
Object Detection = 79.9 [mAP@50]
Pose Estimation = 68.7 [Acc@pi/6]
Each accuracy threshold was determined by training the baseline models (see paper and Github repo) five times, followed by computing the mean performance and adding three standard deviations.

You can find instructions on the submission, data and baseline models on our Github page: https://github.com/eccv22-ood-workshop/ROBIN-dataset

The winners of each track will be invited to submit a paper to an IJCV special issue. The top-3 submissions across all challenge tracks will share a prize pool of 10'000 USD which is sponsored by the Future Fund Regranting Program. All prize winners will be required to write a summary of their method and share it publicly on arXiv.

Image Classification Track
🥇 1st place: $2,000
🥈 2nd place: $1,000
🥉 3rd place: $300

Object Detection Track
🥇 1st place: $2,000
🥈 2nd place: $1,000
🥉 3rd place: $300

Pose Estimation Track
🥇 1st place: $2,000
🥈 2nd place: $1,000
🥉 3rd place: $300

If you find the dataset or challenge useful for your research, please cite the paper.

			
                                    
                                    @article{zhao22oodcv,
                                        author  = {Bingchen Zhao and Shaozuo Yu and Wufei Ma and Mingxin Yu and Shenxiao Mei and Angtian Wang and Ju He and Alan Yuille and Adam Kortylewski},
                                        title   = {OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images},
                                        journal = {Proceedings of the European Conference on Computer Vision (ECCV)},
                                        year    = {2022}
                                    }			
                                    
                                

To be eligible for prizes, winning teams are required to share their methods, code, and models with the organizers as well as the names and associations of each team member. This workshop is sponsored by the FTX Future Fund regranting program. We cannot give awards to teams on US terrorist lists or those subject to sanctions. Sponsor may confirm the legality of sending prize money to winners who are residents of countries outside of the United States. Winners will be announced prior to the workshop, and workshop organizers will judge the submissions. Only authors on awarded papers are winners. All decisions of judges are final. The legality of accepting the prize in his or her country is the responsibility of the winners. All taxes are the responsibility of the winners. Employees or current contractors of FTX and contest organizers are not eligible to win prizes. Entrants must be over the age of 18. By entering the contest, entrants agree to the Terms & Conditions. Entrants agree that FTX shall not be liable to entrants for any type of damages that arise out of or are related to the contest and/or the prizes. By submitting an entry, entrant represents and warrants that, consistent with the terms of the Terms and Conditions: (a) the entry is entrant’s original work; (b) entrant owns any copyright applicable to the entry; (c) the entry does not violate, in whole or in part, any existing copyright, trademark, patent or any other intellectual property right of any other person, organization or entity; (d) entrant has confirmed and is unaware of any contractual obligations entrant has which may be inconsistent with these Terms and Conditions and the rights entrant is required to have in the entry, including but not limited to any prohibitions, obligations or limitations arising from any current or former employment arrangement entrant may have; (e) entrant is not disclosing the confidential, trade secret or proprietary information of any other person or entity, including any obligation entrant may have in connection arising from any current or former employment, without authorization or a license; and (f) entrant has full power and all legal rights to submit an entry in full compliance with these Terms and Conditions.