TorchVision 0.12, including new Models, Datasets, GPU Video Decoding, and more

Highlights

New Models

Four new model families have been released in the latest version along with pre-trained weights for their variants: FCOS, RAFT, Vision Transformer (ViT) and ConvNeXt.

Object Detection

FCOS is a popular, fully convolutional, anchor-free model for object detection. In this release we include a community-contributed model implementation as well as pre-trained weights. The model was trained on COCO train2017 and can be used as follows:

import torch
from torchvision import models

x = [torch.rand(3, 224, 224)]
fcos = models.detection.fcos_resnet50_fpn(pretrained=True).eval()
predictions =  fcos(x)

The box AP of the pre-trained model on COCO val2017 is 39.2 (see #4961 for more details).

We would like to thank Hu Ye and Zhiqiang Wang for contributing to the model implementation and initial training. This was the first community-contributed model in a long while, and given its success, we decided to use the learnings from this process and create a new model contribution guidelines.

Optical Flow support and RAFT model

Torchvision now supports optical flow! Optical flow models try to predict movement in a video: given two consecutive frames, the model predicts where each pixel of the first frame ends up in the second frame. Check out our new tutorial on Optical Flow!

We implemented a torchscript-compatible RAFT model with pre-trained weights (both normal and “small” versions), and added support for training and evaluating optical flow models. Our training scripts support distributed training across processes and nodes, leading to much faster training time than the original implementation. We also added 5 new optical flow datasets: Flying Chairs, Flying Things, Sintel, Kitti, and HD1K.

Image Classification

Vision Transformer (ViT) and ConvNeXt are two popular architectures which can be used as image classifiers or as backbones for downstream vision tasks. In this release we include 8 pre-trained weights for their classification variants. The models were trained on ImageNet and can be used as follows:

import torch
from torchvision import models

x = torch.rand(1, 3, 224, 224)
vit = models.vit_b_16(pretrained=True).eval()
convnext = models.convnext_tiny(pretrained=True).eval()
predictions1 = vit(x)
predictions2 = convnext(x)

The accuracies of the pre-trained models obtained on ImageNet val are seen below:

Model	Acc@1	Acc@5
vit_b_16	81.072	95.318
vit_b_32	75.912	92.466
vit_l_16	79.662	94.638
vit_l_32	76.972	93.07
convnext_tiny	82.52	96.146
convnext_small	83.616	96.65
convnext_base	84.062	96.87
convnext_large	84.414	96.976

The above models have been trained using an adjusted version of our new training recipe and this allows us to offer models with accuracies significantly higher than the ones on the original papers.

GPU Video Decoding

In this release, we add support for GPU video decoding in the video reading API. To use hardware-accelerated decoding, we just need to pass a cuda device to the video reading API as shown below:

import torchvision

reader = torchvision.io.VideoReader(file_name, device='cuda:0')
for frame in reader:
    print(frame)

We also support seeking to anyframe or a keyframe in the video before reading, as shown below:

reader.seek(seek_time)

New Datasets

We have implemented 14 new classification datasets: CLEVR, GTSRB, FER2013, SUN397, Country211, Flowers102, fvgc_aircraft, OxfordIIITPet, DTD, Food 101, Rendered SST2, Stanford cars, PCAM, and EuroSAT.

As part of our work on Optical Flow support (see above for more details), we also added 5 new optical flow datasets: Flying Chairs, Flying Things, Sintel, Kitti, and HD1K.

Documentation

New documentation layout

We have updated our documentation pages to be more compact and easier to browse. Each function / class is now documented in a separate page, clearing up some space in the per-module pages, and easing the discovery of the proposed APIs. Compare e.g. our previous docs vs the new ones. Please let us know if you have any feedback!

Model contribution guidelines

New model contribution guidelines have been published following the success of the FCOS model which was contributed by the community. These guidelines aim to be an overview of the model contribution process for anyone who would like to suggest, implement and train a new model.

Upcoming Prototype APIs

We are currently working on a prototype API which adds Multi-weight support on all of our model builder methods. This will enable us to offer multiple pre-trained weights, associated with their meta-data and inference transforms. The API is still under review and thus was not included in the release but you can read more about it on our blogpost and provide your feedback on the dedicated Github issue.

Changes in our deprecation policy

Up until now, torchvision would almost never remove deprecated APIs. In order to be more aligned and consistent with pytorch core, we are updating our deprecation policy. We are now following a 2-release deprecation cycle: deprecated APIs will raise a warning for 2 versions, and will be removed after that. To reflect these changes and to smooth the transition, we have decided to:

• Remove all APIs that had been deprecated before or on v0.8, released 1.5 years ago.
• Update the removal timeline of all other deprecated APIs to v0.14, to reflect the new 2-cycle policy starting now in v0.12.

Backward-incompatible changes

[models.quantization] Removed the Quantized shufflenet_v2_x1_5 and shufflenet_v2_x2_0 model builders which had no associated weights, rendering them useless. Additionally we added pre-trained weights for the shufflenet_v2_x0_5 quantized variant.. (#4854)
[ops] Change to stable sort in nms implementations - this change can lead to different behavior in rare cases therefore it has been flagged as backwards-incompatible (#4767)
[transforms] Changed the center and the parametrization of shear X/Y in Auto Augment transforms to align with the original papers (#5285) (#5384)

Deprecations

Note: in order to be more aligned with pytorch core, we are updating our deprecation policy. Please read more above in the “Highlights” section.

[ops] The ops.poolers.MultiScaleRoIAlign public methods setup_setup_scales, convert_to_roi_format, and infer_scale have been deprecated and will be removed in 0.14 (#4951) (#4810)

New Features

[datasets] New optical flow datasets added: FlyingChairs, Kitti, Sintel, FlyingThings3D, and HD1K (#4860) (#4845) (#4858) (#4890) (#5004) (#4889) (#4888) (#4870)
[datasets] New classification datasets support for FLAVA: CLEVR, GTSRB, FER2013, SUN397, Country211, Flowers102, fvgc_aircraft, OxfordIIITPet, DTD, Food 101, Rendered SST2, Stanford cars, PCAM, and EuroSAT (#5120) (#5130) (#5117) (#5132) (#5138) (#5177) (#5178) (#5116) (#5115) (#5119) (#5220) (#5166) (#5203) (#5114) (#5164) (#5280)
[models] Add VisionTransformer model (#5173) (#5210) (#5172) (#5085) (#5226) (#5025) (#5086) (#5159)
[models] Add ConvNeXt model (#5330) (#5253)
[models] Add RAFT models and support for optical flow model training (#5022) (#5070) (#5174) (#5381) (#5078) (#5076) (#5081) (#5079) (#5026) (#5027) (#5082) (#5060) (#4868) (#4657) (#4732)
[models] Add FCOS model (#4961) (#5267)
[utils] Add utility to convert optical flow to an image (#5134) (#5308)
[utils] Add utility to draw keypoints (#4216)
[video] Add video GPU decoder (#5019) (#5191) (#5215) (#5256) (#4474) (#3179) (#4878) (#5328) (#5327) (#5183) (#4947) (#5192)

Improvements

[datasets] Migrate mnist dataset from np.frombuffer (#4598)
[io, tests] Switch from np.frombuffer to torch.frombuffer (#4578)
[models] Update ResNet-50 accuracy with Repeated Augmentation (#5201)
[models] Add regnet_y_128gf factory function, and several regnet model weights (#5176) (#4530)
[models] Adding min_size to classification and video models (#5223)
[models] Remove in-place mutation in DefaultBoxGenerator (#5279)
[models] Added Dropout parameter to Models Constructors (#4580)
[models] Allow to use custom norm_layer (#4621)
[models] Add IntermediateLayerGetter on segmentation (#5298)
[models] Use FX feature extractor for segm model (#4563)
[models, ops, io] Add model, ops and io usage logging (#4956) (#4735) (#4736) (#4737) (#5044) (#4799) (#5095) (#5038)
[models.quantization] Implement is_qat in TorchVision (#5299)
[models.quantization] Cleanup Quantized ShuffleNet (#4854)
[models.quantization] Adding new Quantized models (#4969)
[ops] [FBcode->GH] Fix missing kernel guards (#4620) (#4743)
[ops] Expose misc ops at package level (#4812)
[ops] Fix giou naming bug (#5270)
[ops] Change batched NMS threshold to choose for-loop version (#4990)
[ops] Add bias parameter to ConvNormActivation (#5012)
[ops] Feature extraction default arguments - ops (#4810)
[ops] Change to stable sort in nms implementations (#4767)
[reference scripts] Support amp training (#4923) (#4933) (#4994) (#4547) (#4570)
[reference scripts] Add types and improve descriptions to ArgumentParser parameters (#4724)
[reference scripts] Replaced all 'no_grad()' instances with 'inference_mode()' (#4629)
[reference scripts] Adding Repeated Augment Sampler (#5051)
[reference scripts] Reduce variance of classification references evaluation (#4609)
[reference scripts] Avoid inplace modification of target boxes in detection references (#5289)
[reference scripts] Allow variable number of repetitions for RA (#5084)
[reference scripts, classification] Adding gradient clipping (#4824)
[reference scripts, models.quantization] Add --prototype flag to quantization scripts. (#5334)
[reference scripts, ops] Additional SOTA ingredients on Classification Recipe (#4493)
[transforms] Added center arg to F.affine and RandomAffine ops (#5208)
[transforms] Explicitly copying array in pil_to_tensor (#4566)
[transforms] Update functional_tensor.py (#4852)
[transforms] Add api usage log to transforms (#5007)
[utils] Support random colors by default for draw_bounding_boxes (#5127)
[utils] Add API usage calls to utils (#5077)
Various documentation improvements (#4913) (#4892) (#5305) (#5273) (#5089) (#4653) (#5302) (#4647) (#4922) (#5124) (#4972) (#5165) (#4843) (#5238) (#4846) (#4823) (#5316) (#5195) (#5153) (#4783) (#4798) (#4797) (#5368) (#5037) (#4830) (#4681) (#4579) (#4520) (#4586) (#4536) (#4574)) (#4565) (#4822) (#5315) (#4546) (#4522) (#5312) (#5372) (#4833)
[tests] Set seed on several tests to reduce flakiness (#4911) (#4764) (#4762) (#4759) (#4766) (#4763) (#4758) (#4761)
[tests]Other tests improvements (#4756) (#4775) (#4867) (#4929) (#4632) (#5029) (#4597)
Added script to sync fbcode changes with main branch (#4769)
[ci] Various CI improvements (#4662) (#4669) (#4791) (#4626) (#5021) (#4739) (#3973)(#4618) (#4788) (#4946) (#5112) (#5099) (#5288) (#5152) (#4696) (#5122) (#4793) (#4998) (#4498)
[build] Various build improvements (#5261) (#5190) (#4945) (#4920) (#5024) (#4571) (#4742) (#4944) (#4989) (#5179) (#4516) (#4661) (#4695) (#4939) (#4954)
[io] decode_* returns contiguous tensors (#4898)
[io] Revert "decode_* returns contiguous tensors (#4898)" (#4901)

Bug Fixes

[datasets] fix Caltech datasets (#4556)
[datasets] fix UCF101 on Windows (#5129)
[datasets] remove extracted archive if flag was set (#5055)
[datasets] Reverted folder.py back to using complete path to file for make_dataset and is_valid_file rather than just the filename (#4885)
[datasets] fix fromfile on windows (#4980)
[datasets] fix WIDERFace download links (#4649)
[datasets] fix target_type selection for Caltech101 (#4637)
[io] Skip jpeg comparison tests with PIL (#5169)
[io] [Windows] Workaround for loading bundled DLLs (#4893)
[models] Adding missing named param check on ViT (#5196)
[models] Modifying keypoint_rcnn.py for keypoint_predictor issue (#5180)
[models] Fixing bug on SSD backbone freezing (#4590)
[models] [FBcode->GH] Removed type annotations from rcnn (#4883)
[models.quantization] Amend the weights only if quantize=True (#4966)
[models.quantization] fix mobilenetv3 quantization state dict loading (#4997)
[ops] Adding masks_to_boxes to all in ops (#4779)
[ops] Update the error message on DeformConv2d (#4908)
[ops, onnx] RoiAlign aligned=True (#4692)
[reference scripts] Fix reduce_across_processes inconsistent return type (#4733)
[reference scripts] Fix bug on EMA n_averaged estimation (#4544)
[reference scripts] support random seed for RA sampler (#5053)
[reference scripts] fix bug in training model by amp (#4874)
[reference scripts, transforms] Fix a bug on RandomZoomOut (#5278)
[tests] Skip expected checks for quantized resnet50 due to flakiness (#4686)
[transforms] Fix bug on autocontrast when min==max (#4999)
[transforms] Fix augmentation space to be uint8 compatible (#4806)
[utils] Fix draw_bounding_boxes and draw_keypoints for tensors on GPU (#5101) (#5102)
[build] fix formatting CIRCLECI_TAG when building docs (#4693)
[build] Fix nvjpeg packaging into the wheel (#4752)
[build] Switch Android app to pytorch_android stable (#4926)
[ci] Add libtinfo5 dependency (#4931)
[ci] Revert vit_h_14 as it breaks our CI (#5259)
[ci] Remove pager on git diff (#4800)
[ci] Fix failing CI job for android (#4912)
[ci] Add numpy as explicit dependency to build_cmake.sh (#4987)

Code Quality

Various typing improvements (#4603) (#4172) (#4173) (#4631) (#4619) (#4583) (#4602) (#5182)
Add ufmt (usort + black) as code formatter (#4384)
Fix formatting issues (#4535) (#4747)
Add pre-commit hook to fix line endings (#5021)
Various imports cleanups/improvements (#4533) (#4879)
Use f-strings almost everywhere, and other cleanups by applying pyupgrade (#4585)
Update code to Python 3.7 compliance and remove Python 3.6 references (#5125) (#5161)
Consolidate repr methods throughout the repo (#5392)
Set allow_redefinition = True for mypy (#4531)
Use is to compare type of objects (#4605)
Various typos fixed (#5031) (#5092)
Fix annotations for Python >= 3.8 (#5301)
Revamp log api usage method (#5072)
[deprecation] Update deprecation messages stating APIs will be removed in 0.14 and remove APIs that were deprecated before 0.8 (#5387) (#5386)
[build] Updated setup.py to use TorchVersion object for version comparison (#4307)
[ops] remove debugging asserts (#5332)
[c++frontend] Fix missing Torch includes (#5118)
[ci] Cleanup and removing unnecessary references and parameters (#4983) (#4930) (#5042)
[datasets] [FBcode->GH] remove unused requests functionality (#5014)
[datasets] allow single extension as str in make_dataset (#5229)
[datasets] use helper function to extract archive in CelebA (#4557)
[datasets] simplify QMNIST download logic (#4562)
[documentation] fix make html-noplot docs build command (#5389)
[models] Move all weight initializations from private methods to constructors (#5331)
[models] simplify model builders (#5001)
[models] Replace asserts with ValueErrors (#5275)
[models] Use enumerate to get index of ModuleList (#4534)
[models] Simplify efficientnet code by removing _efficientnet_conf (#4690)
[models] Refactor Segmentation models (#4646)
[models] Pass indexing param to meshgrid to avoid warning in detection models (#4645)
[models] Refactor the backbone builders of detection (#4656)
[models.quantization] Switch torch.quantization to torch.ao.quantization (#5296) (#4554)
[ops] Fixed unused variables in ops (#4666)
[ops] Refactor poolers (#4951)
[reference scripts] Simplify the gradient clipping code (#4896)
[reference scripts] only set random generator if shuffle=true (#5135)
[tests] Refactor BoxOps tests to use parameterize (#5380)
[tests] rename TestWeights to appease pytest (#5054)
[tests] fix and add test for sequence_to_str (#5213)
[tests] remove get_bool_env_var (#5222)
[models, tests] remove custom code for model output comparison (#4971)
[utils, documentation] Fix annotation of draw_segmentation_masks (#4527)
[video] Fix error message in demuxer (#5293)

Contributors

We're grateful for our community, which helps us improve torchvision by submitting issues and PRs, and providing feedback and suggestions. The following persons have contributed patches for this release:

Abhijit Deo, Aditya Oke, Alexander Soare, Alexander Unnervik, Allen Goodman, Andrey Talman, Brian Johnson, Bruno Korbar, buckage, Carlosbogo, Chungman Lee, Daniel Falbel, David Fan, Dmytro, Eli Uriegas, Ethan White, Eugene Yurtsev, F-G Fernandez, Fedor, Francisco Massa, Guo, Harish Kulkarni, HeungwooLee, Hu Ye, Jane (Yuan) Xu, Jirka Borovec, Jithun Nair, Joao Gomes, Jopo, Kai Zhang, kbozas, Kevin Tse, Khushi Agrawal, Konstantinos Bozas, Kotchin, Kushashwa Ravi Shrimali, KyleCZH, Mark Harfouche, Marko Kohtala, Masahiro Masuda, Matti Picus, Mengwei Liu, Mohammad (Moe) Rezaalipour, Mriganka Nath, Muhammed Abdullah, Nicolas Granger, Nicolas Hug, Nikita Shulga, peterbell10, Philip Meier, Piyush Singh, Prabhat Roy, ProGamerGov, puhuk, Richard Barnes, rvandeghen, Sai Krishna, Santiago Castro, Saswat Das, Sepehr Sameni, Sergii Khomenko, Stephen Matthews, Sumanth Ratna, Sumukh Aithal, Tal Ben-Nun, Vasilis Vryniotis, vfdev, Xiaolin Wang, Yi Zhang, Yiwen Song, Yoshitomo Matsubara, Yuchen Huang, Yuxin Wu, zhiqiang, and Zhiqiang Wang.