o ϯiI!@sddlmZmZddlZddlmZddlmZddlm Z Gdddej Z ej dddd d dd d d e de dedee dee dee dee dedededej fddZdd ddd e de dedededej f ddZde ej dddd ejd dd d d e de d ed!ed"e d#ededee dee dee dee d$ededededej f d%d&ZGd'd(d(ej ZGd)d*d*ej ZGd+d,d,ej ZdS)-)CallableTupleN)set_attributes)RoIAligncs@eZdZdZdeddffdd Zdejdejfdd ZZ S) SequencePoolz Sequence pool produces a single embedding from a sequence of embeddings. Currently it supports "mean" and "cls". modereturnNcs$t|dvs Jd||_dS)a  Args: mode (str): Optionals include "cls" and "mean". If set to "cls", it assumes the first element in the input is the cls token and returns it. If set to "mean", it returns the mean of the entire sequence. clsmeanz"Unsupported mode for SequencePool.N)super__init__r)selfr __class__L/home/ubuntu/.local/lib/python3.10/site-packages/pytorchvideo/models/head.pyr s  zSequencePool.__init__xcCs:|jdkr|dddf}|S|jdkr|d}|St)Nr rr )rr NotImplementedErrorrrrrrforwards  zSequencePool.forward) __name__ __module__ __qualname____doc__strr torchTensorr __classcell__rrrrr s r)rrr)rr )rrrg?T)pool output_sizepool_kernel_size pool_stride pool_padding dropout_rate activationoutput_with_global_average in_features out_featuresr!r"r#r$r%r&r'r(rc Cs|durd} n|tjkr|dd} n|} |durd} n|tjkr&||} n||||d} | r5td} nd} tt||| | |dkrKt|| dSd| dS)u Creates ResNet basic head. This layer performs an optional pooling operation followed by an optional dropout, a fully-connected projection, an activation layer and a global spatiotemporal averaging. :: Pooling ↓ Dropout ↓ Projection ↓ Activation ↓ Averaging Activation examples include: ReLU, Softmax, Sigmoid, and None. Pool3d examples include: AvgPool3d, MaxPool3d, AdaptiveAvgPool3d, and None. Args: in_features: input channel size of the resnet head. out_features: output channel size of the resnet head. pool (callable): a callable that constructs resnet head pooling layer, examples include: nn.AvgPool3d, nn.MaxPool3d, nn.AdaptiveAvgPool3d, and None (not applying pooling). pool_kernel_size (tuple): pooling kernel size(s) when not using adaptive pooling. pool_stride (tuple): pooling stride size(s) when not using adaptive pooling. pool_padding (tuple): pooling padding size(s) when not using adaptive pooling. output_size (tuple): spatial temporal output size when using adaptive pooling. activation (callable): a callable that constructs resnet head activation layer, examples include: nn.ReLU, nn.Softmax, nn.Sigmoid, and None (not applying activation). dropout_rate (float): dropout rate. output_with_global_average (bool): if True, perform global averaging on temporal and spatial dimensions and reshape output to batch_size x out_features. Nrdim kernel_sizestridepaddingr)projr'r!dropout output_pool)nnSoftmaxAdaptiveAvgPool3dResNetBasicHeadLinearDropout) r)r*r!r"r#r$r%r&r'r(activation_model pool_modelr3rrrcreate_res_basic_head's2A      r<r ) seq_pool_typer&r'r=cCs|dvsJ|dvrt|}n |dkrd}nt|durd}n|tjkr*|dd}n|}t||dkr8t|ndt|||dS) u Creates vision transformer basic head. :: Pooling ↓ Dropout ↓ Projection ↓ Activation Activation examples include: ReLU, Softmax, Sigmoid, and None. Pool type examples include: cls, mean and none. Args: in_features: input channel size of the resnet head. out_features: output channel size of the resnet head. pool_type (str): Pooling type. It supports "cls", "mean " and "none". If set to "cls", it assumes the first element in the input is the cls token and returns it. If set to "mean", it returns the mean of the entire sequence. activation (callable): a callable that constructs vision transformer head activation layer, examples include: nn.ReLU, nn.Softmax, nn.Sigmoid, and None (not applying activation). dropout_rate (float): dropout rate. )r r noner r>Nrr+g sequence_poolr2r1r')rrr4r5VisionTransformerBasicHeadr9r8)r)r*r=r&r'seq_pool_modelr:rrrcreate_vit_basic_heads" -    rC) sampling_ratioroir!r"r#r$r% pool_spatialr&r'r( resolution spatial_scalerDrErFc Cs| durd}n| tjkr| dd}n| }|durd}n|tjkr&||}n||| | d}|r5td}nd}tt||||| rG| |ddnd||||d| dkrZt| |dSd|dS) u0 Creates ResNet RoI head. This layer performs an optional pooling operation followed by an RoI projection, an optional 2D spatial pool, an optional dropout, a fully-connected projection, an activation layer and a global spatiotemporal averaging. Pool3d ↓ RoI Align ↓ Pool2d ↓ Dropout ↓ Projection ↓ Activation ↓ Averaging Activation examples include: ReLU, Softmax, Sigmoid, and None. Pool3d examples include: AvgPool3d, MaxPool3d, AdaptiveAvgPool3d, and None. RoI examples include: detectron2.layers.ROIAlign, detectron2.layers.ROIAlignRotated, tochvision.ops.RoIAlign and None Pool2d examples include: MaxPool2e, AvgPool2d, and None. Args: Projection related configs: in_features: input channel size of the resnet head. out_features: output channel size of the resnet head. RoI layer related configs: resolution (tuple): h, w sizes of the RoI interpolation. spatial_scale (float): scale the input boxes by this number sampling_ratio (int): number of inputs samples to take for each output sample interpolation. 0 to take samples densely. roi (callable): a callable that constructs the roi interpolation layer, examples include detectron2.layers.ROIAlign, detectron2.layers.ROIAlignRotated, and None. Pooling related configs: pool (callable): a callable that constructs resnet head pooling layer, examples include: nn.AvgPool3d, nn.MaxPool3d, nn.AdaptiveAvgPool3d, and None (not applying pooling). pool_kernel_size (tuple): pooling kernel size(s) when not using adaptive pooling. pool_stride (tuple): pooling stride size(s) when not using adaptive pooling. pool_padding (tuple): pooling padding size(s) when not using adaptive pooling. output_size (tuple): spatial temporal output size when using adaptive pooling. pool_spatial (callable): a callable that constructs the 2d pooling layer which follows the RoI layer, examples include: nn.AvgPool2d, nn.MaxPool2d, and None (not applying spatial pooling). Activation related configs: activation (callable): a callable that constructs resnet head activation layer, examples include: nn.ReLU, nn.Softmax, nn.Sigmoid, and None (not applying activation). Dropout related configs: dropout_rate (float): dropout rate. Output related configs: output_with_global_average (bool): if True, perform global averaging on temporal and spatial dimensions and reshape output to batch_size x out_features. Nrr+r-)r/)r"rHrDr)r1r'r!rF roi_layerr2r3)r4r5r6 ResNetRoIHeadr8r9)r)r*rGrHrDrEr!r"r#r$r%rFr&r'r(r:r;r3rrrcreate_res_roi_pooling_heads>[       rKc sfeZdZdZ     ddejdejdejdejdejddf fd d Zd ejdejfd d Z Z S)r7u ResNet basic head. This layer performs an optional pooling operation followed by an optional dropout, a fully-connected projection, an optional activation layer and a global spatiotemporal averaging. :: Pool3d ↓ Dropout ↓ Projection ↓ Activation ↓ Averaging The builder can be found in `create_res_basic_head`. Nr!r2r1r'r3rc(tt|t|jdusJdS)a< Args: pool (torch.nn.modules): pooling module. dropout(torch.nn.modules): dropout module. proj (torch.nn.modules): project module. activation (torch.nn.modules): activation module. output_pool (torch.nn.Module): pooling module for output. Nr r rlocalsr1)rr!r2r1r'r3rrrr _s  zResNetBasicHead.__init__rcCs|jdur ||}|jdur||}|jdur(|d}||}|d}|jdur2||}|jdurE||}||jdd}|S)NrrrrRrrPrQr)r!r2r1permuter'r3viewshaperrrrrss            zResNetBasicHead.forward)NNNNN rrrrr4Moduler rrrrrrrrr7Js*r7cs|eZdZdZ       ddejdejdejdejdejdejd ejd dffd d Zd ejdejd ejfddZ Z S)rJu ResNet RoI head. This layer performs an optional pooling operation followed by an RoI projection, an optional 2D spatial pool, an optional dropout, a fully-connected projection, an activation layer and a global spatiotemporal averaging. Pool3d ↓ RoI Align ↓ Pool2d ↓ Dropout ↓ Projection ↓ Activation ↓ Averaging The builder can be found in `create_res_roi_pooling_head`. Nr!rFrIr2r1r'r3rcrL)a Args: pool (torch.nn.modules): pooling module. pool_spatial (torch.nn.modules): pooling module. roi_spatial (torch.nn.modules): RoI (Ex: Align, pool) module. dropout(torch.nn.modules): dropout module. proj (torch.nn.modules): project module. activation (torch.nn.modules): activation module. output_pool (torch.nn.Module): pooling module for output. NrM)rr!rFrIr2r1r'r3rrrr s  zResNetRoIHead.__init__rbboxescCs|jdur ||}|jdur7|jd}|dkrtdt|d}|||}|jdur2||}|d}|jdurA||}|j durU| d}| |}| d}|j dur_| |}|j durr| |}| |jdd}|S) aC Args: x (torch.tensor): input tensor bboxes (torch.tensor): Accociated bounding boxes. The format is N*5 (Index, X_1,Y_1,X_2,Y_2) if using RoIAlign and N*6 (Index, x_ctr, y_ctr, width, height, angle_degrees) if using RoIAlignRotated. NrzBTemporal dimension should be 1. Consider modifying the pool layer.rOrSrrT)r!rIrW ExceptionrsqueezerF unsqueezer2r1rUr'r3rV)rrrZ temporal_dimrrrrs2                  zResNetRoIHead.forward)NNNNNNNrXrrrrrJs6 $rJc s^eZdZdZ    d dejdejdejdejddf fdd Zd ejdejfd d Z Z S)rAu Vision transformer basic head. :: SequencePool ↓ Dropout ↓ Projection ↓ Activation The builder can be found in `create_vit_basic_head`. Nr@r2r1r'rcrL)z Args: sequence_pool (torch.nn.modules): pooling module. dropout(torch.nn.modules): dropout module. proj (torch.nn.modules): project module. activation (torch.nn.modules): activation module. NrM)rr@r2r1r'rrrr s  z#VisionTransformerBasicHead.__init__rcCsJ|jdur ||}|jdur||}||}|jdur#||}|S)Nr?rrrrr s       z"VisionTransformerBasicHead.forward)NNNNrXrrrrrAs$rA)typingrrrtorch.nnr4pytorchvideo.layers.utilsrtorchvision.opsrrYr AvgPool3dintfloatboolr<rrC MaxPool2drKr7rJrArrrrs   "    e  M     @[