o ϯiMn7@sjddlZddlmZmZddlZddlZddlmZddl m Z ddl m Z ddl mZddlmZmZmZddlmZddlmZdd lmZmZmZdd lmZd d d ejddejddededeedeedeedede de dedej!fddZ"dd ejdddejedd ed!ed"edeedeedede de d#e ded$edej!fd%d&Z#e#d'dd ejdddejed( d ed!ed"ed)ed*e$deedeedede de d#e ded$edej!fd+d,Z%e#dd ejdddejed- d.ed ed!ed"ed)edeedeedede de d#e ded$edej!fd/d0Z&ejd1ejddd2d3ej'd'd4 d ed!ed"ed5ed6ed7eedede de d8e ded9e$dej!fd:d;Z(dd?d3d@dAejddejdBd d dCdDdEe#dFdedGejd2ej'd'dHdIedJedKedLed8e dMe dNe dede de dedOedPeedQeedReeedSeedTeed)edUe d#e d$edVedWedXe$dYedZe$dej!f6d[d\Z)Gd]d^d^ej!Z*dS)_N)CallableTuple)SqueezeExcitation) Conv2plus1d)Swish) round_repeats round_widthset_attributes)ResNetBasicHead)Net)BottleneckBlockResBlockResStage)ResNetBasicStem)r)r)rrrgh㈵>g?)conv_kernel_size conv_stride conv_paddingnormnorm_eps norm_momentum activation in_channels out_channelsrrrrrrrreturnc  Cstj||d|d|dfd|d|dfd|d|dfdd} tj|||dddf|dddf|dddfd|d} t| dd| d} |durIdn||||d } |durVdn|} t| | | dd S) u Creates the stem layer for X3D. It performs spatial Conv, temporal Conv, BN, and Relu. :: Conv_xy ↓ Conv_t ↓ Normalization ↓ Activation Args: in_channels (int): input channel size of the convolution. out_channels (int): output channel size of the convolution. conv_kernel_size (tuple): convolutional kernel size(s). conv_stride (tuple): convolutional stride size(s). conv_padding (tuple): convolutional padding size(s). norm (callable): a callable that constructs normalization layer, options include nn.BatchNorm3d, None (not performing normalization). norm_eps (float): normalization epsilon. norm_momentum (float): normalization momentum. activation (callable): a callable that constructs activation layer, options include: nn.ReLU, nn.Softmax, nn.Sigmoid, and None (not performing activation). Returns: (nn.Module): X3D stem layer. rrrF)rr kernel_sizestridepaddingbias)rrrrr r!groupsN)conv_trrconv_xy num_featuresepsmomentum)convrrpool)nnConv3drr)rrrrrrrrrconv_xy_module conv_t_modulestacked_conv_module norm_moduleactivation_moduler2K/home/ubuntu/.local/lib/python3.10/site-packages/pytorchvideo/models/x3d.pycreate_x3d_stemsD/      r4rrrg?)rrrrrse_ratior inner_actdim_in dim_innerdim_outr6r7c  Cstj||ddd} |durdn||||d} | durdn| } tj||||dd|Dd|dd}|d kr?t|t||d d nt}t|durMtn||||d|}| dur\dn| }tj||ddd}|durndn||||d}t| | | |||||d S) u; Bottleneck block for X3D: a sequence of Conv, Normalization with optional SE block, and Activations repeated in the following order: :: Conv3d (conv_a) ↓ Normalization (norm_a) ↓ Activation (act_a) ↓ Conv3d (conv_b) ↓ Normalization (norm_b) ↓ Squeeze-and-Excitation ↓ Activation (act_b) ↓ Conv3d (conv_c) ↓ Normalization (norm_c) Args: dim_in (int): input channel size to the bottleneck block. dim_inner (int): intermediate channel size of the bottleneck. dim_out (int): output channel size of the bottleneck. conv_kernel_size (tuple): convolutional kernel size(s) for conv_b. conv_stride (tuple): convolutional stride size(s) for conv_b. norm (callable): a callable that constructs normalization layer, examples include nn.BatchNorm3d, None (not performing normalization). norm_eps (float): normalization epsilon. norm_momentum (float): normalization momentum. se_ratio (float): if > 0, apply SE to the 3x3x3 conv, with the SE channel dimensionality being se_ratio times the 3x3x3 conv dim. activation (callable): a callable that constructs activation layer, examples include: nn.ReLU, nn.Softmax, nn.Sigmoid, and None (not performing activation). inner_act (callable): whether use Swish activation for act_b or not. Returns: (nn.Module): X3D bottleneck block. rrrFrrrr!Nr%cSg|]}|dqSrr2.0sizer2r2r3 z/create_x3d_bottleneck_block..)rrrrr r!r"dilationT) num_channelsnum_channels_reducedis_3d)conv_anorm_aact_aconv_bnorm_bact_bconv_cnorm_c)r+r,rrIdentity Sequentialr )r8r9r:rrrrrr6rr7rIrJrKrLserMrNrOrPr2r2r3create_x3d_bottleneck_blockisd@     rTT) bottleneck use_shortcutrrrrrr6rr7rUrVc Csd} |dur||kr||d} t||kst|dkr'|r'tj||d|ddnd||kr0|r0| nd||||||||| | | | d | durEdn| dd d S) u# Residual block for X3D. Performs a summation between an identity shortcut in branch1 and a main block in branch2. When the input and output dimensions are different, a convolution followed by a normalization will be performed. :: Input |-------+ ↓ | Block | ↓ | Summation ←-+ ↓ Activation Args: dim_in (int): input channel size to the bottleneck block. dim_inner (int): intermediate channel size of the bottleneck. dim_out (int): output channel size of the bottleneck. bottleneck (callable): a callable for create_x3d_bottleneck_block. conv_kernel_size (tuple): convolutional kernel size(s) for conv_b. conv_stride (tuple): convolutional stride size(s) for conv_b. norm (callable): a callable that constructs normalization layer, examples include nn.BatchNorm3d, None (not performing normalization). norm_eps (float): normalization epsilon. norm_momentum (float): normalization momentum. se_ratio (float): if > 0, apply SE to the 3x3x3 conv, with the SE channel dimensionality being se_ratio times the 3x3x3 conv dim. activation (callable): a callable that constructs activation layer, examples include: nn.ReLU, nn.Softmax, nn.Sigmoid, and None (not performing activation). inner_act (callable): whether use Swish activation for act_b or not. Returns: (nn.Module): X3D block layer. N)r&rr;F)rrr!) r8r9r:rrrrrr6rr7cSs||SNr2)xyr2r2r3Asz&create_x3d_res_block..) branch1_conv branch1_normbranch2r branch_fusion)r npprodr+r,)r8r9r:rUrVrrrrrr6rr7 norm_modelr2r2r3create_x3d_res_blocks<=  rb) rUrrrrrr6rr7depthc Cstg} t|D]+}t|dkr|n||||||dkr|nd||| |ddr&| nd| | d }| |qtt| dS)u Create Residual Stage, which composes sequential blocks that make up X3D. :: Input ↓ ResBlock ↓ . . . ↓ ResBlock Args: depth (init): number of blocks to create. dim_in (int): input channel size to the bottleneck block. dim_inner (int): intermediate channel size of the bottleneck. dim_out (int): output channel size of the bottleneck. bottleneck (callable): a callable for create_x3d_bottleneck_block. conv_kernel_size (tuple): convolutional kernel size(s) for conv_b. conv_stride (tuple): convolutional stride size(s) for conv_b. norm (callable): a callable that constructs normalization layer, examples include nn.BatchNorm3d, None (not performing normalization). norm_eps (float): normalization epsilon. norm_momentum (float): normalization momentum. se_ratio (float): if > 0, apply SE to the 3x3x3 conv, with the SE channel dimensionality being se_ratio times the 3x3x3 conv dim. activation (callable): a callable that constructs activation layer, examples include: nn.ReLU, nn.Softmax, nn.Sigmoid, and None (not performing activation). inner_act (callable): whether use Swish activation for act_b or not. Returns: (nn.Module): X3D stage layer. rr;rrrE) r8r9r:rUrrrrrr6rr7) res_blocks)rangerbappendrr+ ModuleList)rcr8r9r:rUrrrrrr6rr7rdidxblockr2r2r3create_x3d_res_stageEs$?  rj) rrFg?) pool_actpool_kernel_sizerrr bn_lin5_on dropout_rateroutput_with_global_average num_classesrlrmrorpc  Cs4tj||ddd} ||||d}|durdn|}|dur#td}ntj|dd}tj||ddd}| r=||||d}nd}|durEdn|}t| ||||||d}| durZd}n| tjkre| dd }n| tjkrn| }ntd | | r}td}nd}t tj ||d d ||| d krt | |dSd|dS)u Creates X3D head. This layer performs an projected pooling operation followed by an dropout, a fully-connected projection, an activation layer and a global spatiotemporal averaging. :: ProjectedPool ↓ Dropout ↓ Projection ↓ Activation ↓ Averaging Args: dim_in (int): input channel size of the X3D head. dim_inner (int): intermediate channel size of the X3D head. dim_out (int): output channel size of the X3D head. num_classes (int): the number of classes for the video dataset. pool_act (callable): a callable that constructs resnet pool activation layer such as nn.ReLU. pool_kernel_size (tuple): pooling kernel size(s) when not using adaptive pooling. norm (callable): a callable that constructs normalization layer, examples include nn.BatchNorm3d, None (not performing normalization). norm_eps (float): normalization epsilon. norm_momentum (float): normalization momentum. bn_lin5_on (bool): if True, perform normalization on the features before the classifier. dropout_rate (float): dropout rate. activation (callable): a callable that constructs resnet head activation layer, examples include: nn.ReLU, nn.Softmax, nn.Sigmoid, and None (not applying activation). output_with_global_average (bool): if True, perform global averaging on temporal and spatial dimensions and reshape output to batch_size x out_features. Returns: (nn.Module): X3D head layer. r;Fr<r%Nr)rpre_convpre_normpre_actr* post_conv post_normpost_act)dimz-{} is not supported as an activationfunction.T)r!r)projrr*dropout output_pool) r+r,AdaptiveAvgPool3d AvgPool3d ProjectedPoolSoftmaxSigmoidNotImplementedErrorformatr LinearDropout)r8r9r:rqrlrmrrrrnrorrppre_conv_modulepre_norm_modulepre_act_module pool_modulepost_conv_modulepost_norm_modulepost_act_moduleprojected_pool_moduler1r|r2r2r3create_x3d_heads`E      rrrki@g@ )r5r5r5r5)rrrr)rrrrg@i) input_channelinput_clip_lengthinput_crop_sizemodel_num_classro width_factor depth_factorrrrr stem_dim_instem_conv_kernel_sizestem_conv_stridestage_conv_kernel_sizestage_spatial_stridestage_temporal_striderUbottleneck_factorr6r7 head_dim_out head_pool_acthead_bn_lin5_onhead_activationhead_output_with_global_averagerrrrrrrrrrrrrrrrrrc/Cstjdg}t| |}t||| | dd| D||| | d }||gd}d}| }t||dd} t| |dd}!t|!|dd}"|| |!|"g}#|}$tt|D];}%t|#|%|}&t||&}'t ||%|}(||%||%||%f})t |(|$|'|&|||%|)||| || |d }*||*|&}$qO| d t |}+| d t |},||,ksJd ||+ksJd ||,tt ||+tt ||+f}-t|&|'||||-||| ||||d }.||.tt|dS)u4 X3D model builder. It builds a X3D network backbone, which is a ResNet. Christoph Feichtenhofer. "X3D: Expanding Architectures for Efficient Video Recognition." https://arxiv.org/abs/2004.04730 :: Input ↓ Stem ↓ Stage 1 ↓ . . . ↓ Stage N ↓ Head Args: input_channel (int): number of channels for the input video clip. input_clip_length (int): length of the input video clip. Value for different models: X3D-XS: 4; X3D-S: 13; X3D-M: 16; X3D-L: 16. input_crop_size (int): spatial resolution of the input video clip. Value for different models: X3D-XS: 160; X3D-S: 160; X3D-M: 224; X3D-L: 312. model_num_class (int): the number of classes for the video dataset. dropout_rate (float): dropout rate. width_factor (float): width expansion factor. depth_factor (float): depth expansion factor. Value for different models: X3D-XS: 2.2; X3D-S: 2.2; X3D-M: 2.2; X3D-L: 5.0. norm (callable): a callable that constructs normalization layer. norm_eps (float): normalization epsilon. norm_momentum (float): normalization momentum. activation (callable): a callable that constructs activation layer. stem_dim_in (int): input channel size for stem before expansion. stem_conv_kernel_size (tuple): convolutional kernel size(s) of stem. stem_conv_stride (tuple): convolutional stride size(s) of stem. stage_conv_kernel_size (tuple): convolutional kernel size(s) for conv_b. stage_spatial_stride (tuple): the spatial stride for each stage. stage_temporal_stride (tuple): the temporal stride for each stage. bottleneck_factor (float): bottleneck expansion factor for the 3x3x3 conv. se_ratio (float): if > 0, apply SE to the 3x3x3 conv, with the SE channel dimensionality being se_ratio times the 3x3x3 conv dim. inner_act (callable): whether use Swish activation for act_b or not. head_dim_out (int): output channel size of the X3D head. head_pool_act (callable): a callable that constructs resnet pool activation layer such as nn.ReLU. head_bn_lin5_on (bool): if True, perform normalization on the features before the classifier. head_activation (callable): a callable that constructs activation layer. head_output_with_global_average (bool): if True, perform global averaging on the head output. Returns: (nn.Module): the X3D network. zPYTORCHVIDEO.model.create_x3dcSr=r>r2r?r2r2r3rBrCzcreate_x3d..) rrrrrrrrr)rrrrr)divisor) rcr8r9r:rUrrrrrr6rr7rrz*Clip length doesn't match temporal stride!z'Crop size doesn't match spatial stride!) r8r9r:rqrlrmrrrrnrorrp)blocks)torch_C_log_api_usage_oncerr4rfrelenintrrjr_r`mathceilrr r+rg)/rrrrrorrrrrrrrrrrrrUrr6r7rrrrrr stem_dim_outstem stage_depths exp_stage stage_dim1 stage_dim2 stage_dim3 stage_dim4 stage_dimsr8rhr:r9rcstage_conv_stridestagetotal_spatial_stridetotal_temporal_stridehead_pool_kernel_sizeheadr2r2r3 create_x3ds m         rcsxeZdZdZdddddddddejdejdejdejdejd ejd ejd dffd d Zdejd ejfddZ Z S)ru. A pooling module augmented with Conv, Normalization and Activation both before and after pooling for the head layer of X3D. :: Conv3d (pre_conv) ↓ Normalization (pre_norm) ↓ Activation (pre_act) ↓ Pool3d ↓ Conv3d (post_conv) ↓ Normalization (post_norm) ↓ Activation (post_act) Nrrrsrtrur*rvrwrxrcsDtt|t|jdusJ|jdusJ|jdus JdS)a Args: pre_conv (torch.nn.modules): convolutional module. pre_norm (torch.nn.modules): normalization module. pre_act (torch.nn.modules): activation module. pool (torch.nn.modules): pooling module. post_conv (torch.nn.modules): convolutional module. post_norm (torch.nn.modules): normalization module. post_act (torch.nn.modules): activation module. N)super__init__r localsrsr*rv)selfrsrtrur*rvrwrx __class__r2r3rs  zProjectedPool.__init__rXcCsr||}|jdur||}|jdur||}||}||}|jdur-||}|jdur7||}|SrWrr)rrXr2r2r3forwards           zProjectedPool.forward) __name__ __module__ __qualname____doc__r+ModulerrTensorr __classcell__r2r2rr3rs6  r)+rtypingrrnumpyr_rtorch.nnr+fvcore.nn.squeeze_excitationr pytorchvideo.layers.convolutionsrpytorchvideo.layers.swishrpytorchvideo.layers.utilsrrr pytorchvideo.models.headr pytorchvideo.models.netr pytorchvideo.models.resnetr r rpytorchvideo.models.stemr BatchNorm3dReLUrfloatrr4rTboolrbrjrrrrr2r2r2r3s             \          f    \           !#$%&'( L