o ϯiz;A@sddlmZddlmZmZmZmZddlZddlm Z ddl m Z m Z ddl mZmZddlmZddlmZddlmZmZd d lmZGd d d e jZd d ddd dddddddd dd ddddddddddedddddedededed ed!ed"ed#ed$ed%eed&eed'eed(ed)ed*ed+ed,ed-ed.ed/ed0ed1eeeed2eeeed3eeeed4eeeed5eed6eed7eed8ed9ed:ed;e jf@d)partial)CallableListOptionalTupleN)MultiScaleBlock#SpatioTemporalClsPositionalEncoding) round_widthset_attributes)create_vit_basic_head)init_net_weights) _size_2_t _size_3_t)create_conv_patch_embedcszeZdZdZdeejdejdeejdeejdejdeejdeejd d ffd d Zd e j d e j fddZ Z S)MultiscaleVisionTransformersu Multiscale Vision Transformers Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, Christoph Feichtenhofer https://arxiv.org/abs/2104.11227 :: PatchEmbed ↓ PositionalEncoding ↓ Dropout ↓ Normalization ↓ Block 1 ↓ . . . ↓ Block N ↓ Normalization ↓ Head The builder can be found in `create_mvit`. patch_embedcls_positional_encodingpos_dropnorm_patch_embedblocks norm_embedheadreturnNcs:tt|tt|dsJdt|ddddS)a Args: patch_embed (nn.Module): Patch embed module. cls_positional_encoding (nn.Module): Positional encoding module. pos_drop (Optional[nn.Module]): Dropout module after patch embed. blocks (nn.ModuleList): Stack of multi-scale transformer blocks. norm_layer (nn.Module): Normalization layer before head. head (Optional[nn.Module]): Head module. patch_embed_shapez@cls_positional_encoding should have attribute patch_embed_shape.g{Gz?vit)init_stdstyleN)super__init__r localshasattrr )selfrrrrrrr __class__[/home/ubuntu/.local/lib/python3.10/site-packages/pytorchvideo/models/vision_transformers.pyr2s  z%MultiscaleVisionTransformers.__init__xcCs|jdur ||}||}|jdur||}|jdur#||}|jj}|jD] }|||\}}q*|jdur>||}|jdurH||}|S)N)rrrrrrrr)r"r'thwblkr%r%r&forwardMs            z$MultiscaleVisionTransformers.forward) __name__ __module__ __qualname____doc__rnnModule ModuleListrtorchTensorr* __classcell__r%r%r#r&rs(   rT layernorm`)r7r9)r;)rr7r7Fg@convg?i) cls_embed_on sep_pos_embeddepthnormenable_patch_embedinput_channelspatch_embed_dimconv_patch_embed_kernelconv_patch_embed_strideconv_patch_embed_paddingenable_patch_embed_norm use_2d_patch num_heads mlp_ratioqkv_biasdropout_rate_blockdroppath_rate_block pooling_mode pool_first embed_dim_mulatten_head_mulpool_q_stride_sizepool_kv_stride_sizepool_kv_stride_adaptivepool_kvq_kernelrhead_dropout_ratehead_activationhead_num_classes spatial_size temporal_sizer>r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSrTrUrVrrWrXrYrc1sF| r |dks Jd|dur|dusJd|dkr"ttjdd}ntdt|tr/||f}| r4tjntj} |rCt||| | | | d nd}!||d |dg| rVd t | n| |rhfd d t t Dn}"t ||"||d}#dd t d ||D}$|dkrtj|d}%t |dt |d}&}'|durt t |D]|d|&|d <q|durt t |D]|d|'|d <q| r||nd}(t})dd t |D}*dd t |D}+dd t |Ddd t |D},|dur8t t |D]3|dd|d <|dur#||*|d <qdd |ddD|*|d <q|duri|g}t |D]#t d kr_fdd t t D|gqE|durt t |D]3|dd|,|d <|dur||+|d <qtdd |ddD|+|d <qtt |D]Ht||'ddd}t||&|d}t||&dt||'dd}-|)t||-|||||$||*|+|,|||dq|-}.||.}/|dur||.||rdnd||d}0nd}0t|!|#|dkr|%nd|(|)|/|0dS) a Build Multiscale Vision Transformers (MViT) for recognition. A Vision Transformer (ViT) is a specific case of MViT that only uses a single scale attention block. Args: spatial_size (_size_2_t): Input video spatial resolution (H, W). If a single int is given, it assumes the width and the height are the same. temporal_size (int): Number of frames in the input video. cls_embed_on (bool): If True, use cls embed in the model. Otherwise features are average pooled before going to the final classifier. sep_pos_embed (bool): If True, perform separate spatiotemporal embedding. depth (int): The depth of the model. norm (str): Normalization layer. It currently supports "layernorm". enable_patch_embed (bool): If true, patchify the input video. If false, it assumes the input should have the feature dimension of patch_embed_dim. input_channels (int): Channel dimension of the input video. patch_embed_dim (int): Embedding dimension after patchifing the video input. conv_patch_embed_kernel (Tuple[int]): Kernel size of the convolution for patchifing the video input. conv_patch_embed_stride (Tuple[int]): Stride size of the convolution for patchifing the video input. conv_patch_embed_padding (Tuple[int]): Padding size of the convolution for patchifing the video input. enable_patch_embed_norm (bool): If True, apply normalization after patchifing the video input. use_2d_patch (bool): If True, use 2D convolutions to get patch embed. Otherwise, use 3D convolutions. num_heads (int): Number of heads in the first transformer block. mlp_ratio (float): Mlp ratio which controls the feature dimension in the hidden layer of the Mlp block. qkv_bias (bool): If set to False, the qkv layer will not learn an additive bias. Default: False. dropout_rate_block (float): Dropout rate for the attention block. droppath_rate_block (float): Droppath rate for the attention block. pooling_mode (str): Pooling mode. Option includes "conv" (learned pooling), "avg" (average pooling), and "max" (max pooling). pool_first (bool): If set to True, pool is applied before qkv projection. Otherwise, pool is applied after qkv projection. Default: False. embed_dim_mul (Optional[List[List[int]]]): Dimension multiplication at layer i. If X is used, then the next block will increase the embed dimension by X times. Format: [depth_i, mul_dim_ratio]. atten_head_mul (Optional[List[List[int]]]): Head dimension multiplication at layer i. If X is used, then the next block will increase the head by X times. Format: [depth_i, mul_dim_ratio]. pool_q_stride_size (Optional[List[List[int]]]): List of stride sizes for the pool q at each layer. Format: [[i, stride_t_i, stride_h_i, stride_w_i], ...,]. pool_kv_stride_size (Optional[List[List[int]]]): List of stride sizes for the pool kv at each layer. Format: [[i, stride_t_i, stride_h_i, stride_w_i], ...,]. pool_kv_stride_adaptive (Optional[_size_3_t]): Initial kv stride size for the first block. The stride size will be further reduced at the layer where q is pooled with the ratio of the stride of q pooling. If pool_kv_stride_adaptive is set, then pool_kv_stride_size should be none. pool_kvq_kernel (Optional[_size_3_t]): Pooling kernel size for q and kv. It None, the kernel_size is [s + 1 if s > 1 else s for s in stride_size]. head (Callable): Head model. head_dropout_rate (float): Dropout rate in the head. head_activation (Callable): Activation in the head. head_num_classes (int): Number of classes in the final classification head. Example usage (building a MViT_B model for Kinetics400): spatial_size = 224 temporal_size = 16 embed_dim_mul = [[1, 2.0], [3, 2.0], [14, 2.0]] atten_head_mul = [[1, 2.0], [3, 2.0], [14, 2.0]] pool_q_stride_size = [[1, 1, 2, 2], [3, 1, 2, 2], [14, 1, 2, 2]] pool_kv_stride_adaptive = [1, 8, 8] pool_kvq_kernel = [3, 3, 3] head_num_classes = 400 MViT_B = create_multiscale_vision_transformers( spatial_size=spatial_size, temporal_size=temporal_size, embed_dim_mul=embed_dim_mul, atten_head_mul=atten_head_mul, pool_q_stride_size=pool_q_stride_size, pool_kv_stride_adaptive=pool_kv_stride_adaptive, pool_kvq_kernel=pool_kvq_kernel, head_num_classes=head_num_classes, ) rz-If use_2d_patch, temporal_size needs to be 1.NzEpool_kv_stride_size should be none if pool_kv_stride_adaptive is set.r6gư>)epszOnly supports layernorm.) in_channels out_channelsconv_kernel_size conv_stride conv_paddingr=rrcsg|] }||qSr%r%.0i) input_dims input_stirder%r& sz9create_multiscale_vision_transformers..) embed_dimrr?has_clscSsg|]}|qSr%)item)rdr'r%r%r&rh sr<)pcSg|]}gqSr%r%rcr%r%r&rh cSrmr%r%rcr%r%r&rh!rncSrmr%r%rcr%r%r&rh"rncSrmr%r%rcr%r%r&rh#rncS g|] }|dkr |dn|qSrbr%rdsr%r%r&rh+cs&g|]}t||dqSrb)max)rdd) _stride_kvrestride_qr%r&rh5scSrorbr%rpr%r%r&rhArr) min_widthdivisor)rx)dimdim_outrJrKrL dropout_rate droppath_rate norm_layerkernel_q kernel_kvrv stride_kv pool_mode has_cls_embedrPclsmean) in_features out_features seq_pool_typer{ activation)rrrrrrr)rr/ LayerNormNotImplementedError isinstanceintConv2dConv3drtuplerangelenrr2linspaceDropoutonesr1appendr rr)1rZr[r>r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSrTrUrVrrWrXrYr} conv_patch_oprrrdprrdim_mulhead_mulr mvit_blockspool_qpool_kvrrzrir head_modelr%)rurerfrgrvr&%create_multiscale_vision_transformersbs{                  r) functoolsrtypingrrrrr2torch.nnr/pytorchvideo.layersrrpytorchvideo.layers.utilsr r pytorchvideo.models.headr pytorchvideo.models.weight_initr torch.nn.common_typesr rstemrr0rrboolstrfloatrr%r%r%r&s     U      !"#$