o ϯiwG@s$ddlmZmZmZmZmZddlZddlmZddl m Z ddl m Z m Z ddlmZmZmZddlmZmZddlmZdd d d d Zd ddddddddejejeefdddejejfddddddddeeeefeeeeffe ejdddd d!d"eeeefd#ed$eed%eed&eeegejfd'eed(ed)ed*ed+ed,ed-eed.eed/eeed0eeed1eeeefd2eeed3eeed4eeeed5eeeed6eeed7eeeed8eeed9eeed:eeeeefd;edeed?ed@edAejf@dBdCZ d dddddddDdejejeefdddejejfddddddEdFdeeeefeeeeffe ejdGdej!dHdIdJddK"d"eeeefd#ed$eed%eed&eeegejfd'eed(ed)ed*ed+ed,ed-eed.eed/eeed0eeed1eeeefd2eeed3eeed4eeeed5eeeed6eeed7eeeed8eeed9eeed:eeeeefd;edeed?ed@edLeedMedNedAejfFdOdPZ"GdQdRdRejZ#GdSdTdTZ$GdUdVdVejZ%dS)W)CallableListOptionalTupleUnionN)set_attributes)create_res_basic_headcreate_res_roi_pooling_head)DetectionBBoxNetworkMultiPathWayWithFuseNet)create_bottleneck_blockcreate_res_stage)create_res_basic_stemrrr)r)rrr)r$r)2e)r)rr)rrr)rrrig?)@r))rrr)rr)rrrr )rrrr!))rrrr"rrrr#)r#r#r#r#)r!r!r!r!r$)rr)r"r"r"r"r%)rrrrr&))rrr) rrr"T) slowfast_channel_reduction_ratio"slowfast_conv_channel_fusion_ratio slowfast_fusion_conv_kernel_sizeslowfast_fusion_conv_stridefusion_builderinput_channels model_depthmodel_num_class dropout_ratenorm activation stem_function stem_dim_outsstem_conv_kernel_sizesstem_conv_strides stem_poolstem_pool_kernel_sizesstem_pool_stridesstage_conv_a_kernel_sizesstage_conv_b_kernel_sizesstage_conv_b_num_groupsstage_conv_b_dilationsstage_spatial_stridesstage_temporal_strides bottleneckhead head_poolhead_pool_kernel_sizeshead_output_sizehead_activationhead_output_with_global_averager(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErFreturnc3s&tjdt|}|tvsJ|dtt|} t|tr'|f}t|tr1|f|}t|trB|ft| }|f|}|durXt |d|||| | t| ddj }g}!t |D]2}"|! | |"||"| |"| |"||"dd| |"D||"||"||"d d||"D| | d q^g}#|# t t|!|| ddd d | d}$|$d }%t t| D]}&|$|$||dg}'|%d g}(|%g})|D]}*|'|$|*g}'|(|%d |*g}(|)|%|*g})qg}+t |D]}"| |&},||"|&}-||"|&ddf}.t|-dtrdd|-Dndd|-D}/d||"|&||"|&f}0|+ t|,|'|"|(|"|)|"||"|&|-|.|/||"|&|0||"|&dd||"|&ddkrY||"|&dn ||"|&dd||"|&ddkru||"|&dn ||"|&ddf||"|&||"|&| | dq|# t t|+||%|&dd d |%}$|%d}%qdurd}1n-tjkrćfddt |D}1ntjkrׇfddt |D}1ntd|1|# tdt|1d|$}2|D] }*|2|$|*}2q|dur |# ||2|d|||dtt|#dS)u_ Build SlowFast model for video recognition, SlowFast model involves a Slow pathway, operating at low frame rate, to capture spatial semantics, and a Fast pathway, operating at high frame rate, to capture motion at fine temporal resolution. The Fast pathway can be made very lightweight by reducing its channel capacity, yet can learn useful temporal information for video recognition. Details can be found from the paper: Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. "SlowFast networks for video recognition." https://arxiv.org/pdf/1812.03982.pdf :: Slow Input Fast Input ↓ ↓ Stem Stem ↓ ⭠ Fusion- ↓ Stage 1 Stage 1 ↓ ⭠ Fusion- ↓ . . ↓ ↓ Stage N Stage N ↓ ⭠ Fusion- ↓ ↓ Head Args: slowfast_channel_reduction_ratio (int): Corresponds to the inverse of the channel reduction ratio, $eta$ between the Slow and Fast pathways. slowfast_conv_channel_fusion_ratio (int): Ratio of channel dimensions between the Slow and Fast pathways. DEPRECATED slowfast_fusion_conv_kernel_size (tuple): the convolutional kernel size used for fusion. DEPRECATED slowfast_fusion_conv_stride (tuple): the convolutional stride size used for fusion. fusion_builder (Callable[[int, int], nn.Module]): Builder function for generating the fusion modules based on stage dimension and index input_channels (tuple): number of channels for the input video clip. model_depth (int): the depth of the resnet. model_num_class (int): the number of classes for the video dataset. dropout_rate (float): dropout rate. norm (callable): a callable that constructs normalization layer. activation (callable): a callable that constructs activation layer. stem_function (Tuple[Callable]): a callable that constructs stem layer. Examples include create_res_basic_stem. Indexed by pathway stem_dim_outs (tuple): output channel size to stem. stem_conv_kernel_sizes (tuple): convolutional kernel size(s) of stem. stem_conv_strides (tuple): convolutional stride size(s) of stem. stem_pool (Tuple[Callable]): a callable that constructs resnet head pooling layer. Indexed by pathway stem_pool_kernel_sizes (tuple): pooling kernel size(s). stem_pool_strides (tuple): pooling stride size(s). stage_conv_a_kernel_sizes (tuple): convolutional kernel size(s) for conv_a. stage_conv_b_kernel_sizes (tuple): convolutional kernel size(s) for conv_b. stage_conv_b_num_groups (tuple): number of groups for groupwise convolution for conv_b. 1 for ResNet, and larger than 1 for ResNeXt. stage_conv_b_dilations (tuple): dilation for 3D convolution for conv_b. stage_spatial_strides (tuple): the spatial stride for each stage. stage_temporal_strides (tuple): the temporal stride for each stage. bottleneck (Tuple[Tuple[Callable]]): a callable that constructs bottleneck block layer. Examples include: create_bottleneck_block. Indexed by pathway and stage index head (callable): a callable that constructs the resnet-style head. Ex: create_res_basic_head head_pool (callable): a callable that constructs resnet head pooling layer. head_output_sizes (tuple): the size of output tensor for head. head_activation (callable): a callable that constructs activation layer. head_output_with_global_average (bool): if True, perform global averaging on the head output. Returns: (nn.Module): SlowFast model. z"PYTORCHVIDEO.model.create_slowfastz is not in Nrr)r(conv_fusion_channel_ratioconv_kernel_size conv_strider1r2 max_stage_idxcSg|]}|dqSr.0sizerNrNP/home/ubuntu/.local/lib/python3.10/site-packages/pytorchvideo/models/slowfast.py z#create_slowfast..cSrLrMrNrOrNrNrRrSrT) in_channels out_channelsrIrJ conv_paddingpoolpool_kernel_size pool_stride pool_paddingr1r2) fusion_dim_in stage_idx)multipathway_blocksmultipathway_fusionrcSrLrMrNrOrNrNrRrScSsg|] }dd|DqS)cSrLrMrNrOrNrNrRrSr`z.create_slowfast...rN)rPsizesrNrNrRrSsr)depthdim_in dim_innerdim_outr@conv_a_kernel_size conv_a_strideconv_a_paddingconv_b_kernel_size conv_b_strideconv_b_paddingconv_b_num_groupsconv_b_dilationr1r2csg|]}|qSrNrNrPidx)rDrBrNrRrSHscsg|] }|dddqS)r")rrr) kernel_sizestridepaddingrNrn)rBrCrNrRrSJszUnsupported pool_model type F) retain_listrX) in_features out_featuresrX output_sizer0r2output_with_global_average)blocks)torch_C_log_api_usage_oncelen_MODEL_STAGE_DEPTHkeys isinstanceintrFastToSlowFusionBuilder create_modulerangeappendr nn ModuleListrAdaptiveAvgPool3d AvgPool3dNotImplementedErrorPoolConcatPathwayr )3r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErF _num_pathway stage_depthsstems pathway_idxstages stage_dim_in stage_dim_outropathway_stage_dim_inpathway_stage_dim_innerpathway_stage_dim_outreduction_ratiostagerbstage_conv_a_kernelstage_conv_a_stridestage_conv_a_paddingstage_conv_b_stride pool_modelhead_in_featuresrN)rDrBrCrRcreate_slowfasts<                           rP)r"r"r"r r)rrrrr))rrr)r'rrF)rrg?)"r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErFhead_spatial_resolutionhead_spatial_scalehead_sampling_ratiorrrc"& Cstd id|d|d|d|d|d|d|d|d | d | d | d | d |d|d|d|d|d|d|d|d|d|dtddd|d|}"| ddtt|d}#| d| d}$|#|#|$}%t|%|d|||||| |!d }t|"|S)!u Build SlowFast model for video detection, SlowFast model involves a Slow pathway, operating at low frame rate, to capture spatial semantics, and a Fast pathway, operating at high frame rate, to capture motion at fine temporal resolution. The Fast pathway can be made very lightweight by reducing its channel capacity, yet can learn useful temporal information for video recognition. Details can be found from the paper: Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. "SlowFast networks for video recognition." https://arxiv.org/pdf/1812.03982.pdf :: Slow Input Fast Input Bounding Box Input ↓ ↓ ↓ Stem Stem ↓ ↓ ⭠ Fusion- ↓ ↓ Stage 1 Stage 1 ↓ ↓ ⭠ Fusion- ↓ ↓ . . ↓ ↓ ↓ ↓ Stage N Stage N ↓ ↓ ⭠ Fusion- ↓ ↓ ↓ ↓ ↓----------> Head <--------↓ Args: slowfast_channel_reduction_ratio (int): Corresponds to the inverse of the channel reduction ratio, $eta$ between the Slow and Fast pathways. slowfast_conv_channel_fusion_ratio (int): Ratio of channel dimensions between the Slow and Fast pathways. DEPRECATED slowfast_fusion_conv_kernel_size (tuple): the convolutional kernel size used for fusion. DEPRECATED slowfast_fusion_conv_stride (tuple): the convolutional stride size used for fusion. fusion_builder (Callable[[int, int], nn.Module]): Builder function for generating the fusion modules based on stage dimension and index input_channels (tuple): number of channels for the input video clip. model_depth (int): the depth of the resnet. model_num_class (int): the number of classes for the video dataset. dropout_rate (float): dropout rate. norm (callable): a callable that constructs normalization layer. activation (callable): a callable that constructs activation layer. stem_function (Tuple[Callable]): a callable that constructs stem layer. Examples include create_res_basic_stem. Indexed by pathway stem_dim_outs (tuple): output channel size to stem. stem_conv_kernel_sizes (tuple): convolutional kernel size(s) of stem. stem_conv_strides (tuple): convolutional stride size(s) of stem. stem_pool (Tuple[Callable]): a callable that constructs resnet head pooling layer. Indexed by pathway stem_pool_kernel_sizes (tuple): pooling kernel size(s). stem_pool_strides (tuple): pooling stride size(s). stage_conv_a_kernel_sizes (tuple): convolutional kernel size(s) for conv_a. stage_conv_b_kernel_sizes (tuple): convolutional kernel size(s) for conv_b. stage_conv_b_num_groups (tuple): number of groups for groupwise convolution for conv_b. 1 for ResNet, and larger than 1 for ResNeXt. stage_conv_b_dilations (tuple): dilation for 3D convolution for conv_b. stage_spatial_strides (tuple): the spatial stride for each stage. stage_temporal_strides (tuple): the temporal stride for each stage. bottleneck (Tuple[Tuple[Callable]]): a callable that constructs bottleneck block layer. Examples include: create_bottleneck_block. Indexed by pathway and stage index head (callable): a a callable that constructs the detection head which can take in the additional input of bounding boxes. Ex: create_res_roi_pooling_head head_pool (callable): a callable that constructs resnet head pooling layer. head_output_sizes (tuple): the size of output tensor for head. head_activation (callable): a callable that constructs activation layer. head_output_with_global_average (bool): if True, perform global averaging on the head output. head_spatial_resolution (tuple): h, w sizes of the RoI interpolation. head_spatial_scale (float): scale the input boxes by this number. head_sampling_ratio (int): number of inputs samples to take for each output sample interpolation. 0 to take samples densely. Returns: (nn.Module): SlowFast model. r(r)r*r+r-r.r/r0r1r2r4r5r6r7r8r9r:r;r<r=r>r?r@rANrBrCrrr) rtrurXrvr0r2rw resolution spatial_scalesampling_ratiorN)rr r|r}r r )&r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErFrrrmodelrslow_fast_betarrNrNrRcreate_slowfast_with_roi_headhs'     !"%  rc sZeZdZdZ   ddedeejdeddffd d Z d e e j de j fd d Z ZS)rz Given a list of tensors, perform optional spatio-temporal pool and concatenate the tensors along the channel dimension. FNrrsrXdimrGctt|tdS)a7 Args: retain_list (bool): if True, return the concatenated tensor in a list. pool (nn.module_list): if not None, list of pooling models for different pathway before performing concatenation. dim (int): dimension to performance concatenation. Nsuper__init__rlocals)selfrsrXr __class__rNrRrLs zPoolConcatPathway.__init__xcCs|jdurt|t|jksJg}tt|D]&}||dur>|jdur7|j|dur7|j|||||<|||q|jrIt|dgSt|dS)Nr)rXr|rrrsrycat)rroutputindrNrNrRforward\s   zPoolConcatPathway.forward)FNr)__name__ __module__ __qualname____doc__boolrrrrrrryTensorr __classcell__rNrNrrRrFs"rc@sneZdZejddejdfdededeedeede d ed ed e d ed dfddZ deded ej fddZ dS)rgh㈵>g?rr(rHrIrJr1norm_eps norm_momentumr2rKrGNc Cst|tdS)a Given a list of two tensors from Slow pathway and Fast pathway, fusion information from the Fast pathway to the Slow on through a convolution followed by a concatenation, then return the fused list of tensors from Slow and Fast pathway in order. Args: slowfast_channel_reduction_ratio (int): Reduction ratio from the stage dimension. Used to compute conv_dim_in = fusion_dim_in // slowfast_channel_reduction_ratio conv_fusion_channel_ratio (int): channel ratio for the convolution used to fuse from Fast pathway to Slow pathway. conv_kernel_size (int): kernel size of the convolution used to fuse from Fast pathway to Slow pathway. conv_stride (int): stride size of the convolution used to fuse from Fast pathway to Slow pathway. norm (callable): a callable that constructs normalization layer, examples include nn.BatchNorm3d, None (not performing normalization). norm_eps (float): normalization epsilon. norm_momentum (float): normalization momentum. activation (callable): a callable that constructs activation layer, examples include: nn.ReLU, nn.Softmax, nn.Sigmoid, and None (not performing activation). max_stage_idx (int): Returns identity module if we exceed this N)rr) rr(rHrIrJr1rrr2rKrNrNrRrls#z FastToSlowFusionBuilder.__init__r\r]cCs||jkr tS||j}tj|t||j|j|jdd|jDdd}|j dur-dn |j ||j|j |j d}|j durAdn| }t |||dS)z Creates the module for the given stage Args: fusion_dim_in (int): input stage dimension stage_idx (int): which stage this is cSrLrMrN)rPk_sizerNrNrRrSr`z9FastToSlowFusionBuilder.create_module..F)rprqrrbiasN) num_featuresepsmomentum)conv_fast_to_slowr1r2)rKrIdentityr(Conv3drrHrIrJr1rrr2FuseFastToSlow)rr\r] conv_dim_inr norm_moduleactivation_modulerNrNrRrs2     z%FastToSlowFusionBuilder.create_module) rrrr BatchNorm3dReLUrfloatrrrModulerrNrNrNrRrks8   %rc sNeZdZdZ  d dejdeejdeejddffdd Zd d ZZ S) ra  Given a list of two tensors from Slow pathway and Fast pathway, fusion information from the Fast pathway to the Slow on through a convolution followed by a concatenation, then return the fused list of tensors from Slow and Fast pathway in order. Nrr1r2rGcr)z Args: conv_fast_to_slow (nn.module): convolution to perform fusion. norm (nn.module): normalization module. activation (torch.nn.modules): activation module. Nr)rrr1r2rrNrRrs zFuseFastToSlow.__init__cCsZ|d}|d}||}|jdur||}|jdur!||}t||gd}||gS)Nrr)rr1r2ryr)rrx_sx_ffusex_s_fuserNrNrRrs     zFuseFastToSlow.forward)NN) rrrrrrrrrrrNrNrrRrs r)&typingrrrrrrytorch.nnrpytorchvideo.layers.utilsrpytorchvideo.models.headrr pytorchvideo.models.netr r r pytorchvideo.models.resnetr rpytorchvideo.models.stemrr}rr MaxPool3drrrrrrSigmoidrrrrrNrNrNrRs       ! " #$ % &(, 01 5 67FG HIJKL W   ! " #$ % &(, 01 5 67FG HIJKLMNO _%J