o ϯi2@sddlmZmZmZddlZddlmZddlmZddlm Z  Gdddejj Z Gdd d ej Z Gd d d ej Z Gd d d ej ZGdddej ZGdddejZGdddej ZdS))ListOptionalTupleN)set_attributes)nn)pack_padded_sequencecsJeZdZdZdeffdd Z d dejdeejdejfd d Z Z S) MaskedTemporalPoolingz} Applies temporal pooling operations on masked inputs. For each pooling operation all masked values are ignored. methodcs t|dvs J||_dS)a method (str): the method of pooling to use. Options: 'max': reduces temporal dimension to each valid max value. 'avg': averages valid values in the temporal dimension. 'sum': sums valid values in the temporal dimension. Note if all batch row elements are invalid, the temporal dimension is pooled to 0 values. )maxavgsumN)super__init___method)selfr  __class__Z/home/ubuntu/.local/lib/python3.10/site-packages/pytorchvideo/models/masked_multistream.pyr)s  zMaskedTemporalPooling.__init__NxmaskreturncCsZ|dks Jd|jd|jd}}|dur#tj||ftjd}|jdkrQtd||ddf<||d jd d }d||ddf<tj |dd d}|S|jd kr|| d }|||d jd d }|j d d  }|j dd }| |jdd  d |}|S|jd kr|| d }|j dd }|St|jd)aA Args: x (torch.Tensor): tensor with shape (batch_size, seq_len, feature_dim) mask (torch.Tensor): bool tensor with shape (batch_size, seq_len). Sequence elements that are False are invalid. Returns: Tensor with shape (batch_size, feature_dim) 5Requires x shape (batch_size x seq_len x feature_dim)rNdtyper z-infdimr )minr z/ not available options are: 'max', 'avg', 'sum')rshapetorchonesboolrfloatviewanyr unsqueezer intdivclampexpandsizeNotImplementedError)rrrbtinvalid_first_dim valid_lengthsrrrforward6s0    &   zMaskedTemporalPooling.forwardN) __name__ __module__ __qualname____doc__strrr"Tensorrr3 __classcell__rrrrr#srcsheZdZdZddedeffdd Zedeej fdd Z dd ej d eej dej fd dZ Z S)TransposeMultiheadAttentiona  Wrapper for nn.MultiheadAttention which first transposes the input tensor from (batch_size, seq_len, feature_dim) to (seq_length, batch_size, feature_dim), then applies the attention and transposes the attention outputs back to the input shape. r feature_dim num_headscs$ttj||d|_d|_dS)z Args: feature_dim (int): attention embedding dimension num_heads (int): number of attention heads ) embed_dimr>N)r rrMultiheadAttention _attention_attention_weights)rr=r>rrrrhs  z$TransposeMultiheadAttention.__init__rcCs|jS)zD Contains attention weights from last forward call. )rB)rrrrattention_weightstsz-TransposeMultiheadAttention.attention_weightsNrrcCsf|dks Jd|durd|dddf<|}|dd}|j||||d\}|_|dd}|S)aH Args: x (torch.Tensor): tensor of shape (batch_size, seq_len, feature_dim) mask (torch.Tensor): bool tensor with shape (batch_size, seq_len). Sequence elements that are False are invalid. Returns: Tensor with shape (batch_size, seq_len, feature_dim) rrNTrr)key_padding_mask)r transposerArB)rrr attn_outputrrrr3{s   z#TransposeMultiheadAttention.forward)rr4) r5r6r7r8r)rpropertyrr"r:rCr3r;rrrrr<`s r<csNeZdZdZ ddededeffdd Zd ej d ej d ej fd d Z Z S)LearnMaskedDefaultaU Learns default values to fill invalid entries within input tensors. The invalid entries are represented by a mask which is passed into forward alongside the input tensor. Note the default value is only used if all entries in the batch row are invalid rather than just a portion of invalid entries within each batch row. gaussianFr= init_methodfreezecsnt|dkrtjt|| d|_dS|dkr0tjt|| d|_tj |jdSt |d)al Args: feature_dim (int): the size of the default value parameter, this must match the input tensor size. init_method (str): the initial default value parameter. Options: 'guassian' 'zeros' freeze (bool): If True, the learned default parameter weights are frozen. zeros) requires_gradrIz2 not available. Options are: 'zeros' or 'gaussian'N) r rr Parameterr"rL_learned_defaultsr:initnormal_r.)rr=rJrKrrrrs    zLearnMaskedDefault.__init__rrrcCsZ||jddjdd}td|D]}||}q|||jd|}|S)a Args: x (torch.Tensor): tensor of shape (batch_size, feature_dim). mask (torch.Tensor): bool tensor of shape (batch_size, seq_len) If all elements in the batch dimension are False the learned default parameter is used for that batch element. Returns: Tensor with shape (batch_size, feature_dim) rrrr)r&r!r'rangerr(r%rO)rrrirrrr3s   zLearnMaskedDefault.forward)rIF) r5r6r7r8r)r9r$rr"r:r3r;rrrrrHs$rHc s\eZdZdZ  ddedededeffdd Z dd ej d e ej d ej fddZ Z S)LSTMz? Wrapper for torch.nn.LSTM that handles masked inputs. Fdim_in hidden_dimdropout bidirectionalcsFttj||d||d|_|j|rd|n||_||_dS)z Args: dim_in (int): input feature dimension hidden_dim (int): hidden dimesion of lstm layer dropout (float): dropout rate - 0.0 if no dropout bidirectional (bool): bidirectional or forward only T) batch_firstrXrYN)r rrrTlstmflatten_parameters output_dimrY)rrVrWrXrYrrrrs   z LSTM.__init__Ndatarrc Cs|dksJ|jd|jd}}|dur!tj||ftjd}|jdd}t||d|dddd }| |\}\}}|j r_tj |dddddf|dddddfgd d } | S|d ddddf} | S) a Args: data (torch.Tensor): tensor with shape (batch_size, seq_len, feature_dim) mask (torch.Tensor): bool tensor with shape (batch_size, seq_len). Sequence elements that are False are invalid. Returns: Tensor with shape (batch_size, output_dim) - outoput_dim is determined by hidden_dim and whether bidirectional or not rrrNr)axisTF)rZenforce_sortedrr) rr!r"r#r$r rr+r-r\rYcat) rr_rr/r0lengthsx_packed_houtrrrr3s"  6z LSTM.forward)rUFr4) r5r6r7r8r)r%r$rr"r:rr3r;rrrrrTs*rTcsXeZdZdZ  ddededeffdd Z dd ejd eejd ejfd d Z Z S)TransposeTransformerEncoderzM Wrapper for torch.nn.TransformerEncoder that handles masked inputs. rrVr> num_layerscs$ttt||||_dS)z Args: dim_in (int): input feature dimension num_heads (int): number of heads in the nn.MultiHeadAttention layers num_layers (int): the number of sub-encoder-layers in the encoder N)r rrTransformerEncoderTransformerEncoderLayerencoder)rrVr>rirrrrs   z$TransposeTransformerEncoder.__init__Nr_rrcCsR|durd|dddf<|}|j|dd|ddd}|dddddfS)aD Args: data (torch.Tensor): tensor with shape (batch_size, seq_len, feature_dim) mask (torch.Tensor): bool tensor with shape (batch_size, seq_len). Sequence elements that are False are invalid. Returns: Tensor with shape (batch_size, feature_dim) NTrr)srcsrc_key_padding_mask)rlrE)rr_rrgrrrr3)s  z#TransposeTransformerEncoder.forward)rrr4) r5r6r7r8r)rr"r:rr3r;rrrrrhs&rhc@s:eZdZdZeeeeegZ de j de j de j fddZ dS)MaskedSequentialz A sequential container that overrides forward to take a mask as well as the usual input tensor. This mask is only applied to modules in _MASK_MODULES (which take the mask argument). inputrrcs<|D]tfdd|jDr||d}q|}q|S)Nc3s|]}t|VqdSr4) isinstance).0 mask_typemodulerr Ssz+MaskedSequential.forward..)r)r' _MASK_MODULES)rrprrrtrr3Qs  zMaskedSequential.forwardN) r5r6r7r8rrHr<rTrhrwr"r:r3rrrrroBs rocsZeZdZdZdejdeejddffdd Zde e e j e j fde j fd d Z ZS) MaskedMultiPathWayu Masked multi-pathway is composed of a list of stream nn.Modules followed by a fusion nn.Module that reduces these streams. Each stream module takes a mask and input tensor. :: Pathway 1 ... Pathway N ↓ ↓ Block 1 Block N ↓⭠ --Fusion----↓ multipathway_blocksmultipathway_fusionrNcstt|tdS)z Args: multipathway_blocks (nn.module_list): list of models from all pathways. multipathway_fusion (nn.module): fusion model. N)r rrlocals)rryrzrrrris zMaskedMultiPathWay.__init__ x_and_maskcCsHg}tt|jD]}||j|||q |jdur"||}|Sr4)rRlenryappendrz)rr|rg pathway_idxrrrrr3ws   zMaskedMultiPathWay.forward)r5r6r7r8r ModuleListrModulerrrr"r:r3r;rrrrrx[s rx)typingrrrr"pytorchvideo.layers.utilsrrtorch.nn.utils.rnnrrrr<rHrTrh Sequentialrorxrrrrs   =:6C/