o ߥi*3@shddlZddlZddlmZddlmmZGdddejZGdddejZ GdddejZ dS)Ncs@eZdZ     d fdd ZddZd d Zd d ZZS) SelfAttention'Ncstt|ddg|_|dur$|}||jvs$Jdg|jR||_||_||_||_||_ t t t |_ |_ |_t|jD],}|j t j|||dd|j t j|||dd|jt j|||ddqIt j||dd|_t jdd|_t jd d |_dS) a The basic (multi-head) Attention 'cell' containing the learnable parameters of Q, K and V :param int input_size: Feature input size of Q, K, V. :param int output_size: Feature -hidden- size of Q, K, V. :param int freq: The frequency of the sinusoidal positional encoding. :param int heads: Number of heads for the attention module. :param str | None pos_enc: The type of the positional encoding [supported: Absolute, Relative]. absoluterelativeNzSupported encodings: F) in_features out_featuresbiasdim?p)superr__init__permitted_encodingslower input_size output_sizeheadspos_encfreqnn ModuleListWkWqWvrangeappendLinearoutSoftmaxsoftmaxDropoutdrop)selfrrrrr_ __class__d/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/cv/video_summarization/pgl_sum.pyr sP   zSelfAttention.__init__cCs|j}|j}tjddt|D|jjjd}tjddt|dD|jjjd}||j dd}|j |j ddd}| |j dd}tj |||jjjd}t ||d||||d|f<t||d||||d|df<|S) a.Calculate the sinusoidal positional encoding based on the absolute position of each considered frame. Based on 'Attention is all you need' paper (https://arxiv.org/abs/1706.03762) :param int T: Number of frames contained in Q, K and V :return: Tensor with shape [T, T] cSg|]}|qSr+r+.0kr+r+r, Iz5SelfAttention.getAbsolutePosition..devicecSr-r+r+r.r+r+r,r1Kr2rrr )rrtorchtensorrr"weightr4reshapeshaperepeat_interleaverepeatzerossincos)r'TrdposiAPr+r+r,getAbsolutePosition?s&*z!SelfAttention.getAbsolutePositionc Cs|j}d|}|d }tjddt|D|jjjd}tjddt|D|jjjd}||jdd}|j |jddd}| |jdd}|||}tj |||jjjd}tjd dt|dD|jjjd} t |d d d| f||d d d| f|d d d| f||d d d| f<t |d d d| df||d d d| df|d d d| df||d d d| df<|S) a+Calculate the sinusoidal positional encoding based on the relative position of each considered frame. r_pos calculations as here: https://theaisummer.com/positional-embeddings/ :param int T: Number of frames contained in Q, K and V :return: Tensor with shape [T, T] r5rcSr-r+r+r.r+r+r,r1cr2z5SelfAttention.getRelativePosition..r3cSr-r+r+r.r+r+r,r1dr2rr cSr-r+r+r.r+r+r,r1or2N)rr6r7rr"r8r4r9r:r;r<r=r>r?) r'r@rrAmin_rposrCjr_posRPidxr+r+r,getRelativePositionXs, "" B6z!SelfAttention.getRelativePositionc Csg}t|jD]`}|j||}|j||}|j||}t||dd}|jdurR|jdkr@|j |j dd}||}n|jdkrR|j |j dd} || }| |} | | } t| |} || q|tj|dd} | | fS)a Compute the weighted frame features, based on either the global or local (multi-head) attention mechanism. :param torch.tensor x: Frame features with shape [T, input_size] :return: A tuple of: y: Weighted features based on the attention weights, with shape [T, input_size] att_weights : The attention weights (before dropout), with shape [T, T] rrNr)r@rr )rrrrrr6matmul transposerrEr:rKr$r&r r"catclone) r'xoutputsheadKQVenergiesrDrI att_weights _att_weightsyr+r+r,forwardxs&         zSelfAttention.forward)rrrrN)__name__ __module__ __qualname__rrErKrZ __classcell__r+r+r)r,r s2 rc4eZdZ       d fdd ZddZZS) MultiAttentionrrNrc stt|t|||||d|_||_|jdur>|jdks"Jdt|_t |jD]}|j t|||||ddq,gd|_ ||_ |j durb|j |_ |j |j vsdJdg|j RdSdS)a} Class wrapping the MultiAttention part of PGL-SUM; its key modules and parameters. :param int input_size: The expected input feature size. :param int output_size: The hidden feature size of the attention mechanisms. :param int freq: The frequency of the sinusoidal positional encoding. :param None | str pos_enc: The selected positional encoding [absolute, relative]. :param None | int num_segments: The selected number of segments to split the videos. :param int heads: The selected number of global heads. :param None | str fusion: The selected type of feature fusion. )rrrrrNr5znum_segments must be None or 2+)addmultavgmaxzFusion method must be: )rr`rr attention num_segmentsrrlocal_attentionrr permitted_fusionsfusionr) r'rrrrrgrrjr(r)r+r,rs:     "zMultiAttention.__init__c CsT||\}}|jdur|jdurt|jd|j}t|jD]}||}|d|}|||}|j||\} } tj ||| ddd|||<tj | ddd} |jdkrf|||| 7<q!|jdkrv|||| 9<q!|jdkr|||| 7<|||d<q!|jd krt ||| | |||<q!||fS) a Compute the weighted frame features, based on the global and locals (multi-head) attention mechanisms. :param torch.Tensor x: Tensor with shape [T, input_size] containing the frame features. :return: A tuple of: weighted_value: Tensor with shape [T, input_size] containing the weighted frame features. attn_weights: Tensor with shape [T, T] containing the attention weights. Nrrr5)rr rbrcrdre) rfrgrjmathceilr:rrhF normalizerOr6re) r'rPweighted_value attn_weights segment_sizesegmentleft_pos right_poslocal_xweighted_local_valueattn_local_weightsr+r+r,rZs@       zMultiAttention.forwardrrrNNrNr[r\r]rrZr^r+r+r)r,r`s/r`cr_) PGL_SUMrrNrc stt|t|||||||d|_tj||d|_tj|jjdd|_ tj dd|_ tj |dd|_ tj |jjdd|_t|_t|_dS) al Class wrapping the PGL-SUM model; its key modules and parameters. :param int input_size: The expected input feature size. :param int output_size: The hidden feature size of the attention mechanisms. :param int freq: The frequency of the sinusoidal positional encoding. :param None | str pos_enc: The selected positional encoding [absolute, relative]. :param None | int num_segments: The selected number of segments to split the videos. :param int heads: The selected number of global heads. :param None | str fusion: The selected type of feature fusion. )rrrrrgrrj)rr rrrgư>)normalized_shapeepsN)rrzrr`rfrr!linear_1r linear_2r%r& LayerNormnorm_y norm_linearReLUreluSigmoidsigmoid)r'rrrrrgrrjr)r+r,rs. zPGL_SUM.__init__cCs|d|jd}|}||\}}||}||}||}||}||}||}||}||}| |}| dd}||fS)a Produce frames importance scores from the frame features, using the PGL-SUM model. :param torch.Tensor frame_features: Tensor of shape [T, input_size] containing the frame features produced by using the pool5 layer of GoogleNet. :return: A tuple of: y: Tensor with shape [1, T] containing the frames importance scores in [0, 1]. attn_weights: Tensor with shape [T, T] containing the attention weights. r r) r9r:rfr&rr}rrr~rview)r'frame_featuresresidualrorprYr+r+r,rZs          zPGL_SUM.forwardrxryr+r+r)r,rzs(rz) rkr6torch.nnrtorch.nn.functional functionalrmModulerr`rzr+r+r+r,s X