o iW@sdZddlZddlZddlZddlmmZddlm Z ddlm Z d,ddZ Gd d d ejj Z Gd d d ejj ZGd ddejj ZGdddejj ZGdddejj ZGdddejj ZGdddejj ZGdddejj ZGdddejj ZGdddejj ZGdddejj ZGdd d ejj ZGd!d"d"ejj ZGd#d$d$ejj Zd-d&d'Z %d.d(d)Zd*d+ZdS)/zAttention modules for RNN.N) make_pad_mask) to_devicecCst|ddkr td||}||}|dkr$td |ddd|f<||dkr8td |dd|df<|S)aApply monotonic attention constraint. This function apply the monotonic attention constraint introduced in `Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning`_. Args: e (Tensor): Attention energy before applying softmax (1, T). last_attended_idx (int): The index of the inputs of the last attended [0, T]. backward_window (int, optional): Backward window size in attention constraint. forward_window (int, optional): Forward window size in attetion constraint. Returns: Tensor: Monotonic constrained attention energy (1, T). .. _`Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning`: https://arxiv.org/abs/1710.07654 rrz2Batch attention constraining is not yet supported.infN)sizeNotImplementedErrorfloat)elast_attended_idxbackward_windowforward_window backward_idx forward_idxr_/home/ubuntu/.local/lib/python3.10/site-packages/funasr/models/language_model/rnn/attentions.py_apply_attention_constraint srcs0eZdZdZfddZddZddZZS)NoAttz No attentioncs*tt|d|_d|_d|_d|_dSN)superr__init__h_lengthenc_hpre_compute_enc_hcself __class__rrr/s  zNoAtt.__init__cCd|_d|_d|_d|_dSz reset statesN)rrrrrrrrreset6 z NoAtt.resetcCst|}|jdur||_|jd|_|dur@dt|}|||d}| |j}t j |j| ||jddd|_ |j |fS)aNoAtt forward :param torch.Tensor enc_hs_pad: padded encoder hidden state (B, T_max, D_enc) :param list enc_hs_len: padded encoder hidden state length (B) :param torch.Tensor dec_z: dummy (does not use) :param torch.Tensor att_prev: dummy (does not use) :return: attention weighted encoder state (B, D_enc) :rtype: torch.Tensor :return: previous attention weights :rtype: torch.Tensor Nr?dim)lenrrrrrr new unsqueezetotorchsumviewr)r enc_hs_pad enc_hs_lendec_zatt_prevbatchmaskrrrforward=s  " z NoAtt.forward__name__ __module__ __qualname____doc__rr!r4 __classcell__rrrrr,s  rc4eZdZdZd fdd ZddZd dd ZZS) AttDotaDot product attention :param int eprojs: # projection-units of encoder :param int dunits: # units of decoder :param int att_dim: attention dimension :param bool han_mode: flag to swith on mode of hierarchical attention and not store pre_compute_enc_h Fcsbtt|tj|||_tj|||_||_||_ ||_ d|_ d|_ d|_ d|_||_dSr)rr<rr+nnLinearmlp_encmlp_decdunitseprojsatt_dimrrrr3han_moderrBrArCrDrrrrds zAttDot.__init__cCrr rrrr3rrrrr!rr"z AttDot.reset@c Cs|d}|jdus |jr!||_|jd|_t||j|_|dur-|||j }n| ||j }tj |jt| | |d|j dd}|jdurWt|t||_||jtd tj||dd}tj |j| ||jddd} | |fS)a AttDot forward :param torch.Tensor enc_hs_pad: padded encoder hidden state (B x T_max x D_enc) :param list enc_hs_len: padded encoder hidden state length (B) :param torch.Tensor dec_z: dummy (does not use) :param torch.Tensor att_prev: dummy (does not use) :param float scaling: scaling parameter before applying softmax :return: attention weighted encoder state (B, D_enc) :rtype: torch.Tensor :return: previous attention weight (B x T_max) :rtype: torch.Tensor rNrr%r)rrrDrrr+tanhr? new_zerosrAr-r,r@rCr3rr masked_fill_r Fsoftmax) rr.r/r0r1scalingr2r wrrrrr4ys$    zAttDot.forwardFrGr5rrrrr<Z  r<cr;) AttAddaAdditive attention :param int eprojs: # projection-units of encoder :param int dunits: # units of decoder :param int att_dim: attention dimension :param bool han_mode: flag to swith on mode of hierarchical attention and not store pre_compute_enc_h Fcsvtt|tj|||_tjj||dd|_tj|d|_||_ ||_ ||_ d|_ d|_ d|_d|_||_dSNFbiasr)rrSrr+r=r>r?r@gvecrArBrCrrrr3rDrErrrrs zAttAdd.__init__cCrr rFrrrrr!r"z AttAdd.resetrGc Cst|}|jdus |jr||_|jd|_||j|_|dur)|||j}n| ||j}| | |d|j }| t |j|d}|jdurVt|t||_||jtd tj||dd} t j|j| ||jddd} | | fS)a-AttAdd forward :param torch.Tensor enc_hs_pad: padded encoder hidden state (B x T_max x D_enc) :param list enc_hs_len: padded encoder hidden state length (B) :param torch.Tensor dec_z: decoder hidden state (B x D_dec) :param torch.Tensor att_prev: dummy (does not use) :param float scaling: scaling parameter before applying softmax :return: attention weighted encoder state (B, D_enc) :rtype: torch.Tensor :return: previous attention weights (B x T_max) :rtype: torch.Tensor NrrHrr%)r'rrDrrrr?rJrAr-r@rCrWr+rIsqueezer3rrrKr rLrMr,) rr.r/r0r1rNr2 dec_z_tiledr rOrrrrr4s   zAttAdd.forwardrPrQr5rrrrrSrRrScs<eZdZdZd fdd ZddZ   dd d ZZS)AttLocalocation-aware attention module. Reference: Attention-Based Models for Speech Recognition (https://arxiv.org/pdf/1506.07503.pdf) :param int eprojs: # projection-units of encoder :param int dunits: # units of decoder :param int att_dim: attention dimension :param int aconv_chans: # channels of attention convolution :param int aconv_filts: filter size of attention convolution :param bool han_mode: flag to swith on mode of hierarchical attention and not store pre_compute_enc_h Fcstt|tj|||_tjj||dd|_tjj||dd|_tjj d|dd|dfd|fdd|_ tj|d|_ ||_ ||_ ||_d|_d|_d|_d|_||_dSNFrUrrHrpaddingrV)rrZrr+r=r>r?r@mlp_attConv2dloc_convrWrArBrCrrrr3rDrrBrArC aconv_chans aconv_filtsrDrrrrs( zAttLoc.__init__cCrr rFrrrrr!r"z AttLoc.resetrGNrrc Cs~t|} |jdus |jr||_|jd|_||j|_|dur)|| |j}n| | |j}|durKdt |j |j |j d}|||d}|| | dd|j} | ddd} || } || | d|j} |t| |j| d} |jdurt|t ||_| |jtd |durt| |||} tj|| dd} tj |j| | |jddd}|| fS) aCalculate AttLoc forward propagation. :param torch.Tensor enc_hs_pad: padded encoder hidden state (B x T_max x D_enc) :param list enc_hs_len: padded encoder hidden state length (B) :param torch.Tensor dec_z: decoder hidden state (B x D_dec) :param torch.Tensor att_prev: previous attention weight (B x T_max) :param float scaling: scaling parameter before applying softmax :param torch.Tensor forward_window: forward window size when constraining attention :param int last_attended_idx: index of the inputs of the last attended :param int backward_window: backward window size in attention constraint :param int forward_window: forward window size in attetion constraint :return: attention weighted encoder state (B, D_enc) :rtype: torch.Tensor :return: previous attention weights (B x T_max) :rtype: torch.Tensor Nrr#)devicedtyper$rHrr%)!r'rrDrrrr?rJrAr-rr*rdrer(r)r`rX transposer^r@rCrWr+rIr3rrKr rrLrMr,)rr.r/r0r1rNr r r r2att_convrYr rOrrrrr4"s0    zAttLoc.forwardrP)rGNrrr5rrrrrZs rZcr;) AttCovaCoverage mechanism attention Reference: Get To The Point: Summarization with Pointer-Generator Network (https://arxiv.org/abs/1704.04368) :param int eprojs: # projection-units of encoder :param int dunits: # units of decoder :param int att_dim: attention dimension :param bool han_mode: flag to swith on mode of hierarchical attention and not store pre_compute_enc_h Fcstt|tj|||_tjj||dd|_tjd||_tj|d|_ ||_ ||_ ||_ d|_ d|_d|_d|_||_dSrT)rrhrr+r=r>r?r@wvecrWrArBrCrrrr3rDrErrrrs zAttCov.__init__cCrr rFrrrrr!r"z AttCov.resetrGc CsTt|}|jdus |jr||_|jd|_||j|_|dur)|||j}n| ||j}|durJt |dt | }|| |dg}t|}||d}|| |d|j}|t||j|d} |jdur~t |t ||_| |jt d tj|| dd} || g7}tj|j| ||jddd} | |fS)a+AttCov forward :param torch.Tensor enc_hs_pad: padded encoder hidden state (B x T_max x D_enc) :param list enc_hs_len: padded encoder hidden state length (B) :param torch.Tensor dec_z: decoder hidden state (B x D_dec) :param list att_prev_list: list of previous attention weight :param float scaling: scaling parameter before applying softmax :return: attention weighted encoder state (B, D_enc) :rtype: torch.Tensor :return: list of previous attention weights :rtype: list Nrr#r$rHrr%)r'rrDrrrr?rJrAr-rrr r(r)r,rir@rCrWr+rIrXr3rKrLrM) rr.r/r0 att_prev_listrNr2cov_vecrYr rOrrrrr4s,    zAttCov.forwardrPrQr5rrrrrhrs  rhcr;) AttLoc2Da2D location-aware attention This attention is an extended version of location aware attention. It take not only one frame before attention weights, but also earlier frames into account. :param int eprojs: # projection-units of encoder :param int dunits: # units of decoder :param int att_dim: attention dimension :param int aconv_chans: # channels of attention convolution :param int aconv_filts: filter size of attention convolution :param int att_win: attention window size (default=5) :param bool han_mode: flag to swith on mode of hierarchical attention and not store pre_compute_enc_h Fcstt|tj|||_tjj||dd|_tjj||dd|_tjj d||d|dfd|fdd|_ tj|d|_ ||_ ||_ ||_d|_d|_d|_||_||_d|_||_dSr[)rrlrr+r=r>r?r@r^r_r`rWrArBrCrrrrbatt_winr3rD)rrBrArCrmrbrcrDrrrrs, zAttLoc2D.__init__cCrr rFrrrrr!r"zAttLoc2D.resetrGc Cst|}|jdus |jr||_|jd|_||j|_|dur)|||j}n| ||j}|durTt |dt | }|| |d}|dd|jd}||d}|ddd}||}|| |d|j}|t||j|d} |jdurt |t ||_| |jt d tj|| dd} tj|j| ||jddd} tj|| dgdd}|ddddf}| |fS)aTAttLoc2D forward :param torch.Tensor enc_hs_pad: padded encoder hidden state (B x T_max x D_enc) :param list enc_hs_len: padded encoder hidden state length (B) :param torch.Tensor dec_z: decoder hidden state (B x D_dec) :param torch.Tensor att_prev: previous attention weight (B x att_win x T_max) :param float scaling: scaling parameter before applying softmax :return: attention weighted encoder state (B, D_enc) :rtype: torch.Tensor :return: previous attention weights (B x att_win x T_max) :rtype: torch.Tensor Nrr#r$rHrr%) r'rrDrrrr?rJrAr-rrr r(r)expandrmr`rXrfr^r@rCrWr+rIr3rKrLrMr,cat) rr.r/r0r1rNr2rgrYr rOrrrrr4s2    zAttLoc2D.forwardrPrQr5rrrrrls rlcr;) AttLocRecaPlocation-aware recurrent attention This attention is an extended version of location aware attention. With the use of RNN, it take the effect of the history of attention weights into account. :param int eprojs: # projection-units of encoder :param int dunits: # units of decoder :param int att_dim: attention dimension :param int aconv_chans: # channels of attention convolution :param int aconv_filts: filter size of attention convolution :param bool han_mode: flag to swith on mode of hierarchical attention and not store pre_compute_enc_h Fcstt|tj|||_tjj||dd|_tjjd|dd|dfd|fdd|_ tjj ||dd|_ tj|d|_ ||_ ||_||_d|_d|_d|_d|_||_dSr[)rrprr+r=r>r?r@r_r`LSTMCellatt_lstmrWrArBrCrrrr3rDrarrrrWs( zAttLocRec.__init__cCrr rFrrrrr!nr"zAttLocRec.resetrGcCst|}|jdus |jr||_|jd|_||j|_|dur)|||j}n| ||j}|dur\t |dt | }|| |d}|||j}|||j} || f} n|d}|d} || |dd|j} t| } t| d| df |d} || | \}} || |d|j} |t|d|j| d} |jdurt |t ||_| |jt d tj|| dd }tj|j| ||jddd }|||| fffS) aAttLocRec forward :param torch.Tensor enc_hs_pad: padded encoder hidden state (B x T_max x D_enc) :param list enc_hs_len: padded encoder hidden state length (B) :param torch.Tensor dec_z: decoder hidden state (B x D_dec) :param tuple att_prev_states: previous attention weight and lstm states ((B, T_max), ((B, att_dim), (B, att_dim))) :param float scaling: scaling parameter before applying softmax :return: attention weighted encoder state (B, D_enc) :rtype: torch.Tensor :return: previous attention weights and lstm states (w, (hx, cx)) ((B, T_max), ((B, att_dim), (B, att_dim))) :rtype: tuple Nrr#r$rrrHrr%)r'rrDrrrr?rJrAr-rrr r(r)rCr`rLrelu max_pool2drrr@rWr+rIrXr3rKrMr,)rr.r/r0att_prev_statesrNr2r1att_hatt_c att_statesrgrYr rOrrrrr4us@    zAttLocRec.forwardrPrQr5rrrrrpGs rpcr;) AttCovLocaCoverage mechanism location aware attention This attention is a combination of coverage and location-aware attentions. :param int eprojs: # projection-units of encoder :param int dunits: # units of decoder :param int att_dim: attention dimension :param int aconv_chans: # channels of attention convolution :param int aconv_filts: filter size of attention convolution :param bool han_mode: flag to swith on mode of hierarchical attention and not store pre_compute_enc_h Fcstt|tj|||_tjj||dd|_tjj||dd|_tjj d|dd|dfd|fdd|_ tj|d|_ ||_ ||_ ||_d|_d|_d|_||_d|_||_dSr[)rryrr+r=r>r?r@r^r_r`rWrArBrCrrrrbr3rDrarrrrs* zAttCovLoc.__init__cCrr rFrrrrr!r"zAttCovLoc.resetrGcCsxt|}|jdus |jr||_|jd|_||j|_|dur)|||j}n| ||j}|durJdt | }t ||| |dg}t|}|| |dd|j} | ddd} || } || |d|j} |t| |j| d} |jdurt |t ||_| |jt d tj|| dd} || g7}tj|j| ||jddd} | |fS)a.AttCovLoc forward :param torch.Tensor enc_hs_pad: padded encoder hidden state (B x T_max x D_enc) :param list enc_hs_len: padded encoder hidden state length (B) :param torch.Tensor dec_z: decoder hidden state (B x D_dec) :param list att_prev_list: list of previous attention weight :param float scaling: scaling parameter before applying softmax :return: attention weighted encoder state (B, D_enc) :rtype: torch.Tensor :return: list of previous attention weights :rtype: list Nrr#r$rHrr%)r'rrDrrrr?rJrAr-rr rr(r)r,r`rXrfr^r@rCrWr+rIr3rKrLrM)rr.r/r0rjrNr2r3rkrgrYr rOrrrrr4s0     zAttCovLoc.forwardrPrQr5rrrrrys  ryc2eZdZdZd fdd ZddZddZZS) AttMultiHeadDota Multi head dot product attention Reference: Attention is all you need (https://arxiv.org/abs/1706.03762) :param int eprojs: # projection-units of encoder :param int dunits: # units of decoder :param int aheads: # heads of multi head attention :param int att_dim_k: dimension k in multi head attention :param int att_dim_v: dimension v in multi head attention :param bool han_mode: flag to swith on mode of hierarchical attention and not store pre_compute_k and pre_compute_v Fcstt|tj|_tj|_tj|_t j |D]-}|jtj ||g7_|jtjj ||ddg7_|jtjj ||ddg7_qtjj |||dd|_ ||_||_||_||_||_dt||_d|_d|_d|_d|_d|_||_dS)NFrUr#)rr{rr+r= ModuleListmlp_qmlp_kmlp_vsixmovesranger>mlp_orArBaheads att_dim_k att_dim_vmathsqrtrNrr pre_compute_k pre_compute_vr3rDrrBrArrrrD_rrrr=s*     zAttMultiHeadDot.__init__cC"d|_d|_d|_d|_d|_dSr rrrrr3rrrrr!T  zAttMultiHeadDot.resetc s|d}jdus jr&|_jd_fddtjjD_j dus.jrG|_jd_fddtjjD_ |durS| |j }n| |j }g}g}tjjD]W}t jj|t j|| |djdd} jdurt|t|_| jtd  |tjj| ddg7}|t jj ||| |jdddg7}qet j|dd}||fS) aAttMultiHeadDot forward :param torch.Tensor enc_hs_pad: padded encoder hidden state (B x T_max x D_enc) :param list enc_hs_len: padded encoder hidden state length (B) :param torch.Tensor dec_z: decoder hidden state (B x D_dec) :param torch.Tensor att_prev: dummy (does not use) :return: attention weighted encoder state (B x D_enc) :rtype: torch.Tensor :return: list of previous attention weight (B x T_max) * aheads :rtype: list rNrcs"g|] }tj|jqSr)r+rIr~r.0hrrr osz+AttMultiHeadDot.forward..cg|] }j|jqSrrrrrrrrwrHr%r)rrrDrrrrrrrrJrAr-r+r,rIr}rr3rrrKr rLrMrNrro rr.r/r0r1r2rrOrr rrrr4\s<    0zAttMultiHeadDot.forwardrPr5rrrrr{.s r{crz) AttMultiHeadAdda^Multi head additive attention Reference: Attention is all you need (https://arxiv.org/abs/1706.03762) This attention is multi head attention using additive attention for each head. :param int eprojs: # projection-units of encoder :param int dunits: # units of decoder :param int aheads: # heads of multi head attention :param int att_dim_k: dimension k in multi head attention :param int att_dim_v: dimension v in multi head attention :param bool han_mode: flag to swith on mode of hierarchical attention and not store pre_compute_k and pre_compute_v Fcs.tt|tj|_tj|_tj|_tj|_ t j |D]:}|jtj ||g7_|jtjj ||ddg7_|jtjj ||ddg7_|j tj |dg7_ q%tjj |||dd|_||_||_||_||_||_dt||_d|_d|_d|_d|_d|_||_dS)NFrUrr#)rrrr+r=r|r}r~rrWrrrr>rrArBrrrrrrNrrrrr3rDrrrrrs.     zAttMultiHeadAdd.__init__cCrr rrrrrr!rzAttMultiHeadAdd.resetc s|d}jdus jr&|_jd_fddtjjD_j dus.jrG|_jd_fddtjjD_ |durS| |j }n| |j }g}g}tjjD]Z}j |tj|j|| |djd} jdurt|t|_| jtd |tjj| dd g7}|tjj ||| |jddd g7}qetj|dd }||fS) aAttMultiHeadAdd forward :param torch.Tensor enc_hs_pad: padded encoder hidden state (B x T_max x D_enc) :param list enc_hs_len: padded encoder hidden state length (B) :param torch.Tensor dec_z: decoder hidden state (B x D_dec) :param torch.Tensor att_prev: dummy (does not use) :return: attention weighted encoder state (B, D_enc) :rtype: torch.Tensor :return: list of previous attention weight (B x T_max) * aheads :rtype: list rNrcrrr~rrrrrrrz+AttMultiHeadAdd.forward..crrrrrrrrrrHrr%)rrrDrrrrrrrrJrAr-rWr+rIr}rrXr3rrrKr rLrMrNr,rrorrrrr4s: " 0zAttMultiHeadAdd.forwardrPr5rrrrrs rcs6eZdZdZ d fdd ZddZd dd ZZS) AttMultiHeadLocaMulti head location based attention Reference: Attention is all you need (https://arxiv.org/abs/1706.03762) This attention is multi head attention using location-aware attention for each head. :param int eprojs: # projection-units of encoder :param int dunits: # units of decoder :param int aheads: # heads of multi head attention :param int att_dim_k: dimension k in multi head attention :param int att_dim_v: dimension v in multi head attention :param int aconv_chans: # channels of attention convolution :param int aconv_filts: filter size of attention convolution :param bool han_mode: flag to swith on mode of hierarchical attention and not store pre_compute_k and pre_compute_v Fc stt|tj|_tj|_tj|_tj|_ tj|_ tj|_ t j |D]b} |jtj||g7_|jtjj||ddg7_|jtjj||ddg7_|j tj|dg7_ |j tjjd|dd|dfd|fddg7_ |j tjj||ddg7_ q1tjj|||dd|_||_||_||_||_||_dt||_d|_d|_d|_d|_d|_||_dSNFrUrrHrr\r#) rrrr+r=r|r}r~rrWr`r^rrrr>r_rrArBrrrrrrNrrrrr3rD) rrBrArrrrbrcrDrrrrrsF        zAttMultiHeadLoc.__init__cCrr rrrrrr!FrzAttMultiHeadLoc.resetrGc s.|d}jdus jr&|_jd_fddtjjD_j dus.jrG|_jd_fddtjjD_ |durS| |j }n| |j }|durg}tjjD]}dt |}|t||||dg7}qgg} g} tjjD]{} j| ||  |ddj} | d dd } j| | } j| tj| | j| | |djd } jdurt|t |_| jtd  | tj|| dd g7} | tjj | | |  |jddd g7} q tj!| dd } | | fS) amAttMultiHeadLoc forward :param torch.Tensor enc_hs_pad: padded encoder hidden state (B x T_max x D_enc) :param list enc_hs_len: padded encoder hidden state length (B) :param torch.Tensor dec_z: decoder hidden state (B x D_dec) :param torch.Tensor att_prev: list of previous attention weight (B x T_max) * aheads :param float scaling: scaling parameter before applying softmax :return: attention weighted encoder state (B x D_enc) :rtype: torch.Tensor :return: list of previous attention weight (B x T_max) * aheads :rtype: list rNrcrrrrrrrrcrz+AttMultiHeadLoc.forward..crrrrrrrrirr#r$rHrr%)"rrrDrrrrrrrrJrAr-rr rr(r)r`rXrfr^rWr+rIr}rr3rKrLrMr,rro)rr.r/r0r1rNr2rr3rrOrrgr rrrr4NsR "  0zAttMultiHeadLoc.forwardrPrQr5rrrrrs /rcs4eZdZdZ d fdd ZddZddZZS) AttMultiHeadMultiResLocaMulti head multi resolution location based attention Reference: Attention is all you need (https://arxiv.org/abs/1706.03762) This attention is multi head attention using location-aware attention for each head. Furthermore, it uses different filter size for each head. :param int eprojs: # projection-units of encoder :param int dunits: # units of decoder :param int aheads: # heads of multi head attention :param int att_dim_k: dimension k in multi head attention :param int att_dim_v: dimension v in multi head attention :param int aconv_chans: maximum # channels of attention convolution each head use #ch = aconv_chans * (head + 1) / aheads e.g. aheads=4, aconv_chans=100 => filter size = 25, 50, 75, 100 :param int aconv_filts: filter size of attention convolution :param bool han_mode: flag to swith on mode of hierarchical attention and not store pre_compute_k and pre_compute_v Fc stt|tj|_tj|_tj|_tj|_ tj|_ tj|_ t j |D]j} |jtj||g7_|jtjj||ddg7_|jtjj||ddg7_|j tj|dg7_ || d|} |j tjjd|dd| dfd| fddg7_ |j tjj||ddg7_ q1tjj|||dd|_||_||_||_||_||_dt||_d|_d|_d|_d|_d|_||_dSr) rrrr+r=r|r}r~rrWr`r^rrrr>r_rrArBrrrrrrNrrrrr3rD) rrBrArrrrbrcrDrafiltsrrrrs@        z AttMultiHeadMultiResLoc.__init__cCrr rrrrrr!rzAttMultiHeadMultiResLoc.resetc s0|d}jdus jr&|_jd_fddtjjD_j dus.jrG|_jd_fddtjjD_ |durS| |j }n| |j }|durg}tjjD]}dt |}|t||||dg7}qgg}g} tjjD]|} j| ||  |ddj} | d dd } j| | } j| tj| | j| | |djd } jdurt|t |_| jtd  | tjj| dd g7} |tj j | | |  |jddd g7}q!tj"|dd }|| fS) a-AttMultiHeadMultiResLoc forward :param torch.Tensor enc_hs_pad: padded encoder hidden state (B x T_max x D_enc) :param list enc_hs_len: padded encoder hidden state length (B) :param torch.Tensor dec_z: decoder hidden state (B x D_dec) :param torch.Tensor att_prev: list of previous attention weight (B x T_max) * aheads :return: attention weighted encoder state (B x D_enc) :rtype: torch.Tensor :return: list of previous attention weight (B x T_max) * aheads :rtype: list rNrcrrrrrrrrrz3AttMultiHeadMultiResLoc.forward..crrrrrrrrrr#r$rHrr%)#rrrDrrrrrrrrJrAr-rr rr(r)r`rXrfr^rWr+rIr}rr3rKrLrMrNr,rro) rr.r/r0r1r2rr3rrOrrgr rrrr4sR "  0zAttMultiHeadMultiResLoc.forwardrPr5rrrrrs ,rc:eZdZdZfddZddZ    d d d ZZS) AttForwardaForward attention module. Reference: Forward attention in sequence-to-sequence acoustic modeling for speech synthesis (https://arxiv.org/pdf/1807.06736.pdf) :param int eprojs: # projection-units of encoder :param int dunits: # units of decoder :param int att_dim: attention dimension :param int aconv_chans: # channels of attention convolution :param int aconv_filts: filter size of attention convolution cstt|tj|||_tjj||dd|_tjj||dd|_tjj d|dd|dfd|fdd|_ tj|d|_ ||_ ||_ ||_d|_d|_d|_d|_dSr[)rrrr+r=r>r?r@r^r_r`rWrArBrCrrrr3)rrBrArCrbrcrrrr7s& zAttForward.__init__cCrr rFrrrrr!Lr"zAttForward.resetr#Nrrc Cst|} |jdur||_|jd|_||j|_|dur&|| |j}n|| |j}|durD|j|dd}d|dddf<| || dd|j} | d dd} | | } | |d} |t|j| |  d} |jdurt|t||_| |jtd |durt| |||} tj|| dd} t|ddddd f}||| } tjt| d ddd } tj|j| d dd}|| fS) aDCalculate AttForward forward propagation. :param torch.Tensor enc_hs_pad: padded encoder hidden state (B x T_max x D_enc) :param list enc_hs_len: padded encoder hidden state length (B) :param torch.Tensor dec_z: decoder hidden state (B x D_dec) :param torch.Tensor att_prev: attention weights of previous step :param float scaling: scaling parameter before applying softmax :param int last_attended_idx: index of the inputs of the last attended :param int backward_window: backward window size in attention constraint :param int forward_window: forward window size in attetion constraint :return: attention weighted encoder state (B, D_enc) :rtype: torch.Tensor :return: previous attention weights (B x T_max) :rtype: torch.Tensor NrrHr#rrr%rrr$ư>pr&)r'rrrrr?rJrAr-r`rXrfr^r@r)rWr+rIr3rrrKr rrLrMpad normalizeclampr,)rr.r/r0r1rNr r r r2rgrYr rOatt_prev_shiftrrrrr4Ss6     zAttForward.forwardr#Nrrr5rrrrr)s  rcr) AttForwardTAaForward attention with transition agent module. Reference: Forward attention in sequence-to-sequence acoustic modeling for speech synthesis (https://arxiv.org/pdf/1807.06736.pdf) :param int eunits: # units of encoder :param int dunits: # units of decoder :param int att_dim: attention dimension :param int aconv_chans: # channels of attention convolution :param int aconv_filts: filter size of attention convolution :param int odim: output dimension cstt|tj|||_tjj||dd|_tj|||d|_tjj||dd|_ tjj d|dd|dfd|fdd|_ tj|d|_ ||_ ||_||_d|_d|_d|_d|_d|_dS)NFrUrrHrr\?)rrrr+r=r>r?r@mlp_tar^r_r`rWrAeunitsrCrrrr3trans_agent_prob)rrrArCrbrcodimrrrrs* zAttForwardTA.__init__cCs"d|_d|_d|_d|_d|_dS)Nr)rrrr3rrrrrr!s  zAttForwardTA.resetr#Nrrc Cst|} |jdur||_|jd|_||j|_|dur&|| |j}n|| |j}|durD|j|dd}d|dddf<| || dd|j} | d dd} | | } | || d|j} |t| |j|  d} |jdurt|t||_| |jtd |durt| ||| } tj|| dd}t|ddddd f}|j|d|j||}tjt|d ddd }tj|j|| |jddd}t| tj!|||gdd|_||fS) aCalculate AttForwardTA forward propagation. :param torch.Tensor enc_hs_pad: padded encoder hidden state (B, Tmax, eunits) :param list enc_hs_len: padded encoder hidden state length (B) :param torch.Tensor dec_z: decoder hidden state (B, dunits) :param torch.Tensor att_prev: attention weights of previous step :param torch.Tensor out_prev: decoder outputs of previous step (B, odim) :param float scaling: scaling parameter before applying softmax :param int last_attended_idx: index of the inputs of the last attended :param int backward_window: backward window size in attention constraint :param int forward_window: forward window size in attetion constraint :return: attention weighted encoder state (B, dunits) :rtype: torch.Tensor :return: previous attention weights (B, Tmax) :rtype: torch.Tensor NrrHr#rrr%rr$rr)"r'rrrrr?rJrAr-r`rXrfr^r@rCrWr+rIr3rrrKr rrLrMrrrrr,sigmoidrro)rr.r/r0r1out_prevrNr r r r2rgrYr rOrrrrrr4s8     "zAttForwardTA.forwardrr5rrrrrs rFc Cs,tj}t|dd}t|dd}t|dd}t|dd}t|dd}|dkrDt|D]} t|j|j|j||j |||} | | q+|S|dkr|rat|j |j|j|j |j |j|j|jdd } | Stj}t|D]"} t|j| |j|j|| |j | || || || } | | qj|Std |) afInstantiates an attention module given the program arguments :param Namespace args: The arguments :param int num_att: number of attention modules (in multi-speaker case, it can be 2 or more) :param bool han_mode: switch on/off mode of hierarchical attention network (HAN) :rtype torch.nn.Module :return: The attention module num_encsrrNawinrbrcT)rDz0Number of encoders needs to be more than one. {})r+r=r|getattrr initial_attatyperBrAadimappendhan_type han_headshan_dimhan_winhan_conv_chanshan_conv_filts ValueErrorformat) argsnum_attrDatt_listrrrrbrciattidxrrratt_for-sb          rc CsZ|dkr t} | S|dkrt||||} | S|dkr#t||||} | S|dkr2t||||||} | S|dkrBt|||||||} | S|dkrQt||||||} | S|dkr^t||||} | S|dkrmt||||||} | S|d kr|t||||||} | S|d krt ||||||} | S|d krt ||||||||} | S|d krt ||||||||} | S) a(Instantiates a single attention module :param str atype: attention type :param int eprojs: # projection-units of encoder :param int dunits: # units of decoder :param int aheads: # heads of multi head attention :param int adim: attention dimension :param int awin: attention window size :param int aconv_chans: # channels of attention convolution :param int aconv_filts: filter size of attention convolution :param bool han_mode: flag to swith on mode of hierarchical attention :return: The attention module noattdotaddlocation location2dlocation_recurrentcoveragecoverage_locationmulti_head_dotmulti_head_addmulti_head_locmulti_head_multi_res_loc) rr<rSrZrlrprhryr{rrr) rrBrArrrrbrcrDrrrrrlsP   rcs t|trtjdd|Ddd}|St|ttfr2tjddt|Ddd}|St|t rItjdd|Ddd}|St|t t t t frt|d}g}tj|D]tjfdd|Ddd}||g7}q`tj|dd}|Stj|dd}|S) zConverts attention weights to a numpy array given the attention :param list att_ws: The attention weights :param torch.nn.Module att: The attention :rtype: np.ndarray :return: The numpy array of the attention weights cSsg|] }|dddfqS)Nr$rrawrrrrrz att_to_numpy..rr%cSsg|]\}}||qSrr)rrrrrrrscSsg|]}|dqS)rrrrrrrrcsg|]}|qSrrrrrrrr) isinstancerlr+stackcpunumpyrhry enumeraterpr{rrrr'rrr)att_wsrn_headsatt_ws_sorted_by_head att_ws_headrrr att_to_numpys, $    r)rr)rFrP)r9rrr+torch.nn.functionalr= functionalrL*funasr.models.transformer.utils.nets_utilsrrrModulerr<rSrZrhrlrpryr{rrrrrrrrrrrrs:   .LO}`uynjl~ @ 0