o i@s2ddlZddlmZmZmZddlZdgZdejdejfddZ d#dejd ejd ejdejd ejd eejdeejfd dZ de dej j fddZ dee dedeeefddZdeedeededejdejf ddZGdddej j ZGdddej j ZGd d!d!ej j ZGd"ddeZdS)$N)ListOptionalTupleEmformerlengthsreturncCsF|jd}tt|}tj||j|jd||| dk}|S)Nr)devicedtype) shapeinttorchmaxitemarangerr expand unsqueeze)r batch_size max_length padding_maskrX/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/torchaudio/models/emformer.py_lengths_to_padding_mask s r utterance right_contextsummarymemsleft_context_keyc Cs|d|d|d}|d}|dkrd}|S|t||d} |dur3|dnd} ||d| | } t| d}|S)Nrr )r)sizer rr r) rrrrrrTBrright_context_blocks_lengthleft_context_blocks_lengthklengthsrrr_gen_padding_masks  r$ activationcCsD|dkr tjS|dkrtjS|dkrtjStd|)NrelugelusiluzUnsupported activation )r nnReLUGELUSiLU ValueError)r%rrr_get_activation_module's   r.weight_init_scale_strategy num_layerscCs\|dur ddt|DS|dkrddt|DS|dkr'ddt|DStd|)NcSsg|]}dqSNr).0_rrr 4sz*_get_weight_init_gains.. depthwisecSsg|] }dt|dqS)?r mathsqrtr2 layer_idxrrrr46sconstantcSsg|] }dtdqS)r6r7r:rrrr48sz-Unsupported weight_init_scale_strategy value )ranger-)r/r0rrr_get_weight_init_gains2sr? col_widthscol_masknum_rowsrcs@t|t|kr tdfddt||D}tj|ddS)Nz0Length of col_widths must match that of col_maskcs4g|]\}}|rtj|dntj|dqS)r)r oneszeros)r2 col_width is_ones_colrrBrrr4Cs z-_gen_attention_mask_block..r dim)lenr-zipr cat)r@rArBr mask_blockrrHr_gen_attention_mask_block=s  rOcseZdZdZ    d$dededed eed ed ef fd d Zde j de j de e j e j ffddZ de j de j dee j de j fddZ  d%de j de j de j de j de j de j dee j dee j de e j e j e j e j ffddZde j de j de j de j de j de j de e j e j ffd d!Ze jjde j de j de j de j de j de j de j de e j e j e j e j ffd"d#ZZS)&_EmformerAttentiona_Emformer layer attention module. Args: input_dim (int): input dimension. num_heads (int): number of attention heads in each Emformer layer. dropout (float, optional): dropout probability. (Default: 0.0) weight_init_gain (float or None, optional): scale factor to apply when initializing attention module parameters. (Default: ``None``) tanh_on_mem (bool, optional): if ``True``, applies tanh to memory elements. (Default: ``False``) negative_inf (float, optional): value to use for negative infinity in attention weights. (Default: -1e8) NFח input_dim num_headsdropoutweight_init_gain tanh_on_mem negative_infcst||dkrtd|d|d||_||_||_||_||_|j|jd|_t j j |d|dd|_ t j j ||dd|_ t j j ||dd|_|rht j jj|j j|d t j jj|j j|d dSdS) Nrz input_dim (z") is not a multiple of num_heads (z).gr=T)bias)gain)super__init__r-rSrTrUrWrXscalingr r)Linearemb_to_key_value emb_to_queryout_projinitxavier_uniform_weight)selfrSrTrUrVrWrX __class__rrr\Ys z_EmformerAttention.__init__inputrrc CsX|j\}}}|dd}|d||}t||g}||jddd\}} || fS)Nrr r=chunksrJ)r rr rMr_chunk) rerhrrr3summary_lengthright_ctx_utterance_blockmems_right_ctx_utterance_blockkeyvaluerrr_gen_key_valuews z!_EmformerAttention._gen_key_valueattention_weightsattention_maskrcCs|}||d|j}|d}|d|j}|durC|||j|d}||ddtj |j}|||j|d}tj j j |dd |}tj j j|t|j|jdS)Nrr r=rI)ptraining)float masked_fillrrXrrTviewtor boolr) functionalsoftmaxtype_asrUrv)rerrrsrattention_weights_floatrr attention_probsrrr_gen_attention_probss z'_EmformerAttention._gen_attention_probsrrrrrleft_context_valc s|d|d|d|d} t|||g} t|||gjddd\} } |dur{|dur{| t||d} t| d|d| || |d| dg} t| d|d| || |d| dg} fdd| | | fD\}}}t|j | dd}t ||||||} |||}t||}|j j| jjfkrtd| dd| j}|}|d}|d| |}|| |d}jrt|}ntj|dd d }||| | fS) Nr rr=rics4g|]}|djjjddqS)rtrr ) contiguousryrTrS transpose)r2tensorr rerrr4s&z4_EmformerAttention._forward_impl..z+Computed attention has incorrect dimensionsi )minr)rr`r rMr_rkrr bmmr]rr$rr rTrSAssertionErrorrryrarWtanhclamp)rerrrrrrsrrrqueryrorpr!reshaped_query reshaped_keyreshaped_valuerrrr attentionoutput_right_context_memsrloutput_right_context output_memsrrr _forward_implsP $        z _EmformerAttention._forward_implc Cs,|||||||\}}} } ||ddfS)acForward pass for training. B: batch size; D: feature dimension of each frame; T: number of utterance frames; R: number of right context frames; S: number of summary elements; M: number of memory elements. Args: utterance (torch.Tensor): utterance frames, with shape `(T, B, D)`. lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``utterance``. right_context (torch.Tensor): right context frames, with shape `(R, B, D)`. summary (torch.Tensor): summary elements, with shape `(S, B, D)`. mems (torch.Tensor): memory elements, with shape `(M, B, D)`. attention_mask (torch.Tensor): attention mask for underlying attention module. Returns: (Tensor, Tensor): Tensor output frames corresponding to utterance and right_context, with shape `(T + R, B, D)`. Tensor updated memory elements, with shape `(M, B, D)`. Nrt)r) rerrrrrrsoutputrr3rrrforwards"z_EmformerAttention.forwardc Cs|d|d|d}|d|d|d|d} t|| jtj|jd} d| dd|df<|j|||||| ||d\} } } }| | | |d|dd||d|ddfS)aForward pass for inference. B: batch size; D: feature dimension of each frame; T: number of utterance frames; R: number of right context frames; S: number of summary elements; M: number of memory elements. Args: utterance (torch.Tensor): utterance frames, with shape `(T, B, D)`. lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``utterance``. right_context (torch.Tensor): right context frames, with shape `(R, B, D)`. summary (torch.Tensor): summary elements, with shape `(S, B, D)`. mems (torch.Tensor): memory elements, with shape `(M, B, D)`. left_context_key (torch.Tensor): left context attention key computed from preceding invocation. left_context_val (torch.Tensor): left context attention value computed from preceding invocation. Returns: (Tensor, Tensor, Tensor, and Tensor): Tensor output frames corresponding to utterance and right_context, with shape `(T + R, B, D)`. Tensor updated memory elements, with shape `(M, B, D)`. Tensor attention key computed for left context and utterance. Tensor attention value computed for left context and utterance. rr rTrtN)rr)rr rErzr{rr)rerrrrrrr query_dimkey_dimrsrrrorprrrinfers&)( z_EmformerAttention.infer)rQNFrR)NN)__name__ __module__ __qualname____doc__r rwrr{r\r Tensorrrqrrrjitexportr __classcell__rrrfrrPLs&   I % rPcsheZdZdZ       d6ded ed ed ed ed edededeededeffdd Z dedee j de e j fddZde e j dee j e j e j ffddZde j de j ded e j de e j de e j f d!d"Zd#e j d$e j d%e j de j fd&d'Zd$e j d%e j dee j e j ffd(d)Zd#e j d$e j d%e j dee j e j ffd*d+Zd$e j d,e j d%e j d e j d-ee j dee j e j ff d.d/Zd$e j d,e j d%e j d e j dee e j dee j e j e e j ff d0d1Zd$e j d,e j d%e j d e j d-e j dee j e j e j ff d2d3Ze jjd$e j d,e j d%e j dee e j d e j dee j e j e e j e j ff d4d5ZZS)7_EmformerLayera$Emformer layer that constitutes Emformer. Args: input_dim (int): input dimension. num_heads (int): number of attention heads. ffn_dim: (int): hidden layer dimension of feedforward network. segment_length (int): length of each input segment. dropout (float, optional): dropout probability. (Default: 0.0) activation (str, optional): activation function to use in feedforward network. Must be one of ("relu", "gelu", "silu"). (Default: "relu") left_context_length (int, optional): length of left context. (Default: 0) max_memory_size (int, optional): maximum number of memory elements to use. (Default: 0) weight_init_gain (float or None, optional): scale factor to apply when initializing attention module parameters. (Default: ``None``) tanh_on_mem (bool, optional): if ``True``, applies tanh to memory elements. (Default: ``False``) negative_inf (float, optional): value to use for negative infinity in attention weights. (Default: -1e8) rQr&rNFrRrSrTffn_dimsegment_lengthrUr%left_context_lengthmax_memory_sizerVrWrXc stt|||| | | d|_tj||_tjj||dd|_ t |} tj tj |tj ||| tj|tj ||tj||_tj ||_tj ||_||_||_||_||_|dk|_dS)N)rSrTrUrVrWrXT kernel_sizestride ceil_moder)r[r\rPrr r)DropoutrU AvgPool1d memory_opr. Sequential LayerNormr^pos_fflayer_norm_inputlayer_norm_outputrrrrSuse_mem) rerSrTrrrUr%rrrVrWrXactivation_modulerfrrr\Rs6      z_EmformerLayer.__init__rrrcCsbtj|j||j|d}tj|j||j|d}tj|j||j|d}tjd|tj|d}||||gS)NrCr r)r rErrSrint32)rerr empty_memoryrr past_lengthrrr _init_states  z_EmformerLayer._init_statestatecCs|ddd}t|j|}t|jt||j}|d|j|d}|d|j|d}|d|j|d}|||fS)Nrr r=)rrrrr8ceilr)rerrpast_left_context_lengthpast_mem_lengthpre_memslc_keylc_valrrr _unpack_states  z_EmformerLayer._unpack_statenext_knext_v update_lengthrcCst|d|g}t|d|g}t|d|g|j d|d<||jd|jd|d<||jd|jd|d<|d||d<|S)Nr r=rr)r rMrr r)rerrrrrnew_knew_vrrr _pack_states"z_EmformerLayer._pack_state rc_outputrrcCs4||t||g}|||}||}|Sr1)rUr rMrr)rerrrresultrrr_process_attention_outputs z(_EmformerLayer._process_attention_outputcCs8|t||g}||dd|d|dfSNr)rr rMr)rerrrrrr_apply_pre_attention_layer_normsz._EmformerLayer._apply_pre_attention_layer_normcCs2||||}||dd|d|dfSr)rr)rerrrrrr_apply_post_attention_ffns$z(_EmformerLayer._apply_post_attention_ffnrrsc Csp|durtd|jr||dddddd}n tdj|j|jd}|j ||||||d\}}||fS)Nz;attention_mask must be not None when for_inference is Falser r=rr)rrrrrrs) r-rrpermuter emptyrzr rr) rerrrrrsrrnext_mrrr_apply_attention_forwards  z'_EmformerLayer._apply_attention_forwardc Cs|dur|j|d|jd}||\}}}|jr0||dddddd} | dd} n tdj |j |jd} |j j |||| |||d\} } } } | | | |d||}| | |fS)Nr rCr=rr)rrrrrrr)rrrrrrrr rrzr rrr)rerrrrrrrrrrrrrrrr_apply_attention_infers$ z%_EmformerLayer._apply_attention_inferc CsB|||\}}||||||\}} ||||\} } | | | fS)a1Forward pass for training. B: batch size; D: feature dimension of each frame; T: number of utterance frames; R: number of right context frames; M: number of memory elements. Args: utterance (torch.Tensor): utterance frames, with shape `(T, B, D)`. lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``utterance``. right_context (torch.Tensor): right context frames, with shape `(R, B, D)`. mems (torch.Tensor): memory elements, with shape `(M, B, D)`. attention_mask (torch.Tensor): attention mask for underlying attention module. Returns: (Tensor, Tensor, Tensor): Tensor encoded utterance frames, with shape `(T, B, D)`. Tensor updated right context frames, with shape `(R, B, D)`. Tensor updated memory elements, with shape `(M, B, D)`. )rrr) rerrrrrslayer_norm_utterancelayer_norm_right_contextrroutput_utterancerrrrrs $ z_EmformerLayer.forwardc CsF|||\}}||||||\}} } ||||\} } | | | | fS)a2Forward pass for inference. B: batch size; D: feature dimension of each frame; T: number of utterance frames; R: number of right context frames; M: number of memory elements. Args: utterance (torch.Tensor): utterance frames, with shape `(T, B, D)`. lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``utterance``. right_context (torch.Tensor): right context frames, with shape `(R, B, D)`. state (List[torch.Tensor] or None): list of tensors representing layer internal state generated in preceding invocation of ``infer``. mems (torch.Tensor): memory elements, with shape `(M, B, D)`. Returns: (Tensor, Tensor, List[torch.Tensor], Tensor): Tensor encoded utterance frames, with shape `(T, B, D)`. Tensor updated right context frames, with shape `(R, B, D)`. List[Tensor] list of tensors representing layer internal state generated in current invocation of ``infer``. Tensor updated memory elements, with shape `(M, B, D)`. )rrr) rerrrrrrrrr output_staterrrrrrs )   z_EmformerLayer.infer)rQr&rrNFrR)rrrrr rwstrrr{r\r rrrrrrrrrrrrrrrrrrrrfrr?s    ".(           / rcseZdZ   ddejjdedededef fdd Zd ejd ejfd d Z d eded e efddZ d ejd ejfddZ d ejdejd e ejejffddZejj dd ejdejdee e ejd e ejeje e ejffddZZS) _EmformerImplremformer_layersrrright_context_lengthrcsJt|dk|_tjj||dd|_||_||_||_ ||_ ||_ dS)NrTr) r[r\rr r)rrrrrrr)rerrrrrrfrrr\Ps   z_EmformerImpl.__init__rhrcCs|jd}t||j|j}g}t|dD]}|d|j}||j}||||q||||jdt|SNrr ) r r8rrrr>appendr rM)rerhrnum_segsright_context_blocksseg_idxstartendrrr_gen_right_contextfs   z _EmformerImpl._gen_right_contextrutterance_lengthc Cst||j}|j}|j}||}||}t||j|d}t|d|j|} |j|} |jrUt||jd} |d} | || | |||| ||| ||| g } | S||| ||| ||| g} | Sr) r8rrrrrrrr)rerrrrclcrc_startrc_end seg_startseg_end rc_lengthm_start mem_lengthr@rrr_gen_attention_mask_col_widthsqs<  z,_EmformerImpl._gen_attention_mask_col_widthsc Cs*|d}t||j}g}g}g}|jr0d}ddt|D}ddt|D} |||g} nd}ddt|D}d} ||g} t|D]=} || |} t| ||j|j } | | t| |t |j|| |j|j }| || durt| | d|j }| |qEdt d d| Dt j}|S) Nr cSg|]}|dvqS))r rr2idxrrrr4z5_EmformerImpl._gen_attention_mask..cSr))rrrrrrrr4rcSr))r rrrrrrr4rr cSsg|]}t|qSr)r rM)r2maskrrrr4s)rr8rrrr>rrOrrrrr rMrzr{)rerhrrrc_mask query_mask summary_masknum_colsrc_q_cols_mask s_cols_maskmasks_to_concatrr@ rc_mask_blockquery_mask_blocksummary_mask_blockrsrrr_gen_attention_masksH          z!_EmformerImpl._gen_attention_maskrc Cs|ddd}||}|d|d|j}||}|jr3||ddddddddn tdj |j |j d}|}|j D] }||||||\}}}qD|ddd|fS)aGForward pass for training and non-streaming inference. B: batch size; T: max number of input frames in batch; D: feature dimension of each frame. Args: input (torch.Tensor): utterance frames right-padded with right context frames, with shape `(B, T + right_context_length, D)`. lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid utterance frames for i-th batch element in ``input``. Returns: (Tensor, Tensor): Tensor output frames, with shape `(B, T, D)`. Tensor output lengths, with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in output frames. r rr=Nrtr) rrrrrrrr rrzr rr) rerhrrrrsrrlayerrrrrs  ( z_EmformerImpl.forwardNstatescCs$|d|j|jkrtd|j|jd|dd|ddd}|d|j}||d}|d|}tj||jdd}|jrT||ddddddn t dj |j |j d }|} g} t |jD]\} } | | |||durxdn|| |\} }} }| | qi| ddd|| fS) aForward pass for streaming inference. B: batch size; D: feature dimension of each frame. Args: input (torch.Tensor): utterance frames right-padded with right context frames, with shape `(B, segment_length + right_context_length, D)`. lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``input``. states (List[List[torch.Tensor]] or None, optional): list of lists of tensors representing internal state generated in preceding invocation of ``infer``. (Default: ``None``) Returns: (Tensor, Tensor, List[List[Tensor]]): Tensor output frames, with shape `(B, segment_length, D)`. Tensor output lengths, with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in output frames. List[List[Tensor]] output states; list of lists of tensors representing internal state generated in current invocation of ``infer``. r zIPer configured segment_length and right_context_length, expected size of z# for dimension 1 of input, but got .rr=N)rr)rrrr-rr rrrrrzr r enumeraterrr)rerhrrright_context_start_idxrroutput_lengthsrr output_statesr;rrrrrrs:     z_EmformerImpl.infer)rrrr1)rrrr r) ModuleListr r\rrrrrrrrrrrrrrrfrrOs< $&0#rcsleZdZdZ        dded ed ed ed ed edededededeededeffdd Z Z S)ra_Emformer architecture introduced in *Emformer: Efficient Memory Transformer Based Acoustic Model for Low Latency Streaming Speech Recognition* :cite:`shi2021emformer`. See Also: * :func:`~torchaudio.models.emformer_rnnt_model`, :func:`~torchaudio.models.emformer_rnnt_base`: factory functions. * :class:`torchaudio.pipelines.RNNTBundle`: ASR pipelines with pretrained model. Args: input_dim (int): input dimension. num_heads (int): number of attention heads in each Emformer layer. ffn_dim (int): hidden layer dimension of each Emformer layer's feedforward network. num_layers (int): number of Emformer layers to instantiate. segment_length (int): length of each input segment. dropout (float, optional): dropout probability. (Default: 0.0) activation (str, optional): activation function to use in each Emformer layer's feedforward network. Must be one of ("relu", "gelu", "silu"). (Default: "relu") left_context_length (int, optional): length of left context. (Default: 0) right_context_length (int, optional): length of right context. (Default: 0) max_memory_size (int, optional): maximum number of memory elements to use. (Default: 0) weight_init_scale_strategy (str or None, optional): per-layer weight initialization scaling strategy. Must be one of ("depthwise", "constant", ``None``). (Default: "depthwise") tanh_on_mem (bool, optional): if ``True``, applies tanh to memory elements. (Default: ``False``) negative_inf (float, optional): value to use for negative infinity in attention weights. (Default: -1e8) Examples: >>> emformer = Emformer(512, 8, 2048, 20, 4, right_context_length=1) >>> input = torch.rand(128, 400, 512) # batch, num_frames, feature_dim >>> lengths = torch.randint(1, 200, (128,)) # batch >>> output, lengths = emformer(input, lengths) >>> input = torch.rand(128, 5, 512) >>> lengths = torch.ones(128) * 5 >>> output, lengths, states = emformer.infer(input, lengths, None) rQr&rr5FrRrSrTrr0rrUr%rrrr/rWrXc sVt| | tj f ddt|D}tj|| ddS)Ncs.g|]}t | d qS))rUr%rrrVrWrX)rr: r%rUrrSrrrXrTrrWweight_init_gainsrrr4]s z%Emformer.__init__..)rrr)r?r r)r r>r[r\)rerSrTrr0rrUr%rrrr/rWrXrrfr rr\Ks  zEmformer.__init__)rQr&rrrr5FrR) rrrrr rwrrr{r\rrrrfrr&sJ+     r1)r8typingrrrr __all__rrr$rr)Moduler.r rwr?r{rrOrPrrrrrrrsT "   tX