o }oi@spddlmZmZmZmZmZmZddlZddlm Z ddl m Z ddl m Z ddlmZmZddlmZddlmZmZdd lmZdd lmZdd lmZmZmZmZmZm Z m!Z!m"Z"m#Z#dd l$m%Z%Gd dde j&eZ'Gddde j&eeZ(Gddde j)eeZ*Gdddej+j,eZ-Gdddej+j,Z.Gddde*Z/e(e*e/fD]Z0e1e0dure2e0e0qdS))AnyDictListOptionalTupleUnionN) DictConfig) rnnt_abstract) stateless_net) adapter_utils rnnt_utils)rnn)adapter_mixins typecheck) Exportable)AdapterModuleMixin) AcousticEncodedRepresentation ElementTypeEmbeddedTextType LabelsType LengthsType LogprobsTypeLossType NeuralTypeSpectrogramType)loggingcs6eZdZdZeddZeddZdGddZfd d Z  dHd e e e fd e de de e ffdd ZedIddZ  dJde ejde ejdede e deejeejff ddZddZdejde ee e fdeejeejejffd d!Zdejdeejfd"d#Zd$eeejfd%d&Zd'eejd(e deeejfd)d*Zd'eeejdeejfd+d,Ze dId-e ejejfe!ejBd.e ejejfe!ejBd/ejd0e e ejejfe!ejBfd1d2Z"e dId-e!ejd.e!ejde d Bfd3d4Z#ede!ejde!ejfd5d6Z$ed'e!ejde!e!ejfd7d8Z%e dKd'e!e!ejde!ejfd9d:Z& dId;eejde e'deejf d?d@Z(dAe eejd/ejde eejfdBdCZ)dDeejde ee e fdeeejeeejffdEdFZ*Z+S)LStatelessTransducerDecoderaA Stateless Neural Network Transducer Decoder / Prediction Network. An RNN-T Decoder/Prediction stateless network that simply takes concatenation of embeddings of the history tokens as the output. Args: prednet: A dict-like object which contains the following key-value pairs. pred_hidden: int specifying the hidden dimension of the prediction net. dropout: float, set to 0.0 by default. Optional dropout applied at the end of the final LSTM RNN layer. vocab_size: int, specifying the vocabulary size of the embedding layer of the Prediction network, excluding the RNNT blank token. context_size: int, specifying the size of the history context used for this decoder. normalization_mode: Can be either None, 'layer'. By default, is set to None. Defines the type of normalization applied to the RNN layer. cCs.tdtttdttdtddgdS)*Returns definitions of module input ports.BTrToptionaltargets target_lengthstates)rrtuplerselfr*U/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/modules/rnnt.py input_typesK z&StatelessTransducerDecoder.input_typescC.tdtttdttdtddgdS)+Returns definitions of module output ports.rDr rrTr!outputsprednet_lengthsr&)rrr'rrr(r*r*r+ output_typesTr-z'StatelessTransducerDecoder.output_typescCn|}tj|j||ftjdt|j}tjd||ftjdt|j}t | | }|||fSs Generates input examples for tracing etc. Returns: A tuple of input examples. ) fill_valuesizedtyper)r;r< torchfull blank_idxint32tonext parametersdevicerandintr'initialize_statefloatr) max_batchmax_dimlengthr$r%r&r*r*r+ input_example]   z(StatelessTransducerDecoder.input_examplec d|_tjdi|dSNTr* _rnnt_exportsuper_prepare_for_exportr)kwargs __class__r*r+rTmz.StatelessTransducerDecoder._prepare_for_exportNprednet vocab_size context_sizenormalization_modec sd|d|_||_||_tj||jdd|dd}|jdi|||j|j||d|_d|_dS) N pred_hiddenTr[r@ blank_as_paddropout)r\r[emb_dimr@r]raFr*) r^r@r\rS__init__get_predict_modules predictionrR)r)rZr[r\r]rarWr*r+rdqs   z#StatelessTransducerDecoder.__init__cCsDt|}|jr d}nd}|j|||d\}}|dd}|||fSNFT)stateadd_sosr6r label_collaterRpredict transpose)r)r$r%r&yrjgrir*r*r+forward   z"StatelessTransducerDecoder.forwardTrprirj batch_sizereturnc Cst|}|j}|j}|dur#|j|kr||}|||\}}n |dur5|dur-dn|dd}n|}tj|d|j f||d}|rd|j \}} } tj|d| f|j|jd} tj | |gdd }nd} ~ ||fS)a Stateful prediction of scores and state for a tokenset. Here: B - batch size U - label length C - context size for stateless decoder D - total embedding size Args: y: Optional torch tensor of shape [B, U] of dtype long which will be passed to the Embedding. If None, creates a zero tensor of shape [B, 1, D] which mimics output of pad-token on Embedding. state: An optional one-element list of one tensor. The tensor is used to store previous context labels. The tensor uses type long and is of shape [B, C]. add_sos: bool flag, whether a zero vector describing a "start of signal" token should be prepended to the above "y" tensor. When set, output size is (B, U + 1, D). batch_size: An optional int, specifying the batch size of the `y` tensor. Can be infered if `y` and `state` is None. But if both are None, then batch_size cannot be None. Returns: A tuple (g, state) such that - If add_sos is False: g: (B, U, D) state: [(B, C)] storing the history context including the new words in y. If add_sos is True: g: (B, U + 1, D) state: [(B, C)] storing the history context including the new words in y. Nr6rrEr<dim) rCrDrEr<rBrgr;r>zerosr^shapecat contiguous) r)rprirjrt_prEr<rUr1startr*r*r+rns$ 2   z"StatelessTransducerDecoder.predictcKstjdi|}|S)aA Prepare the trainable parameters of the Prediction Network. Args: vocab_size: Vocab size (excluding the blank token). pred_n_hidden: Hidden size of the RNNs. norm: Type of normalization to perform in RNN. dropout: Whether to apply dropout to RNN. Nr*)r StatelessNet)r)rVnetr*r*r+rfs z+StatelessTransducerDecoder._predict_modules hypothesiscachec C|jdur |jdj}n t|}|j}t|jdkr'|jd|jkr'd}nd}tjddg|jd|tj d}|dddf}t |j}||vrP||\} } n-|r^|j ddddd\} } n |j ||jddd\} } | ddddddf} | | f||<| | |fS a Similar to the predict() method, instead this method scores a Hypothesis during beam search. Hypothesis is a dataclass representing one hypothesis in a Beam Search. Args: hypothesis: Refer to rnnt_utils.Hypothesis. cache: Dict which contains a cache to avoid duplicate computations. Returns: Returns a tuple (y, states, lm_token) such that: y is a torch.Tensor of shape [1, 1, H] representing the score of the last token in the Hypothesis. state is a list of RNN states, each of shape [L, 1, H]. lm_token is the final integer token of the hypothesis. NrTFr6)r:rEr<rirjrt dec_staterErCrDlen y_sequencer@r>r?longr'rn r)rrrEr} blank_statetargetlm_tokensequencerp new_stater*r*r+score_hypothesis(       z+StatelessTransducerDecoder.score_hypothesiscCs2|d}tj||jdg|jtj|jdg}|S)Nrr6)r:r<rE)r;r>r?r\r@rrEr)rpbatchrir*r*r+rG5s  z+StatelessTransducerDecoder.initialize_statedecoder_statescCstdd|D}|gS)a Creates a stacked decoder states to be passed to prediction network. Args: decoder_states (list of list of torch.Tensor): list of decoder states [B, 1, C] - B: Batch size. - C: Dimensionality of the hidden state. Returns: batch_states (list of torch.Tensor): batch of decoder states [[B x C]] cSg|]}|dqSrr*).0sr*r*r+ KzFStatelessTransducerDecoder.batch_initialize_states..r>stack)r)rrr*r*r+batch_initialize_states>s z2StatelessTransducerDecoder.batch_initialize_states batch_statesidxcCs&|dur|d|}|}|gSdS)a<Get decoder state from batch of states, for given id. Args: batch_states (list): batch of decoder states [(B, C)] idx (int): index to extract state from batch of states Returns: (tuple): decoder states for given id [(C)] Nr)r)r)rrr&r*r*r+batch_select_stateOs  z-StatelessTransducerDecoder.batch_select_statecCsLg}g}tt|D]}t||}||q t|d}|||S)zConcatenate a batch of decoder state to a packed state. Args: batch_states (list): batch of decoder states B x ([(C)] Returns: (tuple): decoder states [(B x C)] r)rangerr>rappendr{)r)r state_list batch_list sample_idtensor state_tensorr*r*r+batch_concat_stateses    z.StatelessTransducerDecoder.batch_concat_states src_states dst_statesmaskother_src_statescCs8|dur|n|}tj|d|d|d|dddS)a Replaces states in `dst_states` with states from `src_states` based on the given `mask`. Args: mask (torch.Tensor): When True, selects values from `src_states`, otherwise `out` or `other_src_states`(if provided). src_states (tuple[torch.Tensor, torch.Tensor]): Values selected at indices where `mask` is True. dst_states (tuple[torch.Tensor, torch.Tensor], optional): The output states. other_src_states (tuple[torch.Tensor, torch.Tensor], optional): Values selected at indices where `mask` is False. Note: This operation is performed without CPU-GPU synchronization by using `torch.where`. Nrrout)r>where unsqueeze)clsrrrrotherr*r*r+batch_replace_states_mask{s(z4StatelessTransducerDecoder.batch_replace_states_maskcCsD|dur|d|ddS|dd||dd|dS)8Replace states in dst_states with states from src_statesNrcopy_rrrrtr*r*r+batch_replace_states_alls&z3StatelessTransducerDecoder.batch_replace_states_allcCdd|DS)Return copy of the statescSsg|]}|qSr*cloner sub_stater*r*r+rrz:StatelessTransducerDecoder.clone_state..r*rrir*r*r+ clone_statesz&StatelessTransducerDecoder.clone_statecCr) Split states into a list of states. Useful for splitting the final state for converting results of the decoding algorithm to Hypothesis class. cSsg|] }|jdddqS)r6rrw)splitrr*r*r+rszAStatelessTransducerDecoder.batch_split_states..r*rrr*r*r+batch_split_statessz-StatelessTransducerDecoder.batch_split_statescCs$tjdd|Dddj||dgS)zj Concatenate a batch of decoder state to a packed state. Inverse of `batch_split_states`. cSrrr*rrir*r*r+rrzCStatelessTransducerDecoder.batch_unsplit_states..rrwrvr>rrBrrrEr<r*r*r+batch_unsplit_statess z/StatelessTransducerDecoder.batch_unsplit_states old_states new_statesidsvaluecCs0|dur|d|ddf|d|ddf<|S)a:Copy states from new state to old state at certain indices. Args: old_states: packed decoder states single element list of (B x C) new_states: packed decoder states single element list of (B x C) ids (list): List of indices to copy states at. value (optional float): If a value should be copied instead of a state slice, a float should be provided Returns: batch of decoder states with partial copy at ids (or a specific value). (B x C) Nrr*)r)rrrrr*r*r+batch_copy_statess$z,StatelessTransducerDecoder.batch_copy_statesr&cCs|durdS|d|gS) Return states by mask selection Args: states: states for the batch mask: boolean mask for selecting states; batch dimension should be the same as for states Returns: states filtered by mask Nrr*r)r&rr*r*r+mask_select_statess z-StatelessTransducerDecoder.mask_select_states hypothesescCs`t|}|dkr tdt|}|j}g}g}ddt|D}t|D]$\} } t| j} | |vr:|| || <q&| | jd| | | j fq&|rt|} t j ||t j d| d}|dd|D} |j|| d| d \}} d}t|D](} |r|| d ur|| |}|||f|| <|||f|||d<|d 7}qyd d|Dd d|DfS)n Used for batched beam search algorithms. Similar to score_hypothesis method. Args: hypothesis: List of Hypotheses. Refer to rnnt_utils.Hypothesis. cache: Dict which contains a cache to avoid duplicate computations. Returns: Returns a tuple (batch_dec_out, batch_dec_states) such that: batch_dec_out: a list of torch.Tensor [1, H] representing the prediction network outputs for the last tokens in the Hypotheses. batch_dec_states: a list of list of RNN states, each of shape [L, B, H]. Represented as B x List[states]. r)No hypotheses was provided for the batch!cSg|]}dqSNr*r_r*r*r+rzEStatelessTransducerDecoder.batch_score_hypothesis..rrvcSg|]\}}|qSr*r*rrd_stater*r*r+rrFrNr6cSg|]\}}|qSr*r*rdec_outrr*r*r+r%rcSrr*r*rr dec_statesr*r*r+r%rr ValueErrorrCrDrEr enumerater'rrrr>rrviewrrnr)r)rr final_batchr}rEtokens to_processfinal final_idxhyprrr dec_outputs processed_idxrr*r*r+batch_score_hypothesiss<     z1StatelessTransducerDecoder.batch_score_hypothesisr6r6)r6NrNNTNNN),__name__ __module__ __qualname____doc__propertyr,r5rMrTrstrrintrrdrrrr>Tensorboolrrrnrfr HypothesisrrGrrr classmethodr'listrrrrrrHrrr __classcell__r*r*rWr+r7s       R 5 &"   $      rcseZdZdZeddZeddZdSddZfd d Z dTde e e fde de e dedef fdd ZedUddZ dVde ejde eejdede e deejeejff ddZddZdejdeejejffd d!Zd"ejd#e ee e fdeejeejejffd$d%Zd&eejd#e ee e fdeeejeeejffd'd(Zd)eeejdeejfd*d+Zd,eejd-e deeejfd.d/Ze dUd0e ejejfde d1e d2ejd3e e ejejfde ejejff d4d5Z!d,eeejdeejfd6d7Z"e dUd0eejejfd3eejejfd8ejd9e eejejffd:d;Z#e dUd0eejejfd3eejejfde d Bfdd?Z%ed,e ejejfde&e ejejffd@dAZ'e dWd,e&e ejejfde ejejffdBdCZ( dUdDeejdEeejdFee dGe e)deejf dHdIZ*dJeejejfd8ejdeejejffdKdLZ+dMe dNe,ffdOdP Z-dNe,fdQdRZ.Z/S)X RNNTDecoderaU A Recurrent Neural Network Transducer Decoder / Prediction Network (RNN-T Prediction Network). An RNN-T Decoder/Prediction network, comprised of a stateful LSTM model. Args: prednet: A dict-like object which contains the following key-value pairs. pred_hidden: int specifying the hidden dimension of the prediction net. pred_rnn_layers: int specifying the number of rnn layers. Optionally, it may also contain the following: forget_gate_bias: float, set by default to 1.0, which constructs a forget gate initialized to 1.0. Reference: [An Empirical Exploration of Recurrent Network Architectures](http://proceedings.mlr.press/v37/jozefowicz15.pdf) t_max: int value, set to None by default. If an int is specified, performs Chrono Initialization of the LSTM network, based on the maximum number of timesteps `t_max` expected during the course of training. Reference: [Can recurrent neural networks warp time?](https://openreview.net/forum?id=SJcKhk-Ab) weights_init_scale: Float scale of the weights after initialization. Setting to lower than one sometimes helps reduce variance between runs. hidden_hidden_bias_scale: Float scale for the hidden-to-hidden bias scale. Set to 0.0 for the default behaviour. dropout: float, set to 0.0 by default. Optional dropout applied at the end of the final LSTM RNN layer. vocab_size: int, specifying the vocabulary size of the embedding layer of the Prediction network, excluding the RNNT blank token. normalization_mode: Can be either None, 'batch' or 'layer'. By default, is set to None. Defines the type of normalization applied to the RNN layer. random_state_sampling: bool, set to False by default. When set, provides normal-distribution sampled state tensors instead of zero tensors during training. Reference: [Recognizing long-form speech using streaming end-to-end models](https://arxiv.org/abs/1910.11455) blank_as_pad: bool, set to True by default. When set, will add a token to the Embedding layer of this prediction network, and will treat this token as a pad token. In essence, the RNNT pad token will be treated as a pad token, and the embedding layer will return a zero tensor for this token. It is set by default as it enables various batch optimizations required for batched beam search. Therefore, it is not recommended to disable this flag. cCr.)rrrr1rr1Tr!r#)rrr'rrr(r*r*r+r,br-zRNNTDecoder.input_typescCr.)r/r0rrTr!r2)rrr'rrr(r*r*r+r5kr-zRNNTDecoder.output_typesr6cCr7r8r=rIr*r*r+rMtrNzRNNTDecoder.input_examplec rOrPrQrUrWr*r+rTrYzRNNTDecoder._prepare_for_exportNFTrZr[r]random_state_samplingr`c s|d|_|d|_||_tj||j|d|dd}|dd}|dd}|dd } |d d } ||_|j||j|j||||| | |d d d |_d|_ dS)Nr^pred_rnn_layersr_forget_gate_bias?t_maxweights_init_scalehidden_hidden_bias_scalerbrarnn_hidden_sizer) r[ pred_n_hiddenrrrnormrr rar F) r^rr@rSrdrerrfrgrR) r)rZr[r]rr`rrrr rarWr*r+rds.        zRNNTDecoder.__init__cCsDt|}|jr d}nd}|j|||d\}}|dd}|||fSrhrl)r)r$r%r&rprjrqr*r*r+rrrszRNNTDecoder.forwardrprirjrtrucCs4t|}|j}|j}|dur"|j|kr||}|jd|}n |dur4|dur,dn|dd}n|}tj|d|j f||d}|rc|j \}} } tj|d| f|j|jd} tj | |gdd }nd} |durt|j rt|jrt||}|dd}|jd||\} } | dd} ~~ ~|r|| } | | fS)at Stateful prediction of scores and state for a (possibly null) tokenset. This method takes various cases into consideration : - No token, no state - used for priming the RNN - No token, state provided - used for blank token scoring - Given token, states - used for scores + new states Here: B - batch size U - label length H - Hidden dimension size of RNN L - Number of RNN layers Args: y: Optional torch tensor of shape [B, U] of dtype long which will be passed to the Embedding. If None, creates a zero tensor of shape [B, 1, H] which mimics output of pad-token on EmbeddiNg. state: An optional list of states for the RNN. Eg: For LSTM, it is the state list length is 2. Each state must be a tensor of shape [L, B, H]. If None, and during training mode and `random_state_sampling` is set, will sample a normal distribution tensor of the above shape. Otherwise, None will be passed to the RNN. add_sos: bool flag, whether a zero vector describing a "start of signal" token should be prepended to the above "y" tensor. When set, output size is (B, U + 1, H). batch_size: An optional int, specifying the batch size of the `y` tensor. Can be infered if `y` and `state` is None. But if both are None, then batch_size cannot be None. Returns: A tuple (g, hid) such that - If add_sos is False: g: (B, U, H) hid: (h, c) where h is the final sequence hidden state and c is the final cell state: h (tensor), shape (L, B, H) c (tensor), shape (L, B, H) If add_sos is True: g: (B, U + 1, H) hid: (h, c) where h is the final sequence hidden state and c is the final cell state: h (tensor), shape (L, B, H) c (tensor), shape (L, B, H) Nembedr6rrvrwdec_rnn)rCrDrEr<rBrgr;r>ryr^rzr{r|rtrainingrGrois_adapter_availableforward_enabled_adapters)r)rprirjrtr}rEr<rr~Hrrqhidr*r*r+rns4 ?        zRNNTDecoder.predictc Csv|jrtjj|d||jd} ntj||} tj| tj|| dkr$| n|||||| |||| kr2|ndd d} | S)a Prepare the trainable parameters of the Prediction Network. Args: vocab_size: Vocab size (excluding the blank token). pred_n_hidden: Hidden size of the RNNs. pred_rnn_layers: Number of RNN layers. forget_gate_bias: Whether to perform unit forget gate bias. t_max: Whether to perform Chrono LSTM init. norm: Type of normalization to perform in RNN. weights_init_scale: Float scale of the weights after initialization. Setting to lower than one sometimes helps reduce variance between runs. hidden_hidden_bias_scale: Float scale for the hidden-to-hidden bias scale. Set to 0.0 for the default behaviour. dropout: Whether to apply dropout to RNN. rnn_hidden_size: the hidden size of the RNN, if not specified, pred_n_hidden would be used r6) padding_idxr) input_size hidden_size num_layersr rrrarr  proj_size)r r)r`r>nn Embeddingr@ ModuleDictr ) r)r[r rrrr rr rar r layersr*r*r+rf.s(zRNNTDecoder._predict_modulescCs|d}|jr)|jr)tj|j||j|j|jdtj|j||j|j|jdf}|Stj |j||j|j|jdtj |j||j|j|jdf}|S)a Initialize the state of the LSTM layers, with same dtype and device as input `y`. LSTM accepts a tuple of 2 tensors as a state. Args: y: A torch.Tensor whose device the generated states will be placed on. Returns: Tuple of 2 tensors, each of shape [L, B, H], where L = Number of RNN layers B = Batch size H = Hidden size of RNN. r)r<rE) r;rrr>randnrr^r<rEryrr*r*r+rGds   zRNNTDecoder.initialize_staterrc Crrrrr*r*r+rrzRNNTDecoder.score_hypothesisrcCs\t|}|dkr tdt|}|j}g}g}ddt|D}t|D]$\} } t| j} | |vr:|| || <q&| | jd| | | j fq&|rt|} t j ||t j d| d}|dd|D} |j|| d| d \}} d}t|D]&} || d ur|| |}|||f|| <|||f|||d<|d 7}qyd d|Dd d|DfS)rrrcSrrr*rr*r*r+rrz6RNNTDecoder.batch_score_hypothesis..rrvcSrr*r*rr*r*r+rrFrNr6cSrr*r*rr*r*r+rrcSrr*r*rr*r*r+rrr)r)rrrr}rErrrrrrrrrrrr*r*r+rs<      z"RNNTDecoder.batch_score_hypothesisrcCs0tdd|D}|dddd}t|S)aM Creates a stacked decoder states to be passed to prediction network Args: decoder_states (list of list of list of torch.Tensor): list of decoder states [B, C, L, H] - B: Batch size. - C: e.g., for LSTM, this is 2: hidden and cell states - L: Number of layers in prediction RNN. - H: Dimensionality of the hidden state. Returns: batch_states (list of torch.Tensor): batch of decoder states [C x torch.Tensor[L x B x H] cSsg|]}t|qSr*r)r decoder_stater*r*r+r z7RNNTDecoder.batch_initialize_states..r6rkr)r>rpermuterr|)r)rstacked_statespermuted_statesr*r*r+rs z#RNNTDecoder.batch_initialize_statesrrcs|dur fdd|DSdS)agGet decoder state from batch of states, for given id. Args: batch_states (list): batch of decoder states ([L x (B, H)], [L x (B, H)]) idx (int): index to extract state from batch of states Returns: (tuple): decoder states for given id ([L x (1, H)], [L x (1, H)]) Ncsg|] }|ddfqSrr*rrr*r+rsz2RNNTDecoder.batch_select_state..r*)r)rrr*r$r+rs zRNNTDecoder.batch_select_stater beam_sizeindicesrc Cs|djd}|djd}t||||f}t||||f} |ddddddf|} |dur\tj|d|d| |d|dtj|d|d| |d|d|Stj|d|d| d| tj|d|d| d| fS)a} Aggregates decoder states based on the given indices. Args: src_states (Tuple[torch.Tensor, torch.Tensor]): source states of shape `([L x (batch_size * beam_size, H)], [L x (batch_size * beam_size, H)])` batch_size (int): The size of the batch. beam_size (int): The size of the beam. indices (torch.Tensor): A tensor of shape `(batch_size, beam_size)` containing the indices in beam that map the source states to the destination states. dst_states (Optional[Tuple[torch.Tensor, torch.Tensor]]): If provided, the method updates these tensors in-place. Returns: Tuple[torch.Tensor, torch.Tensor]: Note: - The `indices` tensor is expanded to match the shape of the source states during the gathering operation. rrNrk)rxindexrr6)rxr')rzr>Sizeexpandgatherr) rrrtr%r&r layers_num layers_dim beam_shape flat_shapeindices_expandedr*r*r+batch_aggregate_states_beam"s z'RNNTDecoder.batch_aggregate_states_beamcCsg}tt|dD]A}g}tt|D]%}t|||tjs)t|||n|||}|d}||qt|d}| dd}||q |S)aConcatenate a batch of decoder state to a packed state. Args: batch_states (list): batch of decoder states B x ([L x (H)], [L x (H)]) Returns: (tuple): decoder states (L x B x H, L x B x H) rr6) rr isinstancer>rrrrr{ro)r)rrstate_idrrrrr*r*r+rUs       zRNNTDecoder.batch_concat_statesrrcCs|dur|n|}|dj}tj|dd|d||d||ddtj|dd|d||d||dddS)a Replaces states in `dst_states` with states from `src_states` based on the given `mask`. Args: mask (torch.Tensor): When True, selects values from `src_states`, otherwise `out` or `other_src_states`(if provided). src_states (Tuple[torch.Tensor, torch.Tensor]): Values selected at indices where `mask` is True. dst_states (Tuple[torch.Tensor, torch.Tensor])): The output states. other_src_states (Tuple[torch.Tensor, torch.Tensor], optional): Values selected at indices where `mask` is False. Note: This operation is performed without CPU-GPU synchronization by using `torch.where`. Nrrrr6)r<r>rrrB)rrrrrrr<r*r*r+rss 6:z%RNNTDecoder.batch_replace_states_maskcCs|dur|d|d|d|ddS|dddd|f|dddd|f|dddd|f|dddd|fdS)rNrr6rrr*r*r+rs 26z$RNNTDecoder.batch_replace_states_allcCs|d|dfS)rrr6rrr*r*r+rszRNNTDecoder.clone_statecCs0ddt|djddd|djdddDS)rcSs$g|]\}}|d|dfqSr6)squeeze)r sub_state_1 sub_state_2r*r*r+rsz2RNNTDecoder.batch_split_states..rr6rw)ziprrr*r*r+rs$zRNNTDecoder.batch_split_statescCsDtjdd|Dddj||dtjdd|Dddj||dfS)a: Concatenate a batch of decoder state to a packed state. Inverse of `batch_split_states`. Args: batch_states (list): batch of decoder states B x ([L x (H)], [L x (H)]) Returns: (tuple): decoder states (L x B x H, L x B x H) cSrrr*rr*r*r+rrz4RNNTDecoder.batch_unsplit_states..r6rwrvcSrr3r*rr*r*r+rrrrr*r*r+rs  z RNNTDecoder.batch_unsplit_statesrrrrcCstt|D]A}|dur%||dd|ddf||dd|ddf<q||dd|ddfd9<||dd|ddf|7<q|S)aCCopy states from new state to old state at certain indices. Args: old_states(list): packed decoder states (L x B x H, L x B x H) new_states: packed decoder states (L x B x H, L x B x H) ids (list): List of indices to copy states at. value (optional float): If a value should be copied instead of a state slice, a float should be provided Returns: batch of decoder states with partial copy at ids (or a specific value). (L x B x H, L x B x H) Nrb)rr)r)rrrrr2r*r*r+rs 2"$zRNNTDecoder.batch_copy_statesr&cCs(|ddd|f|ddd|ffS)rrNr6r*rr*r*r+rs( zRNNTDecoder.mask_select_statesnamecfgc||}tj||ddSN)r8r9_update_adapter_cfg_input_dimrS add_adapterr)r8r9rWr*r+r> zRNNTDecoder.add_adaptercCtj|||jd}|SN) module_dim)r update_adapter_cfg_input_dimr^r)r9r*r*r+r=z)RNNTDecoder._update_adapter_cfg_input_dimr)NFTrrr)0rrrrrr,r5rMrTrrrrrrrdrrrr>rrrrnrfrGr rrrrrrr'r0rrrrrrrrHrrrr>r=rr*r*rWr+r(s9     &   o 6 5 "A& "2  ,    ! rcseZdZdZeddZeddZfddZdMd d Zed d Z        dNde e e fde de deedeedededee de deffdd Ze    dOdejdeejd eejd!eejd"eejd#ed$eejeeejffd%d&Zd'ejd$ejfd(d)Zd*ejd$ejfd+d,Zd-ejd.ejd$ejfd/d0Zd1d2Zd3e d4effd5d6 Zd4efd7d8Zed9d:Zed;d<Z ed=d>Z!d?d@Z"edAdBZ#dCdDZ$edEdFZ%dPdGdHZ&edIdJZ'dKdLZ(Z)S)Q RNNTJointaA Recurrent Neural Network Transducer Joint Network (RNN-T Joint Network). An RNN-T Joint network, comprised of a feedforward model. Args: jointnet: A dict-like object which contains the following key-value pairs. encoder_hidden: int specifying the hidden dimension of the encoder net. pred_hidden: int specifying the hidden dimension of the prediction net. joint_hidden: int specifying the hidden dimension of the joint net activation: Activation function used in the joint step. Can be one of ['relu', 'tanh', 'sigmoid']. Optionally, it may also contain the following: dropout: float, set to 0.0 by default. Optional dropout applied at the end of the joint net. num_classes: int, specifying the vocabulary size that the joint network must predict, excluding the RNNT blank token. vocabulary: Optional list of strings/tokens that comprise the vocabulary of the joint network. Unused and kept only for easy access for character based encoding RNNT models. log_softmax: Optional bool, set to None by default. If set as None, will compute the log_softmax() based on the value provided. preserve_memory: Optional bool, set to False by default. If the model crashes due to the memory intensive joint step, one might try this flag to empty the tensor cache in pytorch. Warning: This will make the forward-backward pass much slower than normal. It also might not fix the OOM if the GPU simply does not have enough memory to compute the joint. fuse_loss_wer: Optional bool, set to False by default. Fuses the joint forward, loss forward and wer forward steps. In doing so, it trades of speed for memory conservation by creating sub-batches of the provided batch of inputs, and performs Joint forward, loss forward and wer forward (optional), all on sub-batches, then collates results to be exactly equal to results from the entire batch. When this flag is set, prior to calling forward, the fields `loss` and `wer` (either one) *must* be set using the `RNNTJoint.set_loss()` or `RNNTJoint.set_wer()` methods. Further, when this flag is set, the following argument `fused_batch_size` *must* be provided as a non negative integer. This value refers to the size of the sub-batch. When the flag is set, the input and output signature of `forward()` of this method changes. Input - in addition to `encoder_outputs` (mandatory argument), the following arguments can be provided. - decoder_outputs (optional). Required if loss computation is required. - encoder_lengths (required) - transcripts (optional). Required for wer calculation. - transcript_lengths (optional). Required for wer calculation. - compute_wer (bool, default false). Whether to compute WER or not for the fused batch. Output - instead of the usual `joint` log prob tensor, the following results can be returned. - loss (optional). Returned if decoder_outputs, transcripts and transript_lengths are not None. - wer_numerator + wer_denominator (optional). Returned if transcripts, transcripts_lengths are provided and compute_wer is set. fused_batch_size: Optional int, required if `fuse_loss_wer` flag is set. Determines the size of the sub-batches. Should be any value below the actual batch size per GPU. masking_prob: Optional float, indicating the probability of masking out decoder output in HAINAN (Hybrid Autoregressive Inference Transducer) model, described in https://arxiv.org/pdf/2410.02597 Default to -1.0, which runs standard Joint network computation; if > 0, then masking out decoder output with the specified probability. c CsTtdttdtttdtddtdtddttdtddtdddS)rr0rTr!rencoder_outputsdecoder_outputsencoder_lengths transcriptstranscript_lengths compute_wer)rrrr'rrr(r*r*r+r,Gs  zRNNTJoint.input_typescCsL|js dtdtiSttddttddttddttdddS)r/r3rr r r1T) elements_typer")losswer wer_numer wer_denom)_fuse_loss_werrrrrr(r*r*r+r5Ss     zRNNTJoint.output_typesc s"d|_d|_tjdi|dS)NFr*)rU log_softmaxrSrTrUrWr*r+rTcszRNNTJoint._prepare_for_exportr6 cCsX|||}}}t||j|t|j}t||j|t|j}||fS)r9)r>rencoder_hiddenrBrCrDrEr^)r)rJrKrr r~rIrJr*r*r+rMhs  zRNNTJoint.input_examplecCs tgdS)zHImplement this method to return a set of input names disabled for export)rKrLrMrN)setr(r*r*r+disabled_deployment_input_namesss z)RNNTJoint.disabled_deployment_input_namesrNFjointnet num_classesnum_extra_outputs vocabularyrVpreserve_memory fuse_loss_werfused_batch_sizeexperimental_fuse_loss_wer masking_probc st||_||_||_|d||_| |_|jdkr&|jdks&Jd| dur,| }||_||_|r<|dur