o }oi^@sddlZddlmZddlmZddlZddlmmZ ddl m Z ddl m Z ddlmZddlmZer=ddlmZd efd d ZGd d d e ZGdddeZdS)N) nullcontext)ContextManager)Loss)k2)TRITON_AVAILABLE)logging)rnnt_logprobs_tritonreturncCs tr tjjdtjdStS)z@Get context manager to force float32 precision in autocast mode.cuda)dtype)torchis_autocast_enabledampautocastfloat32rrrb/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/parts/k2/graph_transducer.pyforce_float32_contextsrc seZdZdZ ddeffdd Zejdej de dd fd d Z ejd e de d ej dd fddZ ejdej d e de dd fddZdej d e de dd fddZdej dej dej de dd f ddZdejdej fddZdejdejdej fddZZS) GraphTransducerLossBasea Base class for graph transducer losses. Implementation of the approach described in "Powerful and Extensible WFST Framework for RNN-Transducer Losses" https://ieeexplore.ieee.org/document/10096679 Compose-Transducer: compose the unit (target text) and temporal schemas (graphs) into lattice. Subclass should implement `get_unit_schema` and `get_temporal_schema` methods. Grid-Transducer: construct the RNN-T lattice (grid) directly in code. Subclass should implement `get_grid` method. Fuse_grid_implementationcs&t||_||_||_||_dS)a Args: use_grid_implementation: Whether to use the grid implementation (Grid-Transducer). connect_composed: Connect graph after composing unit and temporal schemas (only for Compose-Transducer). `connect` operation is slow, it is useful for visualization, but not necessary for loss computation. double_scores: Use calculation of loss in double precision (float64) in the lattice. Does not significantly affect memory usage since the lattice is ~V/2 times smaller than the joint tensor. cast_to_float32: Force cast joint tensor to float32 before log-softmax calculation. N)super__init__rconnect_composed double_scorescast_to_float32)selfrrrr __class__rrr2s  z GraphTransducerLossBase.__init__ units_tensor vocab_sizer k2.FsacCdS)a Get unit schema (target text) graph for Compose-Transducer. Args: units_tensor: tensor with target text vocab_size: number of labels (including blank). Needed to construct additional eps-arcs (in some cases). Returns: unit schema graph (k2.Fsa). Labels: :: (k2.Fsa: labels, aux_labels, unit_positions) Nr)rrrrrrget_unit_schemaFs z'GraphTransducerLossBase.get_unit_schema num_framesdevicecCr!)a Get temporal schema graph for Compose-Transducer. Args: num_frames: length of the sequence (in frames) vocab_size: number of labels (including blank) device: device for tensor to construct Returns: temporal schema graph (k2.Fsa). Labels: :. is a unit from vocab + special units (e.g., additional eps). Nr)rr#rr$rrrget_temporal_schemaUz+GraphTransducerLossBase.get_temporal_schemacCr!)a Construct the transducer lattice (grid) directly for Grid-Transducer. Args: units_tensor: tensor with target text num_frames: length of the sequence (in frames) vocab_size: number of labels (including blank) Returns: transducer lattice (k2.Fsa). Labels: :: (k2.Fsa: labels, aux_labels, unit_positions) Nr)rrr#rrrrget_grider&z GraphTransducerLossBase.get_gridcCs@|||}||||j}tj||dd}|jrt|}|S)a Get composed lattice (unit and temporal schemas) for Compose-Transducer. Useful for visualization. Should be equivalent to the lattice from `get_grid` method. Args: units_tensor: tensor with target text num_frames: length of the sequence (in frames) vocab_size: vocab size (including blank) Returns: composed lattice (k2.Fsa) from unit and temporal schemas Ftreat_epsilons_specially)r"r%r$rcomposerconnect)rrr#rfsa_text fsa_temporalcomposedrrrget_composed_latticeus  z,GraphTransducerLossBase.get_composed_latticesource_lengthstargetstarget_lengthsc sjd}tgjr.tfddt|DWdSfddt|D}fddt|D}tjt|t|dd}j rht |}Wd|SWd|S1sswY|S) a{ Get batched lattice (grid or composed) for the batch of sequences. Args: source_lengths: tensor with lengths of logits targets: tensor with target units target_lengths: tensor with lengths of targets vocab_size: vocab size (including blank) Returns: batched lattice - FsaVec (k2.Fsa) rcs0g|]}j|d|f|dqS)N)rr#r)r'.0i)rsource_lengths_listtarget_lengths_listr1rrr sz>GraphTransducerLossBase.get_graphs_batched..Ncs.g|]}j|d|fdqS)N)rr)r"itemr3)rr2r1rrrr8scs&g|]}j|jdqS))r#rr$)r%r9r$r3)rr0r1rrrr8s Fr() shaper no_gradrtolistrcreate_fsa_vecranger*rr+) rr0r1r2r batch_size text_fsas temporal_fsastarget_fsas_vecr)rr0r6r2r7r1rrget_graphs_batcheds:    " ""z*GraphTransducerLossBase.get_graphs_batchedrBcsJjd}j}ttj||tjdtjfddt|D|d}|S)z Get batch indices (for logits) for each arc in the lattices. Args: target_fsas_vec: batch of target FSAs with lattices Returns: 1d tensor with indices r)r$r cs(g|]}jd|djdqS)r)arcsindexvaluesr:r3rBrrr8s(z=GraphTransducerLossBase.get_batch_indices..r$)r:r$r repeat_interleavearangeint64tensorr>)rrBr?r$scores_to_batch_irrGrget_batch_indicess z)GraphTransducerLossBase.get_batch_indices logits_shapecCsn|j|d}||d|d|d|jtj|d|d|jtj|d|jtj}|S)a Get indices of flatten logits for each arc in the lattices. Args: target_fsas_vec: batch of target FSAs with lattices logits_shape: shape of the logits tensor Returns: 1d tensor with indices rG)rN aux_labelstor rKunit_positionslabels)rrBrOrMindicesrrrget_logits_indicess  z*GraphTransducerLossBase.get_logits_indices)FFF)__name__ __module__ __qualname____doc__boolrabcabstractmethodr Tensorintr"r$r%r'r/rCrFsarNSizerX __classcell__rrrrr&s4  4$rc seZdZdZ      d$deffdd Zdejded d fd d Zd ededej d d fddZ e dejdeded ejfddZ dejd eded d fddZ d%dejdejdejdejd d f ddZdejdejd ejd!ejd ejeejd fBf d"d#ZZS)& GraphRnntLossz RNN-T loss implementation based on WFST according to "Powerful and Extensible WFST Framework for RNN-Transducer Losses" https://ieeexplore.ieee.org/document/10096679 TFblankcsBtj||||d||_||_|ot|_|jstddSdS)a Init method Args: blank: blank label index use_grid_implementation: Whether to use the grid implementation (Grid-Transducer). connect_composed: Connect graph after composing unit and temporal schemas (only for Compose-Transducer). `connect` operation is slow, it is useful for visualization, but not necessary for loss computation. double_scores: Use calculation of loss in double precision (float64) in the lattice. Does not significantly affect memory usage since the lattice is ~V/2 times smaller than the joint tensor. cast_to_float32: Force cast joint tensor to float32 before log-softmax calculation. return_graph: Return graph (along with loss) from `forward` function use_triton: use optimized log probs calculations with Triton (faster and more memory efficient) )rrrrz.Triton is disabled, memory usage can be largerN)rrrf return_graphr use_tritonrwarning)rrfrrrrrgrhrrrrs zGraphRnntLoss.__init__rrr r c Cs|j}|j}|jd}tj|dddftj|d}tjd|dtj|d}||ddddf<||ddddf<||ddddf<||ddddf<|d|ddddf<||ddddf<d|d<|dddf}t ||} | dd dd | _d| jd<| S) a Get unit schema (target text) graph for RNN-T loss (Compose-Transducer). Forward arcs represent text labels. Example graph: text [1, 2], blank=0. graph:: 0:0:0 0:0:1 0:0:2 +-------+ +-------+ +-------+ v | v | v | +-----------+ 1:1:0 +-----------+ 2:2:1 +-----------+ -1:-1:-1 #===# | 0 | -------> | 1 | -------> | 2 | ---------> H 3 H +-----------+ +-----------+ +-----------+ #===# Args: units_tensor: 1d tensor with text units vocab_size: number of total labels (vocab size including blank) Returns: unit schema graph (k2.Fsa). Labels: :: (k2.Fsa: labels, aux_labels, unit_positions) rrPrQr r$N)rlrQ)rfr$r:r zerosint32rJdetachclonerrbexpand transposeflattenrU) rrrblank_idr$text_lenrD text_indicesolabelsr,rrrr"s"   zGraphRnntLoss.get_unit_schemar#r$c Cs|j}tj||ddftj|d}tjd|tj|d}|||dd}||dddf<||dddf<tjd|tj|d|||dddf<|d||d|df<tj||ddftj|d|dddf<|dddf }d|d<t ||} t | } | S) a Get temporal schema graph for RNN-T loss (Compose-Transducer). Forward arc - blank, self-loops - all labels excluding blank Example graph: blank=0, num_frames=3, vocab_size=3. Labels: :. is a unit from vocab. graph:: 1:0 1:1 1:2 +-----+ +-----+ +-----+ v | v | v | +---------+ 0:0 +---------+ 0:1 +---------+ 0:2 +---+ -1:-1 #===# | 0 | -----> | 1 | -----> | 2 | -----> | 3 | -------> H 4 H +---------+ +---------+ +---------+ +---+ #===# ^ 2:0 | ^ 2:1 | ^ 2:2 | +-----+ +-----+ +-----+ Args: num_frames: length of the sequence (in frames) vocab_size: number of labels (including blank) device: device for tensor to construct Returns: temporal schema graph (k2.Fsa). Labels: :. is a unit from vocab. rPrjrkrNrlrQrR)rfr rmrnrJrqrrrsrLrorprrbarc_sort) rr#rr$rtfsa_temporal_arcssequence_states start_statesrwr-rrrr%Gs(  z!GraphRnntLoss.get_temporal_schemastatesnmc Cs||}tj||dd}t||}t||}||}||d|}||d} ||kr.|n||d} ||||dd?|t||||||dd?|||| ||| ||dd?||} tj|||k|| | d| S)a Relabel states to be in topological order: by diagonals Args: states: tensor with states n: number of rows m: number of columns Returns: tensor with relabeled states (same shape as `states`) floor rounding_moderP)out)r divminmaxlt logical_andgewhere) r|r}r~r5jmin_mnmax_mndiag anti_diagmax_idx cur_diag_idx new_statesrrrrelabel_states~s$    $zGraphRnntLoss.relabel_statescCsT|j}|jd}|j}||d}|d|d}||} tj|| ddftj|d} tj||d} | |d} | | d|df<| | d|df<|| d|df<tj|tj|d||dddddf} | d} | |d} | | |d df<| | |d df<| | |d df<tj |d||ftj|d| d dd f<tj ||ddftj|d| ddd f<tj | dddf|dd d }| dddf|d}d|d<d|d<| | dd df|d|| dd df<| | dd df|d|| dd df<tj | dddfdd\}}| |}||}||}t||}||_|S)a Construct the RNN-T lattice directly (Grid-Transducer). Args: units_tensor: 1d tensor with text units num_frames: length of the sequence (number of frames) vocab_size: number of total labels (vocab size including blank) Returns: transducer lattice (k2.Fsa). Labels: :: (k2.Fsa: labels, aux_labels, unit_positions) rrPrQrjrkrHNrlrRrrdim)rfr:r$r rmrnrJreshapersrqrLrrsortrrbrU)rrr#rrt text_lengthr$num_grid_statesnum_forward_arcs num_text_arcsrD from_states to_statesilabelsrwrU_rW sorted_arcs rnnt_graphrrrr'sJ   (( ** zGraphRnntLoss.get_gridlogitsr1r0r2c Cs|jd}|||||}t4|jdk}|j|d} |jtj } |j tj } |jtj } | |dWdn1sHwY|j rSt nt} | h|jr|jjdkrt|||j||d\}}| |jk}t||| | | f|| | | ftj}d||<ntj|dd}|| | | | ftj}d||<|r|j||_n ||_Wd|SWd|S1swY|S) a Get batch of graphs (FsaVec) for RNN-T loss calculation. Args: logits: activations (joint tensor). NB: raw logits, not after log-softmax targets: target labels source_lengths: lengths of source sequences target_lengths: length of target sequences use_graph_weight: uses weight from graphs (if `get_graphs_batched` returns graphs with weights) Returns: FsaVec containing RNN-T graphs for all utterances. rlrGrNr )rr1rtr0r2gr)r:rCr r;rVrNrSrprTrKrU masked_fill_rrrrhr$typerrfrrF log_softmaxscores)rrr1r0r2use_graph_weightrrBlast_transition_mask batch_indices time_indices unit_indices text_units cast_context unit_scores blank_scorestext_units_blank_maskr log_probsrrrget_weighted_graphssT           z!GraphRnntLoss.get_weighted_graphsactsrVact_lens label_lenscCs:|j||||dd}d|j|jdd}|jr||fS|S)aS Compute forward method for RNN-T. Args: acts: activations (joint tensor). NB: raw logits, not after log-softmax labels: target labels act_lens: lengths of activations label_lens: length of labels sequences Returns: batch of RNN-T scores (loss) F)rr1r0r2rrlT)use_double_scores log_semiring)rget_tot_scoresrrg)rrrVrrrBrrrrforwards zGraphRnntLoss.forward)TFFFFT)F)rYrZr[r\rarr r`r"r$r% staticmethodrr'rtuplerrdrrrrresN %37 I ?re)r^ contextlibrtypingrr torch.nn.functionalnn functionalrnemo.core.classes.lossrnemo.core.utils.k2_guardrnemo.core.utils.optional_libsr nemo.utilsr2nemo.collections.asr.parts.k2.rnnt_logprobs_tritonrrrrerrrrs       C