o }oi?@snddlmZddlmZddlZddlmmZddl m Z m Z ddl m Z ddlmZGddde ZdS) ) nullcontext)UnionN) GraphRnntLossforce_float32_context)k2) PrettyStrEnumc seZdZdZGdddeZdejddddfdeded e ee fffd d Z d e j d eddfddZded ede jddfddZd e j ded eddfddZde j de j de j de j fddZZS)GraphWTransducerLossa W-Transducer loss: RNN-T loss modification for training RNN-T model for the case when some text at the beginning/end of the utterance is missing. The resulting model behaves like the RNN-T model (no modification for decoding is required). For details see "Powerful and Extensible WFST Framework for RNN-Transducer Losses" paper https://ieeexplore.ieee.org/document/10096679 c@seZdZdZdZdS)z"GraphWTransducerLoss.LastBlankMode allow_ignore force_finalN)__name__ __module__ __qualname__ ALLOW_IGNORE FORCE_FINALrr^/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/parts/k2/w_transducer.py LastBlankMode#srgTFblank eps_weightlast_blank_modecs,tj|||||d||_|||_dS)a Init method Args: blank: blank label index eps_weight: weight of epsilon transitions, 0 means no penalty (default) last_blank_mode: allow to skip last blank in the prediction (default) or force it use_grid_implementation: Whether to use the grid implementation (Grid-Transducer). connect_composed: Connect graph after composing unit and temporal schemas (only for Compose-Transducer). `connect` operation is slow, it is useful for visualization, but not necessary for loss computation. double_scores: Use calculation of loss in double precision (float64) in the lattice. Does not significantly affect memory usage since the lattice is ~V/2 times smaller than the joint tensor. cast_to_float32: Force cast joint tensor to float32 before log-softmax calculation. )ruse_grid_implementationconnect_composed double_scorescast_to_float32N)super__init__rrr)selfrrrrrrr __class__rrr'szGraphWTransducerLoss.__init__ units_tensor vocab_sizereturnzk2.Fsac CsX|j}|}|d}|j}|jd}tj|ddddftj|d}tjd|dtj|d} ||d<| |ddddf<| |ddddf<||ddddf<| |ddddf<| d|ddddf<||ddddf<|d|d<||d <||d <d|d <|d d df} t || } t | | _ | dddd| j dd<d| j d<| S) ak Get unit schema (target text) graph for W-Transducer loss (Compose-Transducer). Forward arcs represent text labels. Example graph: text [1, 2], blank=0. Eps ids: 3, 4. graph:: 3:3:0 0:0:1 0:0:2 +-------+ +-------+ +-------+ v | v | v | +-----------+ 1:1:0 +-----------+ 2:2:1 +-----------+ -1:-1:-1 #===# | 0 | -------> | 1 | -------> | 2 | ---------> H 3 H +-----------+ +-----------+ +-----------+ #===# ^ 0:0:0 | ^ 4:4:2 | +-------+ +-------+ Args: units_tensor: 1d tensor with text units vocab_size: number of total labels (vocab size including blank) Returns: unit schema graph (k2.Fsa). Labels: :: (k2.Fsa: labels, aux_labels, unit_positions) rdtypedevice)rr#)r)r")r)r#)r(r#N)rr'shapetorchzerosint32arangedetachclonerFsa zeros_likeunit_positionsexpand transposeflatten) rrr blank_id start_eps_id end_eps_idr'text_lenarcs text_indicesolabelsfsa_textrrrget_unit_schemaJs0 "   " z$GraphWTransducerLoss.get_unit_schema num_framesr'cCs|j}|}|d}d}|||d|d}tj|dftj|d} tjd|tj|d} | d} | |||dd} | dd| d|df<| dd| d|df<tjd||tj|d|||dd| d|df<| | ||||df<d| ||||df<| d| ||||df<| dd| ||||df<|j|j j kr|dn|| ||||df<tj ||ddftj|d| dddf<tj | dddfdd \} }| |} | dddf }d|d<t| |}t|}|S) a Get temporal schema graph for W-Transducer loss (Compose-Transducer). Example graph: blank=0, num_frames=3, vocab_size=3, last_blank_mode="force_final". Labels: :. is a unit from vocab + special eps ids `vocab_size`, `vocab_size+1`. graph for force_final:: 4:0 +--------------------------------------------+ | 4:1 | | +--------------------+ | 1:0 | 1:1 | 1:2 | | +-----+ | +-----+ | +-----+ | | v | | v | | v | v v +--------------+ 0:0 +------------+ 0:1 +------------+ 0:2 +---+ -1:-1 #===# | 0 | ----> | 1 | -----> | 2 | -----> | 3 | -------> H 4 H +--------------+ +------------+ +------------+ +---+ #===# ^ 2:0 | | | ^ 2:1 | ^ ^ 2:2 | ^ +-----+ | | +-----+ | +-----+ | | | 3:0 | | | +------------------+ 3:0 | +-------------------------------------------+ Args: num_frames: length of the sequence (in frames) vocab_size: number of labels (including blank) device: device for tensor to construct Returns: temporal schema graph (k2.Fsa). Labels: :. is a unit from vocab + special units (e.g., additional eps). r"r#r$r%rNr(dim)rr+r,r-r.r4r5r6rrrtensorsortr/r0rr1arc_sort)rr@r r'r7r8r9num_epsnum_sequence_arcsfsa_temporal_arcssequence_statessequence_states_next start_states_indicesr= fsa_temporalrrrget_temporal_schemas<#  (  z(GraphWTransducerLoss.get_temporal_schemacCs|j}|}|jd}|j}||d}|d|d} |dd} | | } ||} tj| | ddftj|d} tj| |d}||d}|| d| df<|| d| df<|| d| df<ttj|d|d|d|tj|d|d|dg}||d}|| | | |dddf<|| | | |dddf<|| | | |ddf<|d| | |d| |dddf<d| | | |ddf<|d| | |d| |dddf<tj|tj|d||dddddf }|d}| |d }|| | d df<|| | d df<|| | d df<tj |d||ftj|d| d dd f<tj ||ddftj|d| ddd f<tj | dddf|dd d }| dddf|d}d|d<d|d<| | dd df|d|| dd df<| | dd df|d|| dd df<|j|jjkr|| | |d| |dddf<tj| dddfdd\}}| |} ||}||}t| |}||_|S)a Construct W-Transducer lattice directly (Grid-Transducer). Args: units_tensor: 1d tensor with text units num_frames: length of the sequence (number of frames) vocab_size: number of total labels (vocab size including blank) Returns: transducer lattice (k2.Fsa). Labels: :: (k2.Fsa: labels, aux_labels, unit_positions) rr"r#r$r%)r'Nr(r)rAfloor) rounding_moderB)rr*r'r+r,r-r.catreshaper6r4rDdivrelabel_statesrrrrErr1r3)rrr@r r7eps_id text_lengthr'num_grid_statesnum_forward_arcs_basenum_forward_arcs_additionalnum_forward_arcs num_text_arcsr; from_states to_statesilabelsr=r3rMrN rnnt_graphrrrget_gridst      ("(( ** zGraphWTransducerLoss.get_gridactslabelsact_lens label_lensc Cs||||f\}}}}|jd} ||||| } |jrtnt} | ^tj|dd} t| | |j} d| | j dk<d| | j | k<Wdn1sNwY| d| }d|| j dk<|j || j | k<|| _d| j|jdd}|WdS1swYdS)z Forward method is similar to RNN-T Graph-Transducer forward method, but we need to assign eps weight to eps-transitions. r(rBrNT)use_double_scores log_semiring)r*get_graphs_batchedrrrF log_softmaxr+no_gradget_logits_indicesrer6 index_selectrscoresget_tot_scoresr)rrdrerfrglogitstargetslogits_lengthstarget_lengthsr target_fsas_vec cast_context log_probsrNrprrrforward3s$  $zGraphWTransducerLoss.forward)r r r __doc__rrrintfloatrstrrr+Tensorr?r'rPrcry __classcell__rrrrrs8 #<P]r) contextlibrtypingrr+torch.nn.functionalnn functionalrk.nemo.collections.asr.parts.k2.graph_transducerrrnemo.core.utils.k2_guardrnemo.utils.enumrrrrrrs