o }oi-@sddlmZddlmZmZmZddlZddlmZddl m Z m Z m Z m Z mZmZddlmZGdddeZGd d d eZdS) )ABC)ListOptionalTupleN)GradExpNormalize)create_supervisionget_arc_weightsget_uniform_rnnt_prune_rangesmake_non_pad_maskmake_non_pad_mask_3dprep_padded_densefsavec)k2c@seZdZdZ  ddejdejdeejdeejdeejejeejeejff dd Zdejd ejdd fd d Z dejdejdejfddZ dddde de dee eje ejffddZ dS) CtcK2Mixinak2 Mixin class that simplifies the construction of various k2-based CTC-like losses. It does the following: - Prepares and adapts the input tensors (method _prepare_log_probs_and_targets). - Creates Emissions graphs (method _prepare_emissions_graphs). - Extracts the labels and probabilities of the best lattice path (method _extract_labels_and_probabilities). N log_probs input_lengthstargetstarget_lengthsreturncCsL|d|jks Jt|}|||dur"t||jk|d||fSd|fS)zcCreates k2-style supervisions and shifts targets by one if the number is not zero. N)size num_classesrtorchwhereblank)selfrrrr supervisionsr]/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/parts/k2/loss_mixins.py_prepare_log_probs_and_targets)s z)CtcK2Mixin._prepare_log_probs_and_targetsrk2.DenseFsaVeccCs|jrt||St||S)a"Creates DenseFsaVec, padding it with frames if the topology is `compact`. In particular, every second frame of the DenseFsaVec is the frame. frame is a frame with log-probability zero and every other log-probability is -inf. ) pad_fsavecr r DenseFsaVec)rrrrrr_prepare_emissions_graphs<s   z$CtcK2Mixin._prepare_emissions_graphscCst|||jdkr dSdS)zPyTorch is doing the log-softmax normalization as part of the CTC computation. More: https://github.com/k2-fsa/k2/issues/575 summeannone)rapply reductionrrrrrr_maybe_normalize_gradientsHsz%CtcK2Mixin._maybe_normalize_gradientsFTshortest_path_fsaszk2.Fsareturn_ilabelsoutput_alignedc Csg}g}t|jdD]`}||}|jdk|j|jk@}|r$|j|} n |j|} |jdkrD|j| | dk<| | dk| |jk@d8<| jtjd} |sV|sV| | |jk} | | | t || j|j dq ||fS)zExtracts the labels and probabilities of the best lattice path, dropping arcs and restoring the targets shift, if needed. rrrdtypedevice) rangeshapelabelsr aux_labelsrtorlongappendrexpr1) rr+r,r-shortest_pathsprobsishortest_path_fsa non_eps_maskr4rrr!_extract_labels_and_probabilitiesNs"    "z,CtcK2Mixin._extract_labels_and_probabilities)NN)FT)__name__ __module__ __qualname____doc__rTensorrrrr#r*boolrr?rrrrr s6   rcseZdZdZdejdejdejdejdeejejejejff fdd Zdejd ejdd fd d Zdejdejdejfd dZ Z S) RnntK2Mixina_k2 Mixin class that simplifies the construction of various k2-based RNNT-like losses. Inherits CtcK2Mixin. It does the following: - Prepares and adapts the input tensors. - Creates Emissions graphs. - Extracts the labels and probabilities of the best lattice path (method _extract_labels_and_probabilities). rrrrrcst|dks J|\}}}}||} |jdkr|j|kr|jd} t||| |j|d} | ddddf| ddddf} t| dk} |jdkrjtjt | j | | ddj } | d| df} tj | ddd\}}| d d t|d| dk}| dt| d d | dd| | d| ddft|d|df}|}t|f|}|||8<tt||}t |}| d t|d| dk}||| |t|d|f}t|d|dft|d|dff|_|||_nd|_d|_t||d||d}|d}t|| }|}tj| ||jd }|}||||<||||<|d|||d|}t||||S) aBefore calling super()._prepare_log_probs_and_targets, this method reshapes the log_probs tensor from (B, T, U+1, D) to (B, T', D) where T' = T*(U+1), shifts paddings along T and U towards the end of T', and recomputes input_lengths. It also calculates indices on which steps should be applied to the log_probs tensor to emulate arcs shift of the Emissions graph for the pruned RNNT variant. rrTNrdim) return_countsr0)lenrpredictor_window_sizemaxr predictor_step_sizerrrepeat_interleavestackTunique unsqueezerepeatarangecumsumfullcat_RnntK2Mixin__step_indices_RnntK2Mixin__supervisions_addr flattenr$r r1clonereshapeviewsuperr)rrrrrBrQUDTUwindow_size_with_blank ranges_begin step_sizesraw_step_indicesrRcount shift_mask step_indices max_count max_count_vecpad_indices_row pad_uniquepad_shift_mask pad_indicesnon_pad_mask_truenon_pad_mask_fakerearranged_indicesrearranged_indices_buffer __class__rrrusn ( $"      z*RnntK2Mixin._prepare_log_probs_and_targetsrr cCs|jdus |jdur#tj|tj|d|ddf|jdfdd}ntj|d|dtt|jd|dftj d}d||j<tj|d|d|ddf|jd}tj t tj j g|ddg|jd||<tj|tj|d|ddf|jdfddd |d d||<|ddd f|j|dddfjtjd}t|dd |dddkstj|dd }||df|dddf<|||ddd f<n||ddd f<d|_d|_t||S) aOverrides super()._prepare_emissions_graphs. Creates DenseFsaVec, adding outputs to the end of the D dimension. If pruning is used, this method also pads the DenseFsaVec with frames according to the steps, calculated before. frame is a frame with log-probability zero and every other log-probability is -inf. Nrrr0rHr.Tr) descending)rYrZrrXzerosrr1intrKrEtensorfinfofloat32minr^r6r7allargsortr r")rrr log_probs_epsmaskrorderrrrr#s6 &. ," &0" z%RnntK2Mixin._prepare_emissions_graphscCs|S)zNot required for RNNT. rr)rrrr*sz&RnntK2Mixin._maybe_normalize_gradients) r@rArBrCrrDrrr#r* __classcell__rrrurrFlsJ$'rF)abcrtypingrrrr(nemo.collections.asr.parts.k2.grad_utilsr#nemo.collections.asr.parts.k2.utilsrrr r r r nemo.core.utils.k2_guardr rrFrrrrs    L