o }oio?@sRddlZddlZddlmZddlmZmZmZmZddl Z ddl m Z ddl m Z de jde jfdd Zd e jde jfd d Zde jd efddZde jde jdedede jf ddZde jdee jfddZdeddfddZdSdd ZdSd!d"Z  dTd ee jd#e jd$eeeeefeeeeffd%eede jf d&d'Zd(e jd)e jdd*fd+d,Zd-edd.efd/d0ZdUd2d3ZdVd1dd5ed6efd7d8Zd1dde jfd9d:Zd;e jdZ ?  @dWdAe jdBe jdCedDedEeedFe!de jfdGdHZ"dIe jdJe jdKe jdee je jffdLdMZ#dXdOddPe$ddfdQdRZ%dS)YN)UnpicklingError)ListOptionalTupleUnion)k2)logging input_lengthsreturncCsZttt|jdt|jd|fdjtjd}|tj |dddfddS)zwCreates a special supervisions tensor from input lengths. These supervisions are required for some k2 methods. rdtypeNT) descending) torchstacktensorrangeshapezeroscputoint32argsort)r supervisionsrW/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/parts/k2/utils.pycreate_supervision(s rindicescCsDtj|j|j|jd}tjd|jd|j|jd||jtjd<|S)zProduces a tensor of reverse permutation for a given indices. Based on https://github.com/k2-fsa/snowfall/blob/master/snowfall/common.py )devicer rr )rrrrr arangerlong)ransrrrinvert_permutation8s*r#seq_lencCsT|jd}tjd||jd}|d||}||jd}||k}|S)z=Converts input_lengths to a non-padding mask. The mask is 2D.rrr) rrr r unsqueezeexpandclonedetachr)r r$ batch_size seq_rangeseq_range_expandseq_length_expandmaskrrrmake_non_pad_maskBs r/ lengths_x lengths_y max_length_x max_length_ycCs4||ks Jt||dt||d@S)zLConverts two orthogonal input_lengths to a non-padding mask. The mask is 3D.r )sizer/r&)r0r1r2r3rrrmake_non_pad_mask_3dLs r6rtcCs^|}g}|D]}t|dkr|dqt|dkr$||dqdStj|tjdS)zWConverts k2.RaggedTensor to torch.Tensor if the RaggedTensor is shallow (has two axes).rr Nr )tolistlenappendrrr)r7rt_list result_listerrrragged_to_tensor_2axes_simpleVs   r> graph_pathk2.Fsac Cs4tj|rg}ztj|dd}tj|}|WSty}zj| |t |ddd }| }Wdn1s;wYtjj dftjj dftjj dffD]/\}}z|||d }|WWYd}~Stttfy}z | |WYd}~qQd}~wwWYd}~t|d}~wwtd |d dS) zLFsa graph loading helper function. Loads graphs stored in different formats.r) map_locationr7zutf-8)encodingNFT)acceptorzNo such file: '')ospathexistsrloadrFsa from_dictrr:openread from_openfstfrom_str TypeError ValueError RuntimeError Exceptionrwarning) r?errors graph_dictgraphr=f graph_txtfuncrCrrr load_graphds2    *   rZ base_graph aux_graphcCs\t|dsJt|drJtt||j}tjt||dd}t|d|j|S)zIntersection helper function. aux_labelsF)treat_epsilons_speciallyphones) hasattrrarc_sortadd_epsilon_self_loopsrr intersectsetattrlabels)r[r\aux_graph_with_self_loopsresultrrrintersect_with_self_loops}s rhcCs*tt||j}tj||dddS)zComposition helper function.Fr_)r^ inner_labels)rrarbrrcompose)r[r\rfrrrcompose_with_self_loopssrkvaluesr5 min_col_indexcsZ|dust|t|ksJt|dkr"tj|d|d|||dSt|dkr|dj|djkrA|djkrAdksDJJ|d|dkrb|dkrb|kseJJ|durt|tspJ|d|kfdd |D}|}|durtjt ||||j |j d Stjt |||j |j d St d t|) zGWraps up k2.create_sparse to create 2- or 3-dimensional sparse tensors.Nr4rr )rowscolsrlr5rmrcsg|]}|qSrr).0i kept_indicesrr sz)create_sparse_wrapped..)r5r requires_grad)rrvzlen(indices) = ) r9r create_sparsendimnumel isinstanceintrsparse_coo_tensorrrrvrP)rrlr5rmrrsrcreate_sparse_wrappeds@  8B r} log_softmaxrzk2.DenseFsaVeccCstj|tj|jd|jddftd |jdgdd}tj|jd|jdd|jdf|jd}||dddddf<|}|dddfd9<t ||}|S) zVPerforms special epsilon-padding required for composition with some of the topologies.rr infr%r)axisr4N) rcatfullrfloatrrr(r DenseFsaVec)r~rlog_softmax_epslog_softmax_paddedsupervisions_paddeddense_log_softmax_paddedrrrprep_padded_densefsavecs,   rlatticesshiftcCsT|D]#}|jdk}|j||7<t|dr%|jdk}|j||7<qt|S)zyShifts lattice labels and aux_labels by a given number. This is an in-place operation, if the lattice is on GPU. rr])rer`r]reset_properties_fsa)rrlatticer.rrrshift_labels_inpls   rrVcCs(d|jd<|jtdkrt|}|S)zResets properties of a graph. In-place (does not create a new graph) if the graph is on GPU. Use this every time you alter a graph in-place. See https://github.com/k2-fsa/k2/issues/978 for more information.N _propertiesr)__dict__rrrra)rVrrrrs  rautolabelmodecCsz|dvsJd|dkst|dsJdtj|dd\}}|dkr)||j|dk<|d kr9t|dr9||j|dk<t|S) zAdds self-loops with given label to a graph. Supported modes are ``input``, ``output``, and ``auto``, Where ``input`` leaves aux_labels zeroes, if present, ``output`` leaves labels zeroes)inputoutputrz?Supported modes are ``input``, ``output``, and ``auto``: {mode}rr]z.Graph must have aux_labels for mode ``output``T ret_arc_maprr)r`rrbrer]r)rVrr new_grapharc_maprrradd_self_loopssrcCsdt|jdkr td|jdddf}tdt|tjdt|g|R}t |S)z:Returns 1d torch.Tensor with arc weights of a given graph.r4z&FsaVec is not supported at the moment.Nrz%sfz%si) r9rNotImplementedErrorarcsrlr8structunpackpackrTensor)rV weights_int weights_floatrrrget_arc_weightss * r tot_scores reductioncCsVt|t|td @}|dkr||}||fS|dkr'||}||fS)aFigures out the total score(log-prob) over all successful supervision segments (i.e. those for which the total score wasn't -infinity). Args: tot_scores: a Torch tensor of shape (num_segments,) containing total scores from forward-backward reduction: a reduction type ('mean', 'sum' or 'none') Returns: Returns a tuple of 2 scalar tensors: (tot_score, finite_mask) where finite_mask is a tensor containing successful segment mask. Based on get_tot_objf_and_num_frames from https://github.com/k2-fsa/snowfall/blob/master/snowfall/objectives/common.py rmeansum)risnannerrr)rr finite_maskrrrget_tot_objf_and_finite_masks  rr Fencoded_lengthstarget_lengthswindow_size_with_blankstep max_seq_len begin_onlyc CsB|dksJ|dks J||ksJt|t|ksJtjt||dur)|nt||ftjd}||kjdddD]N}||}tt||||d|} |||d| d<tjj j | dddj tj d|d d j tjd||d|f<|||df|||df<q?|r|S|ddd|t|S) aICreates the pruning ranges for the Encoder and Predictor of RNNT. The ranges are similar to https://k2-fsa.github.io/k2/python_api/api.html#k2.get_rnnt_prune_ranges but they are constructed under the assumption of the uniform distribution token activations across time frames and without any posterior knowledge. r Nr T)as_tuplerr4rz nearest-exact)r)r9rrmaxr!nonzeror r{nn functional interpolatereshaperrr&repeat) rrrrrr ranges_beginrr encoded_lenranges_begin_rawrrrget_uniform_rnnt_prune_rangess2  "rencoder_outputsdecoder_outputsrangesc Cs|\}}}|d}|\}}} ||dksJ||dks%J||dks.J|d||||f} tj|d|||| fd||||df|||| fd} | | fS)ztPrepares pruned encoder and decoder outputs according to the prune ranges. Based on k2.do_rnnt_pruning(...) rrr r4)dimindex)r5r&r'rgatherr) rrrBTrD1_UD2encoder_outputs_pruneddecoder_outputs_prunedrrrapply_rnnt_prune_rangesDs rx&1fsa ins_del_scorecsd}tdtd|d}|j}|jdk}tjt |dt d}| }d||dd <|}d |d d d<||}t t | d|} || t||dd d df<t |jd kr|jd| jtjd\} } | d| d } tfd dt| d d| d d D} nt} | j | _| j }d||<|| _d | jd<tj| d d\} }| j }|||dk<|| _d | jd<t| } | S)aConstruct the levenshtein graph from a k2-type WFST or a lattice. See also levenshtein_graph from k2. Args: fst: K2-type source WFST or lattice. ins_del_score: Insertion and deletion penalty. Should be more than 0.5 for substitutions to be preferred over insertions/deletions, or less otherwise. Returns: K2-type levenshtein WFST. gz@iz@frrr4r Fr TNrpcs"g|] \}}t||qSr)rrI)rqrrjnew_arcsrrruzs"z(levenshtein_graph_k2..rr)rrrrrlrerrrr9boolr(r repeat_interleavewhererrrr row_splitsrcreate_fsa_vecziprIr]rrbscoresra)rr sub_score sub_score_intr final_indices template_maskno_duplicate_final_masknew_maskduplicate_indices new_shaper new_splitslevenshtein_fsarerrrrrlevenshtein_graph_k2Ys<  0        r)r[r@r\r@r r@)NN)rVr@)rr)r NF)r)&rErpicklertypingrrrrrnemo.core.utils.k2_guardr nemo.utilsrrrr#r{r/r6 RaggedTensorr>strrZrhrkr}rrrrrrrrrrrrrrrs            -  $  '