o Gi> @sddlZddlmZmZddlZddlZddlZddlmZ ddejde deeee ej ffd d Z Gd d d e Zd ededefddZd ede defddZdS)N)ListUnion)FsaF best_paths return_raggedreturncCst|jtjr(|jd}|j|j}|d}|d}t||j }n|jd}t||j}|d}|j dksCJ|rG|S| S)aExtract the texts (as word IDs) from the best-path FSAs. Note: Used by Nbest.build_levenshtein_graphs during MWER computation. Copied from icefall. Args: best_paths: A k2.Fsa with best_paths.arcs.num_axes() == 3, i.e. containing multiple FSAs, which is expected to be the result of k2.shortest_path (otherwise the returned values won't be meaningful). return_ragged: True to return a ragged tensor with two axes [utt][word_id]. False to return a list-of-list word IDs. Returns: Returns a list of lists of int, containing the label sequences we decoded. rr) isinstance aux_labelsk2 RaggedTensorremove_values_leqarcsshapecompose remove_axisvaluesnum_axestolist)rrr aux_shaper          zNbest.from_latticelatscCs|jj|jksJ|jjd|jt|jdks J|j|j|jks9J|jd|jt|}t|j}|j d}tj |||dd}tj |dd}t |}t ||jdS)aPIntersect this Nbest object with a lattice and get 1-best path from the resulting FsaVec. Caution: We assume FSAs in `self.fsa` don't have epsilon self-loops. We also assume `self.fsa.labels` and `lats.labels` are token IDs. Args: lats: An FsaVec. It can be the return value of :func:`whole_lattice_rescoring`. Returns: Return a new Nbest. This new Nbest shares the same shape with `self`, while its `fsa` is the 1-best path from intersecting `self.fsa` and `lats`. rrrT)a_fsasb_fsas b_to_a_mapsorted_match_a)r*rr)rrCrrrr$r arc_sortadd_epsilon_self_loopsrow_idsintersect_device shortest_pathremove_epsilonr)r!rKfsas_with_epsilon_loopspath_to_seq_mapans_latsone_bestrrr intersects$    zNbest.intersectcCs"|jjddd}t|j|S)aGet total scores of the FSAs in this Nbest. Note: Since FSAs in Nbest are just linear FSAs, log-semirng and tropical semiring produce the same total scores. Returns: Return a ragged tensor with two axes [utt][path_scores]. TF)r* log_semiring)rget_tot_scoresr r rfloat)r!r4rrr total_scoress zNbest.total_scoreskc Cs|}|jddd}t|j|}|jddd}t|ds,Jd|j d|d d d |f }t |j |}tjj|jj|d }t||S) a>Get a subset of paths in the Nbest. The resulting Nbest is regular in that each sequence (i.e., utterance) has the same number of paths (k). We select the top-k paths according to the total_scores of each path. If a utterance has less than k paths, then its last path, after sorting by tot_scores in descending order, is repeated so that each utterance has exactly k paths. Args: k: Number of paths in each utterance. Returns: Return a new Nbest with a regular shape. T) descendingr/ replicater2)mode padding_valuerz&Some utterances contain empty n-best: rN)r$dim1)r_sort_r r rpadr7geall row_splitsflattenr> index_fsarr9regular_ragged_shaper$r) r!r` ragged_scoresr3ragged_indexespadded_indexes top_k_indexes top_k_fsas top_k_shaperrrtop_ks"   z Nbest.top_kcCst|jdd}t|S)z-Return an FsaVec with axes [utt][state][arc].T)r)rrr levenshtein_graph)r!word_idsrrrbuild_levenshtein_graphs&s zNbest.build_levenshtein_graphs)N)Tr')__name__ __module__ __qualname____doc__r r RaggedShaper r"r& staticmethodintboolr^rJr[r_rtrwrrrrrAs>  c((rrKG_with_epsilon_loopsc Cs0t|jdks J|jt|dsJ|jdksJ|j|j}|j|j|_|`t|ddus4Jt|}|jd}tj ||tj d} z tj |||dd}Wn5t y}z(t d |d t d |jt|d d}t d |jWYd}~nd}~wwqHtt|}t|}|S)a Rescore the 1st pass lattice with an LM. In general, the G in HLG used to obtain `lats` is a 3-gram LM. This function replaces the 3-gram LM in `lats` with a 4-gram LM. Args: lats: The decoding lattice from the 1st pass. We assume it is the result of intersecting HLG with the network output. G_with_epsilon_loops: An LM. It is usually a 4-gram LM with epsilon self-loops. It should be arc sorted. Returns: Return a new lattice rescored with a given G. r lm_scores)rNNFr)rCdtypeT)rOzCaught exception:  znum_arcs before: gh㈵>znum_arcs after: N)rrhasattrrCr4rr invertr7zerosint32rT RuntimeErrorlogginginfonum_arcsprune_on_arc_posttop_sortconnect) rKrrC inverted_latsnum_seqsrNrescoring_latseinv_rescoring_latsrrrwhole_lattice_rescoring,s>    rr)cCst|jdks Jt|dsJt|drJ|j|_tj||dd}tj|j|}| d}|j ddd\}}}|j d}| d}t |}t||d S) aGenerate an n-best list from a lattice. Args: lats: The decoding lattice from the first pass after LM rescoring. lats is an FsaVec. It can be the return value of :func:`whole_lattice_rescoring` num_paths: Size of n for n-best list. CAUTION: After removing paths that represent the same token sequences, the number of paths in different sequences may not be equal. Return: Return an Nbest object. Note the returned FSAs don't have epsilon self-loops. rphonestokensTr,rFr-rP)rrrrrr r6r9r:rr;r<rrAr)rKr)paths token_seqsunique_token_seqsrGseq_to_path_shape token_fsasrrrgenerate_nbest_listls     r)F)rtypingrrr7_k2r rrrr~r robjectrrrrrrrs$  .l@