o iB@sddlZddlZddlmZddlmZmZmZmZm Z m Z ddl m Z ddl mZmZGdddeZGdd d ejjZdS) N)chain)AnyDictList NamedTupleTupleUnion) end_detect)PartialScorerInterfaceScorerInterfacec@sveZdZUdZejed<dZee ejfed<e Z e e ee ejffed<e Ze e efed<de fdd Zd S) HypothesiszHypothesis data type.yseqrscorescoresstatesreturncCs0|j|jt|jdd|jDdS)z#Convert data to JSON-friendly dict.cSsi|] \}}|t|qS)float).0kvrrU/home/ubuntu/.local/lib/python3.10/site-packages/funasr/models/e_paraformer/search.py z%Hypothesis.asdict..)r rr)_replacer tolistrrritems_asdictselfrrrasdictszHypothesis.asdictN)__name__ __module__ __qualname____doc__torchTensor__annotations__rrrdictrrstrrrr rrrrr s   r cseZdZdZ   d9deeefdeeefdededed ed e ed ed effd d Z de j de e fddZede j dede j fddZde de j deeee j feeefffddZde de j de j deeee j feeefffddZde j de j dee j e j ffddZedeeefd eee j fd!ed"eee j fd#edeee j ff d$d%Zd&ed'ed#edefd(d)Zd*e e de j d+e j de e fd,d-Z . .d:de j d/e j d0ed1ede e f d2d3Zd4ed5ed0ed*e e d6e e de e f d7d8ZZS);BeamSearchParazBeam search implementation.N?scorersweights beam_size vocab_sizesoseos token_listpre_beam_ratiopre_beam_score_keyc sRt||_t|_t|_t|_tj |_ | D]E\} } | | d} | dks0| dur1qt | tsBJ| dt| d| |j| <t | trR| |j| <n| |j| <t | tjjrc| |j | <q||_||_||_t|||_||_||_| dur| dkr| |jvrt| d|j| |_|jduo|j|jkot|jdk|_dS)aTInitialize beam search. Args: scorers (dict[str, ScorerInterface]): Dict of decoder modules e.g., Decoder, CTCPrefixScorer, LM The scorer will be ignored if it is `None` weights (dict[str, float]): Dict of weights for each scorers The scorer will be ignored if its weight is 0 beam_size (int): The number of hypotheses kept during search vocab_size (int): The number of vocabulary sos (int): Start of sequence id eos (int): End of sequence id token_list (list[str]): List of tokens for debug log pre_beam_score_key (str): key of scores to perform pre-beam search pre_beam_ratio (float): beam size in the pre-beam search will be `int(pre_beam_ratio * beam_size)` rNz (z$) does not implement ScorerInterfacefullz is not found in )super__init__r-r(r, full_scorers part_scorersr%nn ModuleDictnn_dictrget isinstancer typer Moduler0r1r2int pre_beam_sizer.n_vocabKeyErrorr4len do_pre_beam) rr,r-r.r/r0r1r2r3r4rrw __class__rrr7&sL            zBeamSearchPara.__init__xrcCsXt}t}|jD]\}}||||<d||<q td||tj|jg|jddgS)zGet an initial hypothesis data. Args: x (torch.Tensor): The encoder output feature Returns: Hypothesis: The initial hypothesis. device)rrrr ) r(r,r init_stater r%tensorr0rM)rrJ init_states init_scoresrdrrrinit_hypps  zBeamSearchPara.init_hypxscCs$tj|g|j|jd}t||fS)zAppend new token to prefix tokens. Args: xs (torch.Tensor): The prefix token x (int): The new token to append Returns: torch.Tensor: New tensor contains: xs + [x] with xs.dtype and xs.device dtyperM)r%rOrVrMcat)rTrJrrr append_tokens zBeamSearchPara.append_tokenhypcCsJt}t}|jD]\}}||j|j||\||<||<q ||fS)aScore new hypothesis by `self.full_scorers`. Args: hyp (Hypothesis): Hypothesis with prefix tokens to score x (torch.Tensor): Corresponding input feature Returns: Tuple[Dict[str, torch.Tensor], Dict[str, Any]]: Tuple of score dict of `hyp` that has string keys of `self.full_scorers` and tensor score values of shape: `(self.n_vocab,)`, and state dict that has string keys and state values of `self.full_scorers` )r(r8rrr r)rrYrJrrrrRrrr score_fulls $zBeamSearchPara.score_fullidscCsLt}t}|jD]\}}||j||j||\||<||<q ||fS)aaScore new hypothesis by `self.part_scorers`. Args: hyp (Hypothesis): Hypothesis with prefix tokens to score ids (torch.Tensor): 1D tensor of new partial tokens to score x (torch.Tensor): Corresponding input feature Returns: Tuple[Dict[str, torch.Tensor], Dict[str, Any]]: Tuple of score dict of `hyp` that has string keys of `self.part_scorers` and tensor score values of shape: `(len(ids),)`, and state dict that has string keys and state values of `self.part_scorers` )r(r9r score_partialr r)rrYr[rJrrrrRrrrr\s &zBeamSearchPara.score_partialweighted_scorescCsz|d|dkr||jd}||fS||}td |dd<|||<||jd}|||jd}||fS)aCompute topk full token ids and partial token ids. Args: weighted_scores (torch.Tensor): The weighted sum scores for each tokens. Its shape is `(self.n_vocab,)`. ids (torch.Tensor): The partial token ids to compute topk Returns: Tuple[torch.Tensor, torch.Tensor]: The topk full token ids and partial token ids. Their shapes are `(self.beam_size,)` rinfN)sizetopkr.r)rr]r[top_idstmp local_idsrrrbeamszBeamSearchPara.beam prev_scoresnext_full_scoresfull_idxnext_part_scorespart_idxcCsVt}|D]\}}||||||<q|D]\}}||||||<q|S)aMerge scores for new hypothesis. Args: prev_scores (Dict[str, float]): The previous hypothesis scores by `self.scorers` next_full_scores (Dict[str, torch.Tensor]): scores by `self.full_scorers` full_idx (int): The next token id for `next_full_scores` next_part_scores (Dict[str, torch.Tensor]): scores of partial tokens by `self.part_scorers` part_idx (int): The new token id for `next_part_scores` Returns: Dict[str, torch.Tensor]: The new score dict. Its keys are names of `self.full_scorers` and `self.part_scorers`. Its values are scalar tensors by the scorers. )r(r)rfrgrhrirj new_scoresrrrrr merge_scoress zBeamSearchPara.merge_scoresr part_statescCsLt}|D]\}}|||<q|jD]\}}||||||<q|S)aMerge states for new hypothesis. Args: states: states of `self.full_scorers` part_states: states of `self.part_scorers` part_idx (int): The new token id for `part_scores` Returns: Dict[str, torch.Tensor]: The new score dict. Its keys are names of `self.full_scorers` and `self.part_scorers`. Its values are states of the scorers. )r(rr9 select_state)rrrmrj new_statesrrrRrrr merge_statess  zBeamSearchPara.merge_states running_hypsam_scorecCshg}tj|j|jd}|D]}tj|j|j|jd}||7}|||\}} |jD] } ||j| || 7}q)|j rO|j dkrA|n||j } t | |j d}| |||\} } |jD]} |||j| | | 7<q[||j7}t|||D]#\}}|t||||j|||j||| ||| | |dqzt|dddd d tt||j}q |S) a"Search new tokens for running hypotheses and encoded speech x. Args: running_hyps (List[Hypothesis]): Running hypotheses on beam x (torch.Tensor): Encoded speech feature (T, D) Returns: List[Hypotheses]: Best sorted hypotheses rLrUr5r^)rr rrcS|jSNrrJrrrIz'BeamSearchPara.search..TkeyreverseN)r%arangerCrMzerosrVrZr8r-rFr4rarBr\r9rzipreappendr rXr rlrrpsortedminrEr.)rrqrJrr best_hypspart_idsrYr]rrrpre_beam_scores part_scoresrmjpart_jrrrsearchs>         zBeamSearchPara.searchrK am_scores maxlenratio minlenratioc s|jd}tdt|jdtdt||}g}t|D]L}tdt|||||} |||| |}|dkrZt dd|D|rZtd|nt |dkrgtd n td t |q%t |d d d d} t | dkrt d|dkrgS ||td|dS| d} | jD]\} } t| ddj| dd| j| dd| qtd| jdtd| jt | jdtdt | jdurtddfdd| jddDd | S)!aWPerform beam search. Args: x (torch.Tensor): Encoded speech feature (T, D) maxlenratio (float): Input length ratio to obtain max output length. If maxlenratio=0.0 (default), it uses a end-detect function to automatically find maximum hypothesis lengths If maxlenratio<0.0, its absolute value is interpreted as a constant max output length. minlenratio (float): Input length ratio to obtain min output length. Returns: list[Hypothesis]: N-best decoding results rzdecoder input length: zmax output length: z position rKcSsg|]}|qSr)r rhrrr rsz*BeamSearchPara.forward..zend detected at zno hypothesis. Finish decoding.zremained hypotheses: cSrsrtrurvrrrrw{rxz(BeamSearchPara.forward..TryzOthere is no N-best results, perform recognition again with smaller minlenratio.g?z6.2fz * 3z = z for ztotal log probability: z.2fznormalized log probability: z"total number of ended hypotheses: N best hypo: cg|] }j|qSrr2itemrrJrrrrrr^ )shapelogginginfor)rSrangedebugr post_processr rErwarningforwardmaxrrr-rr r2join) rrJrrrmaxlenrq ended_hypsibest nbest_hypsrrrrrrNsJ      8 (zBeamSearchPara.forwardrrrc s tdt|jdur'tddfdd|djddD||dkr;td fd d|D}g}|D]D}|jd jkr~tj j D]#\}} | |j |} |j|| 7<|j|jj|| d }qT||q?||q?|S) aPerform post-processing of beam search iterations. Args: i (int): The length of hypothesis tokens. maxlen (int): The maximum length of tokens in beam search. maxlenratio (int): The maximum length ratio in beam search. running_hyps (List[Hypothesis]): The running hypotheses in beam search. ended_hyps (List[Hypothesis]): The ended hypotheses in beam search. Returns: List[Hypothesis]: The new running hypotheses. z"the number of running hypotheses: Nrrcrrrrrrrrrz/BeamSearchPara.post_process..rr^z-adding in the last position in the loopcs$g|]}|j|jjdqS))r )rrXr r1rrrrrsrru)rrrEr2rr rr1rr8rr9 final_scorerrrrr-r) rrrrrqr remained_hypsrYrrRsrrrrs, $     zBeamSearchPara.post_process)Nr+N)rKrK)r!r"r#r$rr)r rrArr7r%r&r rS staticmethodrXrrrZr\rerlrprrr __classcell__rrrHrr*#s     J         : Fr*)r%r itertoolsrtypingrrrrrrfunasr.metrics.commonr 2funasr.models.transformer.scorers.scorer_interfacer r r r:r@r*rrrrs