o iB@sddlmZddlZddlmZddlmZddlmZddlmZddlmZddlm Z ddl Z dd l m Z dd l mZdd l mZGd d d eZGddde jjZdS))chainN)Any)Dict)List) NamedTuple)Tuple)Union) end_detect)PartialScorerInterface)ScorerInterfacec@sveZdZUdZejed<dZee ejfed<e Z e e ee ejffed<e Ze e efed<de fdd Zd S) HypothesiszHypothesis data type.yseqrscorescoresstatesreturncCs0|j|jt|jdd|jDdS)z#Convert data to JSON-friendly dict.cSsi|] \}}|t|qS)float).0kvrrT/home/ubuntu/.local/lib/python3.10/site-packages/funasr/models/transformer/search.py sz%Hypothesis.asdict..)r rr)_replacer tolistrrritems_asdictselfrrrasdictszHypothesis.asdictN)__name__ __module__ __qualname____doc__torchTensor__annotations__rrrdictrrstrrrrrrrrr s   r cseZdZdZ   d7deeefdeeefdededed ed e ed ed effd d Z de j de e fddZede j dede j fddZde de j deeee j feeefffddZde de j de j deeee j feeefffddZde j de j dee j e j ffddZedeeefd eee j fd!ed"eee j fd#edeee j ff d$d%Zd&ed'ed#edefd(d)Zd*e e de j de e fd+d,Z -d8de j d.ed/ede e fd0d1Zd2ed3ed.ed*e e d4e e de e f d5d6ZZS)9 BeamSearchzBeam search implementation.N?scorersweights beam_size vocab_sizesoseos token_listpre_beam_ratiopre_beam_score_keyc sRt||_t|_t|_t|_tj |_ | D]E\} } | | d} | dks0| dur1qt | tsBJ| dt| d| |j| <t | trR| |j| <n| |j| <t | tjjrc| |j | <q||_||_||_t|||_||_||_| dur| dkr| |jvrt| d|j| |_|jduo|j|jkot|jdk|_dS)aTInitialize beam search. Args: scorers (dict[str, ScorerInterface]): Dict of decoder modules e.g., Decoder, CTCPrefixScorer, LM The scorer will be ignored if it is `None` weights (dict[str, float]): Dict of weights for each scorers The scorer will be ignored if its weight is 0 beam_size (int): The number of hypotheses kept during search vocab_size (int): The number of vocabulary sos (int): Start of sequence id eos (int): End of sequence id token_list (list[str]): List of tokens for debug log pre_beam_score_key (str): key of scores to perform pre-beam search pre_beam_ratio (float): beam size in the pre-beam search will be `int(pre_beam_ratio * beam_size)` rNz (z$) does not implement ScorerInterfacefullz is not found in )super__init__r,r'r+ full_scorers part_scorersr$nn ModuleDictnn_dictrget isinstancer typer Moduler/r0r1int pre_beam_sizer-n_vocabKeyErrorr3len do_pre_beam) rr+r,r-r.r/r0r1r2r3rrw __class__rrr6%sL            zBeamSearch.__init__xrcCsXt}t}|jD]\}}||||<d||<q td||tj|jg|jddgS)zGet an initial hypothesis data. Args: x (torch.Tensor): The encoder output feature Returns: Hypothesis: The initial hypothesis. device)rrrr ) r'r+r init_stater r$tensorr/rL)rrI init_states init_scoresrdrrrinit_hypos  zBeamSearch.init_hypxscCs$tj|g|j|jd}t||fS)zAppend new token to prefix tokens. Args: xs (torch.Tensor): The prefix token x (int): The new token to append Returns: torch.Tensor: New tensor contains: xs + [x] with xs.dtype and xs.device dtyperL)r$rNrUrLcat)rSrIrrr append_tokens zBeamSearch.append_tokenhypcCsJt}t}|jD]\}}||j|j||\||<||<q ||fS)aScore new hypothesis by `self.full_scorers`. Args: hyp (Hypothesis): Hypothesis with prefix tokens to score x (torch.Tensor): Corresponding input feature Returns: Tuple[Dict[str, torch.Tensor], Dict[str, Any]]: Tuple of score dict of `hyp` that has string keys of `self.full_scorers` and tensor score values of shape: `(self.n_vocab,)`, and state dict that has string keys and state values of `self.full_scorers` )r'r7rrr r)rrXrIrrrrQrrr score_fulls $zBeamSearch.score_fullidscCsLt}t}|jD]\}}||j||j||\||<||<q ||fS)aaScore new hypothesis by `self.part_scorers`. Args: hyp (Hypothesis): Hypothesis with prefix tokens to score ids (torch.Tensor): 1D tensor of new partial tokens to score x (torch.Tensor): Corresponding input feature Returns: Tuple[Dict[str, torch.Tensor], Dict[str, Any]]: Tuple of score dict of `hyp` that has string keys of `self.part_scorers` and tensor score values of shape: `(len(ids),)`, and state dict that has string keys and state values of `self.part_scorers` )r'r8r score_partialr r)rrXrZrIrrrrQrrrr[s &zBeamSearch.score_partialweighted_scorescCsz|d|dkr||jd}||fS||}td |dd<|||<||jd}|||jd}||fS)aCompute topk full token ids and partial token ids. Args: weighted_scores (torch.Tensor): The weighted sum scores for each tokens. Its shape is `(self.n_vocab,)`. ids (torch.Tensor): The partial token ids to compute topk Returns: Tuple[torch.Tensor, torch.Tensor]: The topk full token ids and partial token ids. Their shapes are `(self.beam_size,)` rinfN)sizetopkr-r)rr\rZtop_idstmp local_idsrrrbeamszBeamSearch.beam prev_scoresnext_full_scoresfull_idxnext_part_scorespart_idxcCsVt}|D]\}}||||||<q|D]\}}||||||<q|S)aMerge scores for new hypothesis. Args: prev_scores (Dict[str, float]): The previous hypothesis scores by `self.scorers` next_full_scores (Dict[str, torch.Tensor]): scores by `self.full_scorers` full_idx (int): The next token id for `next_full_scores` next_part_scores (Dict[str, torch.Tensor]): scores of partial tokens by `self.part_scorers` part_idx (int): The new token id for `next_part_scores` Returns: Dict[str, torch.Tensor]: The new score dict. Its keys are names of `self.full_scorers` and `self.part_scorers`. Its values are scalar tensors by the scorers. )r'r)rerfrgrhri new_scoresrrrrr merge_scoress zBeamSearch.merge_scoresr part_statescCsLt}|D]\}}|||<q|jD]\}}||||||<q|S)aMerge states for new hypothesis. Args: states: states of `self.full_scorers` part_states: states of `self.part_scorers` part_idx (int): The new token id for `part_scores` Returns: Dict[str, torch.Tensor]: The new score dict. Its keys are names of `self.full_scorers` and `self.part_scorers`. Its values are states of the scorers. )r'rr8 select_state)rrrlri new_statesrrrQrrr merge_statess  zBeamSearch.merge_states running_hypscCs`g}tj|j|jd}|D]}tj|j|j|jd}|||\}}|jD] } ||j| || 7}q%|j rK|j dkr=|n||j } t | |j d}| |||\} } |jD]} |||j| | | 7<qW||j7}t|||D]#\} }|t|| ||j| ||j|| | |||| |dqvt|dddd d tt||j}q |S) a"Search new tokens for running hypotheses and encoded speech x. Args: running_hyps (List[Hypothesis]): Running hypotheses on beam x (torch.Tensor): Encoded speech feature (T, D) Returns: List[Hypotheses]: Best sorted hypotheses rKrTr4r])rr rrcS|jSNrrIrrrEz#BeamSearch.search..TkeyreverseN)r$arangerBrLzerosrUrYr7r,rEr3r`rAr[r8rziprdappendr rWr rkrrosortedminrDr-)rrprI best_hypspart_idsrXr\rrrpre_beam_scores part_scoresrljpart_jrrrsearchs<         zBeamSearch.searchrJ maxlenratio minlenratioc s^|dkr |jd}n|dkrdt|}n tdt||d}t||d}tdt|jdtdt|tdt||}g}t|D]I}t dt| ||} |||| |}|dkrt d d |D|rtd |nt |dkrtd n t d t |qSt|dddd} t | dkrtd|dkrgS||td|dS| d} | jD]\} } t| ddj| dd| j| dd| qtd| jdtd| jt | jdtdt | jdur-tddfd d | jddDd!| S)"aWPerform beam search. Args: x (torch.Tensor): Encoded speech feature (T, D) maxlenratio (float): Input length ratio to obtain max output length. If maxlenratio=0.0 (default), it uses a end-detect function to automatically find maximum hypothesis lengths If maxlenratio<0.0, its absolute value is interpreted as a constant max output length. minlenratio (float): Input length ratio to obtain min output length. Returns: list[Hypothesis]: N-best decoding results rr]zdecoder input length: zmax output length: zmin output length: z position rJcSsg|]}|qSr)rrhrrr qsz&BeamSearch.forward..zend detected at zno hypothesis. Finish decoding.zremained hypotheses: cSrqrrrsrtrrrruzrvz$BeamSearch.forward..TrwzOthere is no N-best results, perform recognition again with smaller minlenratio.g?z6.2fz * 3z = z for ztotal log probability: z.2fznormalized log probability: z"total number of ended hypotheses: N best hypo: cg|]}j|qSrr1rrIrrrr )shaper@maxr_logginginfor(rRrangedebugr post_processr rDr~warningforwardrrr,rr r1join) rrIrrmaxlenminlenrp ended_hypsibest nbest_hypsrrrrrrJsV       8 (zBeamSearch.forwardrrrc s tdt|jdur'tddfdd|djddD||dkr;td fd d|D}g}|D]D}|jd jkr~tj j D]#\}} | |j |} |j|| 7<|j|jj|| d }qT||q?||q?|S) aPerform post-processing of beam search iterations. Args: i (int): The length of hypothesis tokens. maxlen (int): The maximum length of tokens in beam search. maxlenratio (int): The maximum length ratio in beam search. running_hyps (List[Hypothesis]): The running hypotheses in beam search. ended_hyps (List[Hypothesis]): The ended hypotheses in beam search. Returns: List[Hypothesis]: The new running hypotheses. z"the number of running hypotheses: Nrrcrrrrrrrrrz+BeamSearch.post_process..rr]z-adding in the last position in the loopcs$g|]}|j|jjdqS))r )rrWr r0rrrrrsrrs)rrrDr1rr rr0rr7rr8 final_scorerrrrr,r}) rrrrrpr remained_hypsrXrrQsrrrrs( (     zBeamSearch.post_process)Nr*N)rJrJ)r r!r"r#rr(r rr@rr6r$r%r rR staticmethodrWrrrYr[rdrkrorrr __classcell__rrrGrr)"s     J         4 Ir)) itertoolsrrtypingrrrrrrr$funasr.metrics.commonr 2funasr.models.transformer.scorers.scorer_interfacer r r r9r?r)rrrrs