o i8,@sdZddlZddlmZddlmZddlmZddlmZddlZddlm Z ddlm Z dd l m Z dd l mZGd d d e ZdS) z2Parallel beam search module for online simulation.N)Any)Dict)List)Tuple)BatchBeamSearch)BatchHypothesis) Hypothesis) end_detectc seZdZdZdddddddfdd Zd d Zd ed ejd e e e ejfe e e fffddZ   dd ejdededed eef ddZddZddZd ejded eefddZZS)BatchBeamSearchOnlineaoOnline beam search implementation. This simulates streaming decoding. It requires encoded features of entire utterance and extracts block by block from it as it shoud be done in streaming processing. This is based on Tsunoo et al, "STREAMING TRANSFORMER ASR WITH BLOCKWISE SYNCHRONOUS BEAM SEARCH" (https://arxiv.org/abs/2006.14941). (Fr) block_sizehop_size look_aheaddisable_repetition_detectionencoded_feat_length_limitdecoder_text_length_limitc sBtj|i|||_||_||_||_||_||_|dS)zInitialize beam search.N) super__init__r rrrrrreset) selfr rrrrrargskwargs __class__X/home/ubuntu/.local/lib/python3.10/site-packages/espnet/nets/batch_beam_search_online.pyrs  zBatchBeamSearchOnline.__init__cCs.d|_d|_g|_g|_d|_d|_d|_dS)zReset parameters.Nr) encbuffer running_hyps prev_hyps ended_hypsprocessed_block process_idx prev_outputrrrrr3s zBatchBeamSearchOnline.resethypxreturncCst}t}|jD]a\}}|jdkr[t|jdkr[t|jd|jkr[|jd|j |j}|j|dddf<dd|j j dD|j j d<| ||j ||\||<||<q | |j|j ||\||<||<q ||fS)aScore new hypothesis by `self.full_scorers`. Args: hyp (Hypothesis): Hypothesis with prefix tokens to score x (torch.Tensor): Corresponding input feature Returns: Tuple[Dict[str, torch.Tensor], Dict[str, Any]]: Tuple of score dict of `hyp` that has string keys of `self.full_scorers` and tensor score values of shape: `(self.n_vocab,)`, and state dict that has string keys and state values of `self.full_scorers` rNcSsg|]}dqSNr).0_rrr Zsz4BatchBeamSearchOnline.score_full..decoder) dict full_scorersitemsrlenyseqnarrowclonesosrstates batch_score)rr%r&scoresr6kd temp_yseqrrr score_full=s$   "$z BatchBeamSearchOnline.score_fullT maxlenratio minlenratiois_finalc Csz|jdur ||_n tj|j|gdd|_|j}|dkr!|jd}n tdt||d}d} |j|j|j |j }||jdkrM| dd|}d} n|rT|}d} nnUt d|j t d|jd|j|jdkr|jd|jkr| d|jd|j|j}|jdur|||_||| ||}t d |j |j d7_ | r|Sq0|dur|jdurgS|jS||_|S) aPerform beam search. Args: x (torch.Tensor): Encoded speech feature (T, D) maxlenratio (float): Input length ratio to obtain max output length. If maxlenratio=0.0 (default), it uses a end-detect function to automatically find maximum hypothesis lengths minlenratio (float): Input length ratio to obtain min output length. Returns: list[Hypothesis]: N-best decoding results Nr)axisr(TFzStart processing block: %dz* Feature length: {}, current position: {}zFinished processing block: %d)rtorchcatshapemaxintsizer rrr!r3loggingdebugformatr"rrinit_hypprocess_one_blockr#) rr&r>r?r@maxlenret cur_end_framehblock_is_finalrrrforwardbsZ      % zBatchBeamSearchOnline.forwardc Csd|||j|j|krtdt|j||j|}|j|dkr0||j||||j|_|j j d}g}|j t ||j df|jk}d} t|j dD]+} || rc||| } || qQ|js|| s||j | df|j | ddfvr||s|d} qQ| rtdnu|r|d krtd d |jD|jrtd |j||jSt|dkr|std nF|j|_||j||||j|_|r|D]} |j| qt|jdkrtd||jStdt|j|jd7_|j|ks |r||jS|jD]} || q||} |jdkr0t|jdkr0|j|_|jd8_g|_| S)zRecognize one block.z position r(rFNTzDetected repetition.r=cSsg|]}|qSr)asdict)r*lhrrrr,sz;BatchBeamSearchOnline.process_one_block..zend detected at z+Detected hyp(s) reaching EOS in this block.zno hypothesis. Finish decoding.zremained hypotheses: )extendrr"rHrIstrsearch post_processr r2rDrBarangelengtheosrange_selectappendrinfor assemble_hypsr1r) rrPr@rMr>bestn_batchlocal_ended_hyps is_local_eos prev_repeatir%retsrrrrLs~             D   z'BatchBeamSearchOnline.process_one_blockc st|dddd}t|dkrtdgS|d}|jD]\}}t|ddj|d d |j|dd |qtd |jd td|jt|j d tdt|j durtdd fdd|j ddDd|S)zAssemble the hypotheses.cSs|jSr))score)r&rrr sz5BatchBeamSearchOnline.assemble_hyps..T)keyreverserzOthere is no N-best results, perform recognition again with smaller minlenratio.z6.2fz * 3z = z for ztotal log probability: z.2fznormalized log probability: z"total number of ended hypotheses: Nz best hypo: csg|]}j|qSr) token_list)r*r&r$rrr,!sz7BatchBeamSearchOnline.assemble_hyps..r(rS ) sortedr1rHwarningr8r0r`weightsrir2rojoin)rr nbest_hypsrbr9vrr$rra s. .  z#BatchBeamSearchOnline.assemble_hypshypscCsL|jD]\}}t|dr||t|dr#||j||j|<qdS)aExtend probabilities and states with more encoded chunks. Args: x (torch.Tensor): The extended encoder output feature hyps (Hypothesis): Current list of hypothesis Returns: Hypothesis: The extended hypothesis extend_prob extend_stateN)scorersr0hasattrrxryr6)rr&rwr9r:rrrrV&s   zBatchBeamSearchOnline.extend)r=r=T)__name__ __module__ __qualname____doc__rrrrBTensorrrrWrr<floatboolrrrRrLrarV __classcell__rrrrr sD  ( QW$r )rrHtypingrrrrrBespnet.nets.batch_beam_searchrrespnet.nets.beam_searchrespnet.nets.e2e_asr_commonr r rrrrs