o i @shdZddlmZddlZddlZddlmZmZGdddeZGdddeeZ Gd d d eeZ dS) zNgram lm implement.)ABCN)BatchScorerInterfacePartialScorerInterfacec@s(eZdZdZddZddZddZdS) Ngrambasez/Ngram base implemented through ScorerInterface.cCs6dd|D|_t|j|_t||_t|_dS)zInitialize Ngrambase. Args: ngram_model: ngram model path token_list: token list from dict or model.json cSsg|] }|dkr |ndqS)zz).0xrrM/home/ubuntu/.local/lib/python3.10/site-packages/espnet/nets/scorers/ngram.py sz&Ngrambase.__init__..N)chardictlencharlenkenlm LanguageModellmState tmpkenlmstate)self ngram_model token_listrrr __init__s  zNgrambase.__init__cCst}|j||S)zInitialize tmp state.)rrrNullContextWrite)rrstaterrr init_states zNgrambase.init_statec Cst}|jddkr|j|dnd}|j|||tj||j|j d}t |D]\}} |j||j| |j ||<q*||fS)Score interface for both full and partial scorer. Args: y: previous char next_token: next token need to be score state: previous state x: encoded feature Returns: tuple[torch.Tensor, List[Any]]: Tuple of batchfied scores for next token with shape of `(n_batch, n_vocab)` and next state list for ys. rz)dtypedevice) rrshaper r BaseScoretorch empty_likerr enumerater) ry next_tokenrr out_stateysscoresijrrr score_partial_!s  zNgrambase.score_partial_N)__name__ __module__ __qualname____doc__rrr+rrrr r s  rc@seZdZdZddZdS)NgramFullScorerzFullscorer for ngram.cCs||tt|j||S)anScore interface for both full and partial scorer. Args: y: previous char state: previous state x: encoded feature Returns: tuple[torch.Tensor, List[Any]]: Tuple of batchfied scores for next token with shape of `(n_batch, n_vocab)` and next state list for ys. )r+r!tensorranger )rr$rrrrr score>szNgramFullScorer.scoreN)r,r-r.r/r3rrrr r0;s r0c@s eZdZdZddZddZdS)NgramPartScorerzPartialscorer for ngram.cCs|||||S)r)r+)rr$r%rrrrr score_partialRszNgramPartScorer.score_partialcCs|S)z(Empty select state for scorer interface.r)rrr)rrr select_statecszNgramPartScorer.select_stateN)r,r-r.r/r5r6rrrr r4Os r4) r/abcrrr!espnet.nets.scorer_interfacerrrr0r4rrrr s 0