o i@sTdZddlmZmZddlZddlmZddlmZddl m Z Gddde Z dS)zESequential implementation of Recurrent Neural Network Language Model.)TupleUnionN)check_argument_types)AbsLMcseZdZdZ       d$d ed ed ed ed edededeffdd ZddZ de j de j de e j e j ffddZ de j dee j e e j e j ffde j de e j ee j e e j e j ffffddZde j d e j d!e j de e j e j ffd"d#ZZS)%SequentialRNNLMzSequential RNNLM. See also: https://github.com/pytorch/examples/blob/4581968193699de14b56527296262dd76ab43557/word_language_model/model.py NFlstmr vocab_sizeunitnhidnlayers dropout_rate tie_weightsrnn_type ignore_idc stsJt|} |dur|}|}t||_tj|| |d|_|dvr9t t|} | | |||dd|_ n!z ddd|} Wn t yMt dwtj | ||| |dd |_ t|||_|rq|| krkt d |jj|j_||_||_||_dS) N) padding_idx)LSTMGRUT)dropout batch_firsttanhrelu)RNN_TANHRNN_RELUzwAn invalid option for `--model` was supplied, options are ['LSTM', 'GRU', 'RNN_TANH' or 'RNN_RELU']) nonlinearityrrz6When using the tied flag, nhid must be equal to emsize)rsuper__init__uppernnDropoutdrop EmbeddingencodergetattrrnnKeyError ValueErrorRNNLineardecoderweightrr r) selfr r r rrrrrninp rnn_classr __class__I/home/ubuntu/.local/lib/python3.10/site-packages/espnet2/lm/seq_rnn_lm.pyrsL         zSequentialRNNLM.__init__cCsht|jtjjr&tj|j|jftjd}tj|j|jftjd}||f}|Stj|j|jftjd}|S)z,Initialize LM state filled with zero values.)dtype) isinstancer&torchr rzerosrr float)r-hcstater2r2r3 zero_stateQszSequentialRNNLM.zero_stateinputhiddenreturncCsz|||}|||\}}||}|||d|d|d}||d|d|d|fS)Nrr)r"r$r&r+ contiguousviewsize)r-r=r>emboutputdecodedr2r2r3forward\s $zSequentialRNNLM.forwardyr;xcCs4||ddd|\}}|jddd}||fS)aHScore new token. Args: y: 1D torch.int64 prefix tokens. state: Scorer state for prefix tokens x: 2D encoder feature that generates ys. Returns: Tuple of torch.float32 scores for next token (n_vocab) and next state for ys r@dim)rB log_softmax)r-rHr;rI new_statelogpr2r2r3scorejszSequentialRNNLM.scoreysstatesxscsddur dn,t|jtjjr.tjddDddtjddDddfntjdd||ddddf\}|ddksPJ|j|d}|j dd}t|jtjjrz\fd dt dD|fSfd dt dD|fS) aScore new token batch. Args: ys (torch.Tensor): torch.int64 prefix tokens (n_batch, ylen). states (List[Any]): Scorer states for prefix tokens. xs (torch.Tensor): The encoder feature that generates ys (n_batch, xlen, n_feat). Returns: tuple[torch.Tensor, List[Any]]: Tuple of batchfied scores for next token with shape of `(n_batch, n_vocab)` and next state list for ys. rNcSsg|]\}}|qSr2r2.0r9r:r2r2r3 z/SequentialRNNLM.batch_score..r@rKcSsg|]\}}|qSr2r2rTr2r2r3rVrWrJcs,g|]}dd|fdd|ffqSNr2rUi)r:r9r2r3rVs,csg|] }dd|fqSrXr2rY)rRr2r3rVs) r5r&r6r rstackrCshapesqueezerMrange)r-rQrRrSrOr2)r:r9rRr3 batch_scores"    zSequentialRNNLM.batch_score)rNrr Fr r)__name__ __module__ __qualname____doc__intr8boolstrrr<r6TensorrrGrrPr_ __classcell__r2r2r0r3r sh  >  " r) rctypingrrr6torch.nnr typeguardrespnet2.lm.abs_modelrrr2r2r2r3s