o i8@sdZddlZddlmZmZmZddlZddlmZddl mm Z ddl m Z ddlmZddlmZddlmZGdd d ee ejZGd d d ejZGd d d ejZdS)z@Default Recurrent Neural Network Languge Model in `lm_train.py`.N)AnyListTuple) LMInterface) to_device)BatchScorerInterface) strtoboolc @s|eZdZdZeddZddZddZdd Zd d Z d d Z ddZ de j deede j dee j eeffddZdS) DefaultRNNLMaDefault RNNLM for `LMInterface` Implementation. Note: PyTorch seems to have memory leak when one GPU compute this after data parallel. If parallel GPUs compute this, it seems to be fine. See also https://github.com/espnet/espnet/issues/1075 cCs|jdtddddgdd|jddtd d d |jd d tddd |jddtdd|jdtddd |jdtddd |jdtddd |S)z.Add arguments to command line argument parser.z--typelstm?gruzWhich type of RNN to use)typedefaultnargschoiceshelpz--layerz-lzNumber of hidden layers)r rrz--unitz-uizNumber of hidden unitsz --embed-unitNznNumber of hidden units in embedding layer, if it is not specified, it keeps the same number with hidden units.)rr rz--dropout-rate?zdropout probabilityz--emb-dropout-ratezemb dropout probabilityz --tie-weightsFzTie input and output embeddings) add_argumentstrintfloatr)parserrZ/home/ubuntu/.local/lib/python3.10/site-packages/espnet/nets/pytorch_backend/lm/default.py add_argumentssH  zDefaultRNNLM.add_argumentsc Csbtj|t|dd}t|dd}t|dd}t|dd}tt||j|j||j||||_ dS)zInitialize class. Args: n_vocab (int): The size of the vocabulary args (argparse.Namespace): configurations. see py:method:`add_arguments` dropout_rater embed_unitNemb_dropout_rate tie_weightsF) nnModule__init__getattrClassifierWithStateRNNLMlayerunitr model)selfn_vocabargsrrrr rrrr#Cs"      zDefaultRNNLM.__init__cCs |jS)zDump state dict.)r) state_dict)r*rrrr-bs zDefaultRNNLM.state_dictcCs|j|dS)zLoad state dict.N)r)load_state_dict)r*drrrr.fszDefaultRNNLM.load_state_dictc Csd}d}td}d}|j\}}t|D]>} |||dd| f|dd| f\}} tj|dd| fdk| jd} || | 7}|t| | 7}|t | 7}q|||| |j fS)a)Compute LM loss value from buffer sequences. Args: x (torch.Tensor): Input ids. (batch, len) t (torch.Tensor): Target ids. (batch, len) Returns: tuple[torch.Tensor, torch.Tensor, torch.Tensor]: Tuple of loss to backward (scalar), negative log-likelihood of t: -log p(t) (scalar) and the number of elements in x (scalar) Notes: The last two return values are used in perplexity: p(t)^{-n} = exp(-log p(t) / n) rN)dtype) torchtensorlongshaperanger)sumr0meanrtodevice) r*xtlosslogpcountstate batch_sizesequence_lengthi loss_batch non_zerosrrrforwardjs  * zDefaultRNNLM.forwardcCs*|j||dd\}}|d|fS)aScore new token. Args: y (torch.Tensor): 1D torch.int64 prefix tokens. state: Scorer state for prefix tokens x (torch.Tensor): 2D encoder feature that generates ys. Returns: tuple[torch.Tensor, Any]: Tuple of torch.float32 scores for next token (n_vocab) and next state for ys r)r)predict unsqueezesqueeze)r*yr?r: new_statescoresrrrscoreszDefaultRNNLM.scorecCs |j|S)zScore eos. Args: state: Scorer state for prefix tokens Returns: float: final score )r)final)r*r?rrr final_scores zDefaultRNNLM.final_scoreysstatesxsreturncst||jjj|jjjdkrdndddurdn fddD|j|dddf\}|fd d tDfS) aScore new token batch. Args: ys (torch.Tensor): torch.int64 prefix tokens (n_batch, ylen). states (List[Any]): Scorer states for prefix tokens. xs (torch.Tensor): The encoder feature that generates ys (n_batch, xlen, n_feat). Returns: tuple[torch.Tensor, List[Any]]: Tuple of batchfied scores for next token with shape of `(n_batch, n_vocab)` and next state list for ys. r ch)rVrNcs(i|]fddtDqS)cs,g|]tfddtDqS)csg|] }|qSrr).0b)rBkrQrr zBDefaultRNNLM.batch_score....)r1stackr5rW)rYn_batchrQ)rBrrZsz7DefaultRNNLM.batch_score...r5r])r^n_layersrQrYr s z,DefaultRNNLM.batch_score..rFcs"g|] fddDqS)cs(i|]fddtDqS)csg|] }|qSrr)rWrB)rXrYrQrrrZr[zBDefaultRNNLM.batch_score....r_r])rXr`rQrarrbs(z7DefaultRNNLM.batch_score...rr])keysr`rQ)rXrrZz,DefaultRNNLM.batch_score..)lenr) predictorr`typrGr5)r*rPrQrRr=r)rcr^r`rQr batch_scores   zDefaultRNNLM.batch_scoreN)__name__ __module__ __qualname____doc__ staticmethodrr#r-r.rErMrOr1Tensorrrrrhrrrrr s& (  r csPeZdZdZejdddffdd ZddZd d Zd d Z dddZ Z S)r%zA wrapper for pytorch RNNLM.none) reductionrFcsNt|ttfstdt|tt|||_d|_ d|_ ||_ ||_ dS)zInitialize class. :param torch.nn.Module predictor : The RNNLM :param function lossfun : The loss function to use :param int/str label_key : z'label_key must be int or str, but is %sN) isinstancerr TypeErrorr superr%r#lossfunrJr< label_keyrf)r*rfrtru __class__rrr#s  zClassifierWithState.__init__cOst|jtrBt| |jkrt|ks nd|j}t|||j}|jdkr1|dd}n.|d|j||jdd}nt|jtr_|j|vrVd|j}t|||j}||j=d|_d|_|j|g|Ri|\}|_| |j||_||jfS)aCompute the loss value for an input and label pair. Notes: It also computes accuracy and stores it to the attribute. When ``label_key`` is ``int``, the corresponding element in ``args`` is treated as ground truth labels. And when it is ``str``, the element in ``kwargs`` is used. The all elements of ``args`` and ``kwargs`` except the groundtruth labels are features. It feeds features to the predictor and compare the result with ground truth labels. :param torch.Tensor state : the LM state :param list[torch.Tensor] args : Input minibatch :param dict[torch.Tensor] kwargs : Input minibatch :return loss value :rtype torch.Tensor zLabel key %d is out of boundsrFNzLabel key "%s" is not found) rqrurre ValueErrorrrJr<rfrt)r*r?r,kwargsmsgr;rrrrEs& "   "     zClassifierWithState.forwardcCsBt|jdr|jjr|||S|||\}}|tj|ddfS)aPredict log probabilities for given state and input x using the predictor. :param torch.Tensor state : The current state :param torch.Tensor x : The input :return a tuple (new state, log prob vector) :rtype (torch.Tensor, torch.Tensor) normalizedrx)dim)hasattrrfr|F log_softmax)r*r?r:zrrrrGs zClassifierWithState.predictc Cs|jjjdkr |||Sg}g}t|D]#}|durdn||}||||d\}}||||q|t|fS)z(Predict new tokens from buffered inputs.r&Nr) rfrwrirGr5rHappendr1cat) r*r?r:nrK new_log_yrBstate_ilog_yrrr buff_predict&s    z ClassifierWithState.buff_predictNcCs4t|jdr|dur|j||S|j|SdS)zPredict final log probabilities for given state using the predictor. :param state: The state :return The final log probabilities :rtype torch.Tensor rNNr)r~rfrN)r*r?indexrrrrN5s  zClassifierWithState.final)N) rirjrkrlr!CrossEntropyLossr#rErGrrN __classcell__rrrvrr%s *r%cs<eZdZdZ     d fdd Zd d Zd d ZZS)r&zA pytorch RNNLM.Nr rrFc s`tt||dur }t|||_|dkrd|_nt||_|dkr?tt |gfddt |dD|_ ntt |gfddt |dD|_ tfddt |dD|_ t||_||_|_||_td |td td ||r|ksJd |jj|j_|D] } | jd dqdS)zInitialize class. :param int n_vocab: The size of the vocabulary :param int n_layers: The number of layers to create :param int n_units: The number of units per layer :param str typ: The RNN type Nrr cg|]}tqSr)r!LSTMCellrW_n_unitsrrrZiz"RNNLM.__init__..rxcrr)r!GRUCellrrrrrZnrcsg|]}tqSr)r!Dropoutr)rrrrZrszTie weights set to {}zDropout set to {}zEmb Dropout set to {}z>Tie Weights: True need embedding and final dimensions to matchgg?)rsr&r#r! Embeddingembed embed_dropr ModuleListrr5rnnrdropoutLinearlor`rrglogginginfoformatweight parametersdatauniform_) r*r+r`rn_embedrgrrr paramrv)rrrr#IsH      zRNNLM.__init__cCs(t|}t||jj|j|jdS)zInitialize state.)r9r0)nextrr1zerosrr8r9r0)r* batchsizeprrr zero_states zRNNLM.zero_statecs|dur,fddtjD}d|i}jdkr,fddtjD}||d}dgj}jdur@}n}jdkrdgj}jdjd||dd|d df\|d<|d<td jD]#}j|j|||d |d||d |f\||<||<qs||d}n7jdjd||dd|d<td jD]}j|j|||d |d|||<qd|i}jd |d }||fS) zForward neural networks.Nc "g|] }tdqSrrrsizerWrr*r:rrrZs"z!RNNLM.forward..rVr c rrrrrrrrZrdrTrrUrxrF)r5r`rgrrrrr)r*r?r:rVrUembrrJrrrrEs8        "* &0z RNNLM.forward)Nr rrF)rirjrkrlr#rrErrrrvrr&Fs>r&)rlrtypingrrrr1torch.nnr!torch.nn.functional functionalrespnet.nets.lm_interfacer#espnet.nets.pytorch_backend.e2e_asrrespnet.nets.scorer_interfacerespnet.utils.cli_utilsrr"r r%r&rrrrs     In