o i@s~ddlmZmZmZddlZddlmmZddl m Z ddl m Z ddl mZddlmZddlmZGdd d eZdS) )DictOptionalTupleN)check_argument_types)AbsLM)force_gatherable)AbsESPnetModel) make_pad_maskc seZdZddededeffdd Z ddejd ejd eed e ejejffd d Z ddejd ejded e ejejffddZ dejd ejd e eje e ejfejffddZdejd ejd e e ejffddZZS)ESPnetLanguageModelrlm vocab_size ignore_idcs8tsJt||_|d|_|d|_||_dS)N)rsuper__init__r soseosr )selfr r r  __class__K/home/ubuntu/.local/lib/python3.10/site-packages/espnet2/lm/espnet_model.pyrs     zESPnetLanguageModel.__init__Ntext text_lengths max_lengthreturnc Cs*|d}|dur|ddd|f}n |ddd|f}t|ddgd|j}t|ddgd|j}t|D] \}}|j|||f<q:|d} ||d\} } tj | d| j d| ddd} |durx| t | | j ddn| t | |dd | j dd| |d} | | fS) zCompute negative log likelihood(nll) Normally, this function is called in batchify_nll. Args: text: (Batch, Length) text_lengths: (Batch,) max_lengths: int rNrconstantnone) reductiong)maxlen)sizemaxFpadrr enumeraterr cross_entropyviewshape masked_fill_r todevice) rrrr batch_sizextil x_lengthsy_nllrrrr4s& $  zESPnetLanguageModel.nlldr,cCs|d}||kr|||\}}nHg}g}|}d} t| ||} || | ddf} || | } |j| | |d\} }|| ||| } | |krOnqt|}t|}|d|kscJ|d|kslJ||fS)aCompute negative log likelihood(nll) from transformer language model To avoid OOM, this fuction seperate the input into batches. Then call nll for each batch and combine and return results. Args: text: (Batch, Length) text_lengths: (Batch,) batch_size: int, samples each batch contain when computing nll, you may change this to avoid OOM or increase rTN)r)r!r4r"minappendtorchcat)rrrr, total_numr4r1nllsr start_idxend_idx batch_textbatch_text_lengths batch_nllbatch_x_lengthsrrr batchify_nllHs2     z ESPnetLanguageModel.batchify_nllc KsT|||\}}|}||}t|d}t|||f|j\}}} ||| fS)N)loss)r4sumdictdetachrr+) rrrkwargsr4 y_lengthsntokensrCstatsweightrrrforwardrs   zESPnetLanguageModel.forwardcKsiSNr)rrrrGrrr collect_featssz!ESPnetLanguageModel.collect_feats)rrM)r5)__name__ __module__ __qualname__rintrr8Tensorrrr4rBrstrrLrN __classcell__rrrrr sH 1 *  r )typingrrrr8torch.nn.functionalnn functionalr# typeguardrespnet2.lm.abs_modelr espnet2.torch_utils.device_funcsrespnet2.train.abs_espnet_modelr&espnet.nets.pytorch_backend.nets_utilsr r rrrrs