o iR@sRddlZddlmZddlmZmZmZmZmZddl Z ddl m Z ddl mZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlm Z ddl!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.e e j/e dkrddl0m1Z1nedddZ1Gddde&Z2dS)N)contextmanager)DictListOptionalTupleUnion)parse)check_argument_types)CTC) AbsDecoder) AbsEncoder) AbsFrontend)AbsPostEncoder) AbsPreEncoder) AbsSpecAug)ErrorCalculatorTransducer)get_transducer_task_io) AbsNormalize)force_gatherable)AbsESPnetModel)ErrorCalculator) th_accuracy) add_sos_eos)LabelSmoothingLossz1.6.0)autocastTccs dVdS)N)enabledrrL/home/ubuntu/.local/lib/python3.10/site-packages/espnet2/asr/espnet_model.pyr s rc+sJeZdZdZ          dAd ed eeed feefd e e d e e de e de e dede ededede ejjdedededededededededef*fdd Zd!ejd"ejd#ejd$ejd%eejeeejfejff d&d'Zd!ejd"ejd#ejd$ejd%eeejff d(d)Zd!ejd"ejd%eejejffd*d+Zd!ejd"ejd%eejejffd,d-Zd.ejd/ejd0ejd1ejd%ejf d2d3Z 4dBd.ejd/ejd0ejd1ejd5ef d6d7Zd.ejd/ejd0ejd1ejfd8d9Z d.ejd/ejd0ejd1ejfd:d;Z!d.ejd/ejdZ"d!ejd"ejd#ejd$ejfd?d@Z#Z$S)CESPnetASRModelz*CTC-attention hybrid Encoder-Decoder model?FT vocab_size token_list.frontendspecaug normalize preencoderencoder postencoderdecoderctc joint_network ctc_weightinterctc_weight ignore_id lsm_weightlength_normalized_loss report_cer report_wer sym_space sym_blankextract_feats_in_collect_statsc stsJd| krdksJ| J| d| kr$dks)J| J| td|_|d|_|d|_||_||_| |_| |_ | |_ ||_ ||_ ||_||_||_||_t|jdshd|j_|jjrxtj||j|j_| du|_d|_|jrddlm}| |_| |_||jdd|_|s|rt | | |||||d |_!n3d|_!|jdkrt"||||||_n!| dkrd|_n| |_t#||||d |_$|s|rt"||||||_| dkrd|_%n| |_%||_&dS) Nr ?rinterctc_use_conditioningF)RNNTLoss)blankfastemit_lambda)r4r5)size padding_idx smoothingnormalize_length)'r super__init__blank_idsoseosr$r1r/r0copyr%r&r'r(r)r+r*hasattrr;torchnnLinear output_sizeconditioning_layeruse_transducer_decodererror_calculatorwarprnnt_pytorchr<r,r.criterion_transducerrerror_calculator_transrr criterion_attr-r8)selfr$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r< __class__rrrD(s $$             zESPnetASRModel.__init__speechspeech_lengthstext text_lengthsreturncKs|dks J|j|jd|jdkr&|jdkr&|jdks3nJ|j|j|j|jf|jd}|ddd|f}|||\}}d} t|tr[|d} |d}d\} } } } d\}}d\}}}t}|jdkr|||||\}}|dur| nd|d<||d <d}|j dkr| dur| D](\}}|||||\}}||}|dur| nd|d |<||d |<q|t | }d|j ||j |}|j r||||\}}}|dur||j|}n|}|dur| nd|d <||d <||d<nI|jdkr|||||\} } } } |jdkr | }n|jdkr)|}n |j|d|j| }| dur>| nd|d<| |d<| |d<| |d<| |d<t|||f|j\}}}|||fS)aFrontend + Encoder + Decoder + Calc loss Args: speech: (Batch, Length, ...) speech_lengths: (Batch, ) text: (Batch, Length) text_lengths: (Batch,) kwargs: "utt_id" is among the input. r:rN)NNNNNN)NNNr loss_ctccer_ctczloss_interctc_layer{}zcer_interctc_layer{}loss_transducercer_transducerwer_transducerr9loss_attacccerwerloss)dimshapemaxencode isinstancetupledictr/_calc_ctc_lossdetachr0formatlenrO_calc_transducer_loss_calc_att_lossrdevice)rUrXrYrZr[kwargs batch_size encoder_outencoder_out_lensintermediate_outsrcacc_attcer_attwer_attr^r_r`rarbstats loss_interctc layer_idxintermediate_outloss_iccer_icrgweightrrrforwards               zESPnetASRModel.forwardcKs>|jr |||\}}ntd|j||}}||dS)NzkGenerating dummy stats for feats and feats_lengths, because encoder_conf.extract_feats_in_collect_stats is )feats feats_lengths)r8_extract_featsloggingwarning)rUrXrYrZr[rvrrrrr collect_featss  zESPnetASRModel.collect_featsc Csdtd-|||\}}|jdur|jr|||\}}|jdur*|||\}}Wdn1s4wY|jdurF|||\}}|jjrW|j|||jd\}}}n |||\}}}d}t |t ro|d}|d}|j dur|| ||\}}| d| dksJ| | df| d| ksJ| | f|dur||f|fS||fS)zFrontend + Encoder. Note that this method is used by asr_inference.py Args: speech: (Batch, Length, ...) speech_lengths: (Batch, ) FN)r-r:r)rrr'trainingr(r)r*r;r-rlrmr+r?rj) rUrXrYrrrxry_rzrrrrk.sD      zESPnetASRModel.encodecCsb|dks J|j|ddd|f}|jdur(|||\}}||fS||}}||fS)Nr:)rhrirjr&)rUrXrYrrrrrris  zESPnetASRModel._extract_featsrxryys_pad ys_pad_lensc Cst||j|j|j\}}|d}|||||\}} |d} |d} tjjj | d| | d|jdd} | | d} | j dd} | d| ksLJ| S)a,Compute negative log likelihood(nll) from transformer-decoder Normally, this function is called in batchify_nll. Args: encoder_out: (Batch, Length, Dim) encoder_out_lens: (Batch,) ys_pad: (Batch, Length) ys_pad_lens: (Batch,) r:rr!none) ignore_index reductionrh) rrFrGr1r,r?rJrK functional cross_entropyviewsum) rUrxryrr ys_in_pad ys_out_pad ys_in_lens decoder_outrrwdecoder_num_classnllrrrr|s"     zESPnetASRModel.nlldrwcCs|d}||kr|||||}nIg}d} t|||} ||| ddddf} ||| } ||| ddf} ||| } || | | | }||| }||krUnqt|}|d|ksdJ|S)a3Compute negative log likelihood(nll) from transformer-decoder To avoid OOM, this fuction seperate the input into batches. Then call nll for each batch and combine and return results. Args: encoder_out: (Batch, Length, Dim) encoder_out_lens: (Batch,) ys_pad: (Batch, Length) ys_pad_lens: (Batch,) batch_size: int, samples each batch contain when computing nll, you may change this to avoid OOM or increase GPU memory usage rTN)r?rminappendrJcat)rUrxryrrrw total_numr start_idxend_idxbatch_encoder_outbatch_encoder_out_lens batch_ys_padbatch_ys_pad_lens batch_nllrrr batchify_nlls2     zESPnetASRModel.batchify_nllcCst||j|j|j\}}|d}|||||\}} |||} t|d|j||jd} |j s5|j dur:d\} } n|j dd}| | | \} } | | | | fS)Nr:r!) ignore_labelr]r) rrFrGr1r,rTrrr$rrPargmaxcpu)rUrxryrrrrrrrrcr{r|r}ys_hatrrrrts      zESPnetASRModel._calc_att_losscCsR|||||}d}|js%|jdur%|j|j}|j||dd}||fS)NT)is_ctc)r-rrPrdatar)rUrxryrrr^r_rrrrros zESPnetASRModel._calc_ctc_losslabelsc Cst|||j|jd\}}}}|j|j||}||d|d} || |||} d\} } |j sB|j durB| ||\} } | | | fS)aCompute Transducer loss. Args: encoder_out: Encoder output sequences. (B, T, D_enc) encoder_out_lens: Encoder output sequences lengths. (B,) labels: Label ID sequences. (B, L) Return: loss_transducer: Transducer loss value. cer_transducer: Character error rate for Transducer. wer_transducer: Word Error Rate for Transducer. )r1rErr:r]N) rr1rEr, set_devicerur. unsqueezerRrrS) rUrxryr decoder_intargett_lenu_lenr joint_outr`rarbrrrrss.  z$ESPnetASRModel._calc_transducer_lossc Cs|jdurdS|dksJ|j|jd|jdkr-|jdkr-|jdks:nJ|j|j|j|jf|ddd|f}|||\}}t|trW|d}|jj}d|j_|||||}||j_|S)Nr:rF)r-rhrirjrkrlrmreduce) rUrXrYrZr[rxry do_reducer^rrr_calc_batch_ctc_loss2s*   z#ESPnetASRModel._calc_batch_ctc_loss) rr r!r FTTr"r#T)r)%__name__ __module__ __qualname____doc__intrrstrrrr rrrr rr r rJrKModulefloatboolrDTensorrrrrkrrrrtrorsr __classcell__rrrVrr%s*     r    ;  , /    0r)T)3r contextlibrtypingrrrrrrJpackaging.versionrV typeguardr espnet2.asr.ctcr espnet2.asr.decoder.abs_decoderr espnet2.asr.encoder.abs_encoderr !espnet2.asr.frontend.abs_frontendr 'espnet2.asr.postencoder.abs_postencoderr%espnet2.asr.preencoder.abs_preencoderrespnet2.asr.specaug.abs_specaugr'espnet2.asr.transducer.error_calculatorrespnet2.asr_transducer.utilsrespnet2.layers.abs_normalizer espnet2.torch_utils.device_funcsrespnet2.train.abs_espnet_modelrespnet.nets.e2e_asr_commonr&espnet.nets.pytorch_backend.nets_utilsr3espnet.nets.pytorch_backend.transformer.add_sos_eosrs6