o if<@sdZddlZddlmZddlmZmZmZmZm Z ddl Z ddl m Z ddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddl m!Z!e e j"e dkrpddl#m$Z$nedddZ$Gddde!Z%dS)zESPnet2 ASR Transducer model.N)contextmanager)DictListOptionalTupleUnion)parse)check_argument_types) AbsFrontend) AbsSpecAug) AbsDecoder)Encoder) JointNetwork)get_transducer_task_io) AbsNormalize)force_gatherable)AbsESPnetModelz1.6.0)autocastTccs dVdS)N)enabledrrb/home/ubuntu/.local/lib/python3.10/site-packages/espnet2/asr_transducer/espnet_transducer_model.pyrs rc+seZdZdZ            d:d ed eeed feefd e e d e e de e de dededededededededededededededdf*fd d! Zd"ejd#ejd$ejd%ejdeejeeejfejff d&d'Zd"ejd#ejd$ejd%ejdeeejff d(d)Zd"ejd#ejdeejejffd*d+Zd"ejd#ejdeejejffd,d-Zd.ejd/ejd0ejd1ejd2ejdeeje ee eff d3d4Zd.ejd0ejd1ejd2ejdejf d5d6Zd7ejd0ejdejfd8d9ZZS);ESPnetASRTransducerModelaiESPnet2ASRTransducerModel module definition. Args: vocab_size: Size of complete vocabulary (w/ EOS and blank included). token_list: List of token frontend: Frontend module. specaug: SpecAugment module. normalize: Normalization module. encoder: Encoder module. decoder: Decoder module. joint_network: Joint Network module. transducer_weight: Weight of the Transducer loss. fastemit_lambda: FastEmit lambda value. auxiliary_ctc_weight: Weight of auxiliary CTC loss. auxiliary_ctc_dropout_rate: Dropout rate for auxiliary CTC loss inputs. auxiliary_lm_loss_weight: Weight of auxiliary LM loss. auxiliary_lm_loss_smoothing: Smoothing rate for LM loss' label smoothing. ignore_id: Initial padding ID. sym_space: Space symbol. sym_blank: Blank Symbol report_cer: Whether to report Character Error Rate during validation. report_wer: Whether to report Word Error Rate during validation. extract_feats_in_collect_stats: Whether to use extract_feats stats collection. ?FT vocab_size token_list.frontendspecaug normalizeencoderdecoder joint_networktransducer_weightfastemit_lambdaauxiliary_ctc_weightauxiliary_ctc_dropout_rateauxiliary_lm_loss_weightauxiliary_lm_loss_smoothing ignore_id sym_space sym_blank report_cer report_werextract_feats_in_collect_statsreturnNcstts J||_||_||_||_||_||_ ||_ ||_ ||_ ||_ ||_d|_d|_| dk|_| dk|_|jrLtj|j||_| |_|jr[tj|j||_||_| |_| |_| |_| |_||_||_ ||_!dS)z-Construct an ESPnetASRTransducerModel object.Nr)"super__init__r rr+copyrr,r-rr r!r"r#r$criterion_transducererror_calculatoruse_auxiliary_ctcuse_auxiliary_lm_losstorchnnLinear output_sizectc_linctc_dropout_ratelm_linlm_loss_smoothingr%r&r'r)r.r/r0)selfrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0 __class__rrr39s<      z!ESPnetASRTransducerModel.__init__speechspeech_lengthstext text_lengthscKs|dks J|j|jd|jdkr&|jdkr&|jdks3nJ|j|j|j|jf|jd}|ddd|f}|||\}}t|||jd\} } } } |j|j|| } | | d| d}| ||| | | \}}}d\}}|j r| || | | }|jr|| | }|j||j||j|}t|||dkr|nd|dkr|nd||d}t|||f|j\}}}|||fS) aForward architecture and compute loss(es). Args: speech: Speech sequences. (B, S) speech_lengths: Speech sequences lengths. (B,) text: Label ID sequences. (B, L) text_lengths: Label ID sequences lengths. (B,) kwargs: Contains "utts_id". Return: loss: Main loss value. stats: Task statistics. weight: Task weights. rN)r+)rrr)lossloss_transducer aux_ctc_loss aux_lm_losscer_transducerwer_transducer)dimshapemaxencoderr+r# set_devicedevicer$ unsqueeze_calc_transducer_lossr7_calc_ctc_lossr8 _calc_lm_lossr%r'r)dictdetachr)rArDrErFrGkwargs batch_size encoder_outencoder_out_lens decoder_intargett_lenu_len decoder_out joint_out loss_trans cer_trans wer_transloss_ctcloss_lmrJstatsweightrrrforward~sp      z ESPnetASRTransducerModel.forwardcKs>|jr |||\}}ntd|j||}}||dS)aCollect features sequences and features lengths sequences. Args: speech: Speech sequences. (B, S) speech_lengths: Speech sequences lengths. (B,) text: Label ID sequences. (B, L) text_lengths: Label ID sequences lengths. (B,) kwargs: Contains "utts_id". Return: {}: "feats": Features sequences. (B, T, D_feats), "feats_lengths": Features sequences lengths. (B,) zkGenerating dummy stats for feats and feats_lengths, because encoder_conf.extract_feats_in_collect_stats is )feats feats_lengths)r0_extract_featsloggingwarning)rArDrErFrGr\rnrorrr collect_featss  z&ESPnetASRTransducerModel.collect_featscCstd-|||\}}|jdur|jr|||\}}|jdur*|||\}}Wdn1s4wY|||\}}|d|dksVJ||df|d|ksiJ||f||fS)aEncoder speech sequences. Args: speech: Speech sequences. (B, S) speech_lengths: Speech sequences lengths. (B,) Return: encoder_out: Encoder outputs. (B, T, D_enc) encoder_out_lens: Encoder outputs lengths. (B,) FNrrH)rrpr trainingr!r"sizerR)rArDrErnror^r_rrrrSs$   zESPnetASRTransducerModel.encodecCsb|dks J|j|ddd|f}|jdur(|||\}}||fS||}}||fS)a?Extract features sequences and features sequences lengths. Args: speech: Speech sequences. (B, S) speech_lengths: Speech sequences lengths. (B,) Return: feats: Features sequences. (B, T, D_feats) feats_lengths: Features sequences lengths. (B,) rHN)rPrQrRr)rArDrErnrorrrrp+s  z'ESPnetASRTransducerModel._extract_featsr^rerarbrcc Cs|jdur(zddlm}|d|jd|_Wnty'tdtdYnw|||||}|jsd|j s9|j rd|j durWddl m }||j|j|j|j|j|j |j d |_ | ||\} } || | fS|ddfS) a(Compute Transducer loss. Args: encoder_out: Encoder output sequences. (B, T, D_enc) joint_out: Joint Network output sequences (B, T, U, D_joint) target: Target label ID sequences. (B, L) t_len: Encoder output sequences lengths. (B,) u_len: Target label ID sequences lengths. (B,) Return: loss_transducer: Transducer loss value. cer_transducer: Character error rate for Transducer. wer_transducer: Word Error Rate for Transducer. Nr)RNNTLossmean) reductionr&zJwarp-rnnt was not installed.Please consult the installation documentation.rH)ErrorCalculator)r.r/)r5warprnnt_pytorchrvr& ImportErrorrqerrorexitrtr.r/r6'espnet2.asr_transducer.error_calculatorryr#r$rr,r-) rAr^rerarbrcrvrKryrNrOrrrrWEsD        z.ESPnetASRTransducerModel._calc_transducer_lossc Cs|tjjj||jd}tj|dddd}|dk}||}tj j j ddtjjj ||||ddd }Wd n1sBwY|| d}|S) aMCompute CTC loss. Args: encoder_out: Encoder output sequences. (B, T, D_enc) target: Target label ID sequences. (B, L) t_len: Encoder output sequences lengths. (B,) u_len: Target label ID sequences lengths. (B,) Return: loss_ctc: CTC loss value. )prrHrrPT) deterministicsum) zero_infinityrxN)r=r9r: functionaldropoutr> log_softmax transposecpubackendscudnnflagsctc_lossru) rAr^rarbrcctc_in target_mask ctc_targetrirrrrXs$  z'ESPnetASRTransducerModel._calc_ctc_lossrdcCs||ddddddfd|j}|dtj}t.|}||j |jd|dk}| |d}| d| dd|j Wdn1sSwYtj jjtj|dd|dd}| | dd|d}|S)zCompute LM loss. Args: decoder_out: Decoder output sequences. (B, U, D_dec) target: Target label ID sequences. (B, L) Return: loss_lm: LM loss value. NrrHrrnone)rx)r?viewrtyper9int64no_gradclonefill_r@ masked_fillscatter_rVr:rkl_divrrru)rArdra lm_loss_in lm_target true_distignorerjrrrrYs$*   z&ESPnetASRTransducerModel._calc_lm_loss) rrrrrrrrrFFT)__name__ __module__ __qualname____doc__intrrstrrrr r rr r rfloatboolr3r9TensorrrmrsrSrprWrXrY __classcell__rrrBrrs$     E _  $ *  A (r)T)&rrq contextlibrtypingrrrrrr9packaging.versionrV typeguardr !espnet2.asr.frontend.abs_frontendr espnet2.asr.specaug.abs_specaugr *espnet2.asr_transducer.decoder.abs_decoderr &espnet2.asr_transducer.encoder.encoderr $espnet2.asr_transducer.joint_networkrespnet2.asr_transducer.utilsrespnet2.layers.abs_normalizer espnet2.torch_utils.device_funcsrespnet2.train.abs_espnet_modelr __version__torch.cuda.amprrrrrrs*