o i]@s,dZddlZddlZddlZddlZddlZddlmZddlZddl Z ddl Z ddl Z ddl Z ddlmZddlmZddlmZddlmZmZddlmZmZmZmZdd lmZmZmZdd lm Z dd l!m"Z"dd l#m$Z$dd l%m&Z&ddl'm(Z(dZ)Gdddej*Z+Gddde&e j,j-Z.dS)z|?|jd krS|j@sW|jAr|jB|jC|jD|jE|jF|jG|jH|jI|j|jdd }tJjKdi||_L|j@|_@|jA|_And|_@d|_A|jMr|jB|jCd |jE|jF|jG|jH|jI|j|jdd }tJjKdi||_N|jM|_Mnd|_Md|_Hd|_Od|_Pd|_QdS)zConstruct an E2E object. :param int idim: dimension of inputs :param int odim: dimension of outputs :param Namespace args: argument Namespace containing options g?zasr_weight should be [0.0, 1.0)zmt_weight should be [0.0, 1.0)zmtlalpha should be [0.0, 1.0] char_listNrstrnn)modearchzUse label smoothing with ) transcript multilingualF replace_sosT)ctc_typereducelocation) padding_idx)pdtyper<) beam_sizepenalty ctc_weight maxlenratio minlenratio lm_weightrnnlmnbestspaceblanktgt_langg _r&)Rsuperr,__init__torchnnModulerr1 asr_weight mt_weightmtlalphaetypeverbosegetattrr@outdir sym_spacerY sym_blankrZrrsoseospadrr<lsm_typeospathisfile train_jsonr!r"rrGrHrr7rattrdecctcatt_asrdec_asrreprojs dropout_raterIcopydeepcopyatype Embeddingeunitsembed_mtDropout dropout_mtr:oneselayersint64enc_mtinit_like_chainer report_cer report_werrQrRrSrTrUrVrWrXargparse Namespace recog_args report_bleu trans_argslogzerorr)r$idimodimargs labeldistargs_asrrr __class__r&r'r]xs &&&             z E2E.__init__cCsLt||jjjjddtjt |jj D] }t |jj |j qdS)amInitialize weight like chainer. chainer basically uses LeCun way: W ~ Normal(0, fan_in ** -0.5), b = 0 pytorch basically uses W, b ~ Uniform(-fan_in**-0.5, fan_in**-0.5) however, there are two exceptions as far as I know. - EmbedID.W ~ Normal(0, 1) - LSTM.upward.b[forget_gate_range] = 1 (but not used in NStepLSTM) rrAN) rrsembedweightdatanormal_sixmovesrangelendecoderrbias_ih)r$ir&r&r'rs  zE2E.init_like_chainerc stjr|ddddf}|ddddf}nd}||\}}}j||||d\__}|||\_} _} } } ||\_ } j sOj sSd_ nd}jj |t||jjjjrn|dndd}g}g}dd|D}t|D]I\}}||}fdd|D}fd d|D}d |jjd }|jjd }d |jjd }||d g7}||d gg7}qtj||d _ j}|jd|j_dj j!jj jj!j _"t#j}t#j}t#j }t#j"}|t$kr1t%&|s1j'(|||| | j| | | j | j"St)*d |j"S)aEE2E forward. :param torch.Tensor xs_pad: batch of padded input sequences (B, Tmax, idim) :param torch.Tensor ilens: batch of lengths of input sequences (B) :param torch.Tensor ys_pad: batch of padded token id sequence tensor (B, Lmax) :return: loss value :rtype: torch.Tensor NrrAlang_idsr?cS g|] }|ddddqSryseqrAr&.0 nbest_hypr&r&r' R zE2E.forward..c&g|]}t|dkrjt|qSrr9r@ridxr=r&r'rVs&crrrrr=r&r'rW  dzloss (=%f) is not correct)+rGr7rsrr forward_asr loss_asr_att loss_asr_ctc forward_mtrtrainingrrrecognize_beam_batchr^tensorrr@rWsqueezetolist enumeratejoinreplacerYrZsplitnltk bleu_score corpus_bleurcrrarbrfloatCTC_LOSS_THRESHOLDmathisnanrr r!warning)r$xs_padilensys_pad ys_pad_src tgt_lang_idshs_padhlens_rrrrrlpz nbest_hyps list_of_refshypsy_hatsry_haty_trueseq_hatseq_true seq_hat_text seq_true_textasr_ctc_weight loss_st_data loss_asr_data loss_mt_data loss_datar&r=r'forwards             z E2E.forwardc"s ddl}d\}}d}d\}} d} jdkr|||| || fSjdkr|||\} } } jsjs5jrjdkrHjjdkrHj |j }nd}ggggf\}}}}j |t ||jjj}dd|D}t|D]l\}}||}fd d|D}fd d|D}d |jjd }|jjd }d |jjd }|}|}|||||t||d d }|d d }|||||t|qpjsdn tt|t|} jsdn tt|t|}jdkr |||}jdurg} j |j }t|D]_\}}!d dt|!D}||}fdd|D}fdd|D}d |jd }|jd }d |jd }|d d }|d d }t|dkry| |||t|q| rt| t| nd} |||| || fS)aForward pass in the auxiliary ASR task. :param torch.Tensor hs_pad: batch of padded source sequences (B, Tmax, idim) :param torch.Tensor hlens: batch of lengths of input sequences (B) :param torch.Tensor ys_pad: batch of padded target sequences (B, Lmax) :return: ASR attention loss value :rtype: torch.Tensor :return: accuracy in ASR attention decoder :rtype: float :return: ASR CTC loss value :rtype: torch.Tensor :return: character error rate from CTC prediction :rtype: float :return: character error rate from attetion decoder prediction :rtype: float :return: word error rate from attetion decoder prediction :rtype: float rN)r?r?)NNrAr?cSrrr&rr&r&r'rrz#E2E.forward_asr..crrrrr=r&r'rrcrrrrr=r&r'rrrrcSsg|]}|dqS)rr&)rxr&r&r'rscrrrrr=r&r'rrcrrrrr=r&r'rr) editdistancerarcrvrrrrrSrt log_softmaxrrr^rr@rWrrrrYrZrappendevalrrsumargmaxr)"r$rrrrloss_attloss_ctcrrrrrrrrword_eds word_ref_lenschar_eds char_ref_lensnbest_hyps_asrrrrrrrrr hyp_words ref_words hyp_chars ref_charscersyr&r=r'rs              zE2E.forward_asrc Csd}d}|jdkr ||fStj|dkdd}dd|D}t||j}||| ||\}} } | || |\}}} ||fS)aNForward pass in the auxiliary MT task. :param torch.Tensor xs_pad: batch of padded source sequences (B, Tmax, idim) :param torch.Tensor ys_pad: batch of padded target sequences (B, Lmax) :return: MT loss value :rtype: torch.Tensor :return: accuracy in MT decoder :rtype: float r?rrrA)dimcSsg|]}||dkqSrr&)rrr&r&r'r sz"E2E.forward_mt..) rbr^rcpunumpyr rlrrr~rs) r$rrrrrys_src xs_zero_padrrrr&r&r'rs    zE2E.forward_mtcCs t|jdS)zScorers.)r)dictrsr=r&r&r'scorerss z E2E.scorerscCsz||jdg}|dd|jdddf}t|}tj||j|jd}| d}| ||\}}}| dS)zEncode acoustic features. :param ndarray x: input acoustic feature (T, D) :return: encoder outputs :rtype: torch.Tensor rN)devicerO) rshaper<next parametersr^ as_tensorrrO contiguous unsqueezer7r)r$rrrMhhsrr&r&r'encodes   z E2E.encodeNcCs\tdt|jd||d}tdt|d|j|dd|||}|S)aIE2E beam search. :param ndarray x: input acoustic feature (T, D) :param Namespace trans_args: argument Namespace containing options :param list char_list: list of characters :param torch.nn.Module rnnlm: language model module :return: N-best decoding results :rtype: list zinput lengths: rzencoder output lengths: rAN) r!r"r#rrrsizersrecognize_beam)r$rrr@rWrrr&r&r' translate*s  z E2E.translatec sj}tjdd|Dtjd}fdd|D}fdd|D}t|d}||\}} } tt t t | } j || d|||} |rO| S) aoE2E batch beam search. :param list xs: list of input acoustic feature arrays [(T_1, D), (T_2, D), ...] :param Namespace trans_args: argument Namespace containing options :param list char_list: list of characters :param torch.nn.Module rnnlm: language model module :return: N-best decoding results :rtype: list css|]}|jdVqdS)rN)rrxxr&r&r' Isz&E2E.translate_batch..rNcs(g|]}|ddjdddfqS)NrrPr r=r&r'rLs(z'E2E.translate_batch..csg|] }tt|qSr&)r r rr r=r&r'rMsr?N)rrr:fromiterrr r7r^rlistmapr9rsrtrain) r$xsrr@rWprevrrrrrrr&r=r'translate_batch=s   zE2E.translate_batchc Cs|t5|jr!|ddddf}|ddddf}nd}|||\}}}|jj||||d} Wdn1s@wY|| S)adE2E attention calculation. :param torch.Tensor xs_pad: batch of padded input sequences (B, Tmax, idim) :param torch.Tensor ilens: batch of lengths of input sequences (B) :param torch.Tensor ys_pad: batch of padded token id sequence tensor (B, Lmax) :param torch.Tensor ys_pad_src: batch of padded token id sequence tensor (B, Lmax) :return: attention weights with the following shape, 1) multi-head case => attention weights (B, H, Lmax, Tmax), 2) other case => attention weights (B, Lmax, Tmax). :rtype: float ndarray NrrAr)rr^no_gradrGr7rscalculate_all_attentionsr) r$rrrrrhpadrratt_wsr&r&r'r]s  zE2E.calculate_all_attentionsc Cs~d}|jdks |jdkr|S|t|||\}}}|j| }Wdn1s4wY| |S)aE2E CTC probability calculation. :param torch.Tensor xs_pad: batch of padded input sequences (B, Tmax) :param torch.Tensor ilens: batch of lengths of input sequences (B) :param torch.Tensor ys_pad: batch of padded token id sequence tensor (B, Lmax) :param torch.Tensor ys_pad_src: batch of padded token id sequence tensor (B, Lmax) :return: CTC probability (B, Tmax, vocab) :rtype: float ndarray Nr) rarcrr^rr7rtsoftmaxrrr) r$rrrrprobsrrrr&r&r'calculate_all_ctc_probs{s  zE2E.calculate_all_ctc_probscCsT|dd|jdddf}|jdg}t|ttj|tjd}|||fS)z&Subsample speeh frames in the encoder.NrrN) r<rr r^ from_numpyr:arrayfloat32r)r$rilenrr&r&r'subsample_framess  zE2E.subsample_frames)N)r(r)r*r+ staticmethodr1r-r.r/r>r]rrrrrrr rrrr __classcell__r&r&rr'r,Ns0     ju   r,)/r+rryr!rrn itertoolsrchainerrrr:rr^respnet.nets.e2e_asr_commonrespnet.nets.pytorch_backend.ctcr*espnet.nets.pytorch_backend.initializationrr&espnet.nets.pytorch_backend.nets_utilsrr r r (espnet.nets.pytorch_backend.rnn.argumentr r r*espnet.nets.pytorch_backend.rnn.attentionsr(espnet.nets.pytorch_backend.rnn.decodersr(espnet.nets.pytorch_backend.rnn.encodersrespnet.nets.st_interfacerespnet.utils.fill_missing_argsrrChainrr_r`r,r&r&r&r's4