o iK@s@dZddlZddlZddlZddlZddlmZddlZddlZ ddl Z ddl Z ddlm Z ddl mZddlmZddlmZddlmZdd lmZdd lmZmZdd lmZmZmZmZdd lm Z m!Z!m"Z"dd l#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,dZ-Gdddej.Z/Gdddee j0j1Z2dS)zsuperr+__init__r8r9Modulerr0mtlalphaetypeverbosegetattrrCoutdir sym_spacerT sym_blankrUrrsoseosr r?lsm_typeospathisfile train_jsonr r!rrfrontendrfeature_transformn_melsrr7rctcrattrdecinit_like_chainer report_cer report_werrLrMrNrOrPrQrRrSargparse Namespace recog_argslogzerorr)r#idimodimargs labeldistrr __class__r%r&rWmsj &           z E2E.__init__cCsLt||jjjjddtjt |jj D] }t |jj |j qdS)amInitialize weight like chainer. chainer basically uses LeCun way: W ~ Normal(0, fan_in ** -0.5), b = 0 pytorch basically uses W, b ~ Uniform(-fan_in**-0.5, fan_in**-0.5) however, there are two exceptions as far as I know. - EmbedID.W ~ Normal(0, 1) - LSTM.upward.b[forget_gate_range] = 1 (but not used in NStepLSTM) rrDN) r rlembedweightdatanormal_sixmovesrangelendecoderr bias_ih)r#ir%r%r&rms  zE2E.init_like_chainerc% sddl}jdurt||\}}}||\}}n||}}||\}}}jdkr4d_n|||_jdkrGd\_} n |||\_} }| _ jdks_j durbd} nwg} j |j } t| D]]\} }ddt|D}|| }fdd|D}fdd|D}d |jd }|jd }d |jd }|d d }|d d }t|dkr| |||t|qo| rt| t| nd} jsjsjsd \}}nĈjjd krj|j }nd}ggggf\}}}}j |t ||jj j!}d d|D} t| D]m\} }|| }fdd|D}fdd|D}d |jjd }|jjd }d |jjd }|"}|"} |||| |t| |d d }|d d }|||||t|qjsd n t#t|t|}jsd n t#t|t|}j}!|!dkrj_$t#j}"d}#n(|!dkrЈj_$d}"t#j}#n|!jd|!j_$t#j}"t#j}#t#j$}$|$t%krt&'|$sj()|#|"| | |||$j$St*+d|$j$S)aEE2E forward. :param torch.Tensor xs_pad: batch of padded input sequences (B, Tmax, idim) :param torch.Tensor ilens: batch of lengths of input sequences (B) :param torch.Tensor ys_pad: batch of padded token id sequence tensor (B, Lmax) :return: loss value :rtype: torch.Tensor rNrD)NNcSsg|]}|dqS)rr%).0xr%r%r& szE2E.forward..c&g|]}t|dkrjt|qSr<rCridxr@r%r&r&crrrrr@r%r&r  )rBrBrBcSs g|] }|ddddqS)ryseqrDrr%)r nbest_hypr%r%r&r)s crrrrr@r%r&r-rcrrrrr@r%r&r.rzloss (=%f) is not correct), editdistancergrrhr7rYrrjrrlrrCargmaxr| enumeraterjoinreplacerTrUrappendevalsumtrainingrnrorrrN log_softmaxrecognize_beam_batchr8tensorrRsplitfloatrCTC_LOSS_THRESHOLDmathisnanrrr warning)%r#xs_padilensys_padrhs_padhlensmask_rrcersy_hatsryy_haty_trueseq_hatseq_true seq_hat_text seq_true_text hyp_chars ref_charsrrlpzword_eds word_ref_lenschar_eds char_ref_lens nbest_hyps hyp_words ref_wordsalpha loss_att_data loss_ctc_data loss_datar%r@r&forwards                       z E2E.forwardcCst|jt|j|jdS)zScorers.)rrj)dictrlrrjrar@r%r%r&scorers`sz E2E.scorersc Cs||jdg}|dd|jdddf}t|}tj||j|jd}| d}|j durF| ||\}}}| ||\}}n||}}| ||\}} } |dS)zEncode acoustic features. :param ndarray x: input acoustic feature (T, D) :return: encoder outputs :rtype: torch.Tensor rN)devicedtype)rshaper?next parametersr8 as_tensorrr contiguous unsqueezergrhr7squeeze) r#rrphhsenhancedrrrr%r%r&encodeds     z E2E.encodeNcCsL||d}|jdkr|j|d}nd}|j|d||||}|S)aIE2E beam search. :param ndarray x: input acoustic feature (T, D) :param Namespace recog_args: argument Namespace containing options :param list char_list: list of characters :param torch.nn.Module rnnlm: language model module :return: N-best decoding results :rtype: list rrBN)rrrNrjrrlrecognize_beam)r#rrrrCrRrrrr%r%r& recognizes  z E2E.recognizec sj}tjdd|Dtjd}fdd|D}fdd|D}t|d}jdurB||\}} } || \} } n||} } | | \} } } |j dkr^j | } d }nd} d }t ttt| } jj| | | ||||d }|r|S) aoE2E batch beam search. :param list xs: list of input acoustic feature arrays [(T_1, D), (T_2, D), ...] :param Namespace recog_args: argument Namespace containing options :param list char_list: list of characters :param torch.nn.Module rnnlm: language model module :return: N-best decoding results :rtype: list cs|]}|jdVqdSrNrrxxr%r%r& z&E2E.recognize_batch..rc(g|]}|ddjdddfqSNrr?rr@r%r&r(z'E2E.recognize_batch..cg|] }tt|qSr%r rrrr@r%r&rrBNFT)normalize_score)rrr=fromiterint64r rgrhr7rNrjrr8rlistmapr<rlrtrain)r#xsrrrCrRprevrrrrrrrrrrr%r@r&recognize_batchs:       zE2E.recognize_batchcsjdur tdj}tjdd|Dtjd}fdd|D}fdd|D}t|d }||\}}}|rC| | |fS) zForward only in the frontend stage. :param ndarray xs: input acoustic feature (T, C, F) :return: enhaned feature :rtype: torch.Tensor NzFrontend does't existcsrrrrr%r%r&rrzE2E.enhance..rcrrrrr@r%r&rrzE2E.enhance..crr%rrr@r%r&rrrB) rg RuntimeErrorrrr=rrr rcpunumpy)r#rrrrrhlensmrr%r@r&enhances  z E2E.enhancec Cs|t7|jdur"|t||\}}}|||\}}n||}}|||\}}}|j|||} Wdn1sBwY| | S)aE2E attention calculation. :param torch.Tensor xs_pad: batch of padded input sequences (B, Tmax, idim) :param torch.Tensor ilens: batch of lengths of input sequences (B) :param torch.Tensor ys_pad: batch of padded token id sequence tensor (B, Lmax) :return: attention weights with the following shape, 1) multi-head case => attention weights (B, H, Lmax, Tmax), 2) other case => attention weights (B, Lmax, Tmax). :rtype: float ndarray N) rr8no_gradrgrrhr7rlcalculate_all_attentionsr) r#rrrrrrhpadratt_wsr%r%r&rs    zE2E.calculate_all_attentionsc Csd}|jdkr |S|t9|jdur+|t||\}}}|||\}}n||}}|||\}}} |j | }Wdn1sMwY| |S)ajE2E CTC probability calculation. :param torch.Tensor xs_pad: batch of padded input sequences (B, Tmax) :param torch.Tensor ilens: batch of lengths of input sequences (B) :param torch.Tensor ys_pad: batch of padded token id sequence tensor (B, Lmax) :return: CTC probability (B, Tmax, vocab) :rtype: float ndarray Nr) rYrr8rrgrrhr7rjsoftmaxrrr) r#rrrprobsrrrrrr%r%r&calculate_all_ctc_probss     zE2E.calculate_all_ctc_probscCsT|dd|jdddf}|jdg}t|ttj|tjd}|||fS)z&Subsample speeh frames in the encoder.Nrr) r?rr r8 from_numpyr=arrayfloat32r)r#rilenrr%r%r&subsample_framess  zE2E.subsample_frames)N)r'r(r)r* staticmethodr0r,r-r.rArWrmrrrrrrrrr __classcell__r%r%rxr&r+@s.     X   5r+)3r*rpr rrc itertoolsrchainerrr=r~r8respnet.nets.asr_interfacerespnet.nets.e2e_asr_commonrespnet.nets.pytorch_backend.ctcr7espnet.nets.pytorch_backend.frontends.feature_transformr.espnet.nets.pytorch_backend.frontends.frontendr*espnet.nets.pytorch_backend.initializationr r &espnet.nets.pytorch_backend.nets_utilsr r r r(espnet.nets.pytorch_backend.rnn.argumentrrr*espnet.nets.pytorch_backend.rnn.attentionsr(espnet.nets.pytorch_backend.rnn.decodersr(espnet.nets.pytorch_backend.rnn.encodersrespnet.nets.scorers.ctcrespnet.utils.fill_missing_argsrrChainrr9rXr+r%r%r%r&s6