o iwN@sddlZddlZddlmZddlZddlmZddlm Z ddl Z ddl Z ddlmmmZddlmZddlmZdZdZGdddejZd d ZdS) N) Namespace)CTCPrefixScore) end_detectg?csNeZdZdZ     dfdd ZddZd d Zdd d Zd dZZ S)DecoderaMDecoder layer. Args: eprojs (int): Dimension of input variables from encoder. odim (int): The output dimension. dtype (str): Decoder type. dlayers (int): Number of layers for decoder. dunits (int): Dimension of input vector of decoder. sos (int): Number to indicate the start of sequences. eos (int): Number to indicate the end of sequences. att (Module): Attention module defined at `espnet.espnet.nets.chainer_backend.attentions`. verbose (int): Verbosity level. char_list (List[str]): List of all characters. labeldist (numpy.array): Distributed array of counted transcript length. lsm_weight (float): Weight to use when calculating the training loss. sampling_probability (float): Threshold for scheduled sampling. rNc stt||Kt|||_|dkrt|||nt ||||_ t j d|D]}t|d||dkr@t||nt ||q/t|||_Wdn1sYwY||_d|_||_||_||_||_||_| |_| |_| |_d|_| |_| |_dS)Nlstmrnn%d)superr__init__ init_scopeDLEmbedIDembedL StatelessLSTM StatelessGRUrnn0sixmovesrangesetattrLinearoutputdtypelossattdlayersdunitssoseosverbose char_list labeldist vlabeldist lsm_weightsampling_probability)selfeprojsodimrrrr r!rr"r#r$r&r'i __class__\/home/ubuntu/.local/lib/python3.10/site-packages/espnet/nets/chainer_backend/rnn/decoders.pyr (s<   zDecoder.__init__c Cs|jdkr=||d|d|\|d<|d<tjd|jD]}|d|||||||d\||<||<q||fS|ddurq|j}tjj |j t |j |jd|jf|jd|d<Wdn1slwY||d||d<tjd|jD]O}||dur|j}tjj |j #t |j ||djd|jf||djd||<Wdn1swY|d|||||d||<q||fS)Nrrr r )r)rrrrrrxpchainerbackendscudaget_device_from_id _device_idVariablezerosshaper)r(eyz_listc_listz_prevc_prevr+r0r.r.r/ rnn_forwardXs8 "      $zDecoder.rnn_forwardcsd_jjgdjjgdfdd|D}fdd|D}tj|jd}tj|dd}|jd}|jd }t j j d t jd d|Dt j j d t jd d|Ddg} dg} t jd jD] } | d| dqzd} g} j|}tj|d d}t j|D]V}|| d| \}} |dkrtjkrt d| d}tjt|d d}|}t||f}n t|||f}|| | | | \} } | | dqtj| d d||j} | }t |t!|_jt"#dd|Dd 9_tj$|t!|dd}t dt jj%j&dkrj'dur|||d}|}t(t)|j%|j%D]V\\}}}|t*krhnJjj||dkd d}||dk}fdd|D}fdd|D}d+|,dd}d+|,dd}t d||t d||q[j-durj.durt/0j1j-_.t2tj3t|j.d d t4|}dj5jj5|_j|fS)aCore function of Decoder layer. Args: hs (list of chainer.Variable | N-dimension array): Input variable from encoder. ys (list of chainer.Variable | N-dimension array): Input variable of decoder. Returns: chainer.Variable: A variable holding a scalar array of the training loss. chainer.Variable: A variable holding a scalar array of the accuracy. Nr+cg|] }tj|gddqSraxisFconcat.0yr r.r/ z$Decoder.__call__..cg|] }tj|gddqSr@rCrFr!r.r/rJrKpaddingrr z input lengths: cSg|]}|jdqSrr8)rGhr.r.r/rJz output lengths: cSrQrRrSrFr.r.r/rJrUrAz scheduled sampling cSsg|]}t|qSr.)lenrGxr.r.r/rJs) ignore_labelz att loss:cg|] }jt|qSr.r#intrGidxr(r.r/rJcrZr.r[r]r_r.r/rJr` zgroundtruth[%d]: zprediction [%d]: ?)6rr0arrayr!r rD pad_sequencer8logginginfor-__name__strrrrrappendrresetrseparaterandomr'rargmax log_softmaxhstackr>stackreshapersoftmax_cross_entropyflattennpmeanaccuracydatar"r#zip enumerateMAX_DECODER_OUTPUTjoinreplacer$r%r1r6asarraysumscalerVr&)r(hsysys_inys_out pad_ys_in pad_ys_outbatcholengthr;r:_att_wz_alleysr+att_cz_outr9y_allaccy_haty_truey_hat_y_true_idx_hatidx_trueseq_hatseq_trueloss_regr.)r!r(r r/__call__ts         "     zDecoder.__call__c' stdt|jddg}dg}tjd|jD] }|d|dqd} |j |j } |j } |j } |jd|jd} |jdkrK|jd}n tdt|j|jd}t|j|jd}tdt|tdt||rd| g||| dd }n d| g||| d }|durt|d|j|j}||d <d|d <| d krt|jdt| t}n|jd}|g}g}tj|D]^}tdt|g}|D]a}||d|}| |g|dd|d\}}t||f}|||||d|d\}}t| |dj!}|r(|"|d|d|\}}||j#|}n|}|dur|jj$|ddddddfd|}||d||d \}} d | |dd|f| ||d }|rr||j#|dd|f7}|jj$|ddddddfd| }!|dd|!f}"||!}n|jj$|ddddddfd| }|dd|f}"tj| D]s}#i}$|dd|$d<|dd|$d<||$d<|d|"d|#f|$d<dgdt%|d|$d<|d|$ddt%|d<|jd||#d|$dt%|d<|r ||$d<|dur | |!|#|$d <||!|#|$d <||$qt&|ddddd| }q|}tdtt%|tdd'fdd|ddddD(d d!||dkrtd"|D]}|d|jd|jdqmg}%|D]A}|dd|jkrt%|d|kr|d|d| 7<|r|d|j#|)|d7<||q|%|qt*||r|jdkrtd#|nI|%}t%|dkrtd$tt%|ntd%n-|D]}td&d'fd'd|dddD(d d!qtd(tt%|qt&|d)ddddtt%||j+}&t%|&dkr\t,d*t-d.it.|}td|jd+|_|/||||Std,t|&ddtd-t|&ddt%|&dd|&S)/aBeam search implementation. Args: h (chainer.Variable): One of the output from the encoder. lpz (chainer.Variable | None): Result of net propagation. recog_args (Namespace): The argument. char_list (List[str]): List of all characters. rnnlm (Module): RNNLM module. Defined at `espnet.lm.chainer_backend.lm` Returns: List[Dict[str,Any]]: Result of recognition. zinput lengths: rNr r+zmax output length: zmin output length: r)scoreyseqr=r<a_prev rnnlm_prev)rrr=r<rctc_state_prevctc_score_prevrdrPz position rr<rr=rrArcS|dSNrr.rXr.r.r/hz(Decoder.recognize_beam..T)keyreverseznumber of pruned hypotheses: z best hypo: racg|]}t|qSr.r\rWr#r.r/rJpz*Decoder.recognize_beam..rbrcz-adding in the last position in the loopzend detected at %dzremaining hypotheses: zno hypothesis. Finish decoding.zhypo: crr.rrWrr.r/rJrznumber of ended hypotheses: cSrrr.rr.r.r/rrzOthere is no N-best results, perform recognition again with smaller minlenratio.g?ztotal log probability: znormalized log probability: r.)0rgrhrjr8rrrrrkrrl beam_sizepenalty ctc_weightr0fullr maxlenratiomaxr\ minlenratiorr! initial_stateminCTC_SCORING_RATIOdebugrrDrqr>rprrypredict lm_weightargsortrVsortedr}r~finalrnbestwarningrvarsrecognize_beam)'r(rTlpz recog_argsr#rnnlmr;r:rabeamrrrHmaxlenminlenhypctc_prefix_scorectc_beamhyps ended_hypsr+hyps_best_keptr9rrlocal_att_scores rnnlm_statelocal_lm_scores local_scoreslocal_best_ids ctc_scores ctc_statesjoint_best_idslocal_best_scoresjnew_hyp remained_hyps nbest_hypsr.rr/rsJ                 (     $zDecoder.recognize_beamcsF|j|jgd|j|jgdfdd|D}fdd|D}tj||jd}tj|dd}|jd}dg}dg} tj d|j D] } | d| dqFd} g} |j ||} tj| dd } tj |D]&}| || d | \}} t| ||f}||| || |\} }| | qntj| dd } | | jS) aZCalculate all of attentions. Args: hs (list of chainer.Variable | N-dimensional array): Input variable from encoder. ys (list of chainer.Variable | N-dimensional array): Input variable of decoder. Returns: chainer.Variable: List of attention weights. r+cr?r@rCrFrIr.r/rJrKz4Decoder.calculate_all_attentions..crLr@rCrFrMr.r/rJrKrNrPr NrAr)r0rer!r rDrfr8rrrrrkrrlrrmrqr>rrto_cpury)r(rrrrrrrr;r:rratt_wsrr+rr9r.)r!r r/calculate_all_attentionss2      z Decoder.calculate_all_attentions)rNNrr)N) ri __module__ __qualname____doc__r r>rrr __classcell__r.r.r,r/rs0 g`rcCs0t|j||j|j|j||||j|j||j|j S)aReturn the decoding layer corresponding to the args. Args: args (Namespace): The program arguments. odim (int): The output dimension. sos (int): Number to indicate the start of sequences. eos (int) Number to indicate the end of sequences. att (Module): Attention module defined at `espnet.nets.chainer_backend.attentions`. labeldist (numpy.array): Distributed array of length od transcript. Returns: chainer.Chain: The decoder module. ) rr)rrrr"r#r&r')argsr*r r!rr$r.r.r/ decoder_forsr)rgrnargparserr1chainer.functions functionsrD chainer.linkslinksrnumpyrvr2espnet.nets.chainer_backend.deterministic_embed_idnetschainer_backenddeterministic_embed_idrespnet.nets.ctc_prefix_scorerespnet.nets.e2e_asr_commonrrr|Chainrrr.r.r.r/s$      `