o iI@sddlZddlZddlZddlmZddlmZmZmZddl m Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZmZddlmZe ej e dkrpddl!m"Z"nedddZ"e #ddGdddej$j%Z&dS)N)contextmanager)DictOptionalTuple) LooseVersion)tables)postprocess_utils) DatadirWriter)force_gatherable)CTCPrefixScorer)LabelSmoothingLoss) LengthBonus)get_transducer_task_io)load_audio_text_image_video extract_fbank)BeamSearchTransducerz1.6.0)autocastTccs dVdSN)enabledrrR/home/ubuntu/.local/lib/python3.10/site-packages/funasr/models/transducer/model.pyrs r model_classes Transducerc7seZdZ                          dCd eed eed eed eededeededeededeededeededededededededededed ed!ed"ed#ed$ef6fd%d& Z d'e j d(e j d)e j d*e j d+e e j eee j fe j ff d,d-Z d'e j d(e j d+e e j e j ffd.d/Zd0e j d1e j d2e j d3e j d4e j d+e e j eeeeff d5d6Zd0e j d2e j d3e j d4e j d+e j f d7d8Zd9e j d2e j d+e j fd:d;Zded?ed@efdAdBZZS)ErN?PrFfrontend frontend_confspecaug specaug_conf normalizenormalize_confencoder encoder_confdecoder decoder_conf joint_networkjoint_network_conftransducer_weightfastemit_lambdaauxiliary_ctc_weightauxiliary_ctc_dropout_rateauxiliary_lm_loss_weightauxiliary_lm_loss_smoothing input_size vocab_size ignore_idblank_idsoseos lsm_weightlength_normalized_lossshare_embeddingc$ st|durtj|}|di|}|dur'tj|}|di|}tj|}|dd|i|}|} tj| }!|!dd|i| } | j}"tj | }#|#|| |"fi| } d|_ d|_ |dk|_ |dk|_ |j r|tj|||_||_|j rtj| j||_||_| |_||_||_||_||_|dur|n|d|_|dur|n|d|_||_||_||_||_||_ ||_!| |_"| |_#t$||||d|_%||_&d|_'d|_(d|_)dS)Nr1r2rr)size padding_idx smoothingnormalize_lengthrr)*super__init__rspecaug_classesgetnormalize_classesencoder_classes output_sizedecoder_classesjoint_network_classescriterion_transducererror_calculatoruse_auxiliary_ctcuse_auxiliary_lm_losstorchnnLinearctc_linctc_dropout_ratelm_linlm_loss_smoothingr+r,r-r/r4r5r6r2r3rr!r#r%r'r)r criterion_attr8 beam_searchctc ctc_weight)$selfrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9kwargs specaug_classnormalize_class encoder_classencoder_output_size decoder_classdecoder_output_sizejoint_network_class __class__rrr?$sx '        zTransducer.__init__speechspeech_lengthstext text_lengthsreturncKst|dkr|dddf}t|dkr |dddf}|jd}|||\}}t|jdrE|jjdurE|jjj||dd\}}t|||j d\} } } } |j |j | | | } | |d| d}|||| | | \}}}d\}}|jr||| | | }|jr|| | }|j||j||j|}t|||d kr|nd|d kr|nd||d }t|||f|j \}}}|||fS) zEncoder + Decoder + Calc loss Args: speech: (Batch, Length, ...) speech_lengths: (Batch, ) text: (Batch, Length) text_lengths: (Batch,) rNroverlap_chunk_cls) chunk_outs)r3r)rrr)lossloss_transducer aux_ctc_loss aux_lm_losscer_transducerwer_transducer)lenr:shapeencodehasattrr%rf remove_chunkrr3r' set_devicedevicer) unsqueeze_calc_transducer_lossrI_calc_ctc_lossrJ _calc_lm_lossr+r-r/dictdetachr )rVrarbrcrdrW batch_size encoder_outencoder_out_lens decoder_intargett_lenu_len decoder_out joint_out loss_trans cer_trans wer_transloss_ctcloss_lmrhstatsweightrrrforwardsj        zTransducer.forwardcKstd%|jdur|jr|||\}}|jdur"|||\}}Wdn1s,wY|||\}}}d}t|trI|d}|d}|durS||f|fS||fS)zFrontend + Encoder. Note that this method is used by asr_inference.py Args: speech: (Batch, Length, ...) speech_lengths: (Batch, ) ind: int FNrr)rr!trainingr#r% isinstancetuple)rVrarbrWr|r}_intermediate_outsrrrrps     zTransducer.encoder|rrrrc Cs|jdur#z ddlm}||_Wnty"tdtdYnwtj|dd}|j||||d|j |j d d }|j sn|j sB|j rn|jdur`dd lm} | |j|j|j|j|j|j |j d |_||||\} } || | fS|ddfS) a(Compute Transducer loss. Args: encoder_out: Encoder output sequences. (B, T, D_enc) joint_out: Joint Network output sequences (B, T, U, D_joint) target: Target label ID sequences. (B, L) t_len: Encoder output sequences lengths. (B,) u_len: Target label ID sequences lengths. (B,) Return: loss_transducer: Transducer loss value. cer_transducer: Character error rate for Transducer. wer_transducer: Word Error Rate for Transducer. Nr) rnnt_losszJwarp-rnnt was not installed.Please consult the installation documentation.rrdimmeanT) reductionblankr,gather)ErrorCalculatorTransducer) report_cer report_wer)rG warp_rnntr ImportErrorloggingerrorexitrK log_softmaxr4r,rrrrHfunasr.metricsrr'r) token_list sym_space sym_blank) rVr|rrrrRNNTLoss log_probsriErrorCalculatorrlrmrrrrv sH         z Transducer._calc_transducer_lossc Cs|tjjj||jd}tj|dddd}|dk}||}tj j j ddtjjj ||||ddd }Wd n1sBwY|| d}|S) aMCompute CTC loss. Args: encoder_out: Encoder output sequences. (B, T, D_enc) target: Target label ID sequences. (B, L) t_len: Encoder output sequences lengths. (B,) u_len: Target label ID sequences lengths. (B,) Return: loss_ctc: CTC loss value. )prrrrT) deterministicsum) zero_infinityrN)rNrKrL functionaldropoutrOr transposecpubackendscudnnflagsctc_lossr:) rVr|rrrctc_in target_mask ctc_targetrrrrrwOs   zTransducer._calc_ctc_lossrcCs||ddddddfd|j}|dtj}t.|}||j |jd|dk}| |d}| d| dd|j Wdn1sSwYtj jjtj|dd|dd}| | dd|d}|S)zCompute LM loss. Args: decoder_out: Decoder output sequences. (B, U, D_dec) target: Target label ID sequences. (B, L) Return: loss_lm: LM loss value. Nrrrrnone)r)rPviewr2typerKint64no_gradclonefill_rQ masked_fillscatter_rurLrkl_divrrr:)rVrr lm_loss_in lm_target true_distignorerrrrrxus *    zTransducer._calc_lm_losscKszi}|jdkrt|j|jd}|j|d|d}|jtt|dd}||d<t|j|j |dddd }||_ dS) N)rTr6)rTr) length_bonusngram beam_sizerr)nbest) rTr r6updaterAr rnrr'r)rS)rVrWscorersrTrrrSrrrinit_beam_searchs"      zTransducer.init_beam_searchdata_in data_lengthskeyc! s|dddkr td|dddkojdk}|dddko(|dddu}td jd&i||d d_i}t} t |j j |d d d } t} | | d|d<t | |ddj d\} } t}|| d|d<| j jj jd|d<| j|dd} | j|dd} | | \}}t|tr|d}j|ddd}|dj}g}|\}}}t|D]}t|D]}\}}d}|ddurtdst|d_j|dd}d}t|jtr|j}n|j}ttfdd |}||}| |}t!"|\}}|||||d!} |#| |durId"$||d#||<||d$||<||d%||<qq||fS)'Nr{rz!batch decoding is not implementeddecoding_ctc_weightrgh㈵> lm_weightlm_filezenable beam_searchrfsi>)raudio_fsz0.3f load_data data_typesound)rr extract_featibatch_data_timert)rtrT)is_final output_dirwriter best_recogrcs|jko|jko|jkSr)r6r5r4)xrVrrsz&Transducer.inference..)rtokenrctext_postprocessed rrcrr)%rANotImplementedErrorrTrinforrtime perf_counterrrrrritem frame_shiftlfr_ntorprrrSr:range enumeraterqr ryseqlistfilter ids2tokens tokens2textrsentence_postprocessappendjoin)!rVrrr tokenizerrW is_use_ctc is_use_lm meta_datatime1audio_sample_listtime2rarbtime3r|r} nbest_hypsresultsbndi nbest_idxhyp ibest_writerlast_pos token_intrrcrrresult_irrr inferences~            &zTransducer.inference)NNNNNNNNNNNNrrrrrrrrrrrrrFF)NNN)__name__ __module__ __qualname__rstrrfloatintboolr?rKTensorrrrprvrwrxrrr  __classcell__rrr_rr"s     !m V $ D & %$)T)'rrKr contextlibrtypingrrrdistutils.versionrfunasr.registerr funasr.utilsrfunasr.utils.datadir_writerr funasr.train_utils.device_funcsr %funasr.models.transformer.scorers.ctcr "funasr.losses.label_smoothing_lossr .funasr.models.transformer.scorers.length_bonusr *funasr.models.transformer.utils.nets_utilsrfunasr.utils.load_utilsrr/funasr.models.transducer.beam_search_transducerr __version__torch.cuda.amprregisterrLModulerrrrrs,