o i)I@sdddlZddlmZddlmZddlmZddlmZddlmZddlm Z ddlm Z ddl Z ddl m mZdd lmZdd lmZmZdd lmZdd lmZdd lmZddlmZddlmZddlm Z ddl!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,ee j-edkrddl.m/Z/nedddZ/Gddde,Z0dS)N)contextmanager) LooseVersion)Dict)List)Optional)Tuple)Union) AbsNormalize)LabelSmoothingLossNllLoss)CTC) AbsDecoder) AbsEncoder) AbsFrontend)AbsPostEncoder) AbsPreEncoder) AbsSpecAug) add_sos_eos)ErrorCalculator) th_accuracy)force_gatherable) FunASRModelz1.6.0)autocastTccs dVdS)N)enabledrrS/home/ubuntu/.local/lib/python3.10/site-packages/funasr/models/sa_asr/e2e_sa_asr.pyr#s rc+s8eZdZdZ           dCd ed ed eeed feefd e e de e de e de dejjdededededededededededededef*fdd Zd!ejd"ejd#ejd$ejd%ejd&ejd'ejd(ejd)eejeeejfejffd*d+Zd!ejd"ejd#ejd$ejd)eeejff d,d-Zd!ejd"ejd)eejejffd.d/Zd!ejd"ejd)eejejffd0d1Zd2ejd3ejd4ejd5ejd)ejf d6d7Z 8dDd2ejd3ejd4ejd5ejd9ef d:d;Zdejd'ejd(ejfd?d@Zd2ejd3ejd4ejd5ejfdAdBZZ S)E SAASRModelz*CTC-attention hybrid Encoder-Decoder model?FT vocab_size max_spk_num token_list.frontendspecaug normalize asr_encoder spk_encoderdecoderctc spk_weight ctc_weightinterctc_weight ignore_id lsm_weightlength_normalized_loss report_cer report_wer sym_space sym_blankextract_feats_in_collect_statscs\d| kr dksJ| J| d| krdks$J| J| td|_d|_d|_||_||_||_| |_| |_ | |_ | |_ ||_ ||_||_||_||_t|jdsbd|j_|jjrrtj||j|j_d|_| dkr}d|_n| |_t||||d|_t|||d |_|s|rt||||||_| dkrd|_ n| |_ ||_!dS) Nr?rinterctc_use_conditioningF)size padding_idx smoothingnormalize_length)r;r<r>)"super__init__blank_idsoseosr"r#r/r,r-r.copyr$r%r&r'r(r)hasattrr:torchnnLinear output_sizeconditioning_layererror_calculatorr*r criterion_attr criterion_spkrr+r6)selfr"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6 __class__rrr@+s^$$      zSAASRModel.__init__speechspeech_lengthstext text_lengthsprofileprofile_lengthstext_idtext_id_lengthsreturnc  Cs|dks J|j|jd|jdkr&|jdkr&|jdks3nJ|j|j|j|jf|jd} |ddd|f}|||\} } } d} t| tr\| d} | d} d\}}}}}}d\}}t}|jdkrz|| | ||\}}d}|j dkr| dur| D](\}}||| ||\}}||}|dur| nd|d |<||d |<q|t | }d|j ||j |}|jd kr| | | | |||||| \}}}}}}|jdkr|}n|jd kr|}n |j|d|j|}|jdkr|}n |j|d|j|}t| | |dur| nd|dur$| nd|dur.| nd|||||d }t||| f|j\}}}|||fS) a$Frontend + Encoder + Decoder + Calc loss Args: speech: (Batch, Length, ...) speech_lengths: (Batch, ) text: (Batch, Length) text_lengths: (Batch,) profile: (Batch, Length, Dim) profile_lengths: (Batch,) r8rN)NNNNNNNNrzloss_interctc_layer{}zcer_interctc_layer{}r7) lossloss_asrloss_attloss_ctcloss_spkaccacc_spkcerwercer_ctc)dimshapemaxencode isinstancetupledictr-_calc_ctc_lossr.detachformatlen_calc_att_lossr,rdevice)rNrQrRrSrTrUrVrWrX batch_sizeasr_encoder_outencoder_out_lensspk_encoder_outintermediate_outsr]r_acc_attracer_attwer_attr^rdstats loss_interctc layer_idxintermediate_outloss_iccer_icr\r[weightrrrforwards:           zSAASRModel.forwardcCs>|jr |||\}}ntd|j||}}||dS)NzkGenerating dummy stats for feats and feats_lengths, because encoder_conf.extract_feats_in_collect_stats is )feats feats_lengths)r6_extract_featsloggingwarning)rNrQrRrSrTrrrrr collect_featss  zSAASRModel.collect_featsc Cstd1|||\}}|}|jdur!|jr!|||\}}|jdur.|||\}}Wdn1s8wY|jjrN|j|||jd\}}}n |||\}}}d} t |t rf|d} |d}| ||d} | d| dkrt j| dd| ddd dd} n| } | d| dksJ| | df| d|ksJ| |f| d| dksJ| | df| dur|| f|| fS||| fS) zFrontend + Encoder. Note that this method is used by asr_inference.py Args: speech: (Batch, Length, ...) speech_lengths: (Batch, ) FN)r+r8rrnearest)r;mode)rrcloner&trainingr'r(r:r+rirjr)r;F interpolate transposerg) rNrQrRrr feats_raw encoder_outrt_rvencoder_out_spk_oriencoder_out_spkrrrrh sN    zSAASRModel.encodecCsb|dks J|j|ddd|f}|jdur(|||\}}||fS||}}||fS)Nr8)rerfrgr%)rNrQrRrrrrrrFs  zSAASRModel._extract_featsrrtys_pad ys_pad_lensc Cst||j|j|j\}}|d}|||||\}} |d} |d} tjjj | d| | d|jdd} | | d} | j dd} | d| ksLJ| S)a,Compute negative log likelihood(nll) from transformer-decoder Normally, this function is called in batchify_nll. Args: encoder_out: (Batch, Length, Dim) encoder_out_lens: (Batch,) ys_pad: (Batch, Length) ys_pad_lens: (Batch,) r8rr9rnone) ignore_index reductionre) rrBrCr/r*r;rFrG functional cross_entropyviewsum) rNrrtrr ys_in_pad ys_out_pad ys_in_lens decoder_outrrrdecoder_num_classnllrrrrYs"     zSAASRModel.nlldrrcCs|d}||kr|||||}nIg}d} t|||} ||| ddddf} ||| } ||| ddf} ||| } || | | | }||| }||krUnqt|}|d|ksdJ|S)a3Compute negative log likelihood(nll) from transformer-decoder To avoid OOM, this fuction seperate the input into batches. Then call nll for each batch and combine and return results. Args: encoder_out: (Batch, Length, Dim) encoder_out_lens: (Batch,) ys_pad: (Batch, Length) ys_pad_lens: (Batch,) batch_size: int, samples each batch contain when computing nll, you may change this to avoid OOM or increase GPU memory usage rTN)r;rminappendrFcat)rNrrtrrrr total_numr start_idxend_idxbatch_encoder_outbatch_encoder_out_lens batch_ys_padbatch_ys_pad_lens batch_nllrrr batchify_nlls2     zSAASRModel.batchify_nllrsru profile_lensc  Cst||j|j|j\} } |d} ||||| | ||\} }}|d}d|j|f}tj||ddd}| | | }| t ||}t |d|j||jd}t | d|j| |jd}|jsd|jdurid\}}n| jdd}|||\}}||||||fS) Nr8rrconstant)rvalue) ignore_labelrZr)rrBrCr/r*r;r#rpadrLrMrFlogrrr"rrKargmaxcpu)rNrsrurtrrrUrrWrXrrrrweights_no_padrspk_num_no_padrweightsr]r_rarwrxryys_hatrrrrps>       zSAASRModel._calc_att_losscCsR|||||}d}|js%|jdur%|j|j}|j||dd}||fS)NT)is_ctc)r+rrKrdatar)rNrrtrrr^rdrrrrrls zSAASRModel._calc_ctc_loss) rrrrrFTTr r!T)r)!__name__ __module__ __qualname____doc__intrrstrrrrrr rrFrGModuler r floatboolr@Tensorrrrrhrrrrprl __classcell__rrrOrr(s.     Y  r   =  , /  ?r)T)1r contextlibrdistutils.versionrtypingrrrrrrFtorch.nn.functionalrGrrfunasr.layers.abs_normalizer "funasr.losses.label_smoothing_lossr r funasr.models.ctcr !funasr.models.decoder.abs_decoderr !funasr.models.encoder.abs_encoderrfunasr.frontends.abs_frontendr)funasr.models.postencoder.abs_postencoderr'funasr.models.preencoder.abs_preencoderr!funasr.models.specaug.abs_specaugr+funasr.models.transformer.utils.add_sos_eosrfunasr.metricsrfunasr.metrics.compute_accrfunasr.train_utils.device_funcsrfunasr.models.base_modelr __version__torch.cuda.amprrrrrrs: