o ie@shdZddlmZddlZddlZddlZddlZddlmZddl m Z m Z m Z m Z mZmZmZddlZddlZddlmZddlmZmZddlmZmZdd lmZdd lm Z dd l!m"Z"dd l#m$Z$dd l%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.ddl/m0Z0m1Z1m2Z2ddl3m4Z4GdddZ5dGd@dAZ6dBdCZ7dHdDdEZ8e9dFkre8dSdS)Iz2 Inference class definition for Transducer models.) annotationsN)Path)AnyDictListOptionalSequenceTupleUnion)parse)check_argument_typescheck_return_type)BeamSearchTransducer Hypothesis)TooShortUttError) DatadirWriter)ASRTransducerTask)LMTask)build_tokenizer)TokenIDConverter) to_device)set_all_random_seed)config_argparse)str2boolstr2triple_str str_or_none)get_commandline_argscseZdZdZ                dEdFfd*d+ ZdGd,d-Z dHdId2d3Ze 4dJdKd7d8Z edLd9d:Z dMd=d>Z e dNdOdCdDZ ZS)P Speech2TextaSpeech2Text class for Transducer models. Args: asr_train_config: ASR model training config path. asr_model_file: ASR model path. beam_search_config: Beam search config path. lm_train_config: Language Model training config path. lm_file: Language Model config path. token_type: Type of token units. bpemodel: BPE model path. device: Device to use for inference. beam_size: Size of beam during search. dtype: Data type. lm_weight: Language model weight. quantize_asr_model: Whether to apply dynamic quantization to ASR model. quantize_modules: List of module names to apply dynamic quantization on. quantize_dtype: Dynamic quantization data type. nbest: Number of final hypothesis. streaming: Whether to perform chunk-by-chunk inference. chunk_size: Number of frames in chunk AFTER subsampling. left_context: Number of frames in left context AFTER subsampling. right_context: Number of frames in right context AFTER subsampling. display_partial_hypotheses: Whether to display partial hypotheses. Ncpufloat32?Fqint8 rasr_train_configUnion[Path, str]asr_model_filebeam_search_configDict[str, Any]lm_train_configlm_file token_typestrbpemodeldevice beam_sizeintdtype lm_weightfloatquantize_asr_modelboolquantize_modules List[str]quantize_dtypenbest streaming chunk_size left_context right_contextdisplay_partial_hypothesesreturnNonec stts Jt|||\}}| rW| dur0tdd| Ds&tdtdd| D}ntj j h}|dkrFt tj t dkrFtdt t|}tjj|||d }n |jt t| d |durst|||\}}|j}nd}|dur{i}t|j|j| f|| |d |}|j}|dur|j}|dur|j}|durd}n|d kr|durt||d }nd}nt|d }t|d}td|||_||_||_ | |_!||_"||_#||_$||_%||_&t'|d|_(t'|d|_)t'|d|_*|r|dkrd|_&d|jj+_,|j-.dd|_/|j-.dd|_0|j-.dddur$|j-d|_1n|j/|_1|j(|j*|_2|jj+3|j2|j0|_4|jj+j5j6|j*d|j0|_7|8dS)zConstruct a Speech2Text object.NcSsg|]}|dvqS))LSTMLinear.0qrErEX/home/ubuntu/.local/lib/python3.10/site-packages/espnet2/bin/asr_transducer_inference.py asz(Speech2Text.__init__..z]Only 'Linear' and 'LSTM' modules are currently supported by PyTorch and in --quantize_modulescSsg|]}ttj|qSrE)getattrtorchnnrFrErErIrJgsfloat16z1.5.0zufloat16 dtype for dynamic quantization is not supported with torch version < 1.5.0. Switching to qint8 dtype instead.r3)lmr4r;bpe)r-r/)r-) token_listzText tokenizer: rFn_ffti hop_length win_lengthr#)9super__init__r rbuild_model_from_fileall ValueErrorsetrLrMrDV __version__rK quantizationquantize_dynamicevaltorrPrdecoder joint_networkrRr-r/rrlogginginfo asr_modelasr_train_argsr0r3r; converter tokenizer beam_searchr<maxr=r>r?encoderdynamic_chunk_training frontend_confgetrSrTfrontend_window_size window_sizeget_encoder_input_raw_size_raw_ctxembedmin_frame_lengthlast_chunk_lengthreset_inference_cache) selfr&r(r)r+r,r-r/r0r1r3r4r6r8r:r;r<r=r>r?r@rgrhq_configq_dtyperP lm_train_args lm_scorerrkrRrjri __class__rErIrX?s            zSpeech2Text.__init__cCs@d|_|jjj|j|jd|jtj dgg|jd|_ dS)zReset Speech2Text parameters.Nr0r) frontend_cachergrmreset_streaming_cacher>r0rkrxrLtensornum_processed_frames)ryrErErIrxs  z!Speech2Text.reset_inference_cachespeech torch.Tensoris_final!Tuple[torch.Tensor, torch.Tensor]c Cs|jdurtj|jd|gdd}|r:|jr5|d|jkr5tj|j|d|jd}tj||gdd}|}d}nF|d|j|j |j }|d|j|j |j }| dd|j|j ||j }| d|d|j|j ||j|j | }| d tt|j}|jdgtj|dd}||d} t| |jd } |jjd i| \} } |jjdur|j| | \} } |r|jdurnj| dtt|j|j d | dtt|j|j d } nE|jdur | dd| dtt|j|j d } n&| dtt|j|j d | dd tt|j|j d } | jdgtj| dd} |rHd|_| | fSd|i|_| | fS) a-Forward frontend. Args: speech: Speech data. (S) is_final: Whether speech corresponds to the final (or only) chunk of data. Returns: feats: Features sequence. (1, T_in, F) feats_lengths: Features sequence length. (1, T_in, F) Nwaveform_bufferr)dimrOr#)r3 fill_value)rspeech_lengthsrrE)rrLcatr<sizerwzerosr3rqrTnarrowclone unsqueezerbrKnew_fulllongrr0rg_extract_feats normalizemathceil) ryrrpadspeech_to_processrn_frames n_residuallengthsbatchfeats feats_lengthsrErErIapply_frontends            zSpeech2Text.apply_frontendTUnion[torch.Tensor, np.ndarray]List[Hypothesis]cCsxt|tjr t|}|j||d\}}|jjj|||j |j |j d}|j |d|d}|j |j 7_ |r:||S)zSpeech2Text streaming call. Args: speech: Chunk of speech data. (S) is_final: Whether speech corresponds to the final chunk of data. Returns: nbest_hypothesis: N-best hypothesis. r)r>r?r) isinstancenpndarrayrLrrrgrm chunk_forwardrr>r?rkr=rx)ryrrr feats_lengthenc_out nbest_hypsrErErIstreaming_decode?s  zSpeech2Text.streaming_decodecCsRtsJt|tjrt|}||\}}|j||\}}| |d}|S)zSpeech2Text call. Args: speech: Speech data. (S) Returns: nbest_hypothesis: N-best hypothesis. r) r rrrrLrrrgrmrk)ryrrrr_rrErErI__call__es  zSpeech2Text.__call__r List[Any]cCslg}|D]/}ttdd|j}|j|}|jdur"|j|}nd}|||||ft|s3Jq|S)zBuild partial or final results from the hypotheses. Args: nbest_hyps: N-best hypothesis. Returns: results: Results containing different representation for the hypothesis. cSs|dkS)NrrExrErErIz3Speech2Text.hypotheses_to_results..N) listfilteryseqri ids2tokensrj tokens2textappendr )ryrresultshyp token_inttokentextrErErIhypotheses_to_results|s   z!Speech2Text.hypotheses_to_results model_tag Optional[str]kwargs Optional[Any]cKs^|dur(zddlm}Wn tytdw|}|jdi||tdi|S)zBuild Speech2Text instance from the pretrained model. Args: model_tag: Model tag of the pretrained models. Return: : Speech2Text instance. Nr)ModelDownloaderzZ`espnet_model_zoo` is not installed. Please install via `pip install -U espnet_model_zoo`.rE)espnet_model_zoo.downloaderr ImportErrorreerrorupdatedownload_and_unpackr)rrrdrErErIfrom_pretraineds zSpeech2Text.from_pretrained)NNNNNNNrrr r!FNr"r#Fr$r%rF)*r&r'r(r'r)r*r+r'r,r'r-r.r/r.r0r.r1r2r3r.r4r5r6r7r8r9r:r.r;r2r<r7r=r2r>r2r?r2r@r7rArB)rArB)F)rrrr7rAr)T)rrrr7rAr)rrrAr)rrrArN)rrrrrAr)__name__ __module__ __qualname____doc__rXrxrrLno_gradrrr staticmethodr __classcell__rErEr~rIr$sH  m%  r output_dirr. batch_sizer2r3r1ngpuseedr4r5r; num_workers log_levelUnion[int, str]data_path_and_name_and_typeSequence[Tuple[str, str, str]]r&rr(r)Optional[dict]r+r,rr-r/key_fileallow_variable_data_keysr7r6Optional[bool]r8Optional[List[str]]r:r<r= Optional[int]r>r?r@rArBc3 CsdtsJ|dkr td|dkrtdtj| dd|dkr#d}nd}t|td3id| d | d | d |d |d |d|d|d|d|d|d|d|d|d|d|d|d|d|}tjd3d|i|}tj | ||||t |j dt |j d|dd } t |}!| D] \}"}#t|#tsJt|#tdd |"DsJ|"ttt|#}$t|"|$ksJt|"d!|$d"d#|#D}#t|#dksJzI|jr|#d$}%t|%|j}&d%}'t|&D]}(|(d|j}'|j|%|(|j|'dd&q|j|%|'t|%dd&})n|d3i|#})||)}*Wn0tyU}+z#td'|"d(|+td)gd*d+},d(d,gd-g|,gg|}*WYd*}+~+nd*}+~+ww|"d%}-ttd|d|*D]9\}.\}/}0}1},|!|.d.}2d( |0|2d/|-<d( t!t"|1|2d0|-<t"|,j#|2d1|-<|/d*ur|/|2d2|-<qdqWd*d*S1swYd*S)4aTransducer model inference. Args: output_dir: Output directory path. batch_size: Batch decoding size. dtype: Data type. beam_size: Beam size. ngpu: Number of GPUs. seed: Random number generator seed. lm_weight: Weight of language model. nbest: Number of final hypothesis. num_workers: Number of workers. log_level: Level of verbose for logs. data_path_and_name_and_type: asr_train_config: ASR model training config path. asr_model_file: ASR model path. beam_search_config: Beam search config path. lm_train_config: Language Model training config path. lm_file: Language Model path. model_tag: Model tag. token_type: Type of token units. bpemodel: BPE model path. key_file: File key. allow_variable_data_keys: Whether to allow variable data keys. quantize_asr_model: Whether to apply dynamic quantization to ASR model. quantize_modules: List of module names to apply dynamic quantization on. quantize_dtype: Dynamic quantization data type. streaming: Whether to perform chunk-by-chunk inference. chunk_size: Number of frames in chunk AFTER subsampling. left_context: Number of frames in left context AFTER subsampling. right_context: Number of frames in right context AFTER subsampling. display_partial_hypotheses: Whether to display partial hypotheses. r#z!batch decoding is not implementedz%only single GPU decoding is supportedz>%(asctime)s (%(module)s:%(lineno)d) %(levelname)s: %(message)s)levelformatcudarr&r(r)r+r,r-r/r0r3r1r4r;r6r8r:r<r=r>r?rFT)r3rrr preprocess_fn collate_fnr inferencecss|]}t|tVqdSr)rr.)rGsrErErI :szinference..z != cSs$i|]\}}|ds||dqS)_lengthsr)endswith)rGkvrErErI >s$zinference..rrrz Utterance  gN)scorer dec_statezr best_recogrrrrrE)$r NotImplementedErrorre basicConfigrdictrrrbuild_streaming_iteratorbuild_preprocess_fnrhbuild_collate_fnrrtyperZlennextitervaluesitemskeysr<rtrangerrrwarningrzipjoinmapr.r)3rrr3r1rrr4r;rrrr&r(r)r+r,rr-r/rrr6r8r:r<r=r>r?r@r0speech2text_kwargs speech2textloaderwriterrr_bsr_steps_endi final_hypsrerkeynrrr ibest_writerrErErIrs A      " "$  $rcCs\tjdtjd}|jddddddd |jd td d |jd tddd|jdtddd|jddgddd|jdtddd|d}|jdtd dd|jdt d |jd!t d"d#|d$}|jd%td&d'|jd(td)d'|jd*td+d'|jd,td-d'|jd.td/d'|d0}|jd1tdd2d|jd3tdd4d|jd5td6d7d|jd8t d9d:d|jd;id}|jd?t d@gdAdBd |jdCt d@dDd|dE}|jdFt d"dGd|jdHdId@dJdK|jdLtdMdNdMgdOd |dP}|jdQt d"dRd|jdStdTdUd|jdVtdWdXd|jdYtddZd|jd[t d"d\d|S)]z&Get Transducer model inference parser.zASR Transducer Decoding) descriptionformatter_classz --log_levelcSs|Sr)upperrrErErIrqrzget_parser..INFO)CRITICALERRORWARNINGrDEBUGNOTSETzThe verbose level of logging)rdefaultchoiceshelpz --output_dirT)rrequiredz--ngpurz(The number of gpus. 0 indicates CPU mode)rrrz--seedz Random seedz--dtyper )rNr float64z Data type)rrrz --num_workersr#z)The number of workers used for DataLoaderzInput data relatedz--data_path_and_name_and_typer)rr actionz --key_file)rz--allow_variable_data_keysF)rrzThe model configuration relatedz--asr_train_configzASR training configuration)rrz--asr_model_filezASR model parameter filez--lm_train_configzLM training configurationz --lm_filezLM parameter filez --model_tagz[Pretrained model tag. If specify this option, *_train_config and *_file will be overwrittenzBeam-search relatedz --batch_sizezThe batch size for inferencez--nbestzOutput N-best hypothesesz --beam_sizerz Beam sizez --lm_weightr!z RNNLM weightz--beam_search_configz1The keyword arguments for transducer beam search.)rrzText converter relatedz --token_typeN)charrQNzIThe token type for ASR model. If not given, refers from the training argsz --bpemodelzLThe model path of sentencepiece. If not given, refers from the training argszDynamic quantization relatedz--quantize_asr_modelz(Apply dynamic quantization to ASR model.z--quantize_modules*a>Module names to apply dynamic quantization on. The module names are provided as a list, where each name is separated by a comma (e.g.: --quantize-config=[Linear,LSTM,GRU]). Each specified name should be an attribute of 'torch.nn', e.g.: torch.nn.Linear, torch.nn.LSTM, torch.nn.GRU, ...)nargsrrz--quantize_dtyper"rNzDtype for dynamic quantization.zStreaming relatedz --streamingz,Whether to perform chunk-by-chunk inference.z --chunk_sizer$z,Number of frames in chunk AFTER subsampling.z--left_contextr%z@Number of frames in left context of the chunk AFTER subsampling.z--right_contextzANumber of frames in right context of the chunk AFTER subsampling.z--display_partial_hypotheseszFWhether to display partial hypotheses during chunk-by-chunk inference.) rArgumentParserargparseArgumentDefaultsHelpFormatter add_argumentr.r2add_argument_grouprrrr5r7)parsergrouprErErI get_parsergs&       r-cCsFtttjdt}||}t|}|ddtdi|dS)N)fileconfigrE) printrsysstderrr- parse_argsvarspopr)cmdr+argsrrErErImains   r8__main__)rr?rr@r7rArBr):r __future__rr'rerr1pathlibrtypingrrrrrr r numpyrrLpackaging.versionr r] typeguardr r -espnet2.asr_transducer.beam_search_transducerrrespnet2.asr_transducer.utilsrespnet2.fileio.datadir_writerrespnet2.tasks.asr_transducerrespnet2.tasks.lmrespnet2.text.build_tokenizerrespnet2.text.token_id_converterr espnet2.torch_utils.device_funcsr'espnet2.torch_utils.set_all_random_seedr espnet2.utilsrespnet2.utils.typesrrrespnet.utils.cli_utilsrrrr-r8rrErErErIsH  $            3 )