o i0~@s@dZddlZddlZddlZddlmZddlmZmZm Z m Z ddl Z ddl Z ddlZddlmZmZmZmZmZddlmZmZddlmZddlmZdd lmZdd lmZmZdd l m!Z!Gd d d Z"GdddZ#  d de e$e%fde%de%dede de de&de&fddZ'ddZ(d!ddZ)e*dkre)dSdS)"z@Perform CTC segmentation to align utterances within audio files.N)Path)ListOptionalTextIOUnion)CtcSegmentationParametersctc_segmentationdetermine_utterance_segments prepare_textprepare_token_list)check_argument_typescheck_return_type)ASRTask) to_device)config_argparse)str2bool str_or_none)get_commandline_argsc@s`eZdZdZdZdZdZdZdZdZ dZ dZ dZ dZ dZdZdZdZddZdd Zd d ZdS) CTCSegmentationTaskuTask object for CTC segmentation. When formatted with str(·), this object returns results in a kaldi-style segments file formatting. The human-readable output can be configured with the printing options. Properties: text: Utterance texts, separated by line. But without the utterance name at the beginning of the line (as in kaldi-style text). ground_truth_mat: Ground truth matrix (CTC segmentation). utt_begin_indices: Utterance separator for the Ground truth matrix. timings: Time marks of the corresponding chars. state_list: Estimated alignment of chars/tokens. segments: Calculated segments as: (start, end, confidence score). config: CTC Segmentation configuration object. name: Name of aligned audio file (Optional). If given, name is considered when generating the text. utt_ids: The list of utterance names (Optional). This list should have the same length as the number of utterances. lpz: CTC posterior log probabilities (Optional). Properties for printing: print_confidence_score: Includes the confidence score. print_utterance_text: Includes utterance text. NFuttTcKs|jdi|dS)zInitialize the module.N)setselfkwargsrrI/home/ubuntu/.local/lib/python3.10/site-packages/espnet2/bin/asr_align.py__init__PszCTCSegmentationTask.__init__cKs>|D]}|dst||r||durt||||qdS)zUpdate properties. Args: **kwargs: Key-value dict that contains all properties with their new values. Unknown properties are ignored. _N) startswithhasattrsetattr)rrkeyrrrrTs zCTCSegmentationTask.setcsd}tj}jdurfddt|D}n |tjks!Jj}tjD]>\}}||djd}||ddd|dd7}jrT|d|d d 7}jra|dj|7}||d 7}q)|S) z0Return a kaldi-style ``segments`` file (string).Ncsg|] }jd|dqS)r04)name).0irrr hz/CTCSegmentationTask.__str__.. rz.2fz3.4f ) lensegmentsutt_idsrange enumerater$print_confidence_scoreprint_utterance_texttext)routputnum_utts utt_namesr&boundary utt_entryrr'r__str__cs  zCTCSegmentationTask.__str__)__name__ __module__ __qualname____doc__r5ground_truth_matutt_begin_indicestimings char_probs state_listr/configdoner$r0lpzr3r4rrr;rrrrr#s& rc@s0eZdZdZdZdZdZddgZdZddgZ dZ e Z    d.d e eefde eefdededededededefddZddZd/ddZd0ddZede ejejffd d!Zd"d#Zd/d$d%Zed&efd'd(Z  d/de ejejfd)e e!eefde"ed*e"ed+ef d,d-Z#dS)1CTCSegmentationu Align text to audio using CTC segmentation. Usage: Initialize with given ASR model and parameters. If needed, parameters for CTC segmentation can be set with ``set_config(·)``. Then call the instance as function to align text within an audio file. Example: >>> # example file included in the ESPnet repository >>> import soundfile >>> speech, fs = soundfile.read("test_utils/ctc_align_test.wav") >>> # load an ASR model >>> from espnet_model_zoo.downloader import ModelDownloader >>> d = ModelDownloader() >>> wsjmodel = d.download_and_unpack( "kamo-naoyuki/wsj" ) >>> # Apply CTC segmentation >>> aligner = CTCSegmentation( **wsjmodel ) >>> text=["utt1 THE SALE OF THE HOTELS", "utt2 ON PROPERTY MANAGEMENT"] >>> aligner.set_config( gratis_blank=True ) >>> segments = aligner( speech, text, fs=fs ) >>> print( segments ) utt1 utt 0.27 1.72 -0.1663 THE SALE OF THE HOTELS utt2 utt 4.54 6.10 -4.9646 ON PROPERTY MANAGEMENT On multiprocessing: To parallelize the computation with multiprocessing, these three steps can be separated: (1) ``get_lpz``: obtain the lpz, (2) ``prepare_segmentation_task``: prepare the task, and (3) ``get_segments``: perform CTC segmentation. Note that the function `get_segments` is a staticmethod and therefore independent of an already initialized CTCSegmentation object. References: CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition 2020, Kürzinger, Winkelbauer, Li, Watzel, Rigoll https://arxiv.org/abs/2007.09127 More parameters are described in https://github.com/lumaku/ctc-segmentation >NautofixedtokenizeclassicFrr+float32Tasr_train_configasr_model_filefsngpu batch_sizedtypekaldi_style_texttext_converter time_stampsc Ks0tsJ|dkr tdd} |dkrd} n |dkr#tdtdt||| \} } | jtt|d t | d|_ t | d rI| j jj}nd }td |td | jjjd |vritd| |_| |_| |_||_| j|_||_| j|_|jd|| ||d| | jdd|j_dS)aXInitialize the CTCSegmentation module. Args: asr_train_config: ASR model config file (yaml). asr_model_file: ASR model file (pth). fs: Sample rate of audio file. ngpu: Number of GPUs. Set 0 for processing on CPU, set to 1 for processing on GPU. Multi-GPU aligning is currently not implemented. Default: 0. batch_size: Currently, only batch size == 1 is implemented. dtype: Data type used for inference. Set dtype according to the ASR model. kaldi_style_text: A kaldi-style text file includes the name of the utterance at the start of the line. If True, the utterance name is expected as first word at each line. If False, utterance names are automatically generated. Set this option according to your input data. Default: True. text_converter: How CTC segmentation handles text. "tokenize": Use ESPnet 2 preprocessing to tokenize the text. "classic": The text is preprocessed as in ESPnet 1 which takes token length into account. If the ASR model has longer tokens, this option may yield better results. Default: "tokenize". time_stamps: Choose the method how the time stamps are calculated. While "fixed" and "auto" use both the sample rate, the ratio of samples to one frame is either automatically determined for each inference or fixed at a certain ratio that is initially determined by the module, but can be changed via the parameter ``samples_to_frames_ratio``. Recommended for longer audio files: "auto". **ctc_segmentation_args: Parameters for CTC segmentation. r+z!Batch decoding is not implementedcpucudazMulti-GPU not yet implemented.z%Only single GPU decoding is supported)rTFencoderUnknownzEncoder module: zCTC module: rnnz6No RNN model detected; memory consumption may be high.)rQrWrUrVNr)r NotImplementedErrorloggingerrorrbuild_model_from_filetogetattrtorchevalbuild_preprocess_fn preprocess_fnrrZ __class__r=infoctclowerwarning asr_modelasr_train_argsdevicerTrU token_list set_configrE char_list)rrOrPrQrRrSrTrUrVrWctc_segmentation_argsrormrnencoder_modulerrrrsJ ,     zCTCSegmentation.__init__cKsd|vr|d|jvrtdt|j|d|_d|vr%t|d|_d|vr0t|d|_d|vrCt|dts=J|d|j _ d|vrOt |d|j _ d|vrat|dt s\J|d|_ d|vr{|d|jvrvtd t|j|d|_d |vrt|d tsJ|d |j _d |vrt|d tsJ|d |j _d |vrt |d |j _|j jr|j j r|jstd d|_d|vrt|dtsJ|d|j _dSdS)asSet CTC segmentation parameters. Parameters for timing: time_stamps: Select method how CTC index duration is estimated, and thus how the time stamps are calculated. fs: Sample rate. samples_to_frames_ratio: If you want to directly determine the ratio of samples to CTC frames, set this parameter, and set ``time_stamps`` to "fixed". Note: If you want to calculate the time stamps as in ESPnet 1, set this parameter to: ``subsampling_factor * frame_duration / 1000``. Parameters for text preparation: set_blank: Index of blank in token list. Default: 0. replace_spaces_with_blanks: Inserts blanks between words, which is useful for handling long pauses between words. Only used in ``text_converter="classic"`` preprocessing mode. Default: False. kaldi_style_text: Determines whether the utterance name is expected as fist word of the utterance. Set at module initialization. text_converter: How CTC segmentation handles text. Set at module initialization. Parameters for alignment: min_window_size: Minimum number of frames considered for a single utterance. The current default value of 8000 corresponds to roughly 4 minutes (depending on ASR model) and should be OK in most cases. If your utterances are further apart, increase this value, or decrease it for smaller audio files. max_window_size: Maximum window size. It should not be necessary to change this value. gratis_blank: If True, the transition cost of blank is set to zero. Useful for long preambles or if there are large unrelated segments between utterances. Default: False. Parameters for calculation of confidence score: scoring_length: Block length to calculate confidence score. The default value of 30 should be OK in most cases. rWu+Parameter ´time_stamps´ has to be one of rQsamples_to_frames_ratio set_blankreplace_spaces_with_blanksrUrVu.Parameter ´text_converter´ has to be one of min_window_sizemax_window_size gratis_blankzBlanks are inserted between words, and also the transition cost of blank is zero. This configuration may lead to misalignments!Tscoring_lengthN)choices_time_stampsr^listrWfloatrQru isinstanceintrEblankboolrwrUchoices_text_converterrVrxryblank_transition_cost_zerowarned_about_misconfigurationr_r`score_min_mean_over_Lrrrrrq sh)      zCTCSegmentation.set_configcCshd|jji}|jdkr|jdur|}||_|j|j}n|jdks%J||}||j}||d<|S)z+Obtain parameters to determine time stamps.index_durationrKNrJ)rErrWru estimate_samples_to_frames_ratiorQ)r speech_lenlpz_len timing_cfgratiorrurrrget_timing_configjs   z!CTCSegmentation.get_timing_configHcCs2t|}||}|jd}|d}||}|S)a>Determine the ratio of encoded frames to sample points. This method helps to determine the time a single encoded frame occupies. As the sample rate already gave the number of samples, only the ratio of samples per encoded CTC frame are needed. This function estimates them by doing one inference, which is only needed once. Args: speech_len: Length of randomly generated speech vector for single inference. Default: 215040. Returns: samples_to_frames_ratio: Estimated ratio. rr+)rdrandget_lpzshape)rr random_inputrGrrurrrr}s   z0CTCSegmentation.estimate_samples_to_frames_ratiospeechcCst|tjr t|}|dtt|j}|j dgtj | dd}||d}t ||j d}|jjdi|\}}t|dksGJt||j|}|d}|S)zObtain CTC posterior log probabilities for given speech data. Args: speech: Speech audio input. Returns: lpz: Numpy vector with CTC log posterior probabilities. rr+)rT fill_value)rspeech_lengths)roNr)rnpndarrayrdtensor unsqueezerbrcrTnew_fulllongsizerrormencoder.rj log_softmaxdetachsqueezerXnumpy)rrlengthsbatchencrrGrrrrs  zCTCSegmentation.get_lpzcCsrd}t|tr |}ttt|}|jr5dd|D}tdd|}t|}dd|D}dd|D}||fS)z/Convert text to list and extract utterance IDs.NcSg|]}|ddqS)r*r+)splitr%rrrrr(z/CTCSegmentation._split_text..cSs t|dkS)Nr,)r.)uirrrs z-CTCSegmentation._split_text..cSg|]}|dqS)rrrrrrr(cSr)r+rrrrrr(r)rstr splitlinesr}filterr.rU)rr5r0utt_ids_and_textrrr _split_texts zCTCSegmentation._split_textc sj}|dur|jd}||}|jd i||\}}jdkrFfdd|D} |jdfdd| D} t|| \} } n'jdksMJfd d|D}fd d|D} d d| D} t || \} } t |||| | ||d } | S)u Preprocess text, and gather text and lpz into a task object. Text is pre-processed and tokenized depending on configuration. If ``speech_len`` is given, the timing configuration is updated. Text, lpz, and configuration is collected in a CTCSegmentationTask object. The resulting object can be serialized and passed in a multiprocessing computation. A minimal amount of text processing is done, i.e., splitting the utterances in ``text`` into a list and applying ``text_cleaner``. It is recommended that you normalize the text beforehand, e.g., change numbers into their spoken equivalent word, remove special characters, and convert UTF-8 characters to chars corresponding to your ASR model dictionary. The text is tokenized based on the ``text_converter`` setting: The "tokenize" method is more efficient and the easiest for models based on latin or cyrillic script that only contain the main chars, ["a", "b", ...] or for Japanese or Chinese ASR models with ~3000 short Kanji / Hanzi tokens. The "classic" method improves the the accuracy of the alignments for models that contain longer tokens, but with a greater complexity for computation. The function scans for partial tokens which may improve time resolution. For example, the word "▁really" will be broken down into ``['▁', '▁r', '▁re', '▁real', '▁really']``. The alignment will be based on the most probable activation sequence given by the network. Args: text: List or multiline-string with utterance ground truths. lpz: Log CTC posterior probabilities obtained from the CTC-network; numpy array shaped as (