o 7wih@sdZddlmZddlmZddlmZmZmZddl Z ddl Z ddl m Z mZddlmZzddlmZmZmZmZmZWneyMed ed weeZGd d d eZGd ddZdS)uPerform CTC segmentation to align utterances within audio files. This uses the ctc-segmentation Python package. Install it with pip or see the installing instructions in https://github.com/lumaku/ctc-segmentation Authors * Ludwig Kürzinger 2021 )Path)SimpleNamespace)ListOptionalUnionN) EncoderASREncoderDecoderASR) get_logger)CtcSegmentationParametersctc_segmentationdetermine_utterance_segments prepare_textprepare_token_listzMImportError: Is the ctc_segmentation module installed and in your PYTHONPATH?z'The ctc_segmentation module is missing.c@sXeZdZdZdZdZdZdZdZdZ dZ dZ dZ dZ dZdZdZdZddZdd ZdS) CTCSegmentationTaskuTask object for CTC segmentation. This object is automatically generated and acts as a container for results of a CTCSegmentation object. When formatted with str(·), this object returns results in a kaldi-style segments file formatting. The human-readable output can be configured with the printing options. Attributes ---------- text : list Utterance texts, separated by line. But without the utterance name at the beginning of the line (as in kaldi-style text). ground_truth_mat : array Ground truth matrix (CTC segmentation). utt_begin_indices : np.ndarray Utterance separator for the Ground truth matrix. timings : np.ndarray Time marks of the corresponding chars. state_list : list Estimated alignment of chars/tokens. segments : list Calculated segments as: (start, end, confidence score). config : CtcSegmentationParameters CTC Segmentation configuration object. name : str Name of aligned audio file (Optional). If given, name is considered when generating the text. Default: "utt". utt_ids : list The list of utterance names (Optional). This list should have the same length as the number of utterances. lpz : np.ndarray CTC posterior log probabilities (Optional). print_confidence_score : bool Include the confidence score. Default: True. print_utterance_text : bool Include utterance text. Default: True. NFuttTcKs|j|dS)zUpdate object attributes.N)__dict__update)selfkwargsrc/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/speechbrain/alignment/ctc_segmentation.pysetjszCTCSegmentationTask.setcsd}tj}jdurfddt|D}n |tjks!Jj}tjD]>\}}||djd}||ddd|dd7}jrT|d|d d 7}jra|dj|7}||d 7}q)|S) z0Return a kaldi-style ``segments`` file (string).Ncsg|] }jd|dqS)_04)name.0irrr ssz/CTCSegmentationTask.__str__.. rz.2fz3.4f ) lensegmentsutt_idsrange enumeraterprint_confidence_scoreprint_utterance_texttext)routputnum_utts utt_namesrboundary utt_entryrrr__str__ns  zCTCSegmentationTask.__str__)__name__ __module__ __qualname____doc__r,ground_truth_matutt_begin_indicestimings char_probs state_listr&configdonerr'lpzr*r+rr2rrrrr+s$- rc@szeZdZdZdZdZdZdZddgZdZ ddgZ d Z e Z   d.d eeefd ed ed efddZ           d/d eedeedeedeedeed eed eedeedeedeedeefddZd0ddZd1ddZedeejejffd d!Zd"d#Z d0d$d%Z!e"d&e#fd'd(Z$ d2deejejee%fd)ee&eefd*eed+e#fd,d-Z'dS)3CTCSegmentationu Align text to audio using CTC segmentation. Usage ----- Initialize with given ASR model and parameters. If needed, parameters for CTC segmentation can be set with ``set_config(·)``. Then call the instance as function to align text within an audio file. Arguments --------- asr_model : EncoderDecoderASR Speechbrain ASR interface. This requires a model that has a trained CTC layer for inference. It is better to use a model with single-character tokens to get a better time resolution. Please note that the inference complexity with Transformer models usually increases quadratically with audio length. It is therefore recommended to use RNN-based models, if available. kaldi_style_text : bool A kaldi-style text file includes the name of the utterance at the start of the line. If True, the utterance name is expected as first word at each line. If False, utterance names are automatically generated. Set this option according to your input data. Default: True. text_converter : str How CTC segmentation handles text. "tokenize": Use the ASR model tokenizer to tokenize the text. "classic": The text is preprocessed as text pieces which takes token length into account. If the ASR model has longer tokens, this option may yield better results. Default: "tokenize". time_stamps : str Choose the method how the time stamps are calculated. While "fixed" and "auto" use both the sample rate, the ratio of samples to one frame is either automatically determined for each inference or fixed at a certain ratio that is initially determined by the module, but can be changed via the parameter ``samples_to_frames_ratio``. Recommended for longer audio files: "auto". **ctc_segmentation_args Parameters for CTC segmentation. The full list of parameters is found in ``set_config``. Example ------- >>> # using example file included in the SpeechBrain repository >>> from speechbrain.inference.ASR import EncoderDecoderASR >>> from speechbrain.alignment.ctc_segmentation import CTCSegmentation >>> # load an ASR model >>> pre_trained = "speechbrain/asr-transformer-transformerlm-librispeech" >>> asr_model = EncoderDecoderASR.from_hparams(source=pre_trained) # doctest: +SKIP >>> aligner = CTCSegmentation(asr_model, kaldi_style_text=False) # doctest: +SKIP >>> # load data >>> audio_path = "tests/samples/single-mic/example1.wav" >>> text = ["THE BIRCH CANOE", "SLID ON THE", "SMOOTH PLANKS"] >>> segments = aligner(audio_path, text, name="example1") # doctest: +SKIP On multiprocessing ------------------ To parallelize the computation with multiprocessing, these three steps can be separated: (1) ``get_lpz``: obtain the lpz, (2) ``prepare_segmentation_task``: prepare the task, and (3) ``get_segments``: perform CTC segmentation. Note that the function `get_segments` is a staticmethod and therefore independent of an already initialized CTCSegmentation object. References ---------- CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition 2020, Kürzinger, Winkelbauer, Li, Watzel, Rigoll https://arxiv.org/abs/2007.09127 More parameters are described in https://github.com/lumaku/ctc-segmentation i>TNautofixedtokenizeclassicF asr_modelkaldi_style_texttext_converter time_stampsc  sttrtdr.tjdr.tjjdr.ttr2tdr.tjdr.tjjds2tdtds;td_jj _ ttrptjj d sStd d jj j j vr_td d tjdtjffdd }|_njj j_jj_jdjj j|||d|fddtjD}|j_tdd|D}t|dkr|dkrtdt|d|ddSdSdS)Nmodsdecoder ctc_weightencoderctc_linz&The given asr_model has no CTC module! tokenizerz.ctc_forward_step)fsrGrErFcsg|]}j|qSr)rM id_to_piecer)rDrrr s z,CTCSegmentation.__init__..cSsg|]}t|qSrr%)rcrrrr %izThe dictionary has z tokens with a max length of z>. This may lead to low alignment performance and low accuracy.r) isinstancerhasattrrHrIrrKAttributeErrorrD encode_batch_encoderRrNrStorchTensor_ctc log_softmaxrM _tokenizer set_config sample_rater( vocab_sizer< char_listmaxr%loggerwarning) rrDrErFrGctc_segmentation_argsrYrm max_char_lenr)rDrr__init__sp             zCTCSegmentation.__init__rZsamples_to_frames_ratio set_blankreplace_spaces_with_blanks gratis_blankmin_window_sizemax_window_sizescoring_lengthc Cs4|dur||jvrtdt|j||_|durt||_|dur(t||_|dur2t||j_ |durrrtrrrrs   z0CTCSegmentation.estimate_samples_to_frames_ratiospeechcCspt|tjr t|}|d|jj}tdg|jj}| ||}| | }| d }|S)a/Obtain CTC posterior log probabilities for given speech data. Arguments --------- speech : Union[torch.Tensor, np.ndarray] Speech audio input. Returns ------- np.ndarray Numpy vector with CTC log posterior probabilities. rg?)r`npndarrayretensor unsqueezetorDdevicerdrgdetachsqueezecpunumpy)rrwav_lensencr>rrrrs   zCTCSegmentation.get_lpzcCsrd}t|tr |}ttt|}|jr5dd|D}tdd|}t|}dd|D}dd|D}||fS)z/Convert text to list and extract utterance IDs.NcSg|]}|ddqS)r!r")splitrrrrrr z/CTCSegmentation._split_text..cSs t|dkS)Nr#r\)uirrrs z-CTCSegmentation._split_text..cSg|]}|dqS)rrrrrrr r^cSr)r"rrrrrr r^)r`str splitlinesr}filterr%rE)rr,r'utt_ids_and_textrrr _split_texts zCTCSegmentation._split_textc sj}|dur|jd}||}|jd i||\}}jdkrFfdd|D} |jdfdd| D} t|| \} } njdksMJfd d|D} d d| D} t || \} } t |||| | ||d } | S) u Preprocess text, and gather text and lpz into a task object. Text is pre-processed and tokenized depending on configuration. If ``speech_len`` is given, the timing configuration is updated. Text, lpz, and configuration is collected in a CTCSegmentationTask object. The resulting object can be serialized and passed in a multiprocessing computation. It is recommended that you normalize the text beforehand, e.g., change numbers into their spoken equivalent word, remove special characters, and convert UTF-8 characters to chars corresponding to your ASR model dictionary. The text is tokenized based on the ``text_converter`` setting: The "tokenize" method is more efficient and the easiest for models based on latin or cyrillic script that only contain the main chars, ["a", "b", ...] or for Japanese or Chinese ASR models with ~3000 short Kanji / Hanzi tokens. The "classic" method improves the the accuracy of the alignments for models that contain longer tokens, but with a greater complexity for computation. The function scans for partial tokens which may improve time resolution. For example, the word "▁really" will be broken down into ``['▁', '▁r', '▁re', '▁real', '▁really']``. The alignment will be based on the most probable activation sequence given by the network. Arguments --------- text : list List or multiline-string with utterance ground truths. lpz : np.ndarray Log CTC posterior probabilities obtained from the CTC-network; numpy array shaped as (