o }oiV@sddlmZmZmZmZmZmZddlZddlm Z ddl m Z ddl m Z ddlmZddlmZddlmZdd lmZmZdd lmZdd lmZmZmZmZGd d d ZGdddeZGdddeZ GdddeZ!dS))CallableDictListOptionalTupleUnionN)_audio_feature_collate_fn)ExternalFeatureLoader)normalize_batch)ChannelSelectorType)load_speech_segments_from_rttm) tokenizers) collectionsparsers)Dataset)AcousticEncodedRepresentation LabelsType LengthsType NeuralTypec@seZdZ       ddedeeefdeedeeded eed eed ed e fd dZ dede e eeffddZ dede e eeffddZdejjde e eeffddZdS)ASRFeatureManifestProcessorNrFmanifest_filepathparser max_duration min_durationmax_uttsbos_ideos_idpad_idindex_by_file_idc Cs4||_tj|||||| d|_||_||_||_dS)N)manifests_filesrrr max_numberr)rrASRFeatureText collectionrrr) selfrrrrrrrrrr$]/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/data/feature_to_text.py__init__s   z$ASRFeatureManifestProcessor.__init__indexreturncCs|j|}||SN)r"process_text_by_sample)r#r'sampler$r$r%process_text_by_id9s  z.ASRFeatureManifestProcessor.process_text_by_idfile_idcCs$|jj|d}|j|}||S)Nr)r"mappingr*)r#r- manifest_idxr+r$r$r%process_text_by_file_id=s  z3ASRFeatureManifestProcessor.process_text_by_file_idr+cCsV|jt|j}}|jdur|jg|}|d7}|jdur'||jg}|d7}||fS)N) text_tokenslenrr)r#r+ttlr$r$r%r*Bs    z2ASRFeatureManifestProcessor.process_text_by_sample)NNrNNrF)__name__ __module__ __qualname__strrrrfloatintboolr&rrr,r0r ASRAudioText OUTPUT_TYPEr*r$r$r$r%rs<   &rc*@s"eZdZdZdZddgZddgZedee e e ffdd Z       d4de de e efdee de e efdede dedeededeedddeedeeded ed!eed"eed#ed$ed%eef(d&d'Zd(d)Zd*d+Zd,d-Zd.d/Zd0d1Zd2d3Zd S)5_FeatureTextDataseta4 Dataset that loads tensors via a json file containing paths to audio feature files, transcripts, durations (in seconds) and optional RTTM files. Each new line is a different sample. Example below: {"feature_filepath": "/path/to/audio_feature.pt", "text_filepath": "/path/to/audio.txt", "rttm_filepath": "/path/to/audio_rttm.rttm", "duration": 23.147} ... {"feature_filepath": "/path/to/audio_feature.pt", "text": "the transcription", "offset": 301.75, "duration": 0.82, "utt": "utterance_id", "ctm_utt": "en_4156", "side": "A"} Args: manifest_filepath (str): Path to manifest json as described above. Can be comma-separated paths. parser: Str for a language specific preprocessor or a callable. normalize (bool): whether and where to normalize feature, must be one of [None, "post_norm", "pre_norm"] normalize_type (Union[str, dict]): how to normalize feature, see `nemo.collections.asr.parts.preprocessing.features.normalize_batch` use_rttm (bool): whether to use RTTM files if there is any, default to False rttm_mode (str): how to use RTTM files, must be one of ['mask', 'drop'], default to 'mask' feat_min_len (int): minimum length of feature when rttm_mode=deop, default to 4. feat_mask_val (Optional[float]): value used to mask features with RTTM files, default to None to use zero mel-spectralgram frame_unit_time_secs (float): time in seconds for each frame sample_rate (int): Sample rate to resample loaded audio to int_values (bool): If true, load samples as 32-bit integers. Defauts to False. augmentor (nemo.collections.asr.parts.perturb.AudioAugmentor): An AudioAugmentor object used to augment loaded audio max_duration (float): If audio exceeds this length, do not include in dataset min_duration (float): If audio is less than this length, do not include in dataset max_utts (int): Limit number of utterances trim (bool): whether or not to trim silence. Defaults to False bos_id (int): Id of beginning of sequence symbol to append if not None eos_id (int): Id of end of sequence symbol to append if not None pad_id (int): Id of pad symbol. Defaults to 0 return_sample_id (bool): whether to return the sample_id as a part of each sample channel_selector (int | Iterable[int] | str): select a single channel or a subset of channels from multi-channel audio. If set to `'average'`, it performs averaging across channels. Disabled if set to `None`. Defaults to `None`. Uses zero-based indexing. g(\0pre_norm post_normmaskdropr(c CsHtdtttdttdtttdtttdtdddS)z+Returns definitions of module output ports.)BDTrD)rDrFT)optional)featuresfeature_length transcriptstranscript_length sample_id)rrtuplerrr#r$r$r% output_typests  z _FeatureTextDataset.output_types per_featureFN{Gz?>rrr normalizenormalize_typeuse_rttm rttm_mode feat_min_len feat_mask_valframe_unit_time_secs sample_rate augmentor3nemo.collections.asr.parts.perturb.FeatureAugmentorrrrtrimrrrreturn_sample_idchannel_selectorc Cst|tkr |d}| |_||_||_||_||_|jr/|j|jvr/t d|jd|d||_ |dur:||_ n |dkrBd|_ n|j |_ |dur[||j vr[t d|j d|d| |_t||| | ||||d|_t| d |_||_||_||_dS) N,z`rttm_mode` must be one of z, got `z ` insteadr@gz`normalize` must be one of )rrrrrrrr)r\)typer9splitr[rTrUrVrW RTTM_MODES ValueErrorrXrYZERO_LEVEL_SPEC_DB_VAL NORM_MODESrZrmanifest_processorr featurizerr^r_r`)r#rrrTrUrVrWrXrYrZr[r\rrrr^rrrr_r`r$r$r%r&s@    z_FeatureTextDataset.__init__cCs |jj|Sr))rhr")r#rLr$r$r%get_manifest_sample z'_FeatureTextDataset.get_manifest_samplec Cs4|jj|}|j}|durd}|j|j}|t|jd }}|jj |d\}}|j durB|j rA|j rA||||j |j}n0|j dkr]|j rW|j rW||||j |j}||}n||}|j rr|j rr||||j |j}|jr||t| t| |f} | S||t| t| f} | S)Nrr1)r+rA)rhr"offsetriprocess feature_filetorchtensorshapelongr*rTrV rttm_fileprocess_features_with_rttmrYnormalize_featurer_) r#r'r+rlrHfflr4r5outputr$r$r% __getitem__s.        " z_FeatureTextDataset.__getitem__c CsPt|}|}d\}}t|dD]j} || |j} |t|dkr@||d| kr@|d7}|t|dkr@||d| ks,||ddksX| ||dksX| ||dkrk|jdkr^q||dd| f<|d7}q|dd| f|dd|f<|d7}q||jkr|jdkr||ddd|jf<|ddd|jfS|ddd|fS)N)rrr1rrC)r clonerangesizerZr3rWrX) r#rHrlrsmask_valsegments new_featuressidfidir4r$r$r%rts&  0   z._FeatureTextDataset.process_features_with_rttmcCs t|jjSr))r3rhr"rNr$r$r%__len__rkz_FeatureTextDataset.__len__cCst||j|jjdS)N) feat_pad_val label_pad_id)rrYrhr)r#batchr$r$r% _collate_fns z_FeatureTextDataset._collate_fncCs6|d}t|t|dg|j\}}}|dS)zH Args: feat: feature tensor of shape [M, T] r) unsqueezer rorpr|rUsqueeze)r#feat_r$r$r%rus " z%_FeatureTextDataset.normalize_feature)rArPFrBrQNrRrSNNNrFNNrFN)r6r7r8__doc__rfrgrdpropertyrrr9rrOrrdictr<r;r:r r&rjryrtrrrur$r$r$r%r?Os         >$ r?c0seZdZdZ               d'd edeeeefdeedeeefde dede dee de dee dddee dee de de de de dee d ee d!e d"eee fd#e d$ee f.fd%d& ZZS)(FeatureToCharDataseta Dataset that loads tensors via a json file containing paths to audio feature files, transcripts, durations (in seconds) and optional RTTM files. Each new line is a different sample. Example below: {"feature_filepath": "/path/to/audio_feature.pt", "text_filepath": "/path/to/audio.txt", "duration": 23.147, "rttm_filepath": "/path/to/audio_rttm.rttm",} ... {"feature_filepath": "/path/to/audio_feature.pt", "text": "the transcription", "offset": 301.75, "duration": 0.82, "utt": "utterance_id", "ctm_utt": "en_4156", "side": "A"} Args: manifest_filepath (str): Path to manifest json as described above. Can be comma-separated paths. labels (str): String containing all the possible characters to map to normalize (str): how to normalize feature, must be one of [None, "post_norm", "pre_norm"] normalize_type (Union[str, dict]): how to normalize feature, see `nemo.collections.asr.parts.preprocessing.features.normalize_batch` use_rttm (bool): whether to use RTTM files if there is any, default to False rttm_mode (str): how to use RTTM files, must be one of ['mask', 'drop'], default to 'mask' feat_min_len (int): minimum length of feature, default to 4 feat_mask_val (Optional[float]): value used to mask features with RTTM files, default to None to use zero mel-spectralgram frame_unit_time_secs: time in seconds for each frame sample_rate (int): Sample rate to resample loaded audio to int_values (bool): If true, load samples as 32-bit integers. Defauts to False. augmentor (nemo.collections.asr.parts.perturb.AudioAugmentor): An AudioAugmentor object used to augment loaded audio max_duration: If audio exceeds this length, do not include in dataset min_duration: If audio is less than this length, do not include in dataset max_utts: Limit number of utterances blank_index: blank character index, default = -1 unk_index: unk_character index, default = -1 bos_id: Id of beginning of sequence symbol to append if not None eos_id: Id of end of sequence symbol to append if not None return_sample_id (bool): whether to return the sample_id as a part of each sample channel_selector (int | Iterable[int] | str): select a single channel or a subset of channels from multi-channel audio. If set to `'average'`, it performs averaging across channels. Disabled if set to `None`. Defaults to `None`. Uses zero-based indexing. rArPFrBrQNrRrSrrenrlabelsrTrUrVrWrXrYrZr[r\r]rrr blank_index unk_indexr^rrrrr_r`cs||_tj|||||d}tjdid|d|d|d|d|d|d|d |d | d | d | d | d| d|d|d|d|d|d|d|dS)N)rnameunk_idblank_id do_normalizerrrTrUrVrWrXrYrZr[r\rrrr^rrrr_r`r$)rr make_parsersuperr&)r#rrrTrUrVrWrXrYrZr[r\rrrrrr^rrrrr_r` __class__r$r%r&8sZ       zFeatureToCharDataset.__init__)rArPFrBrQNrRrSNNNrrrFNNrrFN)r6r7r8rr9rrrrr<r;r:rr r& __classcell__r$r$rr%rs*       rc&seZdZdZ             d"d ed ddeedeeefdedede dee de dee dddee dee de dedededee f$fd d! Z Z S)#FeatureToBPEDataseta Dataset that loads tensors via a json file containing paths to audio feature files, transcripts, durations (in seconds) and optional RTTM files. Each new line is a different sample. Example below: {"audio_filepath": "/path/to/audio.wav", "text_filepath": "/path/to/audio.txt", "duration": 23.147, "rttm_filepath": "/path/to/audio_rttm.rttm",} ... {"audio_filepath": "/path/to/audio.wav", "text": "the transcription", "offset": 301.75, "duration": 0.82, "utt": "utterance_id", "ctm_utt": "en_4156", "side": "A"} In practice, the dataset and manifest used for character encoding and byte pair encoding are exactly the same. The only difference lies in how the dataset tokenizes the text in the manifest. Args: manifest_filepath (str): Path to manifest json as described above. Can be comma-separated paths. tokenizer: A subclass of the Tokenizer wrapper found in the common collection, nemo.collections.common.tokenizers.TokenizerSpec. ASR Models support a subset of all available tokenizers. normalize (str): how to normalize feature, must be one of [None, "post_norm", "pre_norm"] normalize_type (Union[str, dict]): how to normalize feature, see `nemo.collections.asr.parts.preprocessing.features.normalize_batch` use_rttm (bool): whether to use RTTM files if there is any, default to False rttm_mode (str): how to use RTTM files, must be one of ['mask', 'drop'], default to 'mask' feat_min_len (int): minimum length of feature, default to 4 feat_mask_val (Optional[float]): value used to mask features with RTTM files, default to None to use zero mel-spectralgram frame_unit_time_secs: time in seconds for each frame sample_rate (int): Sample rate to resample loaded audio to int_values (bool): If true, load samples as 32-bit integers. Defauts to False. augmentor (nemo.collections.asr.parts.perturb.AudioAugmentor): An AudioAugmentor object used to augment loaded audio max_duration: If audio exceeds this length, do not include in dataset min_duration: If audio is less than this length, do not include in dataset max_utts: Limit number of utterances trim: Whether to trim silence segments use_start_end_token: Boolean which dictates whether to add [BOS] and [EOS] tokens to beginning and ending of speech respectively. return_sample_id (bool): whether to return the sample_id as a part of each sample channel_selector (int | Iterable[int] | str): select a single channel or a subset of channels from multi-channel audio. If set to `'average'`, it performs averaging across channels. Disabled if set to `None`. Defaults to `None`. Uses zero-based indexing. rArPFrBrQNrRrSrTr tokenizerz0nemo.collections.common.tokenizers.TokenizerSpecrTrUrVrWrXrYrZr[r\r]rrruse_start_end_tokenr^r_r`cs|rt|dr|jdkr|j}nd}|r"t|dr"|jdkr"|j}nd}t|dr2|jdkr2|j}nd}Gddd}tjdid|d||d |d |d |d |d |d|d| d| d| d| d| d|d|d|d|d|d|d|dS)Nrrrrc@seZdZddZddZdS)z6FeatureToBPEDataset.__init__..TokenizerWrappercSs&t|tjjr d|_nd|_||_dS)NTF) isinstancer aggregate_tokenizerAggregateTokenizer is_aggregate _tokenizer)r#rr$r$r%r&s z?FeatureToBPEDataset.__init__..TokenizerWrapper.__init__cWsVt|dtr#|jr#g}|dD]}||j|d|dq|S|jj|}|S)Nrr9lang)rrrextendr text_to_ids)r#argsr4spanr$r$r%__call__s  z?FeatureToBPEDataset.__init__..TokenizerWrapper.__call__N)r6r7r8r&rr$r$r$r%TokenizerWrappers rrrrTrUrVrWrXrYrZr[r\rrrr^r_r`r$)hasattrrrrrr&)r#rrrTrUrVrWrXrYrZr[r\rrrrr^r_r`rrrrrr$r%r&sf      zFeatureToBPEDataset.__init__)rArPFrBrQNrRrSNNNrTFFN)r6r7r8rr9rrrr<r;r:r r&rr$r$rr%rpsn/      r)"typingrrrrrrro*nemo.collections.asr.data.feature_to_labelr7nemo.collections.asr.parts.preprocessing.feature_loaderr 1nemo.collections.asr.parts.preprocessing.featuresr 0nemo.collections.asr.parts.preprocessing.segmentr *nemo.collections.asr.parts.utils.vad_utilsr nemo.collections.commonr +nemo.collections.common.parts.preprocessingrrnemo.core.classesrnemo.core.neural_typesrrrrrr?rrr$r$r$r%s        1C_