o }oin@sddlZddlZddlmZmZmZmZddlZddlm Z m Z ddl m Z ddl mZddlmZddlmZmZddlmZmZmZmZmZdd lmZdd lmZdd lm Z d !gd dde"DZ#dej$de%de%dej$fddZ&ddZ'ddZ(ddZ)ddZ*ddZ+Gd d!d!eZ,Gd"d#d#e,Z-Gd$d%d%e,Z.Gd&d'd'eZ/Gd(d)d)e/Z0Gd*d+d+e/Z1Gd,d-d-eZ2Gd.d/d/eZ3Gd0d1d1e.Z4dS)2N)DictListOptionalUnion)cache_datastore_manifestsexpand_sharded_filepaths)WaveformFeaturizer)available_formats) collections)DatasetIterableDataset) AudioSignal LabelsType LengthsType NeuralTypeRegressionValuesType)logging) webdataset)webdataset_split_by_workers;)wavmp3flacopuscCsg|]}|qS)lower).0fmtrr\/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/data/audio_to_label.py rsignalsig_lenrequired_lengthreturncCsltg}t||}t||}tg}t||g}|dkr2|| d}t||f}|S|}|S)a6repeat signal to make short signal to have required_length Args: signal (Tensor): input signal sig_len (int): length of input signal required_length (int): length of generated signal Returns: signal (Tensor): generated signal of required_length by repeating itself. rN)torchtensorintcat)r!r"r#subrepeatremrep_sigrrr repeat_signal"s   r-cCs||}||S)zUnormalize signal Args: signal(FloatTensor): signal to be normalized. )meanabsmax)r!signal_minusmeanrrr normalize8s r2cCs6t}|D]}|dd}||dd||<q|S)a Count number of wav files in Dict manifest_file_id. Use for _TarredAudioToLabelDataset. Args: manifest_file_id (Dict): Dict of files and their corresponding id. {'A-sub0' : 1, ..., 'S-sub10':100} Returns: count (Dict): Dict of wav files {'A' : 2, ..., 'S':10} -subr)dictsplitget)manifest_file_idcountiaudio_filenamerrrcount_occurenceAs r<cCst|\}}}}d}|ddu}|rt|}t|}gg}} |D]@\} } } } |rH| } | |krCd|| f}tjj| |} || | } | |kr`d|| f}tjjj| ||d} | | q%|rst|}t|}nd\}}t| } t|}||| |fS)arcollate batch of audio sig, audio len, tokens, tokens len Args: batch (Optional[FloatTensor], Optional[LongTensor], LongTensor, LongTensor): A tuple of tuples of signal, signal lengths, encoded tokens, and encoded tokens length. This collate func assumes the signals are 1d torch tensors (i.e. mono audio). rN)valueNN) zipr0itemr%nn functionalpadappendstack)batchpad_id_ audio_lengthstokens_lengths max_audio_len has_audiomax_tokens_len audio_signaltokenssigr"tokens_i tokens_i_lenrCrrr_speech_collate_fnOs4             rScCst|\}}}}|ddu}tt|}ggg}}} |D]M\} } } }|rf| } | |} | dkrY|| }|| }|dkrE| | dntg}t|| g}t||f} | t||| || q|ryt|}t| }nd\}}t|}t|}||||fS)aicollate batch of audio sig, audio len, tokens, tokens len Args: batch (Optional[FloatTensor], Optional[LongTensor], LongTensor, LongTensor): A tuple of tuples of signal, signal lengths, encoded tokens, and encoded tokens length. This collate func assumes the signals are 1d torch tensors (i.e. mono audio). rNr>) r?r'r0r@r%r&r(rDrE)selfrFrHrIrJrL fixed_lengthrNrOnew_audio_lengthsrPr"rQ chunck_lenr*r+r)r,rrr_fixed_seq_collate_fnws0          rXcCs~t|jj|j}t|\}}}}tt|t|}t|jj|j}|ddu}ggggf\}} } }|d} ||d} |D]^\} }}}|jrLt | } t | }t | }t || |f} ||7}|rt j |||dd}t|D]}||}||}| ||}||qr| || |g|||g|q?|rt |}t |}nd\}}t | } t | }||| |fS)acollate batch of audio sig, audio len, tokens, tokens len Args: batch (Optional[FloatTensor], Optional[LongTensor], LongTensor, LongTensor): A tuple of tuples of signal, signal lengths, encoded tokens, and encoded tokens length. This collate func assumes the signals are 1d torch tensors (i.e. mono audio). batch size equals to 1. rNtrunc) rounding_moder>)r' featurizer sample_ratewindow_length_in_secr?minr0shift_length_in_secnormalize_audior2r%zerosr(divrangerDextendrEr&)rTrF slice_lengthrHrIrJshiftrLrN num_slicesrOappend_len_startappend_len_endrPr"rQstartendslicesslice_id start_idxend_idxr!rrr_vad_frame_seq_collate_fnsB              rqcseZdZdZedeeeeffddZ dddddddd e ee efd e ed ee d ee d e de eee efde dee ffddZddZddZZS)_AudioLabelDataseta Dataset that loads tensors via a json file containing paths to audio files, labels, and durations and offsets(in seconds). Each new line is a different sample. Example below: and their target labels. JSON files should be of the following format:: {"audio_filepath": "/path/to/audio_wav_0.wav", "duration": time_in_sec_0, "label": target_label_0, "offset": offset_in_sec_0} ... {"audio_filepath": "/path/to/audio_wav_n.wav", "duration": time_in_sec_n, "label": target_label_n, "offset": offset_in_sec_n} Args: manifest_filepath (Union[str, List[str]]): Dataset parameter. Path to JSON containing data. labels (list): Dataset parameter. List of target classes that can be output by the speaker recognition model. featurizer min_duration (float): Dataset parameter. All training files which have a duration less than min_duration are dropped. Note: Duration is read from the manifest JSON. Defaults to 0.1. max_duration (float): Dataset parameter. All training files which have a duration more than max_duration are dropped. Note: Duration is read from the manifest JSON. Defaults to None. trim (bool): Whether to use trim silence from beginning and end of audio signal using librosa.effects.trim(). Defaults to False. channel selector (Union[str, int, List[int]]): string denoting the downmix mode, an integer denoting the channel to be selected, or an iterable of integers denoting a subset of channels. Channel selector is using zero-based indexing. If set to `None`, the original signal will be used. r$cCstd|durt|drt|jdntttdtd}|jr7|ttdtttdtd|S|ttdt ttdtd|S) +Returns definitions of module output ports.BTN _sample_ratefreqrurN a_sig_lengthtargetstargets_lengthlabel label_length rhasattrr rwtupleris_regression_taskupdaterrrT output_typesrrrrs*  z_AudioLabelDataset.output_types皙?NF) min_duration max_durationtrimchannel_selectorrcal_labels_occurrencemanifest_filepathlabelsrrrrrrc sHtt|tr|d}t|ddtj||||| d_|_ |_ |_ |_ |s|r2|njj _jdurAtjnd_ii__ig__tjD]\} } | j| <| j| <| rpjj| j| <qW| rfddtjD_ttjddD]} td | j| qdSg_d_dS) N,T)manifest_filepaths cache_audio)manifests_filesrrrrr4cg|]}j|qSr) id2occurrence)rkrTrrr=z/_AudioLabelDataset.__init__..$ label id {} and its mapped label {})super__init__ isinstancestrr6rr ASRSpeechLabel collectionr\rrr uniq_labelsrlen num_classeslabel2idid2labelrlabels_occurrence enumeratesortedrdrdebugformat) rTrrr\rrrrrrlabel_idridx __class__rrrsB      z_AudioLabelDataset.__init__cC t|jSNrrrrrr__len__F z_AudioLabelDataset.__len__c Cs|j|}|j}|durd}|jj|j||j|j|jd}|t |j d }}|j s9t |j |j }nt |j}t d }||||fS)Nr)offsetdurationrrr4)rrr\process audio_filerrrr%r&shapelongrrrfloat rTindexsamplerfeaturesfflttlrrr __getitem__Is"  z_AudioLabelDataset.__getitem__)__name__ __module__ __qualname____doc__propertyrrrrrrrrboolr'rrr __classcell__rrrrrrs:'   4rrc@eZdZdZddZdS)!AudioToClassificationLabelDatasetaZ Dataset that loads tensors via a json file containing paths to audio files, command class, and durations (in seconds). Each new line is a different sample. Example below: {"audio_filepath": "/path/to/audio_wav_0.wav", "duration": time_in_sec_0, "label": target_label_0, "offset": offset_in_sec_0} ... {"audio_filepath": "/path/to/audio_wav_n.wav", "duration": time_in_sec_n, "label": target_label_n, "offset": offset_in_sec_n} Args: manifest_filepath (Union[str, List[str]]): Path to manifest json as described above. Can be comma-separated paths. labels (Optional[list]): String containing all the possible labels to map to if None then automatically picks from ASRSpeechLabel collection. featurizer: Initialized featurizer class that converts paths of audio to feature tensors max_duration: If audio exceeds this length, do not include in dataset min_duration: If audio is less than this length, do not include in dataset trim: Boolean flag whether to trim the audio cC t|ddSNr)rGrSrTrFrrr _collate_fn| z-AudioToClassificationLabelDataset._collate_fnNrrrrrrrrrres rcseZdZdZdddddddddd deeeefd eed eed eed e d eeee ee fdeedeede de dee ffddZ ddZ ddZ ZS)AudioToSpeechLabelDataseta4 Dataset that loads tensors via a json file containing paths to audio files, command class, and durations (in seconds). Each new line is a different sample. Example below: {"audio_filepath": "/path/to/audio_wav_0.wav", "duration": time_in_sec_0, "label": target_label_0, "offset": offset_in_sec_0} ... {"audio_filepath": "/path/to/audio_wav_n.wav", "duration": time_in_sec_n, "label": target_label_n, "offset": offset_in_sec_n} Args: manifest_filepath (Union[str, List[str]]): Path to manifest json as described above. Can be comma-separated paths. labels (Optional[list]): String containing all the possible labels to map to if None then automatically picks from ASRSpeechLabel collection. min_duration (float): Dataset parameter. All training files which have a duration less than min_duration are dropped. Note: Duration is read from the manifest JSON. Defaults to 0.1. max_duration (float): Dataset parameter. All training files which have a duration more than max_duration are dropped. Note: Duration is read from the manifest JSON. Defaults to None. trim (bool): Whether to use trim silence from beginning and end of audio signal using librosa.effects.trim(). Defaults to False. channel selector (Union[str, int, List[int]]): string denoting the downmix mode, an integer denoting the channel to be selected, or an iterable of integers denoting a subset of channels. Channel selector is using zero-based indexing. If set to `None`, the original signal will be used. window_length_in_sec (float): length of window/slice (in seconds) Use this for speaker recognition and VAD tasks. shift_length_in_sec (float): amount of shift of window for generating the frame for VAD task in a batch Use this for VAD task during inference. normalize_audio (bool): Whether to normalize audio signal. Defaults to False. is_regression_task (bool): Whether the dataset is for a regression task instead of classification. Defaults to False. cal_labels_occurrence (bool): Whether to calculate occurrence of labels Defaults to False. rNFr4) rrrrr^r`rarrrrrrrrr^r`rarrc sX||_| |_| |_td|jtd|jtj|||||||| | d dS)N5Window/slice length considered for collate func is {}.Shift length considered for collate func is {}) rrr\rrrrrr)r^r`rarrrrr) rTrrr\rrrrr^r`rarrrrrrs  z"AudioToSpeechLabelDataset.__init__cC t||SrrXrrrrfixed_seq_collate_fnrz.AudioToSpeechLabelDataset.fixed_seq_collate_fncCrrrqrrrrvad_frame_seq_collate_fnrz2AudioToSpeechLabelDataset.vad_frame_seq_collate_fn)rrrrrrrrrrr'rrrrrrrrrsH.     #rc@seZdZdZddddddddddeeeefd eeeefd eed ed ee d ee de dededede fddZ ddZ ddZ ddZddZdS)_TarredAudioLabelDataseta A similar Dataset to the AudioLabelDataSet, but which loads tarred audio files. Accepts a single comma-separated JSON manifest file (in the same style as for the AudioToSpeechLabelDataset), as well as the path(s) to the tarball(s) containing the wav files. Each line of the manifest should contain the information for one audio file, including at least the label and name of the audio file within the tarball. Valid formats for the audio_tar_filepaths argument include: (1) a single string that can be brace-expanded, e.g. 'path/to/audio.tar' or 'path/to/audio_{1..100}.tar.gz', or (2) a list of file paths that will not be brace-expanded, e.g. ['audio_1.tar', 'audio_2.tar', ...]. Note: For brace expansion in (1), there may be cases where `{x..y}` syntax cannot be used due to shell interference. This occurs most commonly inside SLURM scripts. Therefore we provide a few equivalent replacements. Supported opening braces - { <=> (, [, < and the special tag _OP_. Supported closing braces - } <=> ), ], > and the special tag _CL_. For SLURM based tasks, we suggest the use of the special tags for ease of use. See the documentation for more information about accepted data and input formats. If using multiple processes the number of shards should be divisible by the number of workers to ensure an even split among workers. If it is not divisible, logging will give a warning but training will proceed. In addition, if using mutiprocessing, each shard MUST HAVE THE SAME NUMBER OF ENTRIES after filtering is applied. We currently do not check for this, but your program may hang if the shards are uneven! Notice that a few arguments are different from the AudioLabelDataSet; for example, shuffle (bool) has been replaced by shuffle_n (int). Additionally, please note that the len() of this DataLayer is assumed to be the length of the manifest after filtering. An incorrect manifest length may lead to some DataLoader issues down the line. Args: audio_tar_filepaths: Either a list of audio tarball filepaths, or a string (can be brace-expandable). manifest_filepath (str): Path to the manifest. labels (list): Dataset parameter. List of target classes that can be output by the speaker recognition model. featurizer shuffle_n (int): How many samples to look ahead and load to be shuffled. See WebDataset documentation for more details. Defaults to 0. min_duration (float): Dataset parameter. All training files which have a duration less than min_duration are dropped. Note: Duration is read from the manifest JSON. Defaults to 0.1. max_duration (float): Dataset parameter. All training files which have a duration more than max_duration are dropped. Note: Duration is read from the manifest JSON. Defaults to None. trim(bool): Whether to use trim silence from beginning and end of audio signal using librosa.effects.trim(). Defaults to False. window_length_in_sec (float): length of slice/window (in seconds) # Pass this only for speaker recognition and VAD task shift_length_in_sec (float): amount of shift of window for generating the frame for VAD task. in a batch # Pass this only for VAD task during inference. normalize_audio (bool): Whether to normalize audio signal. Defaults to False. shard_strategy (str): Tarred dataset shard distribution strategy chosen as a str value during ddp. - `scatter`: The default shard strategy applied by WebDataset, where each node gets a unique set of shards, which are permanently pre-allocated and never changed at runtime. - `replicate`: Optional shard strategy, where each node gets all of the set of shards available in the tarred dataset, which are permanently pre-allocated and never changed at runtime. The benefit of replication is that it allows each node to sample data points from the entire dataset independently of other nodes, and reduces dependence on the value of `shuffle_n`. .. warning:: Replicated strategy allows every node to sample the entire set of available tarfiles, and therefore more than one node may sample the same tarfile, and even sample the same data points! As such, there is no assured guarantee that all samples in the dataset will be sampled at least once during 1 epoch. Scattered strategy, on the other hand, on specific occasions (when the number of shards is not divisible with ``world_size``), will not sample the entire dataset. For these reasons it is not advisable to use tarred datasets as validation or test datasets. global_rank (int): Worker rank, used for partitioning shards. Defaults to 0. world_size (int): Total number of processes, used for partitioning shards. Defaults to 0. is_regression_task (bool): Whether it is a regression task. Defualts to False. rrNFscatter) shuffle_nrrrshard_strategy global_rank world_sizeraudio_tar_filepathsrrrrrrrrrrc  Cst|dtj|||dd|_t|jj|_||_||_|r |n|jj |_ t |j |_ ii|_ |_t|j D]\} }| |j |<||j| <q7tt |j ddD]}td||j|qQt|| | | d}ttj|dtt|ttjtdd td d |jt|j |_!dS) N)rT)rrrindex_by_file_idrrsharded_filepathsrrrurls__key__audiokeyrr)"rr rrr<mappingfile_occurencer\rrrrrrrrrdrrrrwds DataPipelineSimpleShardListrshuffletarfile_to_samplesrenameVALID_FILE_FORMATSto_tuple_filtermap _build_sample_dataset)rTrrrr\rrrrrrrrrrrrrrr sD         z!_TarredAudioLabelDataset.__init__c Gfddd}||j|jS)This function is used to remove samples that have been filtered out by ASRSpeechLabel already. Otherwise, we would get a KeyError as _build_sample attempts to find the manifest entry for a sample that was filtered out (e.g. for duration). Note that if using multi-GPU training, filtering may lead to an imbalance in samples in each shard, which may make your code hang as one process will finish before the other. c0eZdZfddZddZddZddZd S) z;_TarredAudioLabelDataset._filter..TarredAudioFilterc |_||_||_||_dSriteratorrr_internal_generator _iterablerTrrrrrrdzD_TarredAudioLabelDataset._filter..TarredAudioFilter.__init__cS||_|Srrrrrrr__iter__j zD_TarredAudioLabelDataset._filter..TarredAudioFilter.__iter__cS:zt|j}W|Sty||_t|j}Y|Swrnextr StopIterationrrTvaluesrrr__next__n   zD_TarredAudioLabelDataset._filter..TarredAudioFilter.__next__cst|jD]8\}}|\}}tjtj|\}}||jvr>td|j|D]}|dkr0|}n|dt|}||fVq'qdSa> WebDataset requires an Iterator, but we require an iterable that yields 1-or-more values per value inside self.iterator. Therefore wrap the iterator with a generator function that will yield 1-or-more values per sample in the iterator. rr3N rrospathsplitextbasenamerrdrrTrHtup audio_bytesr;file_idjrrrrx  zO_TarredAudioLabelDataset._filter..TarredAudioFilter._internal_generatorNrrrrr rrrrrrTarredAudioFilterc   r!rrrTrr!rrrr[)z _TarredAudioLabelDataset._filtercCs|\}}tjtj|\}}|jj|}|j|}|j}|dur$d}t|} |j j | ||j |j d} | | t| jd} } |j|j} d}| | t| t|fS)\Builds the training sample by combining the data from the WebDataset with the manifest info.Nrrrrr4)rrrrrrrioBytesIOr\rrrcloser%r&rrrrrTrrr;rrH manifest_idxmanifest_entryraudio_filestreamrrrrrrrrrs&     z&_TarredAudioLabelDataset._build_samplecC |jSrrr rrrrr rz!_TarredAudioLabelDataset.__iter__cCrrrrrrrrrz _TarredAudioLabelDataset.__len__)rrrrrrrr'rrrrrrr rrrrrrsJS      ;3 rc@r)'TarredAudioToClassificationLabelDataseta A similar Dataset to the AudioToClassificationLabelDataset, but which loads tarred audio files. Accepts a single comma-separated JSON manifest file (in the same style as for the AudioToClassificationLabelDataset), as well as the path(s) to the tarball(s) containing the wav files. Each line of the manifest should contain the information for one audio file, including at least the transcript and name of the audio file within the tarball. Valid formats for the audio_tar_filepaths argument include: (1) a single string that can be brace-expanded, e.g. 'path/to/audio.tar' or 'path/to/audio_{1..100}.tar.gz', or (2) a list of file paths that will not be brace-expanded, e.g. ['audio_1.tar', 'audio_2.tar', ...]. See the WebDataset documentation for more information about accepted data and input formats. If using multiple processes the number of shards should be divisible by the number of workers to ensure an even split among workers. If it is not divisible, logging will give a warning but training will proceed. In addition, if using mutiprocessing, each shard MUST HAVE THE SAME NUMBER OF ENTRIES after filtering is applied. We currently do not check for this, but your program may hang if the shards are uneven! Notice that a few arguments are different from the AudioToBPEDataset; for example, shuffle (bool) has been replaced by shuffle_n (int). Additionally, please note that the len() of this DataLayer is assumed to be the length of the manifest after filtering. An incorrect manifest length may lead to some DataLoader issues down the line. Args: audio_tar_filepaths: Either a list of audio tarball filepaths, or a string (can be brace-expandable). manifest_filepath (str): Path to the manifest. labels (list): Dataset parameter. List of target classes that can be output by the speaker recognition model. featurizer shuffle_n (int): How many samples to look ahead and load to be shuffled. See WebDataset documentation for more details. Defaults to 0. min_duration (float): Dataset parameter. All training files which have a duration less than min_duration are dropped. Note: Duration is read from the manifest JSON. Defaults to 0.1. max_duration (float): Dataset parameter. All training files which have a duration more than max_duration are dropped. Note: Duration is read from the manifest JSON. Defaults to None. trim(bool): Whether to use trim silence from beginning and end of audio signal using librosa.effects.trim(). Defaults to False. shard_strategy (str): Tarred dataset shard distribution strategy chosen as a str value during ddp. - `scatter`: The default shard strategy applied by WebDataset, where each node gets a unique set of shards, which are permanently pre-allocated and never changed at runtime. - `replicate`: Optional shard strategy, where each node gets all of the set of shards available in the tarred dataset, which are permanently pre-allocated and never changed at runtime. The benefit of replication is that it allows each node to sample data points from the entire dataset independently of other nodes, and reduces dependence on value of `shuffle_n`. .. warning:: Replicated strategy allows every node to sample the entire set of available tarfiles, and therefore more than one node may sample the same tarfile, and even sample the same data points! As such, there is no assured guarantee that all samples in the dataset will be sampled at least once during 1 epoch. Scattered strategy, on the other hand, on specific occasions (when the number of shards is not divisible with ``world_size``), will not sample the entire dataset. For these reasons it is not advisable to use tarred datasets as validation or test datasets. global_rank (int): Worker rank, used for partitioning shards. Defaults to 0. world_size (int): Total number of processes, used for partitioning shards. Defaults to 0. is_regression_task (bool): Whether it is a regression task. Defualts to False. cCrrrrrrrrrz3TarredAudioToClassificationLabelDataset._collate_fnNrrrrrr1s Cr1cseZdZdZddddddddddd d eeeefd eeeefd eed edee dee de dee dee de dededeffddZ ddZ ddZ ddZZS)TarredAudioToSpeechLabelDataseta A similar Dataset to the AudioToSpeechLabelDataset, but which loads tarred audio files. Accepts a single comma-separated JSON manifest file (in the same style as for the AudioToSpeechLabelDataset), as well as the path(s) to the tarball(s) containing the wav files. Each line of the manifest should contain the information for one audio file, including at least the transcript and name of the audio file within the tarball. Valid formats for the audio_tar_filepaths argument include: (1) a single string that can be brace-expanded, e.g. 'path/to/audio.tar' or 'path/to/audio_{1..100}.tar.gz', or (2) a list of file paths that will not be brace-expanded, e.g. ['audio_1.tar', 'audio_2.tar', ...]. See the WebDataset documentation for more information about accepted data and input formats. If using multiple processes the number of shards should be divisible by the number of workers to ensure an even split among workers. If it is not divisible, logging will give a warning but training will proceed. In addition, if using mutiprocessing, each shard MUST HAVE THE SAME NUMBER OF ENTRIES after filtering is applied. We currently do not check for this, but your program may hang if the shards are uneven! Notice that a few arguments are different from the AudioToBPEDataset; for example, shuffle (bool) has been replaced by shuffle_n (int). Additionally, please note that the len() of this DataLayer is assumed to be the length of the manifest after filtering. An incorrect manifest length may lead to some DataLoader issues down the line. Args: audio_tar_filepaths: Either a list of audio tarball filepaths, or a string (can be brace-expandable). manifest_filepath (str): Path to the manifest. labels (list): Dataset parameter. List of target classes that can be output by the speaker recognition model. featurizer shuffle_n (int): How many samples to look ahead and load to be shuffled. See WebDataset documentation for more details. Defaults to 0. min_duration (float): Dataset parameter. All training files which have a duration less than min_duration are dropped. Note: Duration is read from the manifest JSON. Defaults to 0.1. max_duration (float): Dataset parameter. All training files which have a duration more than max_duration are dropped. Note: Duration is read from the manifest JSON. Defaults to None. trim(bool): Whether to use trim silence from beginning and end of audio signal using librosa.effects.trim(). Defaults to False. window_length_in_sec (float): time length of window/slice (in seconds) # Pass this only for speaker recognition and VAD task shift_length_in_sec (float): amount of shift of window for generating the frame for VAD task. in a batch # Pass this only for VAD task during inference. normalize_audio (bool): Whether to normalize audio signal. Defaults to False. shard_strategy (str): Tarred dataset shard distribution strategy chosen as a str value during ddp. - `scatter`: The default shard strategy applied by WebDataset, where each node gets a unique set of shards, which are permanently pre-allocated and never changed at runtime. - `replicate`: Optional shard strategy, where each node gets all of the set of shards available in the tarred dataset, which are permanently pre-allocated and never changed at runtime. The benefit of replication is that it allows each node to sample data points from the entire dataset independently of other nodes, and reduces dependence on value of `shuffle_n`. .. warning:: Replicated strategy allows every node to sample the entire set of available tarfiles, and therefore more than one node may sample the same tarfile, and even sample the same data points! As such, there is no assured guarantee that all samples in the dataset will be sampled at least once during 1 epoch. Scattered strategy, on the other hand, on specific occasions (when the number of shards is not divisible with ``world_size``), will not sample the entire dataset. For these reasons it is not advisable to use tarred datasets as validation or test datasets. global_rank (int): Worker rank, used for partitioning shards. Defaults to 0. world_size (int): Total number of processes, used for partitioning shards. Defaults to 0. rrNFrr4r) rrrrr^r`rarrrrrrrrrrr^r`rarrrc sXtd| td| | |_| |_| |_tj||||||||| | |d dS)Nrr) rrrr\rrrrrrr)rinforr^r`rarr)rTrrrr\rrrrr^r`rarrrrrrrCs$ z(TarredAudioToSpeechLabelDataset.__init__cCrrrrrrrrirz4TarredAudioToSpeechLabelDataset.fixed_seq_collate_fncCstr)NotImplementedErrorrrrrsliced_seq_collate_fnlsz5TarredAudioToSpeechLabelDataset.sliced_seq_collate_fncCrrrrrrrrorz8TarredAudioToSpeechLabelDataset.vad_frame_seq_collate_fn)rrrrrrrr'rrrrrr5rrrrrrr2sTL     &r2cseZdZdZedeeeeffddZ dddddddddddd d e ee efd e d ee ed e d ddeedeede dee ee e e fde dee deedeeffddZddZdefddZddZd d!Zd"d#ZZS)$AudioToMultiLabelDataseta Dataset that loads a json file containing paths to audio files, durations (in seconds), and a sequence of labels. Each new line is a different sample. Example below: {"audio_filepath": "/path/to/audio_wav_0.wav", "duration": time_in_sec_0, "label": "0 1 1 0 1", "offset": offset_in_sec_0} ... {"audio_filepath": "/path/to/audio_wav_n.wav", "duration": time_in_sec_n, "label": "0 1 0 0 1", "offset": offset_in_sec_n} Args: manifest_filepath (Union[str, List[str]]): Path to manifest json as described above. Can be comma-separated paths. labels (Optional[list]): String containing all the possible labels to map to if None then automatically picks from ASRSpeechLabel collection. min_duration (float): Dataset parameter. All training files which have a duration less than min_duration are dropped. Note: Duration is read from the manifest JSON. Defaults to 0.1. max_duration (float): Dataset parameter. All training files which have a duration more than max_duration are dropped. Note: Duration is read from the manifest JSON. Defaults to None. trim_silence (bool): Whether to use trim silence from beginning and end of audio signal using librosa.effects.trim(). Defaults to False. channel selector (Union[str, int, List[int]]): string denoting the downmix mode, an integer denoting the channel to be selected, or an iterable of integers denoting a subset of channels. Channel selector is using zero-based indexing. If set to `None`, the original signal will be used. window_length_in_sec (float): length of window/slice (in seconds) Use this for speaker recognition and VAD tasks. shift_length_in_sec (float): amount of shift of window for generating the frame for VAD task in a batch Use this for VAD task during inference. normalize_audio (bool): Whether to normalize audio signal. Defaults to False. is_regression_task (bool): Whether the dataset is for a regression task instead of classification. Defaults to False. cal_labels_occurrence (bool): Whether to calculate occurrence of labels Defaults to False. delimiter (Optional[str]): Delimiter to use when splitting the label string, default to None. normalize_audio_db (Optional[float]): normalize audio signal to a target db, default to None. r$cCstd|durt|drt|jdntttdtd}|jr7|ttdtttdtd|S|tdt ttdtd |S) rsrtNrwrxrurzzB, Tr|rrrrrrrs*   z%AudioToMultiLabelDataset.output_typesNFr) r int_values augmentorrr trim_silencerrr delimiternormalize_audio_dbrr]rr7r81nemo.collections.asr.parts.perturb.AudioAugmentorrrr9rrrr:r;c sFtt|tr|d}| |_| |_tj|||| | | d|_ t |||d|_ ||_ | |_ | |_i|_d|_| s|r>|n||_|jdurMt|jnd|_ii|_|_t|jD]"\}}||j|<||j|<| r~|j j||j|<|j|j|q\tt|jddD]}td||j|qdSg|_d|_dS)Nr)rrrrrr:r]r7r8r4rr)rrrrr6r:r;r rrrr\rrrrr_get_label_setrrrrrrrDrdrrr)rTrr]rr7r8rrr9rrrr:r;rrrrrrrsF       z!AudioToMultiLabelDataset.__init__cCJg}|jD]}|j}|r|jr||jn|}||qtt|Srrrr:r6rersetrTrr label_strlabel_str_listrrrr>   z'AudioToMultiLabelDataset._get_label_setrCcdjr |jn|}jr dd|D}t|}|Sfdd|D}t|}|S)NcSg|]}t|qSrrrsrrrrr zAAudioToMultiLabelDataset._label_str_to_tensor..crrrrIrrrrrr:r6rr%r&rrrTrCrrrr_label_str_to_tensorz-AudioToMultiLabelDataset._label_str_to_tensorcCrrrrrrrr rz AudioToMultiLabelDataset.__len__c Cs|j|}|j}|durd}|jj|j||j|j|j|jd}|t | d }}| |j}t | d }||||fS)Nr)rrrr normalize_db)rrr\rrrrrr;r%r&sizerrNrrrrrrs   z$AudioToMultiLabelDataset.__getitem__cCrrrrrrrr'rz$AudioToMultiLabelDataset._collate_fn)rrrrrrrrrrrrr'rrrr>rNrrrrrrrrr6ss^)%      9  r6c"seZdZdZdddddddddddddd deeeefd eeeefd ed eeed ed e dddee dee de de dedededeedee f fddZ ddZ defddZ d d!Zd"d#Zd$d%Zd&d'Zd(d)ZZS)*TarredAudioToMultiLabelDatasetaS A similar Dataset to the AudioToMultiLabelDataset, but which loads tarred audio files. Accepts a single comma-separated JSON manifest file (in the same style as for the AudioToSpeechLabelDataset), as well as the path(s) to the tarball(s) containing the wav files. Each line of the manifest should contain the information for one audio file, including at least the transcript and name of the audio file within the tarball. Valid formats for the audio_tar_filepaths argument include: (1) a single string that can be brace-expanded, e.g. 'path/to/audio.tar' or 'path/to/audio_{1..100}.tar.gz', or (2) a list of file paths that will not be brace-expanded, e.g. ['audio_1.tar', 'audio_2.tar', ...]. See the WebDataset documentation for more information about accepted data and input formats. If using multiple processes the number of shards should be divisible by the number of workers to ensure an even split among workers. If it is not divisible, logging will give a warning but training will proceed. In addition, if using mutiprocessing, each shard MUST HAVE THE SAME NUMBER OF ENTRIES after filtering is applied. We currently do not check for this, but your program may hang if the shards are uneven! Notice that a few arguments are different from the AudioToBPEDataset; for example, shuffle (bool) has been replaced by shuffle_n (int). Additionally, please note that the len() of this DataLayer is assumed to be the length of the manifest after filtering. An incorrect manifest length may lead to some DataLoader issues down the line. Args: audio_tar_filepaths: Either a list of audio tarball filepaths, or a string (can be brace-expandable). manifest_filepath (str): Path to the manifest. labels (list): Dataset parameter. List of target classes that can be output by the speaker recognition model. shuffle_n (int): How many samples to look ahead and load to be shuffled. See WebDataset documentation for more details. Defaults to 0. min_duration (float): Dataset parameter. All training files which have a duration less than min_duration are dropped. Note: Duration is read from the manifest JSON. Defaults to 0.1. max_duration (float): Dataset parameter. All training files which have a duration more than max_duration are dropped. Note: Duration is read from the manifest JSON. Defaults to None. trim(bool): Whether to use trim silence from beginning and end of audio signal using librosa.effects.trim(). Defaults to False. window_length_in_sec (float): time length of window/slice (in seconds) # Pass this only for speaker recognition and VAD task shift_length_in_sec (float): amount of shift of window for generating the frame for VAD task. in a batch # Pass this only for VAD task during inference. normalize_audio (bool): Whether to normalize audio signal. Defaults to False. shard_strategy (str): Tarred dataset shard distribution strategy chosen as a str value during ddp. - `scatter`: The default shard strategy applied by WebDataset, where each node gets a unique set of shards, which are permanently pre-allocated and never changed at runtime. - `replicate`: Optional shard strategy, where each node gets all of the set of shards available in the tarred dataset, which are permanently pre-allocated and never changed at runtime. The benefit of replication is that it allows each node to sample data points from the entire dataset independently of other nodes, and reduces dependence on value of `shuffle_n`. .. warning:: Replicated strategy allows every node to sample the entire set of available tarfiles, and therefore more than one node may sample the same tarfile, and even sample the same data points! As such, there is no assured guarantee that all samples in the dataset will be sampled at least once during 1 epoch. Scattered strategy, on the other hand, on specific occasions (when the number of shards is not divisible with ``world_size``), will not sample the entire dataset. For these reasons it is not advisable to use tarred datasets as validation or test datasets. global_rank (int): Worker rank, used for partitioning shards. Defaults to 0. world_size (int): Total number of processes, used for partitioning shards. Defaults to 0. delimiter (Optional[str]): Delimiter to use when splitting the label string, default to None. normalize_audio_db (Optional[float]): normalize audio signal to a target db, default to None. NrFrr) rrr7r8rrr9rrrrr:r;rrr]rrr7r8r<rrr9rrrrr:r;c sntt|tr|d}| |_| |_||_||_t j ||| | dd|_ t |j j |_t|||d|_| s|r;|n||_|jdurJt|jnd|_ii|_|_t|jD]\}}||j|<||j|<qYtt|jddD]}td||j|qsng|_d|_t|| || d}ttj|d t t!|t"tj#t$d d t%d d |j&t'|j(|_)dS)NrT)rrrrrr=r4rrrrrrrr)*rrrrr6rrr:r;r rrr<rrrr\r>rrrrrrrdrrrrrrrrrrrrrrrrr)rTrrr]rrr7r8rrr9rrrrr:r;rrrrrrrrsX          z'TarredAudioToMultiLabelDataset.__init__cCr?rr@rBrrrr>rEz-TarredAudioToMultiLabelDataset._get_label_setrCcrF)NcSrGrrHrIrrrrr zGTarredAudioToMultiLabelDataset._label_str_to_tensor..crrrKrIrrrrrrLrMrrrrNrOz3TarredAudioToMultiLabelDataset._label_str_to_tensorcr)rcr) zATarredAudioToMultiLabelDataset._filter..TarredAudioFiltercrrrrrrrrrzJTarredAudioToMultiLabelDataset._filter..TarredAudioFilter.__init__cSrrrrrrrr r zJTarredAudioToMultiLabelDataset._filter..TarredAudioFilter.__iter__cSr rr rrrrrrzJTarredAudioToMultiLabelDataset._filter..TarredAudioFilter.__next__csrrrrrrrrrzUTarredAudioToMultiLabelDataset._filter..TarredAudioFilter._internal_generatorNr rrrrr!r"r!r#r$rrrrr%z&TarredAudioToMultiLabelDataset._filtercCs|\}}tjtj|\}}|jj|}|j|}|j}|dur$d}t|} |j j | ||j |j |j d} | | t| jd} } ||j} t| d}| | | |fS)r&Nr)rrrrP)rrrrrrrr(r)r\rrrr;r*r%r&rrrNrrQr+rrrrs(     z,TarredAudioToMultiLabelDataset._build_samplecCr/rr0rrrrr #rz'TarredAudioToMultiLabelDataset.__iter__cCrrrrrrrr&rz&TarredAudioToMultiLabelDataset.__len__cCrrrrrrrr)rz*TarredAudioToMultiLabelDataset._collate_fn)rrrrrrrr'rrrrr>rNrrr rrrrrrrrR+snL      H  3"rRcseZdZdZedeeeeffddZ ddddd dd d ee eBd e ed e dBde dBde de dBde dBde ffddZ ddZddZZS)AudioPairToLabelDataseta Dataset class for audio pairs classification tasks, such as calculating EER for speaker verification. The input manifest file should contain pairs of audio files and a label. It's format is almost the same as `AudioToSpeechLabelDataset` except that the `audio_filepath` field should be a list of two audio file paths instead of one, and that `offset` and `duration` are not used as the dataset class will load the whole audio. Example of a line in the manifest file: { "audio_filepath": ["/path/to/audio_wav_0.wav", "/path/to/audio_wav_1.wav"], "duration": null, # not used, will load the whole audio "offset": 0.0, # not used, will load the whole audio "label": "0" # label for the pair, can be a string or an integer } r$cCstd|durt|drt|jdntttdttd|dur,t|dr,t|jdntttdtttdtttdtd}|S)rsrtNrwrxru)rNr{audio_signal_2a_sig_length_2rr)rrr rwrrrrrrrr>s$  z$AudioPairToLabelDataset.output_typesrNFrr4)rrrr^r`rarrrrrr^r`rac s&tj||||||||| ddd dS)NF) rrr\rrrr^r`rarr)rr) rTrrr\rrrr^r`rakwargsrrrr[s z AudioPairToLabelDataset.__init__c Cs|j|}|j}|jj|ddd|jd}|t|jd}}|jj|ddd|jd}|t|jd}} t|j |j } td} |||| | | fS)Nrr'r4) rrr\rrr%r&rrrr) rTrr audio_pairrrr features2f2fl2rrrrrrws z#AudioPairToLabelDataset.__getitem__cCslt|\}}}}}}tt||||}t||\} } } } tt||||} t|| \}}}}| | ||| | fSr)r?listrX)rTrFaudio1 audio_len1audio2 audio_len2r label_lenbatch1a_sig1 a_sig_len1 pair_labelpair_label_lenbatch2a_sig2 a_sig_len2rHrrrrs z,AudioPairToLabelDataset.fixed_seq_collate_fn)rrrrrrrrrrrrrrrrrrrrrrS-s:"    rS)5r(rtypingrrrrr%'nemo.collections.asr.data.audio_to_textrr1nemo.collections.asr.parts.preprocessing.featuresr0nemo.collections.asr.parts.preprocessing.segmentr valid_sf_formats+nemo.collections.common.parts.preprocessingr nemo.core.classesr r nemo.core.neural_typesr rrrr nemo.utilsrrrnemo.utils.distributedrjoinkeysrTensorr'r-r2r<rSrXrqrrrrrr1r2r6rRrSrrrrsD         (*2ScHv9