o wiI @sddlZddlZddlZddlmZddlmZddlmZm Z m Z m Z ddl Z ddlZddlmZmZddlmZddlmZmZmZmZmZmZmZmZddlmZdd lm Z m!Z!m"Z"dd l#m$Z$dd l%m&Z&m'Z'dd l(m)Z)m*Z*dd l+m,Z,m-Z-m.Z.ddl/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7ddl8m9Z9ddl:m;Z;ddlm?Z?eGdddZ@deAde)deAfddZB dAde eCe)fdeDdeDdejEjFjGdejEjFjHf ddZI dAde)deDdeDdejEjFjGdejEjFjHf dd ZJ dAd!e)deDdeDdejEjFjGdejEjFjHf d"d#ZKdAdeLe eAffd$d%ZMd&edeLee!ffd'd(ZNd)d*ZOde e)eCfde)fd+d,ZPd-d.ZQd/eRe;Bfd0d1ZSd&ed2eTdefd3d4ZUd&edefd5d6ZVdeWfd7d8ZXd9eAfd:d;ZYdeDeRBdeWfd?d@Z[dS)BN) dataclass)partial)AnyOptionalSequenceUnion)CutSet RecordingSet)Cut)CutConcatenateDynamicBucketingSamplerDynamicCutSamplerIterableDatasetWrapperReverbWithImpulseResponseRoundRobinSampler ZipSamplermake_worker_init_fn) resolve_seed) CutSamplerSamplingConstraintTimeConstraint) LazyFlattener)fastcopyfix_random_seed) DictConfig OmegaConf)IncompleteConfigErrorguess_parse_cutsetread_cutset_from_config)BucketingFilterDurationFilter FixedBucketBatchSizeConstraint2D*MultimodalFixedBucketBatchSizeConstraint2DMultimodalSamplingConstraintTokenCountFilterTokenPerSecondFilterTokenPerTokenFilter)apply_prompt_format_fn)PromptFormatter)TokenizerWrapper)loggingc@seZdZUdZdZeed<dZeed<dZeed<dZ e dBed<dZ eed<dZ e ed <dZe ed <dZedBed <dZedBed <dZedBed <dZe ed<dZeedBed<dZeed<dZeed<dZeed<dZeed<dZe ed<dZe ed<dZedBed<dZe ed<dZee Bed<dZ edBed<dZ!e ed<dZ"e ed<d Z#e ed!<dZ$e%e efdBed"<dZ&e ed#<dZ'e dBed$<dZ(e ed%<dZ)e dBed&<dZ*edBed'<dZ+edBed(<dZ,edBed)<d*Z-edBed+<ed,Z.edBed-<d*Z/eed.<ed,Z0eed/<dZ1edBed0<dZ2edBed1<dZ3e ed2<d*Z4eed3<ed,Z5eed4<dZ6e ed5<d6Z7eed7<d8Z8ee Bed9<d8Z9eed:<dZ:e ed;<dZ;ee BdBed<<dZZ=e>eefed?<d@Z?eedA<dZ@e edB<dZAe edC<dDZBeedE<dFZCeedG<dZDe edH<dIZEeFeedJ<dZGeFeedK<dLZHe edM<dZIeFeedN<dZJeFeedO<dZKe edP<dZLe edQ<dZMe dBedR<d@ZNeedS<dZOeFeedT<dUZPe edV<dWZQe edX<dYZRe edZ<dZSe ed[<dZTe ed\<dZUe ed]<dZVe ed^<dS)_LhotseDataLoadingConfiga  Structured config used for OmegaConf schema validation. It's also a single source of truth for reading default option values. The options not supported anymore but present, e.g., in old configs, will be emitted in a DeprecationWarning and ignored. N input_cfgmanifest_filepathtarred_audio_filepaths cuts_path shar_pathFskip_missing_manifest_entriestarred_random_access batch_sizebatch_durationquadratic_duration use_bucketingbucket_batch_size num_bucketsi'num_cuts_for_bins_estimatebucket_duration_binsbucket_buffer_sizeTconcurrent_bucketingbucketing_2d_strict_modeshuffle_buffer_size drop_lasttrng shard_seedmax_open_streamscuda_expandable_segments multi_config round_robinsampler_fusionsampler_weights pretokenize prompt_formatuse_multimodal_samplingaudio_locator_tagtoken_equivalent_duration batch_tokensquadratic_factor min_durationinf max_durationmin_tpsmax_tps min_tokens max_tokensmeasure_total_lengthmin_tptmax_tptshufflei> sample_raterseed num_workers pin_memorychannel_selector noise_path)g$@g4@ noise_snrg?noise_mix_prob perturb_speedconcatenate_samplesg?concatenate_gap_secondsg?concatenate_duration_factorconcatenate_merge_supervisionsg9db_normtruncate_durationrandomtruncate_offset_typecut_into_windows_durationcut_into_windows_hopkeep_excessive_supervisions rir_enabledrir_pathrir_probpad_min_durationright pad_directiontext text_fieldlang lang_field metadata_only force_finiteforce_map_datasetforce_iterable_dataset)W__name__ __module__ __qualname____doc__r,r__annotations__r-r.r/strr0r1boolr2r3intr4floatr5r6r7listr9r:r;r<r=r>r?r@rBrCrDrErGrHdictrIrJrKrLrMrNrOrQrSrTrUrVrWrXrYrZr[r\r]r^r_r`rarbtuplercrdrerfrgrhrirrjrlrmrnrorprqrrrsrurwryrzr{r|r}rrk/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/common/data/lhotse/dataloader.pyr+>s                                         r+use_iterable_datasetconfigreturncCs*|jr |jr Jd|s|jo|j }|S)zDDetermine whether to use iterable dataset for a given configuration.zKConflicting options: force_map_dataset=True and force_iterable_dataset=True)r|r})rrrrrdetermine_use_iterable_datasetsr global_rank world_sizedatasetcCsVt|ts t|}t|ddd|ddr"t|||||dSt|||||dS)a Set up a Lhotse training dataloder. Expects a typical NeMo dataset configuration format, with additional fields: "use_lhotse=True". Some fields in the original NeMo configuration may be ignored. The ``dataset`` parameter should be an instance of a Lhotse-compatible PyTorch Dataset class. It only needs to define the following method ``__getitem__(self, cuts: CutSet) -> Dict[str, torch.Tensor]``. This dataset is not expected to hold a reference to any actual data; it may be interpreted as a function mapping a Lhotse CutSet into a mini-batch of tensors. For an example, see: :class:`nemo.collections.asr.data.audio_to_text_lhotse.LhotseSpeechToTextBpeDataset`, which is constructed from just a tokenizer and essentially loads and collates audio and tokenizes the transcript. The ``tokenizer`` is used both for audio and text datasets for on-the-fly tokenization. This allows us to stratify the bucketing by the count of input/output tokens (depending on modality). If "prompt_format" is additionally provided in the config, we will also apply a prompt formatter. Note that ``tokenizer`` can be any tokenizer type (e.g. both SentencePiece and Aggregate tokenizers work). rDT)enabledrEF)top_level_configrrr tokenizerrrrrr) isinstancerrcreate"maybe_set_cuda_expandable_segmentsget'get_lhotse_dataloader_from_multi_config(get_lhotse_dataloader_from_single_configrrrr!get_lhotse_dataloader_from_configs    rc Cstdt|}t|j|_t|jt||||d\}}|r5tt||dt |||jd|j dkd}nt||d}t j j jd i|d|j |jd}|S) a Set up a Lhotse training dataloder. Expects a typical NeMo dataset configuration format, with additional fields: "use_lhotse=True". Some fields in the original NeMo configuration may be ignored. The ``dataset`` parameter should be an instance of a Lhotse-compatible PyTorch Dataset class. It only needs to define the following method ``__getitem__(self, cuts: CutSet) -> Dict[str, torch.Tensor]``. This dataset is not expected to hold a reference to any actual data; it may be interpreted as a function mapping a Lhotse CutSet into a mini-batch of tensors. For an example, see: :class:`nemo.collections.asr.data.audio_to_text_lhotse.LhotseSpeechToTextBpeDataset`, which is constructed from just a tokenizer and essentially loads and collates audio and tokenizes the transcript. The ``tokenizer`` is used when text-only datasets are included in dataloading. In these cases we will tokenize ``TextExample``s before sampling mini-batches so that we can account for their number of tokens. Note: this behaviour might eventually be extended to audio datasets too. Note that ``tokenizer`` can be any tokenizer type (e.g. both SentencePiece and Aggregate tokenizers work). z%We will be using a Lhotse DataLoader.rrrrrsamplerrankrr]rrworker_init_fnpersistent_workersNr3r^r_r)r*info$make_structured_with_schema_warningsrr]rget_lhotse_sampler_from_configrrrr^torchutilsdata DataLoaderr_) rrrrrrrdloader_kwargsdloaderrrrrs,       rrc sfdd}|}t|jddD}ig}|D]B\} } zt| } |D]\} } | | | <q-t| |||d\}}WntyX}z td| d| |d}~ww||| <|q tfd d d dDsvJd t}|jd krt | }nK|jdkrt | }n?|jdkrgg}}| D]}||||j dur||j |qt |t|dkr|nd|jd}ntd|j|rtt||dt|||jd|jdkd}nt||d}tjjjdi|d|j|jd}|S)a Set up a Lhotse training dataloder. It works similarly to :func:`get_lhotse_dataloader_from_config`, except that you can provide multiple configs to set up different sampling, batching, and augmentation settings for every dataset and decide how to merge them. The expected format is that the ``configs`` is a dict of group name -> actual config. The first config is treated as a "main" config that determines the RNG, CUDA allocator, and sampler fusion settings. csZtfdddDsJdgd}tttdd<tfdd|DS) z In multi-config setting, the top-level config defines several attributes that overwrite the ones present in sub-configs. c3s|]}|vVqdSNr.0krrr dszVget_lhotse_dataloader_from_multi_config..gather_shared_opts..)r]rBr[zIn a multi-config setting (multi_config=True), the top-level namespace (typically train_ds)must define at least 'seed', 'shard_seed', and 'shuffle' keys that will be shared by all sub-configs.) r]rBr^r_r[rGrHrErzr{r]csi|] }|||qSr)rr)defaultsrrr wszWget_lhotse_dataloader_from_multi_config..gather_shared_opts..)allr structuredr+rr)overwriting_optsr)rrgather_shared_opts_s zCget_lhotse_dataloader_from_multi_config..gather_shared_optscSs(i|]\}}t|tr|dvr||qS))rH)rr)rnamecrrrr|sz;get_lhotse_dataloader_from_multi_config..rzoCannot create a sampler for one of the sub-configs in a multi_config setup.The problematic config is under key=z! and has the following contents: Nc3s|] }|dkVqdS)rNr)rst)source_use_iterable_datasetrrrsz:get_lhotse_dataloader_from_multi_config..zWhen using multiple input_cfg sources ensure they are all tarred or non-tarred (can't mix). You can provide force_iterable_dataset=True to each namespace to fix.ziprFrandomized_round_robinrT) randomizer]z%Unsupported sampler fusion strategy: rrrrr)rr]itemsrrrappendrrGrvaluesrkeysrHlen RuntimeErrorrrrr^rrrrr_)rrrrrr shared_optsconfigssource_samplersrrexpanded_configrvsterr _samplers_weightskeyrrr)rrrrKs                rc Csvt|\}}t||}|jdur!td|j|tt|jd}|jtt|j ddd}t t |jt dd}|j rC|dusCJd|durx|jrx|sQtd|jdurd|jtt||jddd}nt|tsmt|}|jtt|d dd}|jdurt|j}|j|t|j|j|jd d }|jrt ||d |d }|jdur|j|j|j |j!d}|j"dur|j#|j"|j$|j!d}|j%dur|j&|j%|j'd d}|(t)|j*|j+}|(t,|j-|j.|j/d}|dur|jr|(t0|j1|j2}|(t3|j4|j5}t6|}t7|||\}}|j8rHtd|j9d|j:dt;|||j<|j=|j>|j|j?t6||j@|jA|jB|r=dn||rCdn|d } n(td|j9d|j:dtC|||j<|j=|j>|j|rfdn||rldn|d} |jDr| tE|jF|jGd} |jHdur| ttI|jHd} |jJr| tK} |jLr| tM|jNdurtOP|jNnd|jQtRS|jTd} | |fS)z-Create a CutSampler from a dataloader config.NzUsing channel selector %s.)r`) sampling_rate)apply_fnzYou must pass a tokenizer to `get_lhotse_dataloader_from_config` in order toread text-only datasets (enabled via use_multimodal_dataloading)aYou are using a non-tarred dataset and requested tokenization during data sampling (pretokenize=True). This will cause the tokenization to happen in the main (GPU) process,possibly impacting the training speed if your tokenizer is very large.If the impact is noticable, set pretokenize=False in dataloader config.(note: that will disable token-per-second filtering and 2D bucketing features))rrJ)rT)cutssnrmix_probr]random_mix_offsetg?g?)rS offset_typero)durationhopro)r direction preserve_id)rXz>Creating a Lhotse DynamicBucketingSampler (max_batch_duration=z max_batch_size=)rr) constraintr[r@r?r]r9 duration_binsr: buffer_size concurrentrrzPCreating a Lhotse DynamicCutSampler (bucketing is disabled, (max_batch_duration=)rr[r@r?r]rr)gapduration_factor)ri)rir_recordingsprandgen)Urrr`r*rmapr_select_channelresampler\rr_flatten_alt_textrKrIwarningrJtokenize_with_promptrr)tokenizerarmixrrbrcrBrdmuxrjtruncaterlrormcut_into_windowsrnrspadrufilterr rQrSr$rVrWrXr%rTrUr&rYrZdetermine_bucket_duration_binsdetermine_sampling_constraintr6r4r3r r[r@r?r9r:r<r=r rer rfrgri_normalize_loudnessrh_merge_supervisionsrprrqr from_filerrrkRandomr]) rrrrrrnoiser;rrrrrrs                    rrcCs|jrD|jdur2|dusJdt||j|j|jt|jtr!|jnd|jd}| t |}||fSt |j|j |j |j|jd}||fS|jduro|dusQJdt||j|jt|jtr`|jndd}| t |}||fSt|j |j|jd}||fS)a Select an appropriate sampling strategy (constraint) for Lhotse samplers based on the configuration. Sampling constraint affects the batch size (static/dynamic) and bucketing behaviour (1D/2D). It is the appropriate customization point to introduce support of other modalities, as it defines a method for example sequence length measurement (audio duration, text tokens, etc.). Some constraints apply extra filter on ``cuts`` which is why we accept and return the ``CutSet``. Lhotse's default is :class:`TimeConstraint` for regular audio data, other available options are multimodal constraints (joint text + audio) and their 2D bucketing extensions. NzMCannot use bucket_batch_size option if bucket_duration_bins are not provided.)max_seq_len_buckets batch_sizesrM strict_2d max_ratiorX)rMr3rNrOrX)rrrr)max_cutsrSr5)rKr7r"rMr>rrZrrXrrr#r3rNrOr!rUrr4r5)rr;rrrrrrusR   rcCs|jdurt|j}t|dtrdd|D}|S|jr dS|jdurN|jtdkrN|jdur9|jdkr9|jnd}|j}t |||j ddd SdS) a Returns appropriate bucket bins based on configuration. If user provided them explicitly, we just pass them along; otherwise, we try to create provisional bins when min/max duration is available. We might return None if it's impossible to determine the bins without computing data statistics, in which case it will be automatically done at the start of training (but may take a few minutes). NrcSsg|]}t|qSr)r)ritemrrr sz2determine_bucket_duration_bins..rRgrrP) r;r to_containerrrrKrSrrQnplinspacer9tolist)ransbeginendrrrrs   rcCstt}t|tst|}tt|}tt|}||}|d|r6t dd |t |t |}t||}|ddrRt dd|_|jrZt d|S) z Checks the schema and fills missing default option values. Warns the user if any of the fields are not supported by the current schema but does not raise exceptions. use_lhotsezCThe following configuration keys are ignored by Lhotse dataloader: ,r2Fz^Option 'tarred_random_access' is deprecated and replaced with 'skip_missing_manifest_entries'.TzNote: skip_missing_manifest_entries is set to True. If any of your manifests and tar files are mismatched, the entire tar file will be skipped without warning. It's your responsibility to ensure data integrity with this setting.)rrr+rrsetrrdiscardr*rjoin masked_copyrmergerr1)rdefaultsupported_keys received_keysunsupported_keysrrrrs.     rcCsnt|tr|jD]}|jdurt||j|j|_q|St|dr.t |j r.| |}|St dt |)zCReturn the text in the example according to the provided tokenizer.NrzUnsupported type of example: ) rr supervisionsrvrasarraylanguagetokenshasattrcallablerrtype)examplerrrrrrs    rrJcCsDt|tr t||}t||}|D] \}}t|||q|S)zCTokenize the example with the provided tokenizer and prompt format.)rrr(resolver'rsetattr)rrrJencodedrvaluerrrrs  rricCs|j|ddS)NF)target mix_first)normalize_loudness)rrirrrrsrcCs|Sr)merge_supervisions)rrrrrsrcs|g}t|tr|jdus|jddur|S|jdd}|jd}|D]"|fdd}ddd|_| |q'|S) Nalt_textwav) audio_formatcst|dddS)Nrvrx)rvr)r)rrrr%sz#_flatten_alt_text..rvrx)rvrx) rr customrmove_to_memorypoprcopymap_supervisionsr)cutr  paired_text text_instancerr*rrs$    rrcCs||r:tjrs    (  (     3 I |/7#'