o }oiz @sRddlZddlZddlmZddlmZddlmZddlm Z m Z m Z m Z m Z ddlZddlmZmZmZddlmZmZddlmZmZmZdd lmZmZmZdd lmZmZm Z dd l!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(dd l)m*Z*d e ee+fde ee,ffddZ-Gddde.Z/ia0de e1fddZ2de1fddZ3de e1e4e1ffddZ5de6ee,ffddZ7dede+dee,gfdd Z8e5d!d ede6ee,ffd"d#Z9e5d$d ede6ee,ffd%d&Z:e5d'd ede6ee,ffd(d)Z;e5d*d ede6ee,ffd+d,Ze5d4d5gde6ee,ffd6d7Z?e5d8gde6ee,ffd9d:Z@d;e e1efdZAd?ed@e1defdAdBZBe5dCdDgde6ee,ffdEdFZCddGdHdIdJedKe4e eDeEfdLe eDdfdMe e1eDfdNe,def dOdPZFdQe e1e+ejfdefdRdSZGdS)TN)partialrepeat)Path)KeysViewMappingSequenceTupleUnion)CutSetFeatures Recording)Array TemporalArray)CutMixedCut PaddingCut) DictConfig ListConfig OmegaConf)LazyNeMoIteratorLazyNeMoTarredIteratorexpand_sharded_filepaths) AudioTurnLhotseTextAdapterLhotseTextPairAdapterNeMoMultimodalConversation&NeMoMultimodalConversationJsonlAdapterNeMoSFTJsonlAdapterTextTurn) get_full_pathconfigreturncstts tddurt\}}||fStfdddD}|r<ddur2tdt\}}||fSt\}}||fS)z Reads NeMo configuration and creates a CutSet either from Lhotse or NeMo manifests. Returns a tuple of ``CutSet`` and a boolean indicating whether the data is tarred (True) or not (False). input_cfgNc3s|] }|duVqdSN)get).0optr!^/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/common/data/lhotse/cutset.py :sz*read_cutset_from_config..) cuts_path shar_pathmanifest_filepathzCYou must specify either: manifest_filepath, cuts_path, or shar_path) isinstancerr%read_dataset_configallIncompleteConfigErrorread_nemo_manifestread_lhotse_manifest)r!cuts is_tarreduse_nemo_manifestr)r(r*read_cutset_from_config-s     r8c@seZdZdZdS)r2z Placeholder for an error raised.N)__name__ __module__ __qualname____doc__r)r)r)r*r2Esr2cCstS)z Return the names of all registered data type parsers. Example: >>> get_known_config_data_types() ["nemo", "nemo_tarred", "lhotse", ...] )KNOWN_DATA_CONFIG_TYPESkeysr)r)r)r*get_known_config_data_typesNs r?data_type_namecCst|S)a Return the parsing function for a given data type name. Parsing function reads a dataloading config and returns a tuple of lhotse ``CutSet`` and boolean indicating whether we should use iterable dataset (True) or map dataset (False) mechanism ("is tarred"). )r=)r@r)r)r* get_parser_fnZsrAnamecsfdd}|S)a? Decorator used to register data type parser functions. Parsing function reads a dataloading config and returns a tuple of lhotse ``CutSet`` and boolean indicating whether we should use iterable dataset (True) or map dataset (False) mechanism ("is tarred"). Example: >>> @data_type_parser("my_new_format") ... def my_new_format(config): ... return CutSet(read_my_format(**config)), True ... ... fn = get_parser_fn("my_new_format") ... cuts, is_tarred = fn({"my_arg_0": ..., "my_arg_1": ..., ...}) cs,ttr |t<|SD]}|t|<q |Sr$)r/strr=)fnnrBr)r* _decoratorus  z$data_type_parser.._decoratorr))rBrGr)rFr*data_type_parserds  rHcCs|dd|dd|dd|dd|d d|d d|d d |d d |dd|dd|ddd }|j}t|ttfrIt|}t||d\}}||fS)a Input configuration format examples. Example 1. Combine two datasets with equal weights and attach custom metadata in ``tags`` to each cut:: input_cfg: - type: nemo_tarred manifest_filepath: /path/to/manifest__OP_0..512_CL_.json tarred_audio_filepath: /path/to/tarred_audio/audio__OP_0..512_CL_.tar weight: 0.5 tags: lang: en some_metadata: some_value - type: nemo_tarred manifest_filepath: /path/to/manifest__OP_0..512_CL_.json tarred_audio_filepath: /path/to/tarred_audio/audio__OP_0..512_CL_.tar weight: 0.5 tags: lang: pl some_metadata: some_value Example 2. Combine multiple (4) datasets, with 2 corresponding to different tasks (ASR, AST). There are two levels of weights: per task (outer) and per dataset (inner). The final weight is the product of outer and inner weight:: input_cfg: - type: group weight: 0.7 tags: task: asr input_cfg: - type: nemo_tarred manifest_filepath: /path/to/asr1/manifest__OP_0..512_CL_.json tarred_audio_filepath: /path/to/tarred_audio/asr1/audio__OP_0..512_CL_.tar weight: 0.6 tags: lang: en some_metadata: some_value - type: nemo_tarred manifest_filepath: /path/to/asr2/manifest__OP_0..512_CL_.json tarred_audio_filepath: /path/to/asr2/tarred_audio/audio__OP_0..512_CL_.tar weight: 0.4 tags: lang: pl some_metadata: some_value - type: group weight: 0.3 tags: task: ast input_cfg: - type: nemo_tarred manifest_filepath: /path/to/ast1/manifest__OP_0..512_CL_.json tarred_audio_filepath: /path/to/ast1/tarred_audio/audio__OP_0..512_CL_.tar weight: 0.2 tags: src_lang: en tgt_lang: pl - type: nemo_tarred manifest_filepath: /path/to/ast2/manifest__OP_0..512_CL_.json tarred_audio_filepath: /path/to/ast2/tarred_audio/audio__OP_0..512_CL_.tar weight: 0.8 tags: src_lang: pl tgt_lang: en shuffleF shard_seedtrng text_fieldtext lang_fieldlang metadata_only force_finitemax_open_streamsNtoken_equivalent_durationskip_missing_manifest_entriesforce_map_datasetforce_iterable_dataset) rIrJrLrNrPrQrRrSrTrUrVpropagate_attrs)r%r#r/rCrrloadparse_and_combine_datasets)r!rXr#r5r6r)r)r*r0s" ?            r0grp_cfgrXcCs|jtvsJd|j|jdkrt|j}||\}}n t|j|d\}}|d}dur<|jtt|ddd}||fS)zEParse a group configuration, potentially combining multiple datasets.z7Unknown item type in dataset config list: grp_cfg.type=grouprWtagsN)r])apply_fn) typer?rArZr#r%mapr attach_tags)r[rX parser_fnr5r6 extra_tagsr)r)r* parse_groups   rdtxtcCs8tt|j|j|j|jd}|dds|}|dfS)z-Read paths to text files and create a CutSet.pathslanguageshuffle_shardsrJrQFT)r rrgrhrIrJr%rr!r5r)r)r*read_txt_pathss rktxt_pairc CsXtt|j|j|d|d|d|d|j|jd}|dds(|}|dfS) z?Read paths to source and target text files and create a CutSet.source_languagetarget_languagequestions_pathquestions_language) source_paths target_pathsrmrnrorprirJrQFT)r rrqrrr%rIrJrrjr)r)r*read_txt_pair_pathss rsnemo_sft_jsonlcCs<tt|j|d|j|jd}|dds|}|dfS)z7Read paths to Nemo SFT JSONL files and create a CutSet.rhrfrQFT)r rrgr%rIrJrrjr)r)r*read_nemo_sft_jsonl s rumultimodal_conversationc CsHtt|j|d|j|d|j|jd}|dds |}|dfS)zFRead paths to multimodal conversation JSONL files and create a CutSet.tarred_audio_filepathsrS)r.rwaudio_locator_tagrSrirJrQFT)r rr.r%rxrIrJrrjr)r)r*"read_multimodal_conversation_jsonls ryr]cCs"|D] \}}t|||q|S)z'Attach extra tags to a cut dynamically.)itemssetattr)cutr]keyvalr)r)r*ra/srar\ config_listc shg}g}gt|dksJd|D]=}|}|D]\}}||vr)|||<q||||<qt||\}} ||| |d} durO|| qtfddD} | sz|dse|drvtd |dd |dd nt d t|dkst|t|ksJd t|dkrt ||r|nd|d|d|dp|dd}n|\}|dfS)zPParse a list of dataset configurations, potentially combining multiple datasets.rz#Empty group in dataset config list.weightNc3s|] }|dkVqdS)rNr))r&t tarred_statusr)r*r+Tsz-parse_and_combine_datasets..rUrVz]Not all datasets in the group have the same tarred status, using provided force_map_dataset (z) and force_iterable_dataset (z') to determine the final tarred status.zyMixing tarred and non-tarred datasets is not supported when neither force_map_dataset nor force_iterable_dataset is True.z\Missing dataset weight. When weighting datasets, every dataset must have a specified weight.rRrJrQrPweightsrRseedrQ) lencopyrzrdappendr%r1loggingwarning ValueErrormux) rrXr5ritemnext_propagate_attrskv item_cutsitem_is_tarredwall_same_tarred_statusr)rr*rZ6sX        rZlhotse lhotse_sharc Cs|ddu}|r/|dd}|dd}|dd}|ddur(td t|jttfrWtd |jd t j dit |j|d |d }|sS|sS| }||fSt|jt rtdg}g}|jD]e}t|ttfr|} t j dit | |d |d } t| } n0t|t rt|dkrt|dttfsJd|d |\} } t j dit | |d |d } td| d| || || qit|||dd||d}||fSt|jtr!dd|jD} d|jvsJd|j|r d| di} t j | d |d}|s|s| }||fStdt|jd|j|j} t | tt| d}||fS)z8Read paths to Lhotse manifest files and create a CutSet.r-NrJrKrPFrQr,zMNote: lhotse.cuts_path will be ignored because lhotse.shar_path was provided.zEInitializing Lhotse Shar CutSet (tarred) from a single data source: ''T)rirzInitializing Lhotse Shar CutSet (tarred) from multiple data sources with a weighted multiplexer. We found the following sources and weights: rzSupported inputs types for config.shar_path are: str | list[str] | list[tuple[str, number]] where str is a path and number is a mixing weight (it may exceed 1.0). We got: 'z- path= weight=rRrcSsi|] \}}|t|qSr))rr&rrr)r)r* sz(read_lhotse_manifest..r5zInvalid value for key 'shar_path': a dict was provided, but didn't specify key 'cuts' pointing to the manifests. We got the following: config.shar_path=)fieldsrirzUnexpected value for key 'shar_path'. We support string, list of strings, list of tuples[string,float], and dict[string,list[string]], but got: type(config.shar_path)=z config.shar_path= manifest_pathr))r%warningswarnr/r-rCrrinfor from_shar_resolve_shar_inputsrrrintfloatrrrrzr> RuntimeErrorr_r, from_filer`rresolve_relative_paths) r!r6rJrPrQr5cutsetsrrpathcsrrr)r)r*r4ss      <   *       r4lhotse_as_conversationcs4dtdtffdd }t\}}||}||fS)Nr|r"csXt|djdt|jdjddg}t|dr!t|jddg|}t|j|j |j dS)Nuser)r|rolerxr assistant)valuercontext)idturnsrScustom) rrxr supervisionsrMhasattrrrrrSr)r|rr(r)r*cut_to_conversations z8read_lhotse_as_conversation..cut_to_conversation)rrr8r`)r!rr5r6r)r(r*read_lhotse_as_conversations  rr only_metadatacCs*|rtdtt|didSt|dS)Nr5zcuts.*)r)in_dir)dictsortedrglob)rrr)r)r*rs rr|rcst|tr|St|tr|jD] }t|j|_q|Sfdd}fdd|jr0||j|jr8|j |j dur]|j D]\}}t|t rP||qBt|t ttfr\|qB|S)z;Resolve relative paths in a Cut object to their full paths.cs*|jD]}|jdkrt|jd|_qdS)Nfile manifest_file)sourcesr_r source)r audio_sourcerr)r*resolve_recordings  z1resolve_relative_paths..resolve_recordingcst|tr |j|_dS|jdvr0tttt|j|jd}t|j |_t|j |_dS|jdvr?t|jd|_dSdS)N) numpy_files lilcom_filesr)kaldiiochunked_lilcom_hdf5 lilcom_chunky lilcom_hdf5 numpy_hdf5) r/rarray storage_typerr rC storage_path storage_keyparentrB)rabspathr resolve_arrayr)r*rs    z-resolve_relative_paths..resolve_arrayN)r/rrtracksrr| has_recording recording has_featuresfeaturesrrzr rrr )r|rtrackrr}rr)rr*rs(         rnemo nemo_tarredc Csi}dD]}||vr|dkr|||d<q||||<q|dd}|dd}d|i}|ddu}t|jttfrztd |d |jd |ri|sitt|jf|j |d dd |}|se| }||fStt |jfi||}||fStd|dg}g} |r|j nt d} t |j| D]\} } |rt| t ttfr| \} | d} n| } |r|std| | |d dd|}n t | fi||}t| tst| dkrt|}n t| trt| dkrt| dttfsJd| d | d}td| d||ddur$|D]}|t|| |qq|t|| |qt|| |d|dd|pA|d}||fS)z.Read NeMo manifest and return a Lhotse CutSet.)rLrNrIrJ extra_fieldsrIrirPFrQrwNzOInitializing Lhotse CutSet from a single NeMo manifest (is_tarred=z): 'rrT) tar_pathsrTzOInitializing Lhotse CutSet from multiple NeMo manifest (is_tarred=z_) sources with a weighted multiplexer. We found the following sources and weights: r$r)rrrTrrzSupported inputs types for config.manifest_filepath are: str | list[list[str]] | list[tuple[str, number]] where str is a path and number is a mixing weight (it may exceed 1.0). We got: 'z- manifest_path=rrRrJrKrr))r%r/r.rCrrrr rrwrrziplisttuplerrrrr to_shardsrr)r! common_kwargsr}rPrQ notar_kwargsr6r5rrr manifest_infotar_pathr nemo_iterrsubiterr)r)r*r3s    NL       r3rKF)rRrrQrrrRrrQcGsl|dur|r Jdtj||||d}|S|sdd|D}t|dkr+|d}|Stj|||d}|S) a Helper function to call the right multiplexing method flavour in lhotse. The result is always an infinitely iterable ``CutSet``, but depending on whether ``max_open_streams`` is set, it will select a more appropriate multiplexing strategy. NzJmax_open_streams and metadata_only/force_finite options are not compatible)rrrRcSsg|]}|qSr)r)r&rr)r)r* szmux..rr)rr)r infinite_muxrr)rrRrrQrr5r)r)r*rs   rinpc Csddlm}t|ttjfr;z|tdd|D}t |\}}|WSt y:}z t d|d|d}~wwt|t rz| drQ|td |g}n!| d s[| d rg|td |g}n |td |g}t |\}}|St dt|d)a  Utility function that supports opening a CutSet from: * a string path to YAML input spec (see :func:`read_dataset_config` for details) * a string path to Lhotse non-tarred JSONL manifest * a string path to NeMo non-tarred JSON manifest * a dictionary specifying inputs with keys available in :class:`nemo.collections.common.data.lhotse.dataloader.LhotseDataLoadingConfig` It's intended to be used in a generic context where we are not sure which way the user will specify the inputs. r)$make_structured_with_schema_warningscSsg|] \}}|d|qS)=r)rr)r)r*rsz&guess_parse_cutset..z)Couldn't open CutSet based on dict input z1 (is it compatible with LhotseDataLoadingConfig?)Nz.yamlz input_cfg=z.jsonlz .jsonl.gzz cuts_path=zmanifest_filepath=zUnsupported input type: z (expected a dict or a string)).nemo.collections.common.data.lhotse.dataloaderrr/r omegaconfrr from_dotlistrzr8 ExceptionrrCendswithr_)rrr!r5_er)r)r*guess_parse_cutsets.      r)Hrr functoolsr itertoolsrpathlibrtypingrrrr r rrr r r lhotse.arrayrr lhotse.cutrrrrrr1nemo.collections.common.data.lhotse.nemo_adaptersrrr1nemo.collections.common.data.lhotse.text_adaptersrrrrrrr4nemo.collections.common.parts.preprocessing.manifestr rboolr8rr2r=rCr?rArrHrr0rdrkrsruryrarZr4rrrr3rrrrr)r)r)r*s   $ "  S  <\ 1v   "