o wiO@s>ddlZddlZddlZddlmZddlmZddlmZddl m Z m Z m Z m Z mZddlZddlmZmZmZddlmZmZddlmZmZmZdd lmZmZmZdd lmZm Z m!Z!dd l"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*dd l+m,Z,d eee-fde ee.ffddZ/Gddde0Z1ia2de e3fddZ4de3fddZ5dee3e6e3ffddZ7de8ee.ffddZ9dede-dee.gfdd Z:e7d!d ede8ee.ffd"d#Z;e7d$d ede8ee.ffd%d&Ze7d-gde8ee.ffd.d/Z?d0ee3efd1e.de-fd2d3Z@d4e-fd5d6ZAe7d7d8ee6eefde-de8ee.ffd9d:ZBe7d;dZCd?ed@e3dAeDde&fdBdCZEe7dDgde8ee.ffdEdFZFeGdGeGdHfdIe3de3fdJdKZHGdLdMdMZI N O Pdnd?ed@e3dAeDdQe e3dRe e3dSe.de&fdTdUZJe7dVgde8ee.ffdWdXZKd0ee3efd1e.de-fdYd3Z@d?edZe3defd[d\ZLe7d]d^gde8ee.ffd_d`ZMddadbdcddedee6eeNeDfdfeeNdfdgee3eNfdhe.def didjZOdkee3e-ejfdefdldmZPdS)oN)partialrepeat)Path)KeysViewMappingSequenceTupleUnion)CutSetFeatures Recording)Array TemporalArray)CutMixedCut PaddingCut) DictConfig ListConfig OmegaConf)LazyNeMoIteratorLazyNeMoTarredIteratorexpand_sharded_filepaths) AudioTurnLhotseTextAdapterLhotseTextPairAdapterNeMoMultimodalConversation&NeMoMultimodalConversationJsonlAdapter.NeMoMultimodalConversationShareGPTJsonlAdapterNeMoSFTJsonlAdapterTextTurn) get_full_pathconfigreturncstts tddurt\}}||fStfdddD}|r<ddur2tdt\}}||fSt\}}||fS)z Reads NeMo configuration and creates a CutSet either from Lhotse or NeMo manifests. Returns a tuple of ``CutSet`` and a boolean indicating whether the data is tarred (True) or not (False). input_cfgNc3s|] }|duVqdSN)get).0optr"g/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/common/data/lhotse/cutset.py <sz*read_cutset_from_config..) cuts_path shar_pathmanifest_filepathzCYou must specify either: manifest_filepath, cuts_path, or shar_path) isinstancerr&read_dataset_configallIncompleteConfigErrorread_nemo_manifestread_lhotse_manifest)r"cuts is_tarreduse_nemo_manifestr*r)r+read_cutset_from_config/s     r9c@seZdZdZdS)r3z Placeholder for an error raised.N)__name__ __module__ __qualname____doc__r*r*r*r+r3Gsr3cCstS)z Return the names of all registered data type parsers. Example: >>> get_known_config_data_types() ["nemo", "nemo_tarred", "lhotse", ...] )KNOWN_DATA_CONFIG_TYPESkeysr*r*r*r+get_known_config_data_typesPs r@data_type_namecCst|S)a Return the parsing function for a given data type name. Parsing function reads a dataloading config and returns a tuple of lhotse ``CutSet`` and boolean indicating whether we should use iterable dataset (True) or map dataset (False) mechanism ("is tarred"). )r>)rAr*r*r+ get_parser_fn\srBnamecsfdd}|S)a? Decorator used to register data type parser functions. Parsing function reads a dataloading config and returns a tuple of lhotse ``CutSet`` and boolean indicating whether we should use iterable dataset (True) or map dataset (False) mechanism ("is tarred"). Example: >>> @data_type_parser("my_new_format") ... def my_new_format(config): ... return CutSet(read_my_format(**config)), True ... ... fn = get_parser_fn("my_new_format") ... cuts, is_tarred = fn({"my_arg_0": ..., "my_arg_1": ..., ...}) cs,ttr |t<|SD]}|t|<q |Sr%)r0strr>)fnnrCr*r+ _decoratorws  z$data_type_parser.._decoratorr*)rCrHr*rGr+data_type_parserfs  rIcCs|dd|dd|dd|dd|d d|d d|d d |d d |dd |dd|dd|ddd }t|j|d\}}||fS)a Input configuration format examples. Example 1. Combine two datasets with equal weights and attach custom metadata in ``tags`` to each cut:: input_cfg: - type: nemo_tarred manifest_filepath: /path/to/manifest__OP_0..512_CL_.json tarred_audio_filepath: /path/to/tarred_audio/audio__OP_0..512_CL_.tar weight: 0.5 tags: lang: en some_metadata: some_value - type: nemo_tarred manifest_filepath: /path/to/manifest__OP_0..512_CL_.json tarred_audio_filepath: /path/to/tarred_audio/audio__OP_0..512_CL_.tar weight: 0.5 tags: lang: pl some_metadata: some_value Example 2. Combine multiple (4) datasets, with 2 corresponding to different tasks (ASR, AST). There are two levels of weights: per task (outer) and per dataset (inner). The final weight is the product of outer and inner weight:: input_cfg: - type: group weight: 0.7 tags: task: asr input_cfg: - type: nemo_tarred manifest_filepath: /path/to/asr1/manifest__OP_0..512_CL_.json tarred_audio_filepath: /path/to/tarred_audio/asr1/audio__OP_0..512_CL_.tar weight: 0.6 tags: lang: en some_metadata: some_value - type: nemo_tarred manifest_filepath: /path/to/asr2/manifest__OP_0..512_CL_.json tarred_audio_filepath: /path/to/asr2/tarred_audio/audio__OP_0..512_CL_.tar weight: 0.4 tags: lang: pl some_metadata: some_value - type: group weight: 0.3 tags: task: ast input_cfg: - type: nemo_tarred manifest_filepath: /path/to/ast1/manifest__OP_0..512_CL_.json tarred_audio_filepath: /path/to/ast1/tarred_audio/audio__OP_0..512_CL_.tar weight: 0.2 tags: src_lang: en tgt_lang: pl - type: nemo_tarred manifest_filepath: /path/to/ast2/manifest__OP_0..512_CL_.json tarred_audio_filepath: /path/to/ast2/tarred_audio/audio__OP_0..512_CL_.tar weight: 0.8 tags: src_lang: pl tgt_lang: en shuffleF shard_seedtrng text_fieldtext lang_fieldlang metadata_only force_finitemax_open_streamsNaudio_locator_tagtoken_equivalent_durationskip_missing_manifest_entriesforce_map_datasetforce_iterable_dataset) rJrKrMrOrQrRrSrTrUrVrWrXpropagate_attrs)r&parse_and_combine_datasetsr$)r"rZr6r7r*r*r+r1s ?           r1grp_cfgrZcCs|jtvsJd|j|jdkrt|j}||\}}n t|j|d\}}|d}dur<|jtt|ddd}||fS)zEParse a group configuration, potentially combining multiple datasets.z7Unknown item type in dataset config list: grp_cfg.type=grouprYtagsNr^apply_fn) typer@rBr[r$r&mapr attach_tags)r\rZ parser_fnr6r7 extra_tagsr*r*r+ parse_groups   rgtxtcCs8tt|j|j|j|jd}|dds|}|dfS)z-Read paths to text files and create a CutSet.pathslanguageshuffle_shardsrKrRFT)r rrjrkrJrKr&rr"r6r*r*r+read_txt_pathss rntxt_pairc CsXtt|j|j|d|d|d|d|j|jd}|dds(|}|dfS) z?Read paths to source and target text files and create a CutSet.source_languagetarget_languagequestions_pathquestions_language) source_paths target_pathsrprqrrrsrlrKrRFT)r rrtrur&rJrKrrmr*r*r+read_txt_pair_pathss rvnemo_sft_jsonlcCs<tt|j|d|j|jd}|dds|}|dfS)z7Read paths to Nemo SFT JSONL files and create a CutSet.rkrirRFT)r rrjr&rJrKrrmr*r*r+read_nemo_sft_jsonl s rxmultimodal_conversationc CsXtt|j|d|j|d|j|j|didd}|dds(|}|dfS) zFRead paths to multimodal conversation JSONL files and create a CutSet.tarred_audio_filepathsrUr^ system_prompt)r/rzrTrUrlrKr{rRFT)r rr/r&rTrJrKrrmr*r*r+"read_multimodal_conversation_jsonls r| share_gptc CsLtt|j|d|j|j|d|j|jd}|dds"|}|dfS)zURead paths to ShareGPT JSONL files and create a CutSet of NeMoMultimodalConversation.rzrU)r/rzrTaudio_placeholdersrUrlrKrRFT) r rr/r&rTr~rJrKrrmr*r*r+read_share_gpt_as_conversation/s rpath only_metadatacC*|rtdtt|didSt|dSNr6zcuts.*)fields)in_dirdictsortedrglobrrr*r*r+_resolve_shar_inputsB rr^cCs"|D] \}}t|||q|S)z'Attach extra tags to a cut dynamically.)itemssetattr)cutr^keyvalr*r*r+rdIsrdr] config_listc sg}g}gt|ttfrt|}t|dksJd|D]=}|}|D]\}}||vr5|||<q(||||<q(t||\}} | | | | d} dur[| | qt fddD} | s|dsq|drt d |dd |dd ntd t|dkst|t|ksJd t|dkrt||r|nd|d|d|dp|dd}n|\}|dfS)zPParse a list of dataset configurations, potentially combining multiple datasets.rz#Empty group in dataset config list.weightNc3s|] }|dkVqdS)rNr*r't tarred_statusr*r+r,rsz-parse_and_combine_datasets..rWrXz]Not all datasets in the group have the same tarred status, using provided force_map_dataset (z) and force_iterable_dataset (z') to determine the final tarred status.zyMixing tarred and non-tarred datasets is not supported when neither force_map_dataset nor force_iterable_dataset is True.z\Missing dataset weight. When weighting datasets, every dataset must have a specified weight.rSrKrRrQweightsrSseedrR)r0rDrrloadlencopyrrgappendr&r2loggingwarning ValueErrormux) rrZr6ritemnext_propagate_attrskv item_cutsitem_is_tarredwall_same_tarred_statusr*rr+r[Ps\         r[lhotse lhotse_sharc Cs|ddu}|r/|dd}|dd}|dd}|ddur(td t|jttfrWtd |jd t j dit |j|d |d }|sS|sS| }||fSt|jt rtdg}g}|jD]e}t|ttfr|} t j dit | |d |d } t| } n0t|t rt|dkrt|dttfsJd|d |\} } t j dit | |d |d } td| d| || || qit|||dd||d}||fSt|jtr!dd|jD} d|jvsJd|j|r d| di} t j | d |d}|s|s| }||fStdt|jd|j|j} t | tt| d}||fS)z8Read paths to Lhotse manifest files and create a CutSet.r.NrKrLrQFrRr-zMNote: lhotse.cuts_path will be ignored because lhotse.shar_path was provided.zEInitializing Lhotse Shar CutSet (tarred) from a single data source: ''T)rlrzInitializing Lhotse Shar CutSet (tarred) from multiple data sources with a weighted multiplexer. We found the following sources and weights: rzSupported inputs types for config.shar_path are: str | list[str] | list[tuple[str, number]] where str is a path and number is a mixing weight (it may exceed 1.0). We got: 'z- path= weight=rSrcSsi|] \}}|t|qSr*)rr'rrr*r*r+ sz(read_lhotse_manifest..r6zInvalid value for key 'shar_path': a dict was provided, but didn't specify key 'cuts' pointing to the manifests. We got the following: config.shar_path=)rrlrzUnexpected value for key 'shar_path'. We support string, list of strings, list of tuples[string,float], and dict[string,list[string]], but got: type(config.shar_path)=z config.shar_path= manifest_pathr*)r&warningswarnr0r.rDrrinfor from_sharrrrrintfloatrrrrr? RuntimeErrorrbr- from_filercrresolve_relative_paths) r"r7rKrQrRr6cutsetsrrrcsrrr*r*r+r5s      <   *       r5rrTrUcCst|tr|St|d||jdjdt|jdjddg}t|dr,t|jddg|}t|dr;t|jddg|}t|j |||j d S) a? Converts a lhotse Cut into a two-turn NeMoMultimodalConversation, where the user turn contains cut's audio, and assistant turn contains text response from ``cut.supervisions[0].text``. If ``cut`` has a custom field ``context``, it's pre-pended as an extra user text turn before the user's audio turn. userrrrolerTrN assistantvaluercontextr{systemidturnsrUcustom) r0rr supervisionsrNr hasattrrr{rr)rrTrUrr*r*r+cut_to_conversations   rlhotse_as_conversationcCsTt|\}}|d}dur|jtt|ddd}|tt|j|jd}||fS)Nr^r_r`)rTrU)r9r&rcrrdrrTrU)r"r6r7rfr*r*r+read_lhotse_as_conversation s rz <\|\d+\|>z\s+rNcCs|d|}|d|S)z Strips timestamp tokens from text, e.g. turns: '<|0|> Hey <|3|> <|3|> how <|5|> <|7|> are <|8|> <|8|> <|10|> you? <|12|>' into: 'Hey how are you?'  )substrip)rN_TIMESTAMP_PATTERN_SPACE_PATTERNr*r*r+_strip_timestampss rc@s eZdZdS)FailedConversionN)r:r;r<r*r*r*r+r(srrUserr AssistantagentAgentT input_roles output_rolesstrip_timestamp_tokensc Cs<|jdd}g}d}|D]o} t| jdksJd|j| jdj} | jdj} |r/t| } t| jdkrH| jdj||djdjksHJ| |vrX|t| d|| dn| |vrf|t | dd nt d | d |jt S|d7}q t |d rtd d|Drt |jdd g|}t|j|||jdS)a( Converts a lhotse Cut representing multi-turn speech-to-speech conversation (with multiple supervision segments) into a multi-turn NeMoMultimodalConversation, where the user has AudioTurns and assistant responds in TextTurns. Args: cut: lhotse Cut to convert. audio_locator_tag: special token indicating audio will be inserted in this location in the token sequence. token_equivalent_duration: how much speech duration is counted as one token. input_roles: when supervision.speaker is set to one of these values, we consider it user's turn. output_roles: when supervision.speaker is set to one of these values, we consider it assistant's turn. strip_timestamp_tokens: strips tokens like <|0|>, <|1|>, etc indicating timestamps from the text. F)keep_overlappingrrz.rr)trim_to_supervisionsrrrspeakerrNrrrr rrrrr2r{rr) rrTrUrrr turn_cutsridx per_turn_cut turn_speaker turn_textr*r*r+s2s_cut_to_conversation,s8    $  rs2s_as_conversationc Cs\t|\}}|tt|j|j|dddg|dgd|dddd d }||fS) NrrrrrrT)rTrUrrrcSs t|t Sr%)r0r)exr*r*r+os z*read_s2s_as_conversation..)r9rcrrrTrUr&filter)r"r6r7r*r*r+read_s2s_as_conversationcs    rcCrrrrr*r*r+rsrrcst|tr|St|tr|jD] }t|j|_q|Sfdd}fdd|jr0||j|jr8|j |j dur]|j D]\}}t|t rP||qBt|t ttfr\|qB|S)z;Resolve relative paths in a Cut object to their full paths.cs*|jD]}|jdkrt|jd|_qdS)Nfile manifest_file)sourcesrbr!source)r audio_sourcerr*r+resolve_recordings  z1resolve_relative_paths..resolve_recordingcst|tr |j|_dS|jdvr0tttt|j|jd}t|j |_t|j |_dS|jdvr?t|jd|_dSdS)N) numpy_files lilcom_filesr)kaldiiochunked_lilcom_hdf5 lilcom_chunky lilcom_hdf5 numpy_hdf5) r0rarray storage_typerr!rD storage_path storage_keyparentrC)rabspathr resolve_arrayr*r+rs    z-resolve_relative_paths..resolve_arrayN)r0rrtracksrr has_recording recording has_featuresfeaturesrrr rrr )rrtrackrrrr*rr+rzs(         rnemo nemo_tarredc Csi}dD]}||vr|dkr|||d<q||||<q|dd}|dd}d|i}|ddu}t|jttfrztd |d |jd |ri|sitt|jf|j |d dd |}|se| }||fStt |jfi||}||fStd|dg}g} |r|j nt d} t |j| D]\} } |rt| t ttfr| \} | d} n| } |r|std| | |d dd|}n t | fi||}t| tst| dkrt|}n t| trt| dkrt| dttfsJd| d | d}td| d||ddur$|D]}|t|| |qq|t|| |qt|| |d|dd|pA|d}||fS)z.Read NeMo manifest and return a Lhotse CutSet.)rMrOrJrK extra_fieldsrJrlrQFrRrzNzNInitializing Lhotse CutSet from a single NeMo manifest (is_tarred=z): 'rrV) tar_pathsrVzNInitializing Lhotse CutSet from multiple NeMo manifest (is_tarred=z_) sources with a weighted multiplexer. We found the following sources and weights: r%r)rrrVrrzSupported inputs types for config.manifest_filepath are: str | list[list[str]] | list[tuple[str, number]] where str is a path and number is a mixing weight (it may exceed 1.0). We got: 'z- manifest_path=rrSrKrLrr*)r&r0r/rDrrrr rrzrrziplisttuplerrrrr to_shardsrr)r" common_kwargsrrQrR notar_kwargsr7r6rrr manifest_infotar_pathr nemo_iterrsubiterr*r*r+r4s    NL       r4rLF)rSrrRrrrSrrRcGsl|dur|r Jdtj||||d}|S|sdd|D}t|dkr+|d}|Stj|||d}|S) a Helper function to call the right multiplexing method flavour in lhotse. The result is always an infinitely iterable ``CutSet``, but depending on whether ``max_open_streams`` is set, it will select a more appropriate multiplexing strategy. NzJmax_open_streams and metadata_only/force_finite options are not compatible)rrrScSsg|]}|qSr*r)r'rr*r*r+ 0szmux..rr)rr)r infinite_muxrr)rrSrrRrr6r*r*r+rs   rinpc Csddlm}t|ttjfr;z|tdd|D}t |\}}|WSt y:}z t d|d|d}~wwt|t rz| drQ|td |g}n!| d s[| d rg|td |g}n |td |g}t |\}}|St dt|d)a  Utility function that supports opening a CutSet from: * a string path to YAML input spec (see :func:`read_dataset_config` for details) * a string path to Lhotse non-tarred JSONL manifest * a string path to NeMo non-tarred JSON manifest * a dictionary specifying inputs with keys available in :class:`nemo.collections.common.data.lhotse.dataloader.LhotseDataLoadingConfig` It's intended to be used in a generic context where we are not sure which way the user will specify the inputs. r)$make_structured_with_schema_warningscSsg|] \}}|d|qS)=r*rr*r*r+rHsz&guess_parse_cutset..z)Couldn't open CutSet based on dict input z1 (is it compatible with LhotseDataLoadingConfig?)Nz.yamlz input_cfg=z.jsonlz .jsonl.gzz cuts_path=zmanifest_filepath=zUnsupported input type: z (expected a dict or a string)).nemo.collections.common.data.lhotse.dataloaderrr0r omegaconfrr from_dotlistrr9 ExceptionrrDendswithrb)rrr"r6_er*r*r+guess_parse_cutset9s.      r&)rrT)Qrrer functoolsr itertoolsrpathlibrtypingrrrr r r rr r r lhotse.arrayrr lhotse.cutrrrrrr1nemo.collections.common.data.lhotse.nemo_adaptersrrr1nemo.collections.common.data.lhotse.text_adaptersrrrrrrrr 4nemo.collections.common.parts.preprocessing.manifestr!rboolr9rr3r>rDr@rBrrIrr1rgrnrvrxr|rrrdr[r5rrrcompilerrrrrr4rrr&r*r*r*r+s   ( "  P  @\   7 1v   "