o }oiq@sddlZddlZddlZddlZddlZddlZddlZddlZddlm Z m Z ddl m Z ddl mZddlmZddlmZmZmZmZmZmZddlZddlmZmZddlmZmZd Zzdd l m!Z!m"Z"m#Z#dd l m$Z%dd l&m'Z'Wn e(yd ZYnwerddl)m*Z+ddl,m-Z-m.Z.ddl/m0Z0dZ1dZ2Gddde Z3e GdddZ4dee5de5fddZ6de5de!fddZ7dUdeeeeeffddZ8d d!Z9d"d#dee5fd$d%Z:d&e5de5fd'd(Z;d)eee5fd*d+Zej?dWd4d5Z@d"ed6dd#fd7d8ZAdXd:d;ZBdYdd?d@e5fdAdBZDdedCfdDdEZEdFe5dGe5deeeFe5ffdHdIZGdJdKZHdZdLdMZIdNee5efdefdOdPZJdZdQe5dReFddfdSdTZKdS)[N) dataclass is_dataclass)Enum) lru_cache)Path) TYPE_CHECKINGListOptionalTupleTypeUnion)AppStatelogging)is_datastore_pathresolve_cache_dirT) DictConfig ListConfig OmegaConf)errors)versionF)ModelPTPretrainedModelInfo) NemoConfigzmodel_config.yamlds_itemc@seZdZdZdZdZdS)ArtifactPathTypea ArtifactPathType refers to the type of the path that the artifact is located at. LOCAL_PATH: A user local filepath that exists on the file system. TAR_PATH: A (generally flattened) filepath that exists inside of an archive (that may have its own full path). rN)__name__ __module__ __qualname____doc__ LOCAL_PATHTAR_PATHr"r"J/home/ubuntu/.local/lib/python3.10/site-packages/nemo/utils/model_utils.pyr;src@s8eZdZUdZeed<ejZeed<dZ e eed<dS) ArtifactItempath path_typeN hashed_path) rrrr&str__annotations__rr r'r(r r"r"r"r#r$Gs r$namesreturncCs@t|dt}t|dksJd|d|dt}|S)zyDetect model config prefix for a list of file names. Useful to identify prefix used within .nemo tarball checkpoint.*rz/Exactly one model config path expected, found: .r)fnmatchfilter MODEL_CONFIGlen removesuffix)r+ model_configprefixr"r"r# detect_prefixNsr6 model_filec Cstj|rVtB}t|d#}t|}|j |t |dt tj |t }Wdn1s7wYWd|SWd|S1sOwY|Stj|rht tj |t }|St|)z>Load model config from extracted directory or '.nemo' tarball.r:r&N)osr&isfiletempfileTemporaryDirectorytarfileopenr6getnamesextractr1rloadjoinisdirFileNotFoundError)r7tmptarr5r4r"r"r# load_configXs  (   rHmodule_instancescCszd}t|ts |g}d}g}|D]$}|r"t||r!|j}t||sn t|dr/|j}t|ds'||q|s;|dS|S)zBUnwrap model from wrapper classes like Float16Module, for example.TFmoduler) isinstancelistrJhasattrappend)modelrI return_listunwrapped_model model_moduler"r"r# unwrap_modelgs$     rScCst|d p |j S)Nshared)rMrT)paramr"r"r#param_is_not_shared}srVcfgrcCst|vr|tdur|t}t|tr||vr||StS|D]R\}}t|tttfvrYd}|D]}t|}tj |sHtj |sHt |rM|d7}q2|t |krX|Sq!tj t|sotj t|sot t|rs|Sq!dS)a Parses items of the provided sub-config to find the first potential key that resolves to an existing file or directory. # Fast-path Resolution In order to handle cases where we need to resolve items that are not paths, a fastpath key can be provided as defined in the global `_VAL_TEST_FASTPATH_KEY`. This key can be used in two ways : ## _VAL_TEST_FASTPATH_KEY points to another key in the config If this _VAL_TEST_FASTPATH_KEY points to another key in this config itself, then we assume we want to loop through the values of that key. This allows for any key in the config to become a fastpath key. Example: validation_ds: splits: "val" ... <_VAL_TEST_FASTPATH_KEY>: "splits" <-- this points to the key name "splits" Then we can write the following when overriding in hydra: ```python python train_file.py ... model.validation_ds.splits=[val1, val2, dev1, dev2] ... ``` ## _VAL_TEST_FASTPATH_KEY itself acts as the resolved key If this _VAL_TEST_FASTPATH_KEY does not point to another key in the config, then it is assumed that the items of this key itself are used for resolution. Example: validation_ds: ... <_VAL_TEST_FASTPATH_KEY>: "val" <-- this points to the key name "splits" Then we can write the following when overriding in hydra: ```python python train_file.py ... model.validation_ds.<_VAL_TEST_FASTPATH_KEY>=[val1, val2, dev1, dev2] ... ``` # IMPORTANT NOTE: It potentially mismatch if there exist more than 2 valid paths, and the first path does *not* resolve the the path of the data file (but does resolve to some other valid path). To avoid this side-effect, place the data path as the first item on the config file. Args: cfg: DictConfig (Sub-config) that should be parsed. Returns: A str representing the `key` of the config which hosts the filepath(s), or None in case path could not be resolved. Nrr)_VAL_TEST_FASTPATH_KEYrKr)itemstyperLtuplerr:r&existsrDrr2)rW fastpath_keykeyvaluevalues_are_pathsval_ir"r"r#resolve_dataset_name_from_cfgs(<   ,rbnamecCstjt|stjt|stt|rt|j}nt|}|dd}d|vr0|dd}d|vr:|dd}|dkrBt dd|dkrL|d}|S)a; Constructs a valid prefix-name from a provided file path. Args: name: str path to some valid data/manifest file or a python object that will be used as a name for the data loader (via str() cast). Returns: str prefix used to identify uniquely this data/manifest file. -_manifestr%dataseta"Provided dataset / manifest filename was `manifest.json` or `dataset.json`. Such a name is invalid, since multiple datasets/manifests can share the same name, thereby overriding their results during logging. Please pick a more discriptive filename for the provided dataset / manifest file.) r:r&r\r)rDrrstemreplace ValueErrorrcr"r"r#parse_dataset_as_names,     rm name_listcCsD|durdSt}|D]}||vrtd|q ||q dS)z Performs a uniqueness check on the name list resolved, so that it can warn users about non-unique keys. Args: name_list: List of strings resolved for data loaders. NzName resolution has found more than one data loader having the same name ! In such cases, logs will nor be properly generated. Please rename the item to have unique names. Resolved name : )setrwarningadd)rnr+rcr"r"r#unique_names_checks rrrOrc Csts tdtdt|j}g}d|jvr*t |}|d d}t |}nd}||_ t |j}|dus<|dkrMtd|j||jdS|j|}t|tttfr|D]}t|ttfri||_n||j|<||j||jq\||_t|dkrt|dttfrdd |D|_n:|jd g}t|dkrt|t|krtd t|d t|d |d|dd |D|_ndd |D|_t|jddS||j|jd d}|durt|tstd|t|g|_nt|g|_t|jddS)a Helper method that operates on the ModelPT class to automatically support multiple dataloaders for the validation set. It does so by first resolving the path to one/more data files via `resolve_dataset_name_from_cfg()`. If this resolution fails, it assumes the data loader is prepared to manually support / not support multiple data loaders and simply calls the appropriate setup method. If resolution succeeds: Checks if provided path is to a single file or a list of files. If a single file is provided, simply tags that file as such and loads it via the setup method. If multiple files are provided: Inject a new manifest path at index "i" into the resolved key. Calls the appropriate setup method to set the data loader. Collects the initialized data loader in a list and preserves it. Once all data loaders are processed, assigns the list of loaded loaders to the ModelPT. Finally assigns a list of unique names resolved from the file paths to the ModelPT. Args: model: ModelPT subclass, which requires >=1 Validation Dataloaders to be setup. @This function requires Hydra/Omegaconf and it was not installed.r val_dl_idx validation_dsrN_Could not resolve file path from provided config - {}. Disabling support for multi-dataloaders.cSg|]}|jqSr"rl.0dsr"r"r# _z2resolve_validation_dataloaders..rcNumber of names (%) does not match number of datasets (). Got  and cSg|]}t|qSr"rmrynr"r"r#r{hcSrr"rrxr"r"r#r{jrrn1`name` must be a string for single manifest, got ) _HAS_HYDRArerrorexitcopydeepcopy_cfgrur to_containerpopcreate _val_dl_idxrbdebugformatsetup_validation_datarKrLr[rdictrrN_validation_dlr2_validation_namesgetrkrrr)rm)rOrW dataloadersrtds_key ds_valuesds_valueds_namesr"r"r#resolve_validation_dataloaderssl                rc Cs ts tdtdt|j}g}d|jvr*t |}|d d}t |}nd}||_ t |j}|durItd|j||jdS|j|}t|tttfr|D]}t|ttfre||_n||j|<||j||jqX||_t|dkrt|dttfrdd |D|_n:|jd g}t|dkrt|t|krtd t|d t|d |d|dd |D|_ndd |D|_t|jddS||j|jd d}|durt|tstd|t|g|_nt|g|_t|jddS)a Helper method that operates on the ModelPT class to automatically support multiple dataloaders for the test set. It does so by first resolving the path to one/more data files via `resolve_dataset_name_from_cfg()`. If this resolution fails, it assumes the data loader is prepared to manually support / not support multiple data loaders and simply calls the appropriate setup method. If resolution succeeds: Checks if provided path is to a single file or a list of files. If a single file is provided, simply tags that file as such and loads it via the setup method. If multiple files are provided: Inject a new manifest path at index "i" into the resolved key. Calls the appropriate setup method to set the data loader. Collects the initialized data loader in a list and preserves it. Once all data loaders are processed, assigns the list of loaded loaders to the ModelPT. Finally assigns a list of unique names resolved from the file paths to the ModelPT. Args: model: ModelPT subclass, which requires >=1 Test Dataloaders to be setup. rsr test_dl_idxtest_dsrNrvcSrwr"rlrxr"r"r#r{r|z,resolve_test_dataloaders..rcr}r~rrcSrr"rrr"r"r#r{rcSrr"rrxr"r"r#r{rrr) rrrrrrrrrrrr _test_dl_idxrbrrsetup_test_datarKrLr[rrrrN_test_dlr2 _test_namesrrkrrr)rm)rOrWrrrrrrr"r"r#resolve_test_dataloaders{sl                rinstancepl.LightningModulecCsD||i|}t|tr |dur d|vr |d}|j|dd|S)NlogT)on_step)rKrrlog_dict)wrappedrargskwargs output_dictrr"r"r#wrap_training_steps  r)rrcCspts tdtdt|ttfst|rt|}t|ts*t dt |dtj |dd}t |}|S)z Converts its input into a standard DictConfig. Possible input values are: - DictConfig - A dataclass which is a subclass of NemoConfig Args: cfg: A dict-like object. Returns: The equivalent DictConfig rsrzAcfg constructor argument must be of type DictConfig/dict but got z instead.T)resolve) rrrrrKrrr structuredrkrZrr)rWconfigr"r"r##convert_model_config_to_dict_configs    rrc Csts tdtdd|vrd|vr|d|_d|vr/|d}|D]\}}|||<q&z|D] \}}t|trAt |q4Wd St j yd}zt d|d|dWYd }~d Sd }~ww) zURecursive function convertint the configuration from old hydra format to the new one.rsrcls_target_paramsz)Skipped conversion for config/subconfig: z Reason: r.N) rrrrrrrYrKr_convert_configomegaconf_errorsOmegaConfBaseExceptionrp)rWr param_key param_valresub_cfger"r"r#rs&     &rcCsts tdtd|dur*t|ts*z t|}|}Wn tj y)|YSwt |}t |dt |t |d|S)a Recursively convert Hydra 0.x configs to Hydra 1.x configs. Changes include: - `cls` -> `_target_`. - `params` -> drop params and shift all arguments to parent. - `target` -> `_target_` cannot be performed due to ModelPT injecting `target` inside class. Args: cfg: Any Hydra compatible DictConfig Returns: An updated DictConfig that conforms to Hydra 1.x format. rsrNFT)rrrrrKrrrrrrr set_structr)rWtemp_cfgr"r"r#maybe_update_config_versions     ri)maxsizer&cCs@|d}d|dd}|d}t||gd}t||}|S)z3 Recursive import of class by path string. r.Nrh)fromlist)splitrC __import__getattr)r&paths class_namemodr"r"r#import_class_by_pathAs  rrcs,tfdd|ttS)a Recursively traverses the inheritance graph of subclasses to extract all pretrained model info. First constructs a set of unique pretrained model info by performing DFS over the inheritance graph. All model info belonging to the same class is added together. Args: base_class: The root class, whose subclass graph will be traversed. Returns: A list of unique pretrained model infos belonging to all of the inherited subclasses of this baseclass. csd|D]+}||}|dur/t|dkr/|D] }|jdur$||_q|D]}|q'qdS)Nr)__subclasses__list_available_modelsr2class_rq)rsubclasssubclass_models model_infolist_of_modelsrecursive_subclass_walkr"r#r]s   zGresolve_subclass_pretrained_model_info..recursive_subclass_walk)rorLsorted) base_classr"rr#&resolve_subclass_pretrained_model_infoNs   rlib_namechecked_versionc CszNd|vr t|}nt|}t|drDt|j}t|}|||r/d|d}d|fWSd|d|d|jd|d }d |fWSd|d }d |fWStt t fyZYnwd|d }d |fS)a Checks if a library is installed, and if it is, checks the operator(lib.__version__, checked_version) as a result. This bool result along with a string analysis of result is returned. If the library is not installed at all, then returns None instead, along with a string explaining that the library is not installed Args: lib_name: lower case str name of the library that must be imported. checked_version: semver string that is compared against lib.__version__. operator: binary callable function func(a, b) -> bool; that compares lib.__version__ against version in some manner. Must return a boolean. Returns: A tuple of results: - Bool or None. Bool if the library could be imported, and the result of operator(lib.__version__, checked_version) or False if __version__ is not implemented in lib. None is passed if the library is not installed at all. - A string analysis of the check. r. __version__zLib z version is satisfied !Tz version (z ) is not z than required version zI. Please upgrade the lib using either pip or conda to the latest version.FzX does not implement __version__ in its init file. Could not check version compatibility.zI has not been installed. Please use pip or conda to install this package.N) r importlib import_modulerMrVersionrrAttributeError ImportErrorModuleNotFoundError)rroperatorrlib_ver match_vermsgr"r"r#check_lib_versionus.            rcsTttfdddDr(tjtj}tj}tj||SS)Ncsg|]}|vr|qSr"r")rysfilepathr"r#r{sz0uninject_model_parallel_rank..)mp_ranktp_rank fsdp_shard)r)anyr:r&dirnamebasenamerC)rrrr"rr#uninject_model_parallel_ranks rcCst|}t}tj|}tj|}|jdurU|jdkrU|r&d|jdnd}|jdus2|jdkrB|d|j d|d|}|S|d |j dd |j d d|}|S|r^d |jdnd}||d|S) zz Injects tensor/pipeline model parallel ranks into the filepath. Does nothing if not using model parallelism. Nr _fsdp_shard_05dr%z /mp_rank_02d/z /tp_rank_ _pp_rank_03dz /fsdp_shard_) rr r:r&rrmodel_parallel_sizedata_parallel_rankpipeline_model_parallel_sizetensor_model_parallel_rankpipeline_model_parallel_rank)rfsdp_sharded_ckpt app_staterrrr"r"r#inject_model_parallel_ranks  "rrcCsHt|}|jdkr|r|S|jdksJd|d||j}|S)zPTL considers checkpoints as .ckpt files. This method removes the extension and returns a path to be used as a directory for distributed checkpoints z.ckptz filepath: z must have .ckpt extension)rsuffixis_dir with_nameri)rcheckpoint_dirr"r"r# ckpt_to_dirs  r output_dir use_abspathcCst}|j}t|j}|jjdkr|j|t|ds0t|dr.t |jt j |ddSt j |r?tj}|dd}nt j |rKtj}i}nt||d i|f}|durat|} |jD]N\} } | j d\} } t j || }|dur|j| | |d t t j || |n tt j || ||rt j |nt j |}t || |qfWdn1swYt|drt |jt j |ddSdS) zJSave all model artifacts and tokenizer config to a given output directory. huggingface artifacts tokenizerztokenizer_config.yamlNr8)rcmodeznemo:r9r") r model_restore_pathrrrWrlibrarysave_pretrainedrMrsaver:r&rCr;r>r?rD contextlib nullcontextrEr6r@rrYrrArenameshutilabspathrupdate)rOrrrr7 model_cfgmodel_file_handlerr maybe_tarr5 arti_name arti_itemre arti_file arti_pathr"r"r#save_artifactssB          r)N)rOr)rr)rWr)rWr)F)Lr rr/rr:rr>r< dataclassesrrenumr functoolsrpathlibrtypingrrr r r r wrapt nemo.utilsr rnemo.utils.data_utilsrrr omegaconfrrrrr packagingrrlightning.pytorchpytorchplnemo.core.classesrrnemo.core.config.modelPTrr1rXrr$r)r6rHrSrVrbrmrrrr decoratorrrrrrrboolrrrrrr"r"r"r#sn          Z(  _`   ' "'6