o }oim @sddlmZddlZddlZddlZddlZddlZddlmZddlm Z ddlm Z ddl m Z m Z mZmZmZmZmZmZddlZddlZddlmZddlmZmZzdd lmZmZdd lmZd Z Wn e!e"fyud Z Ynwdd l#m$Z$m%Z%ddl&m'Z'm(Z(ddl)m*Z*m+Z+m,Z,ddl-m.Z.ddl/m0Z0ddl1m2Z2ddl3m4Z4ddl5m6Z6m7Z7ddl8m9Z9m:Z:ddl;mZ>ddl?m@Z@ddlAmBZBmCZCdgZDe+jEdddd dGddde$e2ZFdS) ) annotationsN)abstractmethodpath)Path)AnyCallableDictIteratorListOptionalTupleUnion) NeuralModule)import_multistorageclientis_multistorageclient_url)OptimizerConfigget_megatron_optimizer)get_model_configTF)LightningModuleTrainer) model_summaryrank_zero_only) DictConfig OmegaConf open_dict) package_info)optim)Model)SaveRestoreConnector)McoreDistributedOptimizerprepare_lr_scheduler)logging model_utils)AppState)register_debug_hooks)NeMoBaseException)get_rankis_global_rank_zeroModelPTmultiplycCs||SN)xyr,r,M/home/ubuntu/.local/lib/python3.10/site-packages/nemo/core/classes/modelPT.py=sr0)replacecseZdZdZddʇfdd Zdd d Zdˇfd d Z ddddZdddZdddZ dddZ dd!d"Z #ddd'd(Z dd*d+Z e    ,   ddd8d9Zedddd:dՇfd?d@ZeddCdDZeddFdGZddIdJZddKdLZddMdNZddQdRZ  dddVdWZdXdYZdZd[Zd\d]Zddd_d`ZdadbZdcddZdedfZdddidjZ ddkdlZ! mdddrdsZ" mdddtduZ#dddvdwZ$dddxdyZ%ddzd{Z&e'ddd~dZ(dfdd Z)e , ddddZ*dddZ+dddZ,dddZ-ddddZ.dddZ/e0ddZ1e0ddZ2e3j4j5ddZ4e2j6ddZ2e0fddZ7e0ddZ8e8j6ddZ8e0ddZ9e9j6ddZ9e:dddZ;e:ddddZddZ?ddZ@ddZAddddZBddddZCddZDddZEddÄZFddńZGdddȄZHZIS)r)z; Interface for Pytorch-lightning based NeMo models Ncfgrtrainerrcs|durt|tstdt|dt t}t|}t |}d|vr.tdd|vrIt |dd |j j|j j|_t |d d |vret| tj|_Wdn1s`wY||_t|_|d d|_d|_d|_d|_d|_d|_||t |_!|"t#j$%rt#j$&durt#j$&|_'|jdur|(sd |jvr|jj)dur|jj)*d ds|+|jj)d|jvr|jj,dur|jj,*d ds|j-|j,dd|jvr|jj.dur|jj.*d ds|j/|j.dnNd |jvr|jj)durt01dt 2|jj)d|jvr.|jj,dur.t01dt 2|jj,d|jvrH|jj.durHt01dt 2|jj.d|_3d|_4t5|j6|_6|7d|_8d|_9d|_:d|_;|<d|_=dS)a. Base class from which all NeMo models should inherit Args: cfg (DictConfig): configuration object. The cfg object should have (optionally) the following sub-configs: * train_ds - to instantiate training dataset * validation_ds - to instantiate validation dataset * test_ds - to instantiate testing dataset * optim - to instantiate optimizer with learning rate scheduler trainer (Optional): Pytorch Lightning Trainer instance NzWtrainer constructor argument must be either None or lightning.pytorch.Trainer. But got z instead.modelz^Creating model config node is forbidden due to collision problem when loading from checkpoint.targetFz{0}.{1}T nemo_versionr2train_ds defer_setup validation_dsval_data_configtest_dstest_data_configzIf you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader. Train config : zIf you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s). Validation config : zPlease call the ModelPT.setup_test_data() or ModelPT.setup_multiple_test_data() method and provide a valid configuration file to setup the test data loader(s). Test config : )> isinstancer ValueErrortypesuper__init__r$r##convert_model_config_to_dict_configmaybe_update_config_versionr set_structformat __class__ __module____name__r5rr __version__r6_cfgdict$_nemo_submodule_name_to_config_fieldsave_hyperparameters _train_dl_validation_dl_test_dl_optimizer_param_groups _optimizer _scheduler set_trainerr_save_restore_connector_set_model_guidtorchcuda is_availablecurrent_device device_id_is_model_being_restoredr7getsetup_training_datar9setup_multiple_validation_datar<setup_multiple_test_datar"warningto_yaml_validation_step_outputs_test_step_outputswrap_training_step training_step_setup_profiling_nsys_profile_started_nsys_profile_complete_memory_profile_started_memory_profile_complete_setup_chakra_profiling_chakra_profile_in_progress)selfr2r3 app_staterHr,r/rCEs                    zModelPT.__init__returnNonecCs t|_dSr+)rrW)clsr,r,r/__init_subclass__s zModelPT.__init_subclass__cs6|jddrt|j|j|j|jddtS)z' Register debug hooks. dump_debug_infoFdump_debug_info_to_file)r2r_r%r4r3logrB on_fit_startrprrr,r/rzs zModelPT.on_fit_startT config_pathstrsrcverify_src_existsboolcCs||dus|dkr |St|jdkrtdt|dsi|_|jdur%i|_||jvr5td|d|j ||||S)a+Register model artifacts with this function. These artifacts (files) will be included inside .nemo file when model.save_to("mymodel.nemo") is called. How it works: 1. It always returns existing absolute path which can be used during Model constructor call EXCEPTION: src is None or "" in which case nothing will be done and src will be returned 2. It will add (config_path, model_utils.ArtifactItem()) pair to self.artifacts .. code-block:: If "src" is local existing path: then it will be returned in absolute path form. elif "src" starts with "nemo_file:unique_artifact_name": .nemo will be untarred to a temporary folder location and an actual existing path will be returned else: an error will be raised. WARNING: use .register_artifact calls in your models' constructors. The returned path is not guaranteed to exist after you have exited your model's constructor. Args: config_path (str): Artifact key. Usually corresponds to the model config. src (str): Path to artifact. verify_src_exists (bool): If set to False, then the artifact is optional and register_artifact will return None even if src is not found. Defaults to True. Returns: str: If src is not None or empty it always returns absolute path which is guaranteed to exist during model instance life Nz.nemoz|Registering .nemo files as artifacts not supported. If you are trying to make a nested model, use `register_nemo_submodule`. artifactsz3You tried to register an artifact under config key=z4 but an artifact for it has already been registered.) rsuffixr&hasattrrkeysr"rcrWregister_artifact)rpr|r~rr,r,r/rs&   zModelPT.register_artifactcCs"t|do|jduot|jdkS)z.Returns True if model has artifacts registeredrNr)rrlenr{r,r,r/ has_artifacts*s"zModelPT.has_artifactscCsD|D]}t|trt|dr|jdurt|jdkrdSqdS)zHReturns True if it has artifacts or any of the submodules have artifactsrNrTF)modulesr?r)rrr)rpmoduler,r,r/"has_native_or_submodules_artifacts.s  z*ModelPT.has_native_or_submodules_artifactscCst|jdkS)z5Returns True if it has any registered NeMo submodulesr)rrNr{r,r,r/has_nemo_submodules:szModelPT.has_nemo_submodulesname config_fieldr4 'ModelPT'cCsJt|tstdt|jt|dstdt|||||j|<dS)aX Adds a NeMo model as a submodule. Submodule can be accessed via the `name` attribute on the parent NeMo model this submodule was registered on (`self`). In the saving process, the whole parent model (self) is held as a solid model with artifacts from the child submodule, the submodule config will be saved to the `config_field` of the parent model. This method is necessary to create a nested model, e.g. .. code-block:: python class ParentModel(ModelPT): def __init__(self, cfg, trainer=None): super().__init__(cfg=cfg, trainer=trainer) # annotate type for autocompletion and type checking (optional) self.child_model: Optional[ChildModel] = None if cfg.get("child_model") is not None: self.register_nemo_submodule( name="child_model", config_field="child_model", model=ChildModel(self.cfg.child_model, trainer=trainer), ) # ... other code Args: name: name of the attribute for the submodule config_field: field in config, where submodule config should be saved model: NeMo model, instance of ModelPT zBModel is not and instance of ModelPT, so can't be registered. Got rNzYou are trying to register a submodule before the model is initialized. This is not allowed. Did you forget to call `super().__init__`?N)r?r)r&rArJrsetattrrN)rprrr4r,r,r/register_nemo_submodule>s   zModelPT.register_nemo_submoduler prefix_name prefix_config$Iterator[Tuple[str, str, 'ModelPT']]c cst|ds td|||fV|jD]2\}}|r"|d|n|}|r-|d|n|}t||}|j||dD] \}} } || | fVq;qdS)a Returns an iterator over all NeMo submodules recursively, yielding tuples of (attribute path, path in config, submodule), starting from the core module Args: prefix_name: prefix for the name path prefix_config: prefix for the path in config Returns: Iterator over (attribute path, path in config, submodule), starting from (prefix, self) rNzModel is not fully initialized. Calling `named_nemo_modules` before __init__ not allowed. Did you forget to call `super().__init__`?.)rrN)rr&rNitemsgetattrnamed_nemo_modules) rprrrrattribute_pathr|rsubmodule_namesubconfig_path submoduler,r,r/rks    zModelPT.named_nemo_modules save_pathcCsddd}t|st|}t}|jdurE|jdkr)t|jtkr)t dt r0||t j r:t j |j|t|dSt rW|||j|t|dSdS) a Saves model instance (weights and configuration) into .nemo file You can use "restore_from" method to fully restore instance from .nemo file. .nemo file is an archive (tar.gz) with the following: model_config.yaml - model configuration in .yaml format. You can deserialize this into cfg argument for model's constructor model_wights.ckpt - model checkpoint Args: save_path: Path to .nemo file where model instance should be saved rUnion[str, 'pathlib.Path']cSs,t|s|js|jjdddSdSdS)NT)parents)rparentexistsmkdirrr,r,r/maybe_make_save_dirs  z,ModelPT.save_to..maybe_make_save_dirNzDefault NeMo SaveRestoreConnector will not work in model parallel mode. You should use a connector which supports model parallel mode, such as NLPSaveRestoreConnector in NLP. You can also use a custom one.)rr)rr expanduserresolver$model_parallel_sizerArWrr@r(rY distributedis_initializedbarriersave_tor})rprrrqr,r,r/rs&     zModelPT.save_toF restore_pathoverride_config_pathOptional[Union[OmegaConf, str]] map_locationOptional[torch.device]strict return_configsave_restore_connectorrOptional[Trainer]validate_access_integrityc Cs|durt}t|rt} | jj|std|n'|jdur-tjtj |}n tjtj |j}t|sDtd|t } || _ | ||j ||||||||} t| trd|| _ | S)a Restores model instance (weights and configuration) from .nemo file. Args: restore_path: path to .nemo file from which model should be instantiated override_config_path: path to a yaml config that will override the internal config file or an OmegaConf / DictConfig object representing the model config. map_location: Optional torch.device() to map the instantiated model to a device. By default (None), it will select a GPU if available, falling back to CPU otherwise. strict: Passed to load_state_dict. By default True. return_config: If set to true, will return just the underlying config of the restored model as an OmegaConf DictConfig object without instantiating the model. trainer: Optional, a pytorch lightning Trainer object that will be forwarded to the instantiated model's constructor. save_restore_connector (SaveRestoreConnector): Can be overridden to add custom save and restore logic. Example: ``` model = nemo.collections.asr.models.EncDecCTCModel.restore_from('asr.nemo') assert isinstance(model, nemo.collections.asr.models.EncDecCTCModel) ``` Returns: An instance of type cls or its underlying config (if return_config is set). N Can't find )rrrosrrFileNotFoundErrormodel_extracted_dirabspathrr$model_restore_pathupdate_save_restore_connectorrW restore_fromr?r)) rurrrrrrr3rmscrqinstancer,r,r/rs8&    zModelPT.restore_from)r hparams_filercheckpoint_pathAOptional[Union[Dict[str, str], str, torch.device, int, Callable]]r Optional[str]c sNd}z|jddtj|||||d|}W|jdd|S|jddw)z Loads ModelPT from checkpoint, with some maintenance of restoration. For documentation, please refer to LightningModule.load_from_checkpoint() documentation. NT)is_being_restored)rrrrF)_set_model_restore_staterBload_from_checkpoint)rurrrrargskwargs checkpointrrr,r/rs  zModelPT.load_from_checkpointtrain_data_configUnion[DictConfig, Dict]cCdS)z Setups data loader to be used in training Args: train_data_layer_config: training data layer parameters. Returns: Nr,)rprr,r,r/r` zModelPT.setup_training_datar;cCr)z Setups data loader to be used in validation Args: val_data_layer_config: validation data layer parameters. Returns: Nr,rpr;r,r,r/setup_validation_data&rzModelPT.setup_validation_datar>cCst)z (Optionally) Setups data loader to be used in test Args: test_data_layer_config: test data layer parameters. Returns: )NotImplementedErrorrpr>r,r,r/setup_test_data2 zModelPT.setup_test_datacCd|_d|_d|_|jd|dzd|_tj|dWd|_nd|_w|jdurE|jdurGt|jtt fvrIdd t t |jD|_dSdSdSdS) z (Optionally) Setups data loader to be used in validation, with support for multiple data loaders. Args: val_data_layer_config: validation data layer parameters. rN validation dataset_nameconfigTr4FcSg|]}d|qS)zval_{}_rG.0idxr,r,r/ Tz:ModelPT.setup_multiple_validation_data..) _val_dl_idx_validation_namesrQ_update_dataset_config_multi_dataset_moder#resolve_validation_dataloadersrAlisttuplerangerrr,r,r/ra= z&ModelPT.setup_multiple_validation_datacCr) z (Optionally) Setups data loader to be used in test, with support for multiple data loaders. Args: test_data_layer_config: test data layer parameters. rNtestrTrFcSr)ztest_{}_rrr,r,r/rmrz4ModelPT.setup_multiple_test_data..) _test_dl_idx _test_namesrRrrr#resolve_test_dataloadersrArrrrrr,r,r/rbVrz ModelPT.setup_multiple_test_data optim_config!Union[Dict[str, Any], DictConfig]cCsjt|jd}t|j|j|j|d|d|dd|dd|dtj|jj |j |j j ddd }|S) z Setup mcore optimizer config. Args: optim_config: Nemo optim args used to set up Mcore optimizer options. rlr weight_decaybetasreps(overlap_param_gather_with_optimizer_stepF) fp16bf16 params_dtyperr adam_beta1 adam_beta2adam_eps clip_graduse_distributed_optimizerr) rr4rrrrr_rr3gradient_clip_valuse_mcore_dist_optimr2r)rprrmegatron_optim_configr,r,r/setup_megatron_optimizationos"   z#ModelPT.setup_megatron_optimization!Optional[Union[DictConfig, Dict]] optim_kwargsOptional[Dict[str, Any]]c Cs|||}|durtddS|jdur#t|jdr#||j_|dur.tj|dd}|j dur8t dd|vr|j durt |j j t sLtd|jjd kr|j j|dd <|j j |dd <|j j|dd <t}|jdurz|j|dd <n*|jdur|j j|j j|dd <n|j j|j j|j|dd <n|j j|dd<t|}tj|dd}d|vr|d}nd}|dd}|dur|dd}nt|r|j}n |dd}|dd}d|vr|d}t ||}nt!"|}|dd|dd|dd|dur|#||dur!||d<|durt|rA||j$fi|} tdt%| | |_&nzz1td|i}|durSd|i} ni} | #|t'j(j)||j$fi| } tdt%| | |_&WnHt*y} z t+d,|t%| | d} ~ ww|dkr|-|} t.| |j/}t0|} nt1|} | |j$fi|} tdt%| | |_&t2|j&||j3d|_4|j&|j4fS)aaPrepares an optimizer from a string name and its optional config parameters. Args: optim_config: A dictionary containing the following keys: * "lr": mandatory key for learning rate. Will raise ValueError if not provided. * "optimizer": string name pointing to one of the available optimizers in the registry. If not provided, defaults to "adam". * "opt_args": Optional list of strings, in the format "arg_name=arg_value". The list of "arg_value" will be parsed and a dictionary of optimizer kwargs will be built and supplied to instantiate the optimizer. optim_kwargs: A dictionary with additional kwargs for the optimizer. Used for non-primitive types that are not compatible with OmegaConf. Nz@No optimizer config provided, therefore no optimizer was createdrT)rzSTrainer wasn't specified in model constructor. Make sure that you really wanted it.schedzIWe do not currently support gradient acculumation that is not an integer.r t_max_epochst_accumulate_grad_batchest_limit_train_batches t_num_workers max_steps_target_radamrrrruzOptimizer config = %sz4Could not instantiate class path - {} with kwargs {}mcore_distributed_optim) optimizerscheduler_configtrain_dataloader)5setup_optimizer_param_groups_optim_config_copyr"inforLrrr to_container_trainerrcr?accumulate_grad_batchesintr@r3r  max_epochslimit_train_batchesr$data_parallel_sizer num_devices num_nodescreatepopr_inspectisclassrJlowersplitparse_optimizer_argscopydeepcopyupdaterSr}rThydrautils instantiate ExceptionerrorrGrrr4r get_optimizerr!rPrU)rprrrqr optimizer_clsoptimizer_nameroptimizer_argsroptimizer_configoptimizer_instanceer_megatron_optimizerr,r,r/setup_optimizations                                zModelPT.setup_optimizationcCs,t|ds d|_dSg}g}d|jvr|jj}|D]@\}}t||d}|dur0t|dt|drU||dt| i}|D]\}} | ||<qF||qt|dg} | D]\} } d} |D] }| |rsd} qj| r{| | qbt | rd| ig|}n dt| ig}||_dS) a Used to create param groups for the optimizer. As an example, this can be used to specify per-layer learning rates: optim.SGD([ {'params': model.base.parameters()}, {'params': model.classifier.parameters(), 'lr': 1e-3} ], lr=1e-2, momentum=0.9) See https://pytorch.org/docs/stable/optim.html for more information. By default, ModelPT will use self.parameters(). Override this method to add custom param groups. In the config file, add 'optim_param_groups' to support different LRs for different components (unspecified params will use the default LR): model: optim_param_groups: encoder: lr: 1e-4 momentum: 0.8 decoder: lr: 1e-3 optim: lr: 3e-3 momentum: 0.9 parametersNoptim_param_groupsz not found in model.paramsz does not have parameters.TF) rrSr2r6rrr@appendrr5named_parameters startswithr)rp known_groups param_groupsparam_groups_cfggroup group_cfgr new_groupkv other_paramsnp is_unknownr,r,r/r9sB          z$ModelPT.setup_optimizer_param_groupscCs(||jdur |jS|jg|jgfS)z8 Configure the optimizer and scheduler. N)r4rUrTr{r,r,r/configure_optimizersys zModelPT.configure_optimizerscs8dfdd D] \}}}j|_|q dS)zK Propagates the model GUID to all submodules, recursively. r'NeuralModule'cs"j|_|D]}|qdSr+) model_guidchildren)rchildrecursively_propagate_guidrpr,r/rMs  z@ModelPT.propagate_model_guid..recursively_propagate_guidN)rrH)rrI)rp_rr,rLr/propagate_model_guids  zModelPT.propagate_model_guidstagecCsf||dkr;d|jvo|jjduo|jjdd}|dup/t|to/t|dk}|r;|r;||jj|dvrsd|jvoQ|jj duoQ|jj dd}| dupft| toft| dk}|rs|rs|j |jj d |d krd |jvo|jj duo|jj dd}| dupt| tot| dk}|r|r|j|jj d dSdSdSdS) zCalled at the beginning of fit, validate, test, or predict. This is called on every process when using DDP. Args: stage: fit, validate, test or predict fitr7Nr8Fr)rQvalidater9r:rr<r=)rOrLr7r_rr?rrr`r9val_dataloaderrar<test_dataloaderrb)rprPtrain_deferred_setupno_train_dataloaderval_deferred_setupno_val_dataloadertest_deferred_setupno_test_dataloaderr,r,r/setupsH          z ModelPT.setupcCs|jdur|jSdS)z. Get the training dataloader. N)rPr{r,r,r/rs zModelPT.train_dataloadercC|jdurg|_|jS)z0 Get the validation dataloader. N)rQr{r,r,r/rS zModelPT.val_dataloadercCr\)z* Get the test dataloader. N)rRr{r,r,r/rTr]zModelPT.test_dataloader sync_metrics,Optional[Dict[str, Dict[str, torch.Tensor]]]cCs|jdurt|jdkriSt|jdtr8|j|jdd}|dur1d|vr1|j|dd|d|j|Sdii}t|jD]x\}}| |}|j||d}|pTi}d|vrhd|vrh||j krh|d|d<| D]E\}}|dkri} | D] \} } | |dvr||j kr| } | | || <n|| } | | | <qz|d} | | | |d<ql||}|||<ql|j|qAd|vr|j|dd|d|S)a Default DataLoader for Validation set which automatically supports multiple data loaders via `multi_validation_epoch_end`. If multi dataset support is not required, override this method entirely in base class. In such a case, there is no need to implement `multi_validation_epoch_end` either. .. note:: If more than one data loader exists, and they all provide `val_loss`, only the `val_loss` of the first data loader will be used by default. This default can be changed by passing the special key `val_dl_idx: int` inside the `validation_ds` config. Args: outputs: Single or nested list of tensor outputs from one or more data loaders. Returns: A dictionary containing the union of all items from individual data_loaders, along with merged logs from all data loaders. Nrdataloader_idxryT)on_epoch sync_distval_loss) validation_step_outputsrr?rMmulti_validation_epoch_endlog_dictrclear enumerate get_validation_dataloader_prefixrrr&)rpr^ output_dictra val_outputsdataloader_prefixdataloader_logsrArBrgk_logv_log new_k_log output_logsnew_kr,r,r/on_validation_epoch_endsD       zModelPT.on_validation_epoch_endcCs|jdurt|jdkriSt|jdtr7|j|jdd}|dur0d|vr0|j|ddd|j|Sdii}t|jD]z\}}| |}|j||d}|pSi}d|vrgd|vrg||j krg|d|d<| D]G\}}|dkri}| D] \} } | |dvr||j kr| } | ||| <n|| } | || <qy| di} | || |d<qk||} ||| <qk|j|q@d|vr|j|ddd|S)a Default DataLoader for Test set which automatically supports multiple data loaders via `multi_test_epoch_end`. If multi dataset support is not required, override this method entirely in base class. In such a case, there is no need to implement `multi_test_epoch_end` either. .. note:: If more than one data loader exists, and they all provide `test_loss`, only the `test_loss` of the first data loader will be used by default. This default can be changed by passing the special key `test_dl_idx: int` inside the `test_ds` config. Args: outputs: Single or nested list of tensor outputs from one or more data loaders. Returns: A dictionary containing the union of all items from individual data_loaders, along with merged logs from all data loaders. Nrr`ryT)rb test_loss)test_step_outputsrr?rMmulti_test_epoch_endrgrrhriget_test_dataloader_prefixrrr_r&)rprkra test_outputsrmrnrArBrgrorprqrrrsr,r,r/on_test_epoch_end9sD        zModelPT.on_test_epoch_endroutputsList[Dict[str, torch.Tensor]]rarcCtddS)a8 Adds support for multiple validation datasets. Should be overriden by subclass, so as to obtain appropriate logs for each of the dataloaders. Args: outputs: Same as that provided by LightningModule.on_validation_epoch_end() for a single dataloader. dataloader_idx: int representing the index of the dataloader. Returns: A dictionary of values, optionally containing a sub-dict `log`, such that the values in the log will be pre-pended by the dataloader prefix. aEMulti data loader support has been enabled, but `multi_validation_epoch_end(outputs, dataloader_idx) has not been implemented. If you require multi data loader support for validation sets, please override this method. If you do not require multi data loader support, please instead override `on_validation_epoch_end(outputs).Nr"rcrpr{rar,r,r/rfz"ModelPT.multi_validation_epoch_endcCr})a2 Adds support for multiple test datasets. Should be overriden by subclass, so as to obtain appropriate logs for each of the dataloaders. Args: outputs: Same as that provided by LightningModule.on_validation_epoch_end() for a single dataloader. dataloader_idx: int representing the index of the dataloader. Returns: A dictionary of values, optionally containing a sub-dict `log`, such that the values in the log will be pre-pended by the dataloader prefix. a9Multi data loader support has been enabled, but `multi_test_epoch_end(outputs, dataloader_idx) has not been implemented. If you require multi data loader support for validation sets, please override this method. If you do not require multi data loader support, please instead override `on_test_epoch_end(outputs).Nr~rr,r,r/rwrzModelPT.multi_test_epoch_endcC |j|Sz Get the name of one or more data loaders, which will be prepended to all logs. Args: dataloader_idx: Index of the data loader. Returns: str name of the data loader at index provided. )rrprar,r,r/rj z(ModelPT.get_validation_dataloader_prefixcCrr)rrr,r,r/rxrz"ModelPT.get_test_dataloader_prefixc Csg}i}|D]+\}}d} |D] } | |vrd} nq|D]} | |vr,||d} nq| r3|||<q|j|dd|duratd|t|dkr_td|d|td dSdSt|dkrqtd |dSdS) z? Load a part of the state dict into the model. FTrNz)Model checkpoint partially restored from rz9The following parameters were excluded when loading from z : z'Make sure that this is what you wanted!zAThe following parameters were excluded when loading checkpoint : )rr8load_state_dictr"rr) rp state_dictincludeexcludeload_from_stringexcluded_param_names dict_to_loadrArB should_addrEr2r,r,r/load_part_of_state_dictsH   zModelPT.load_part_of_state_dictcpurc sgdfddD}t|dkrdSt|dkr+tdfddt|Dd vrjdurtstjtrbj}|j||d d d }|j | d dt d|d~nBtjt tfrj}|D]-}|j}|j||d d d }|ddg}|dg} || || d|d~qqntdWdn1swYdvrijdurittjtrd} t|dr|jdur|j} t| dr| jjdurt d WddS|j| |d d d }|j | d dt d| d~nDtjt tfrUj}|D].}|j} |j| |d d d }|ddg}|dg} || || d| d~q%ntdWdn 1sdwYdvrjdurtwtjtrd} tj| |d} |j | dd dt d | d~ n?tjt tfrۈj}|D])}|j} tj| |d} |ddg}|dg} || d|| d| d~ qn td!WddSWddS1swYdSdSdS)"a Initializes a given model with the parameters obtained via specific config arguments. The state dict of the provided model will be updated with `strict=False` setting so as to prevent requirement of exact model parameters matching. Initializations: init_from_nemo_model: Str path to a .nemo model in order to load state_dict from single nemo file; if loading from multiple files, pass in a dict where the values have the following fields: path: Str path to .nemo model include: Optional list of strings, at least one of which needs to be contained in parameter name to be loaded from this .nemo file. Default: everything is included. exclude: Optional list of strings, which can be used to exclude any parameter containing one of these strings from being loaded from this .nemo file. Default: nothing is excluded. hydra usage example: init_from_nemo_model: model0: path: include:["encoder"] model1: path: include:["decoder"] exclude:["embed"] init_from_pretrained_model: Str name of a pretrained model checkpoint (obtained via cloud). The model will be downloaded (or a cached copy will be used), instantiated and then its state dict will be extracted. If loading from multiple models, you can pass in a dict with the same format as for init_from_nemo_model, except with "name" instead of "path" init_from_ptl_ckpt: Str name of a Pytorch Lightning checkpoint file. It will be loaded and the state dict will extracted. If loading from multiple files, you can pass in a dict with the same format as for init_from_nemo_model. Args: cfg: The config used to instantiate the model. It need only contain one of the above keys. map_location: str or torch.device() which represents where the intermediate state dict (from the pretrained model or checkpoint) will be loaded. )init_from_nemo_modelinit_from_pretrained_modelinit_from_ptl_ckptcs$g|]}|vr|durdndqS)Nrrr,)rarg)r2r,r/r7s$zAModelPT.maybe_init_from_pretrained_checkpoint..rNrzLCannot pass more than one model initialization arguments to config! Found : csg|] \}}|r|qSr,r,)rr arg_present)rr,r/r@sr init_strictT)rrFrz6Model checkpoint restored from nemo file with path : ``rrrznemo file with path `z=Invalid type: init_from_nemo_model is not a string or a dict!rr3resume_from_checkpointzxModel training is being resumed via Pytorch Lightning. Initialization from pretrained model (via cloud) will be skipped.zBModel checkpoint restored from pretrained checkpoint with name : `z!pretrained checkpoint with name `zCInvalid type: init_from_pretrained_model is not a string or a dict!r)rrzIModel checkpoint restored from pytorch lightning checkpoint with path : `z;Invalid type: init_from_ptl_ckpt is not a string or a dict!)sumr@rirrr?r}rr_rrr"rrrMvaluesrrr TypeErrorrrr3_checkpoint_connectorresume_checkpoint_pathfrom_pretrainedrrrYload)rpr2r arg_matches model_pathrestored_modelmodel_load_dictmodel_load_cfgrr model_namer3 ckpt_pathckptr,)rr2r/%maybe_init_from_pretrained_checkpoints-            3     $z-ModelPT.maybe_init_from_pretrained_checkpointcs.|dkrdtjvrtjdt|dS)zm Called at the end of fit and test. Args: stage: either 'fit' or 'test' rQPL_TRAINER_GPUSN)renvironrrBteardown)rprPrrr,r/rs  zModelPT.teardownsave_dirsplit_by_modulecCsD|durt}t|std||||j|||}|S)a Extract the state dict(s) from a provided .nemo tarfile and save it to a directory. Args: restore_path: path to .nemo file from which state dict(s) should be extracted save_dir: directory in which the saved state dict(s) should be stored split_by_module: bool flag, which determins whether the output checkpoint should be for the entire Model, or the individual module's that comprise the Model save_restore_connector (SaveRestoreConnector): Can be overrided to add custom save and restore logic. Example: To convert the .nemo tarfile into a single Model level PyTorch checkpoint :: state_dict = nemo.collections.asr.models.EncDecCTCModel.extract_state_dict_from('asr.nemo', './asr_ckpts') To restore a model from a Model level checkpoint :: model = nemo.collections.asr.models.EncDecCTCModel(cfg) # or any other method of restoration model.load_state_dict(torch.load("./asr_ckpts/model_weights.ckpt")) To convert the .nemo tarfile into multiple Module level PyTorch checkpoints :: state_dict = nemo.collections.asr.models.EncDecCTCModel.extract_state_dict_from( 'asr.nemo', './asr_ckpts', split_by_module=True ) To restore a module from a Module level checkpoint :: model = nemo.collections.asr.models.EncDecCTCModel(cfg) # or any other method of restoration # load the individual components model.preprocessor.load_state_dict(torch.load("./asr_ckpts/preprocessor.ckpt")) model.encoder.load_state_dict(torch.load("./asr_ckpts/encoder.ckpt")) model.decoder.load_state_dict(torch.load("./asr_ckpts/decoder.ckpt")) Returns: The state dict that was loaded from the original .nemo checkpoint Nr)rrrFileExistsErrorrrWextract_state_dict_from)rurrrrrr,r,r/rs4  zModelPT.extract_state_dict_from 'Trainer'cCsLt|jds tddSd}|dur|jdkrt|dS||dS)a Helper method to check whether the model can safely be tested on a dataset after training (or loading a checkpoint). :: trainer = Trainer() if model.prepare_test(trainer): trainer.test(model) Returns: bool which declares the model safe to test. Provides warnings if it has to return False to guide the user. r<z.No `test_ds` config found within the manifest.Fav During testing, it is currently advisable to construct a new Trainer " "with single GPU and no DDP to obtain accurate results. "Following pattern should be used: " "trainer = Trainer(devices=1, accelerator='gpu')" "if model.prepare_test(trainer):" " trainer.test(model) NrT)rrLr"rrrcrV)rpr3DDP_WARNr,r,r/ prepare_tests     zModelPT.prepare_testcCs||_||_||dS)zz Set an instance of Trainer object. Args: trainer: PyTorch Lightning Trainer object. N)r3rset_world_size)rpr3r,r,r/rV+szModelPT.set_trainercCsPd|_|durt|tr|jr|jr|j|j|_ntdt}|j|_dS)z Determines the world size from the PyTorch Lightning Trainer. And then updates AppState. Args: trainer (Trainer): PyTorch Lightning Trainer object rNz8World size can only be set by PyTorch Lightning Trainer.) world_sizer?rrrr"rcr$)rpr3rqr,r,r/r6s     zModelPT.set_world_sizer max_depthmodel_summary.ModelSummarycCstj||dS)a Summarize this LightningModule. Args: max_depth: The maximum depth of layer nesting that the summary will include. A value of 0 turns the layer summary off. Default: 1. Return: The model summary object )r)r summarize)rprr,r,r/rJs zModelPT.summarizerrcCst|dr |jdur dS|dur?t|tst|}|dvr;t|jd|d}||j|<t|jd|j|_dSt ddS)a Update the config (if not None) of the dataset by given name. Preserves said config after updating. Args: dataset_name: str name of the dataset whose config is being updated. Can be one of `train`, `validation` and `test`. config: Optional DictConfig or dict. If None is passed, this method simply returns. If dict is passed, it is cast into a DictConfig. The internal config is updated with the passed config. rTN)trainrrF_dszL`dataset_name` when updating config must be one of [train, validation, test]) rrr?rrrrFr2rLr@)rprrkey_namer,r,r/rVs     zModelPT._update_dataset_configcCs(d}|D] }|jr||7}q|S)z\ Utility property that returns the total number of parameters of the Model. r)r5 requires_gradnumel)rpnumrEr,r,r/ num_weightsvs   zModelPT.num_weightscC|jS)  Property that holds the finalized internal config of the model. Note: Changes to this config are not reflected in the state of the model. Please create a new model using an updated config to properly update the model. )rLr{r,r,r/r2rz ModelPT.cfgcCr)z) Get the trainer object. )rr{r,r,r/r3szModelPT.trainercCsN||_|td|jit|dr#d|jvr%t|j|jd<dSdSdS)rr2_hparams_initialN)rL _set_hparamsrrrr to_object)rpr2r,r,r/r2s  csX|td|jit|dr(d|jvr(t|jdtr(t|jd|jd<t j S)a) Overwrite default hparams property to return the lastest model config. Without this change, the hparams property would return the old config if there was a direct change to self._cfg (e.g., in self.setup_optimization()) that was not done via `self.cfg = new_cfg`. r2r) rrrrLrrr?rrrBhparamsr{rrr,r/rs zModelPT.hparamscCd|jdur|jSg|_|jdur/t|jttfr/t|jdkr/tt|jD]}|jgq&|jS)z Cached outputs of validation_step. It can be a list of items (for single data loader) or a list of lists (for multiple data loaders). Returns: List of outputs of validation_step. Nr)rerQr?rrrrr8rprNr,r,r/res  zModelPT.validation_step_outputscC ||_dSr+)rerpvaluer,r,r/re cCr)z Cached outputs of test_step. It can be a list of items (for single data loader) or a list of lists (for multiple data loaders). Returns: List of outputs of test_step. Nr)rfrRr?rrrrr8rr,r,r/rvs (zModelPT.test_step_outputscCrr+)rfrr,r,r/rvrcCs t}|jSr+)r$is_model_being_restored)rqr,r,r/r^sz ModelPT._is_model_being_restoredrfoldercCst}||_||_dSr+)r$rnemo_file_folder)rrrqr,r,r/rs z ModelPT._set_model_restore_statecCsJt|ds#t}tt|_|r|j}nd}|j|j|ddSdS)NrI)restoration_path) rr$r}uuiduuid4rIr^rregister_model_guid)rpappstaterr,r,r/rXs zModelPT._set_model_guidcCs$t|dr ||_dSt|d|dS)zB Update the save_restore_connector for the model. rWN)rrWr)rurr,r,r/rs  z%ModelPT.update_save_restore_connectorcCs|jdddur|jjddrddlm}ddlm}d|_|jjd d|_|jjd d|_ |jjd d}|dusDt j |sLt d |d t|}|jjdd}|jjdd}|dd}||d|_||d|_|jjddd|jjdddt|jtrtd|jn t dt|jt|j trtd|j n t dt|j |j |jkrnt d|jjddrtd||_tjjtjjjtjjjgtjj d||d|jd|_!dSdSdS)aEnables chakra profiling To use, add the following options to the model config: ## Chakra profiling options chakra_profile: enabled: False start_step: 2 # Global batch to start profiling end_step: 2 # Global batch to end profiling warmup_steps: 0 # Global batch to start profiling active_steps: 1 # Global batch to start profiling trace_dir: None # Path to store the profile output file chakra_profileNenabledFr)ExecutionTraceObserver) get_envintT start_stepend_step trace_dirzchakra profile output path () is not set or does not exist. warmup_steps active_stepsr SLURM_JOB_ID_chakra_kineto)rexist_okz(chakra profiling setup with start_step: z.chakra start_step must be of type int. Found: z&chakra profiling setup with end_step: z,chakra end_step must be of type int. Found: zBchakra end_step must be greater than or equal to chakra start_stepzZProfiler conflict: Chakra profiling and Nsys profiling cannot be enabled at the same time.)waitwarmupactive) activitiesscheduleexecution_trace_observer)"r2r_rtorch.profilerrnemo.utils.env_var_parsingr_chakra_profile_enabled_chakra_profile_start_step_chakra_profile_end_steprrisdirr@r_chakra_trace_dir_kineto_trace_dirrr?rr"rrA nsys_profiler*_etrYprofilerprofileProfilerActivityCPUCUDAr_prof)rprrrrrjob_idr,r,r/rnsZ        zModelPT._setup_chakra_profilingcCs$|jdddurz|jjddrzd|_|jjdd|_|jjdd|_|jjd dg|_|jjd d|_t|jt krJt d |jn t d t|jt|jt kret d |jn t dt|j|j|jkrvnt d|jdddur |jj ddrd|_|jj dd|_|jj dd|_|jj dd|_|jj dd|_t|jt krt d |jn t dt|jt|jt krt d|jn t dt|j|j|jkrnt d|jdustj|jst d|jddSdSdS)aEnables nsys profiling To use, add the following optoins to the model config: ## Nsys profiling options nsys_profile: False start_step: 10 # Global batch to start profiling end_step: 10 # Global batch to end profiling ranks: [0] # Global rank IDs to profile gen_shape: False # Generate model and kernel details including input shapes And then wrap the model training script with: nsys profile -s none -o -t cuda,nvtx --force-overwrite true --capture-range=cudaProfilerApi --capture-range-end=stop python ./examples/... See more options at: https://docs.nvidia.com/nsight-systems/UserGuide/index.html#cli-profiling Enables CUDA memory profiling To use, add the following options to the model config: ## CUDA memory profiling options memory_profile: enabled: True start_step: 10 # Global batch to start profiling end_step: 10 # Global batch to end profiling rank: 0 # Global rank ID to profile output_path: None # Path to store the profile output file rNrFTrrrranks gen_shapez&Nsys profiling setup with start_step: z,Nsys start_step must be of type int. Found: z$Nsys profiling setup with end_step: z*Nsys end_step must be of type int. Found: z>Nsys end_step must be greater than or equal to nsys start_stepmemory_profilerank output_pathz3CUDA memory start_step must be of type int. Found: z+CUDA memory profiling setup with end_step: z1CUDA memory end_step must be of type int. Found: zGCUDA memory end_step must be greater than or equal to memory start_stepzMemory profile output path (r)r2r_r_nsys_profile_enabled_nsys_profile_start_step_nsys_profile_end_step_nsys_profile_ranks_nsys_profile_gen_shaperArr"rr@r_memory_profile_enabled_memory_profile_start_step_memory_profile_end_step_memory_profile_rank_memory_profile_output_pathrrrr{r,r,r/riYsZ   zModelPT._setup_profilingcCs~t|ds=t|jdr5|jjdur5|jjddr5t|dt|jjdd|jdD|jd <dSt|dddSdS) zPyTorch Lightning hook: https://pytorch-lightning.readthedocs.io/en/stable/common/lightning_module.html#on-train-start We use it here to copy the relevant config for dynamic freezing. _freeze_cfgfreeze_updatesNrFcSsi|]}|dqSFr,)rrAr,r,r/ sz*ModelPT.on_train_start..r is_frozen) rr2rr_rrrrrr{r,r,r/on_train_starts   "zModelPT.on_train_startbatchr batch_idxunused Optional[int]cCsp|jjdkrt|drB|jrB|jsB|jj|jkrB|jj|jkrBt d|jjd|j t |jdtd|jd|_t|drt|jrt|jst||jkrtt|jvrtt d tj|jrqtjjjdd d|_t|d r|jr|j s||j!krt|j"krt d tjj#j$d dd|_ t|dr,|j%dur.|j&r0t|dr2|jdur4|jjd}|j%d'D]r\}}t(|t)rt*|dksJd||dko||dkp|ddk}n||kp|dk}|r|j%d|st+||,t+||-d|j%d|<q|s+|j%d|r+t+||.d|j%d|<qdSdSdSdSdSdS)zPyTorch Lightning hook: https://pytorch-lightning.readthedocs.io/en/stable/common/lightning_module.html#on-train-batch-start We use it here to enable profiling and dynamic freezing. rZrz/====== Start chakra profiling from global_step  ======rank-.jsonTrz"====== Start nsys profiling ======) record_shapesrz)====== Start CUDA memory profiling ======i) max_entriesrNr3rrz>freeze_updates modules list cannot have more than two elementsrr r F)/devicerArrror3 global_steprrr"rrregister_callbackr}rr'rstartrrjrrrYrZcudartcudaProfilerStartrautogradr emit_nvtx __enter__rrlrrmemory_record_memory_historyrtrainingrr?rrrfreezerunfreeze)rpr r r num_updatesmlm_steps should_freezer,r,r/on_train_batch_startsX              & zModelPT.on_train_batch_startcCsr|jjdkrt|drQ|jrQ|jrQ|jjd|jkrCt d|jjd|j |j t |jdtd|jd|_n|jjd|jkrQ|j t|d rv|jrv|jsv||jkrvt|jvrvt d tjd |_t|d r|jr|js||jkrt|jkrt d tjj !|j"d|jdtjj j#ddd |_dSdSdSdSdSdSdS)zPyTorch Lightning hook: https://pytorch-lightning.readthedocs.io/en/stable/common/lightning_module.html#on-train-batch-end We use it here to enable nsys profiling. rZrrz+====== End chakra profiling at global_step rrrFrz ====== End nsys profiling ======Trz'====== End CUDA memory profiling ======z/memory_profile_rankz.pickleN)r)$rrArrror3rrr"rrstopexport_chrome_tracer}rr'runregister_callbackrsteprrkrrrYrZrcudaProfilerStoprrmrrr_dump_snapshotrr )rpr{r r rr,r,r/on_train_batch_ends<              zModelPT.on_train_batch_endcCs$t|dr t|dd|_d|_dS)zX Utility function to clean up the module state at the end of execution. rN)rdelattrrerfr{r,r,r/_cleanup_on_execution_ends   z!ModelPT._cleanup_on_execution_endcC |dS)zPyTorch Lightning hook: https://pytorch-lightning.readthedocs.io/en/stable/common/lightning_module.html#on-train-end We use it here to cleanup the dynamic freezing config. Nr1r{r,r,r/ on_train_end*s zModelPT.on_train_endcCr2zPyTorch Lightning hook: https://pytorch-lightning.readthedocs.io/en/stable/common/lightning_module.html#on-test-end Nr3r{r,r,r/ on_test_end2 zModelPT.on_test_endcCr2r5r3r{r,r,r/on_predict_end9r7zModelPT.on_predict_endOptional[DictConfig]cCsP|dur|jdurt|jdr|jj}|durdSt|tr#t|St|S)zy Return a copy of `optim_config` if provided (and otherwise of the internal optim config, if available). Nr) rLrrr?rr$r%rr)rprr,r,r/r@s   zModelPT._optim_config_copyr+)r2rr3r)rsrt)T)r|r}r~r}rr)rsr)rr}rr}r4rrsrt)rr)rr}rr}rsr)rr})NNTFNNT)rr}rrrrrrrrrrr3rrr)rr}rrrrrr)rr)r;r)r>r)rr)NN)rrrr)rPrr)r^rrsr_)rsr_)r)r{r|rarrsr_)rarrsr})r)r2rrr})rPr})FN)rr}rr}rrrr)r3rrsr)r3r)r)rrrsr)rr}rr)rrrr})r rr rrrrsr)r rr rrrrsrt)rrrsr9)JrJrI __qualname____doc__rCrvrzrrrrrrr classmethodrrrr`rrrarbrr4rrGrOr[rrSrTrtrzrfrwrjrxrrrrrrrVrrrpropertyrr2rr3gettersetterrrerv staticmethodr^rrXrrnrir r(r/r1r4r6r8r __classcell__r,r,rrr/r)@s  =  . (G     0@ , `^   '2 = $             IT  9$ )G __future__rr$rrpathlibrabcrrrtypingrrr r r r r rr'rYnemo.core.classes.modulernemo.utils.msc_utilsrrmegatron.core.optimizerrrmegatron.core.utilsrHAVE_MEGATRON_CORE ImportErrorModuleNotFoundErrorlightning.pytorchrrlightning.pytorch.utilitiesrr omegaconfrrrnemor nemo.corernemo.core.classes.commonr+nemo.core.connectors.save_restore_connectorrnemo.core.optimr r! nemo.utilsr"r#nemo.utils.app_stater$nemo.utils.debug_hookr%nemo.utils.exceptionsr&nemo.utils.get_rankr'r(__all__register_new_resolverr)r,r,r,r/sJ    (