o }oi`2"@sdZddlmZddlmZddlmZmZmZm Z ddl m Z ddl Z ddlmZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lm Z ddl!m"Z"ddl#m$Z$e"d\Z%Z&e"d\Z'Z(e(rzddl)m*Z*e"d\Z'Z+e"d\Z'Z,e(re+re,rddl-m.Z.erddl m Z/ddl0m1Z1ddl2m3Z3ddgZ4dej5dej5fddZ6de ej5ej7fde ej5ej7ffddZ8d e j9d!e:ddfd"dZ; # #   # # # $  %   d>d&e:d'edBd2e>dBd3e>dBde?e ej@ejAfejBffd4dZC d?d ejDd!ee:d5ed6ddeBfd7d8ZEd d9d!e:d:d;fdUtility functions for loading models with modelopt layer spec.) nullcontextpartial) TYPE_CHECKINGCallableOptionalUnionN)_WrappingCheckpointIO)StrictHandling) lightning)llm) _setup_trainer_and_restore_model)ckpt_to_context_subdir)ckpt_to_weights_subdir)logging) safe_import) unwrap_modelzmodelopt.torch.opttransformer_engineget_gpt_modelopt_spec mamba_ssm causal_conv1d)get_mamba_stack_modelopt_spec) CheckpointIO)MegatronParallel#set_modelopt_spec_if_exists_in_ckpt2setup_trainer_and_restore_model_with_modelopt_spec model_cfgreturncCstdt|tjsJdzddlm}ddlm}||d|j d}Wnt y=ddl m }||j d|j d }Ynw||_|S) z9Set model.config.transformer_layer_spec to modelopt spec.z@Setting model.config.transformer_layer_spec to gpt_modelopt_speczmodel_cfg must be a GPTConfigrrrT)remap_te_layernorm qk_l2_norm)get_gpt_layer_modelopt_spec) num_expertsrr )rinfo isinstancer GPTConfig functoolsr4megatron.core.post_training.modelopt.gpt.model_specsrr ImportError8megatron.core.inference.modelopt_support.gpt.model_specsr!num_moe_expertstransformer_layer_spec)rrr modelopt_specr!r-]/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/modelopt/model_utils.py_set_gpt_modelopt_spec:s       r/cCsXtdt|tjrttdd|_|St|tjr#tt dd|_ |St dt |)a Set the model layer spec to a modelopt spec variant. This function updates the model config with the appropriate modelopt layer specification based on the model type. Args: model_cfg (Union[llm.GPTConfig, llm.SSMConfig]): The model config. Returns: Union[llm.GPTConfig, llm.SSMConfig]: The model config updated for the modelopt layer specification. zrxrrbrestore_sharded_modelopt_staterry)r5r6rt core_modelr-r-r.restore_modelopt_states.       rr checkpoint_iorcCstsdSt|}tj|sdSt|tr|jn|}t|ddr+t d|j ddt |dr4| nttjj|gt|dd|jd Wdn1sQwYt d dS) a  Save ModelOpt state to checkpoint. Args: model (nn.Module): The MegatronParallel model to save the state from. path (str): The path to the checkpoint. checkpoint_io (CheckpointIO): The checkpoint IO object from MegatronStrategy. N async_saveFzCModel-Optimizer library in use. Async checkpoint saving is blocked.T)blockingrxr7)sharded_strategyz,Saved Model-Optimizer state into checkpoint.)rzrr}r~rr$r rr{rrCmaybe_finalize_save_checkpointr>rxrrbsave_sharded_modelopt_statersave_sharded_strategyr#)r5r6rrckpt_ior-r-r.save_modelopt_state s"      r) rErENNrErErETNFNNN)NN)G__doc__ contextlibrr&rtypingrrrrlightning.pytorchrTLrhtorch.nnnn$lightning.pytorch.plugins.io.wrapperr +megatron.core.dist_checkpointing.validationr nemor rfnemo.collectionsr #nemo.collections.llm.inference.baser nemo.lightning.ckpt_utilsrnemo.lightning.io.plr nemo.utilsrnemo.utils.import_utilsrnemo.utils.model_utilsrr}rz_HAVE_TEr'rHAVE_MAMBA_SSMHAVE_CAUSAL_CONV1D6megatron.core.post_training.modelopt.mamba.model_specsrpllightning.fabric.pluginsr nemo.lightning.megatron_parallelr__all__r%r/r0r4LightningModuler;rintbooldicttupler?r@rkrModulerrr-r-r-r.s                               l )