o }oi@@sddlZddlZddlmZddlmZmZmZddlm Z m Z ddl m Z ddlmZddlmZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lm Z m!Z!ej"d krfeZ#neZ#ddZ$eddGdddZ%Gddde#Z&dS)N) dataclass)Path PosixPath WindowsPath)OptionalUnion)io)NEMO_MODELS_CACHE)ADAPTER_META_FILENAME) RestoreConfig)logging)AppState)uninject_model_parallel_rank)import_multistorageclientis_multistorageclient_urlntc Csddlm}ddlm}z||d}Wnty/}ztd||WYd}~Sd}~wwt||r?||_|j |j _|Std|S)Nr) TokenizerSpec) load_contextzmodel.tokenizerz`Encountered error while trying to restore tokenizer. Tokenizer is not restored. Original error: zJCheckpoint does not have model.tokenizer field. Tokenizer is not restored.) "nemo.collections.common.tokenizersrnemo.lightning.ior ValueErrorr warning isinstance tokenizer__io__)model ckpt_pathrrrerI/home/ubuntu/.local/lib/python3.10/site-packages/nemo/lightning/resume.py_try_restore_tokenizer(s"     r T)kw_onlyc@seZdZUdZdZeeed<dZee ed<dZ ee ed<dZ e ed<dZ e ed<dZe ed <d Zd efd d ZddeejejffddZde d efddZddZd eefddZddeejd eefddZddeejd eefddZdS) AutoResumeasClass that handles the logic for setting checkpoint paths and restoring from checkpoints in NeMo. Attributes: restore_config (Optional[RestoreConfig]): Optional config for selectively restoring specific parts like model weights, optimizer states, etc. If the config contains a path from HF or another non-NeMo checkpoint format, the checkpoint will be automatically converted to a NeMo compatible format. resume_from_folder or the run's log_dir takes precedence over restore_config. resume_from_directory (str): Path to the checkpointing directory to restore from. resume_from_path (str): Path to a specific checkpoint to restore from. resume_if_exists (bool): Whether this experiment is resuming from a previous run. If True, it sets trainer._checkpoint_connector._ckpt_path so that the trainer should auto-resume. exp_manager will move files under log_dir to log_dir/run_{int}. Defaults to False. resume_past_end (bool): By default, AutoResume throws an error if resume_if_exists is True and a checkpoint matching ``*end.ckpt`` indicating a previous training run fully completed. Setting resume_past_end=True disables this behavior and loads the last checkpoint. resume_ignore_no_checkpoint (bool): AutoResume throws an error if resume_if_exists is True and no checkpoint could be found. Setting resume_ignore_no_checkpoint=True disables this behavior, in which case exp_manager will print a message and continue without restoring. Nrestore_configresume_from_directoryresume_from_pathFresume_if_existsresume_past_endresume_ignore_no_checkpointweightsreturncCs ||jS)zReturns the path to the weights directory within the specified path. Args: path: The checkpoint directory path Returns: Path: A Path object pointing to the weights directory ) WEIGHTS_PATH)selfpathrrrget_weights_pathbs zAutoResume.get_weights_pathtrainercCst|tjr td||}|r?||_||j_t|j ddr=t|t r1|j d}| s0|j }n| |}t||}dSdS|j r|j|j jd}t|t rSJdt||j _|j |j_ |j jrt|t rot|j d}n|d}|sy|}t||dSdSdS)aSets up checkpoint restoration for the Pytorch Lightning trainer. This method configures the trainer with the appropriate checkpoint path for resuming training and handles loading model artifacts like tokenizers when specified. Args: trainer: The PyTorch Lightning trainer or Fabric instance model: Optional model instance to load artifacts into Raises: NotImplementedError: If trainer is a Fabric instance (not yet supported) zFabric is not supported yet.load_artifactsFcontext)r-z/AdapterPath is not supported for restore_configN)rflFabricNotImplementedErrorget_trainer_ckpt_pathrcheckpoint_callbacklast_model_pathgetattrr# AdapterPathbase_model_pathexistsget_context_pathr _extract_pathr-strstrategyr0ris_dir)r,r/rtrainer_ckpt_path context_pathnew_pathrrrsetupms>         zAutoResume.setupr-cCsRd|vr|ds Jd|d\}}tjt|}n|}t|tr't|}|S)Nz://znemo://zDOnly NeMo based paths starting with nemo:// are currently supported.) startswithsplitosr-joinr rr>r)r,r-__pathrCrrrr=s zAutoResume._extract_pathcCsZt|d }t|}Wdn1swYt|d}|s+|r+|j}|S)Nrmodel_ckpt_path)openjsonloadrr@r;parent)r,adapter_meta_pathrfmetadatar:rrr _get_base_model_path_for_adapters   z+AutoResume._get_base_model_path_for_adaptercs@ddlm}m}t}|j}d}|jr(t|jr"t}||j}nt|j}n|dur5tt|d}ndSddt | dD}dd|D} dd|D} t | } || } t | } | dkri| dkrit d t | }|| t }|dkr|dkrt d | rt | dkst dks|jrd |d }|dur|d 7}t||S|jrdS|d |dt | dkr|jst d| ddt | dkrdt| dvr| d}|St d| d|St dkrtfdddDrd}t|}|Stddddd}tdd|Sd}|S)Nr) NotFoundError"_filter_out_unfinished_checkpoints checkpointscSsg|]}|r|qSr)r@.0drrr sz6AutoResume._find_trainer_ckpt_path..*cSg|] }|dr|qS)z*endmatchrXrrrr[cSr])z*lastr^rXrrrr[r`zEnd checkpoint is unfinished and cannot be used to resume the training. Please remove the checkpoint manually to avoid unexpected cosequences, such as restarting from scratch.aJLast checkpoint is unfinished and cannot be used to resume the training. Please remove the checkpoint manually to avoid unexpected cosequences, such as restarting from scratch. Hint: Iteration number can be added to the checkpoint name pattern to maximize chance that there is at least one finished last checkpoint to resume from.z]There were no checkpoints found in checkpoint_dir or no checkpoint folder at checkpoint_dir :z. zTraining from scratch.z. Cannot resume.zFound z= indicating that the last training run has already completed.mp_rankzMultiple checkpoints z that matches *end.ckpt.cs g|] }|tdvr|qS)r)r>)rYslast_checkpointsrrr[s )rbtp_rank fsdp_shardcSs |jSN)lstatst_mtime)pthrrr s z4AutoResume._find_trainer_ckpt_path..T)keyreversezA matches *last.ckpt. Selecting one with the latest modified time.)nemo.utils.exp_managerrUrVr log_dirr$rrrlistgloblenrr;r(r rr#r'r>anyrsorted)r,rUrV app_staterp checkpointmsccheckpoint_dirdist_checkpointsend_dist_checkpointslast_dist_checkpoints end_chkpt_cntend_checkpointsfinished_end_chkpt_cntlast_chkpt_cntfinished_last_chkpt_cntwarnrrdr_find_trainer_ckpt_paths    $    z"AutoResume._find_trainer_ckpt_pathrcCs<d}t}|j|_|jr|}|r|d}|r|}|S)aRetrieves the path to the context directory of a checkpoint. The context directory contains serialized objects like tokenizers. This method handles both cases where the context is directly in the checkpoint directory or in a subdirectory called "context". Args: model: Optional model instance Returns: Optional[Path]: Path to the context directory if found, None otherwise Nr1)r r&restorerr@)r,rrwrvmaybe_context_pathrrrr<s zAutoResume.get_context_pathc Cs|jrr%r&boolr'r(r+rr.rplTrainerr2r3rDBasePathr=rTrrConnectorMixinr<r5rrrrr">s     2  ]"r"csFeZdZUdZeeed<dddeeffddZddZZ S) r9zPath object for adapter paths which include a field for the base model the adapters are trained on to facilitate model loading.r:Nrcs$tj|g|Ri|}||_|Srh)super__new__r:)clsr:argskwargsoutput __class__rrroszAdapterPath.__new__cCsd|jj||jS)Nz{}({!r}, base_model_path={}))formatrras_posixr:)r,rrr__repr__tszAdapterPath.__repr__) rrrrrrrrr __classcell__rrrrr9is  r9)'rNrG dataclassesrpathlibrrrtypingrrlightning.fabricfabricr2lightning.pytorchpytorchrnemo.lightningrnemo.lightning.baser nemo.lightning.ckpt_utilsr 'nemo.lightning.pytorch.strategies.utilsr nemo.utilsr nemo.utils.app_stater nemo.utils.model_utilsrnemo.utils.msc_utilsrrnamerr r"r9rrrrs0           ,