o ॵi@sddlZddlmZddlZddlZddlmZddlm Z ddl m Z ddl m Z ddlmZddlmZdd lmZe je jd Gd d d eZdS) N)Optional)version)Hooks)HOOKS)CheckpointProcessor)Hook)Priority) get_logger) module_namec@sveZdZdZejZdZdZ   dde e de e de e fd d Z d d Z d dZeddZe  dddZdS)LoadCheckpointHookaLoad a checkpoint file at the beginning of training or evaluating. This hook does not need to be configured or saved in the config file. User should use it by: >>> trainer.train('some-checkpoint', load_all_state=True) or >>> trainer.evaluate('some-checkpoint') instead. Args: checkpoint_file (str): The checkpoint file to be loaded. load_all_state (bool): Load all states(optimizer, epoch, lr_scheduler, random_state, etc.) when loading old training state file or not. The model's state dict will only be loaded if False. strict (bool): If strict, any unmatched keys will cause an error. Fz1.3.1NTcheckpoint_fileload_all_statestrictcCs*||_d|_d|_||_||_t|_dS)NF)r rng_stateneed_load_rng_stater rr processor)selfr r rrm/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/trainers/hooks/checkpoint/load_checkpoint_hook.py__init__+s  zLoadCheckpointHook.__init__cCsXt|ds t|_n|j|_|jdur*||j||j|j}|d|_|j|_ dSdS)Nloggerr) hasattrr rr load_checkpointr rgetrr)rtrainermetarrr before_run8s      zLoadCheckpointHook.before_runcCs|jr>|jdur6t|jdtj|jdtj|jdtj r1tjj |jdd|_dS|j ddSdS)NrandomnumpycpucudaFzlRandom state cannot be found in checkpoint file, this may cause a random data order or model initialization.) rrrsetstatenp set_statetorch set_rng_stater is_availableset_rng_state_allrinfo)rrrrrbefore_train_iterDs   z$LoadCheckpointHook.before_train_itercCs|d|j|_|d|j|_|d|j|_d}|jD]5}t|drRt|ddrR|jd|}||vr@|||in|j d |jd |d |d 7}qdS) Nepochiter inner_iterrload_state_dict _should_saveT-zThe state_dict of hook z at index z% is not found in the checkpoint file.) r_epoch_iter _inner_iterhooksrgetattr __class__r-rwarning)rrihookkeyrrr_restore_training_stateSs"  z*LoadCheckpointHook._restore_training_statecCs|j||||}|r||||durG|d}|dur5t|ttjkr5|j dtjd|j d|d|dd||S) aA static method to load checkpoint files. Args: filename(str): An absolute model bin file(pth or bin) or a dir path with a file prefix(like epoch_1). trainer(`EpochBasedTrainer`): The trainer instance. load_all_state(`bool`): Load all states including the trainer states. strict(`bool`): Load module state dict strictly. Returns: A dict containing the train states saved by `_create_training_state` N modelscopezPThe unique pth file is split into a model file and a trainer file since version zd,consider re-training your model or using a converting script to split the single pth file into two.z Checkpoint z saving time: timez, modelscope version: ) rload_checkpointsr;rrparser _TWO_PTH_FILE_VERSIONrr7r()clsfilenamerr rr_versionrrrrfs.    z"LoadCheckpointHook.load_checkpoint)NTF)TF)__name__ __module__ __qualname____doc__rHIGHPRIORITYr.r@rstrboolrrr) staticmethodr; classmethodrrrrrr s.    r )rtypingrrr"r$ packagingrmodelscope.metainfor!modelscope.trainers.hooks.builderr9modelscope.trainers.hooks.checkpoint.checkpoint_processorrmodelscope.trainers.hooks.hookr"modelscope.trainers.hooks.priorityrmodelscope.utils.loggerr register_moduler rrrrs