o ॵi+@snddlZddlZddlZddlmZddlmZmZmZddl m Z ddl m Z ddl mZGdddZdS) N) Pipelines)load_checkpointsave_checkpointsave_configuration) ModelFile) get_logger is_masterc@seZdZdZdZddZeddZeddZ dd d Z d dZ ddZ ddZ ddZ ddZddZeddZddZeddZd S) CheckpointProcessorz_trainer_state.pthz.pthcCsD|j}|dddttDvrd|di|d<||||ddS)zPrepares the output of target folder. This is a strategic function which can be registered by other hook's function. Args: trainer: The trainer instance. output_dir: The target folder used in inference. taskcSs g|] }|dstt|qS)__) startswithgetattrr).0attrrm/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/trainers/hooks/checkpoint/checkpoint_processor.py !s z6CheckpointProcessor.prepare_output..typepipelinez*.binN)cfgdirrcopy_files_and_dump_config)selftrainer output_dirconfigrrrprepare_outputs  z"CheckpointProcessor.prepare_outputcCs||j}Gddd}dD]$}|d|dur"|d||d|dur3|d|q|||}t|d rK|j||d d |j|d |jdurZ|jj||j|d |jduri|jj||j|d | dS)z[Copy useful files to target output folder and dumps the target configuration.json. c@s$eZdZddZddZddZdS)zBCheckpointProcessor.copy_files_and_dump_config..SaveConfigcSs||_||_dSN)rr)rrrrrr__init__2s zKCheckpointProcessor.copy_files_and_dump_config..SaveConfig.__init__cSs ||_dSr)r)r _output_dir_configrrr__call__6s zKCheckpointProcessor.copy_files_and_dump_config..SaveConfig.__call__cSst|j|jdSr)rrr)rrrr save_config9szNCheckpointProcessor.copy_files_and_dump_config..SaveConfig.save_configN)__name__ __module__ __qualname__rr"r#rrrr SaveConfig0s r') push_to_hub hub_repo_id hub_token private_hubztrain.checkpoint.period.Nztrain.checkpoint.periodztrain.checkpoint.best.ztrain.checkpoint.bestsave_pretrainedc_sdSrr)argskwargsrrrLsz@CheckpointProcessor.copy_files_and_dump_config..) save_functionrsave_config_function)r1) unwrap_modulemodelsafe_getpophasattrr,rtrain_preprocessoreval_preprocessorr#)rrrbin_filer3r'pop_keysave_config_fnrrrr*sD       z.CheckpointProcessor.copy_files_and_dump_configcCs,tj}t|drtjt|jvrtj}|S)zGet bin file path. model_dir)rTORCH_MODEL_BIN_FILEr6TORCH_MODEL_FILEoslistdirr<)r3default_bin_filerrr _bin_file\szCheckpointProcessor._bin_fileNTc CsJ||j}||\}}|||||||||||||dS)aSave the state dict for trainer and model. This is a strategic function which can be registered by other hook's function. Args: trainer(`EpochBasedTrainer`): The trainer instance. checkpoint_path_prefix(`str`): The saving dir with a prefix. like: /tmp/test/epoch_0 output_dir(`str`): The output dir for inference. meta: (`dict`): The meta info needed to be saved into files. save_optimizers: (`bool`): Do save the optimizers state N)r2r3_get_state_file_namesave_trainer_statesave_model_statelink) rrcheckpoint_path_prefixrmetasave_optimizersr3 _model_file_train_state_filerrrsave_checkpointsgs   z$CheckpointProcessor.save_checkpointscCsB||\}}tj|rt|tj|rt|dSdS)aBRemove obsolete checkpoint files. This is a strategic function which can be registered by other hook's function. Args: trainer(`EpochBasedTrainer`): The trainer instance. checkpoint_path_prefix(`str`): The saving dir with a prefix. like: /tmp/test/epoch_0 N)rCr?pathisfileremove)rrrGrJrKrrrremove_checkpointss    z&CheckpointProcessor.remove_checkpointscCstS)aUsed in ddp or other distributed training scenario, returns whether do saving in current rank. This is a strategic function which can be registered by other hook's function. Args: trainer(`EpochBasedTrainer`): The trainer instance. r)rrrrrshould_save_on_ranksz'CheckpointProcessor.should_save_on_rankc Cs||}tj||}tj|rt|z t||WdStyI}zt d|d|d|dt ||WYd}~dSd}~ww)zLinks the src bin file to the output folder. Args: model: The model instance. src_file: The src bin file path. output_dir: The target folder used in inference. zLink z to z error: z>, changing to copy the bin file, this may use more disk space.N) rBr?rMjoinrNunlinkrFOSErrorrerrorshutilcopyfile)rr3src_filerr9 dest_fileerrrrFs   zCheckpointProcessor.linkcCs,t|||r|jnd|r|jnd|dddS)atSave the trainer state, including optimizer/lr_scheduler's state dict, random states etc. Args: trainer: The trainer instance. model: The model instance. train_state_file: The target file name for saving trainer states. meta: Some extra meta info. save_optimizers: Save optimizers state or not. NF)rH with_model)r optimizer lr_scheduler)rrr3train_state_filerHrIrrrrDs    z&CheckpointProcessor.save_trainer_statecCst||ddddddS)zSave the model state. Args: model: The model instance. model_file: The target file name for saving model states. NF)rH with_meta)r)rr3 model_filerrrrEs  z$CheckpointProcessor.save_model_statecCsP||\}}i}tj|r||||}ntd|d|||||S)alLoad checkpoint files of trainer state and model state. This is a strategic function which can be registered by other hook's function. Args: checkpoint_path_prefix(str): The checkpoint dir with prefix or a model state file. Example: '/tmp/test/epoch_0' or '/tmp/test/epoch_0.pth' trainer(`EpochBasedTrainer`): The trainer instance. load_all_state(`boolean`): Load all states (else load only module states). strict(`boolean`): If strict, any unmatched keys will cause an error. Returns: The meta info in json. zNo trainer state file z found, skip.)rCr?rMrNload_trainer_stateprintload_model_state)rrGrload_all_statestrictrJrKrHrrrload_checkpointss z$CheckpointProcessor.load_checkpointscCs6|rt|ddnd}|rt|ddnd}t|d||S)z!Load trainer state file. r\Nr])rr)rr^rdr\r]rrrrasz&CheckpointProcessor.load_trainer_statecCst|||jddS)zLoad model state file. N)rr2r3)rrr`rerrrrcs z$CheckpointProcessor.load_model_statecCsltj|\}}t|dkstd|ddr"|tj|tjfS||tj ddd|ddfS)aGet the default file name for state files. If the input is a checkpoint dir with prefix, this function will append suffix for both checkpoint files. If the input is an absolute file name, this function will return it as the model file name, and append suffix for the trainer file name. NOTE: a best checkpoint filename with float or int metric value inside will not be judged as having a extension file name. like: '/tmp/test/epoch_0_accuracy0.85' Args: checkpoint_path_prefix(`str`): The checkpoint dir with prefix or a model state file with extension file name. like: '/tmp/test/epoch_0' Returns: A tuple of model state file name and trainer state file name. rz^\d+$N.) r?rMsplitextlenrematchr MODEL_STATE_SUFFIXTRAINER_STATE_SUFFIXsplit)rGbaseextrrrrCs   z(CheckpointProcessor._get_state_file_name)NT)r$r%r&rnrmr staticmethodrrBrLrPrQrFrDrErfrarcrCrrrrr s, 1      r )r?rkrVmodelscope.metainformodelscope.utils.checkpointrrrmodelscope.utils.constantrmodelscope.utils.loggerrmodelscope.utils.torch_utilsr r rrrrs