o oi8@s8dZddlZddlmZmZmZmZddlmZddl m Z ddl m Z ddl mZeeefZdeeeeegefffd d Zd edefd d Zd edefddZd edefddZd edefddZdefddZd edefddZd edefddZd edefddZd edefddZd edefddZdS) a1Contains migration functions to upgrade legacy checkpoints to the format of the current Lightning version. When Lightning loads a checkpoint, these migrations will be applied on the loaded checkpoint dictionary sequentially, see :func:`~lightning.pytorch.utilities.migration.utils.migrate_checkpoint`. For the Lightning developer: How to add a new migration? 1. Create a new function with a descriptive name and docstring that explains the details of this migration. Include version information as well as the specific commit or PR where the breaking change happened. 2. Add the function to the `_migration_index()` below. The key in the index is the version of Lightning in which the change happened. Any checkpoint with a version greater or equal to that version will apply the given function. Multiple migrations per version get executed in the provided list order. 3. You can test the migration on a checkpoint (backup your files first) by running: cp model.ckpt model.ckpt.backup python -m lightning.pytorch.utilities.upgrade_checkpoint model.ckpt N)AnyCallableDictList)PossibleUserWarning) EarlyStopping)ModelCheckpointrank_zero_warnreturncCs"tgttgtgtgttttgdS)zQMigration functions returned here will get executed in the order they are listed.)z0.10.0z1.6.0z1.6.5z1.9.0z2.0.0) (_migrate_model_checkpoint_early_stopping._migrate_loop_global_step_to_progress_tracking0_migrate_loop_current_epoch_to_progress_tracking"_migrate_loop_batches_that_stepped9_migrate_model_checkpoint_save_on_train_epoch_end_default_drop_apex_amp_state+_migrate_loop_structure_after_tbptt_removal4_migrate_loop_structure_after_optimizer_loop_removal5_migrate_loop_structure_after_dataloader_loop_removalrrc/home/ubuntu/.local/lib/python3.10/site-packages/lightning/pytorch/utilities/migration/migration.py_migration_index,sr checkpointcCstdftdftdftdftdfd}|dpi|d<|D](\}}||vrG||}|\}}|d|p7i|d|<||d||<||=q|S)zcThe checkpoint and early stopping keys were renamed. Version: 0.10.0 Commit: a5d1176 best_model_scorebest_model_path wait_countpatience)$checkpoint_callback_best_model_score#checkpoint_callback_best_model_pathcheckpoint_callback_bestearly_stop_callback_waitearly_stop_callback_patience callbacks)rrgetitems)r keys_mappingkeynew_pathvalue callback_type callback_keyrrrr <s r cCsp|d}|ddti|ddt|ddd}||dddd<|ddd }||dd<|S) zSets the `global_step` value for checkpoints before v1.6 without the progress tracking state. It will be overwritten by the loop's state if it was also saved. Version: 1.6.0 Commit: c67b075 PR: #13645, #11805 global_steploopsfit_loop3epoch_loop.batch_loop.optimizer_loop.optim_progress optimizersteptotal completed5epoch_loop.batch_loop.manual_loop.optim_step_progress setdefault!_get_fit_loop_initial_state_1_6_0)rr+optim_progressoptim_step_progressrrrr Vs  r cCsH|d}|ddti|ddt||ddddd<|S)zSets the `current_epoch` value for checkpoints before v1.6 without the progress tracking state. It will be overwritten by the loop's state if it was also saved. Version: 1.6.0 Commit: aea96e4 PR: #11805 epochr,r-epoch_progresscurrentr2r4)rr9rrrrks  rcCs$|d}|dddd||S)zSets the `_batches_that_stepped` default value for checkpoints before v1.6.5 which don't have this key. Version: 1.6.5 Commit: c67b075 PR: #13645 r+r,r-epoch_loop.state_dict_batches_that_stepped)r5)rr+rrrr{srcCsdddddddidddddddddddddddddddiidddddddddddd dddddddd diddddddddddddddddddd iidddddddddddid S) Nr)r2ready)r;r1)r2r>started)r0 zero_grad)r/optimizer_position)r2 processedr>r?F)r; is_last_batchr1r=)r3,epoch_loop.batch_loop.manual_loop.state_dictr./epoch_loop.batch_loop.optimizer_loop.state_dict epoch_loop.batch_loop.state_dictzepoch_loop.batch_progresszepoch_loop.scheduler_progressr<z'epoch_loop.val_loop.dataloader_progressz-epoch_loop.val_loop.epoch_loop.batch_progressz)epoch_loop.val_loop.epoch_loop.state_dictzepoch_loop.val_loop.state_dictr: state_dictrrrrrr6sF         r6csld|vr|Sdtdtfddt|d}fdd|dD}t||kr0tdtd |S||d<|S) a:The ``save_on_train_epoch_end`` was removed from the state-key of ``ModelCheckpoint`` in 1.9.0, and this migration drops it from the state-keys saved in the checkpoint dict so that the keys match when the Trainer loads the callback state. Version: 1.9.0 Commit: f4ca56 PR: #15300, #15606 r"old_keyr cSs|ds|Stdd|S)Nrz., 'save_on_train_epoch_end': (None|True|False)) startswithresub)rHrrrnew_keys zJ_migrate_model_checkpoint_save_on_train_epoch_end_default..new_keycs$i|]\}}t|tr||qSr) isinstancestr).0rHstaterMrr s zM_migrate_model_checkpoint_save_on_train_epoch_end_default..aNYou have multiple `ModelCheckpoint` callback states in this checkpoint, but we found state keys that would end up colliding with each other after an upgrade, which means we can't differentiate which of your checkpoint callbacks needs which states. At least one of your `ModelCheckpoint` callbacks will not be able to reload the state.)category)rOlenr$r r)rnum_keysnew_callback_statesrrRrrs     rcCsd}||vr td||=|S)zApex support was removed in v2.0.0, and this migration drops it from the state-keys saved in the checkpoint dict. Version: 2.0.0 Commit: e544676ff434ed96c6dd3b4e73a708bcb27ebcf1 PR: #16149 amp_scaling_statezTThis checkpoint contains apex AMP data, but apex support has been removed in v2.0.0.r )rr&rrrrs  rcCsd|vr|Sd|dvr|S|dd}ddddd}t|D]\}}||vr2||||<||=q!d|vrC|drC|d|d d <|dd |S) aXAdjusts the loop structure since it changed when the support for truncated backpropagation was removed. The optimizer loop and the manual loop were previously children of the training batch loop. After its removal, they became the children of the training epoch loop. Version: 2.0.0 Commit: 7807454 PR: #16337, #16172 r,r-*epoch_loop.manual_loop.optim_step_progress!epoch_loop.manual_loop.state_dict(epoch_loop.optimizer_loop.optim_progress$epoch_loop.optimizer_loop.state_dict)r3rDr.rErFr<old_batch_loop_state_dictN)listr$pop)rr-old_key_new_key_mappingoldnewrrrrs&     rcCsd|vr|Sd|dvr|S|dd}d|vr |dddd|vr2|d|d<|d|d<d |vrD|d |d <|d |d <|S) a)Adjusts the loop structure since it changed when the support for multiple optimizers in automatic optimization mode was removed. There is no longer a loop over optimizer, and hence no position to store for resuming the loop. Version: 2.0.0 Commit: 6a56586 PR: #16539, #16598 r,r-r[rANr\z,epoch_loop.automatic_optimization.state_dictz0epoch_loop.automatic_optimization.optim_progressrZz)epoch_loop.manual_optimization.state_dictrYz2epoch_loop.manual_optimization.optim_step_progress)r_)rr-rrrrs$   rcsfd|vr|S|d}dD]$}||vrq ||dddfddtD}|q |S)a<The dataloader loops (``_DataLoaderLoop``, ``_PredictionLoop`, and ``_EvaluationLoop``) were flattened into the ``_EvaluationEpochLoop`` (now ``_EvaluationLoop``) and ``_PredictionEpochLoop`` (now ``_PredictionLoop``). Version: 2.0.0 Commit: ec4f592ecfe238edd83185f6c6905fb1e2406d61 PR: #16726 r,) predict_loop validate_loop test_loopdataloader_progressNz epoch_loop.cs.i|]}|r|td|qS)N)rJrUr_)rPkepoch_loop_keylooprrrSGs.zI_migrate_loop_structure_after_dataloader_loop_removal..)r_r^update)rr,loop_keyepoch_loop_dictrrhrr5s   r)__doc__rKtypingrrrr#lightning.fabric.utilities.warningsr*lightning.pytorch.callbacks.early_stoppingr,lightning.pytorch.callbacks.model_checkpointr%lightning.pytorch.utilities.rank_zeror rO _CHECKPOINTrr r rrr6rrrrrrrrrs&      $ 0%'!