o 8wiJ@sdZddlZddlmZmZddlmZmZmZm Z ddl Z ddl m Z m Z ddlmZddlmZddlmZddlZdd lmZdd lmZdd lmZeeZd ed e fddZ!GdddeZ"Gddde"Z#dS)z_ Finetuning Callback ^^^^^^^^^^^^^^^^^^^^ Freeze and unfreeze models for finetuning purposes. N) GeneratorIterable)AnyCallableOptionalUnion)Module ModuleDict) _BatchNorm) Optimizer)override)Callback)MisconfigurationException)rank_zero_warnepochreturncCsdS)Ng@)rrrc/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/pytorch_lightning/callbacks/finetuning.pymultiplicative'src@s@eZdZdZdde eee eeffdededefddZe de eee eeffddfddZe deddfddZe d?de eee eeffdeddfddZe d ed!edefd"d#Ze  $ d@de eee eeffd ed%eed&ededdf d'd(Zed d d dd)eddfd*d+Ze d,eeeefd-edeeeeffd.d/Zd dd0ed1ed2eeeefddf d3d4Z ed=d5d6Z!d dd7ed eddfd8d9Z"dAd:d;Z#dS)BBaseFinetuningaThis class implements the base logic for writing your own Finetuning Callback. .. warning:: This is an :ref:`experimental ` feature. Override ``freeze_before_training`` and ``finetune_function`` methods with your own logic. ``freeze_before_training``: This method is called before ``configure_optimizers`` and should be used to freeze any modules parameters. ``finetune_function``: This method is called on every train epoch start and should be used to ``unfreeze`` any parameters. Those parameters need to be added in a new ``param_group`` within the optimizer. .. note:: Make sure to filter the parameters based on ``requires_grad``. Example:: >>> from torch.optim import Adam >>> class MyModel(pl.LightningModule): ... def configure_optimizer(self): ... # Make sure to filter the parameters based on `requires_grad` ... return Adam(filter(lambda p: p.requires_grad, self.parameters())) ... >>> class FeatureExtractorFreezeUnfreeze(BaseFinetuning): ... def __init__(self, unfreeze_at_epoch=10): ... super().__init__() ... self._unfreeze_at_epoch = unfreeze_at_epoch ... ... def freeze_before_training(self, pl_module): ... # freeze any module you want ... # Here, we are freezing `feature_extractor` ... self.freeze(pl_module.feature_extractor) ... ... def finetune_function(self, pl_module, current_epoch, optimizer): ... # When `current_epoch` is 10, feature_extractor will start training. ... if current_epoch == self._unfreeze_at_epoch: ... self.unfreeze_and_add_param_group( ... modules=pl_module.feature_extractor, ... optimizer=optimizer, ... train_bn=True, ... ) rNcCsi|_d|_dSNF)_internal_optimizer_metadata _restartingselfrrr__init__Xs zBaseFinetuning.__init__cCs d|jiS)Ninternal_optimizer_metadata)rrrrr state_dict\szBaseFinetuning.state_dictrcCs&d|_d|vr|d|_dS||_dS)NTr)rrrrrrrload_state_dictbs zBaseFinetuning.load_state_dicttrainer pl.Trainer pl_modulepl.LightningModulecCsL|jr$t|}t|jD]\}}||j||}||_qd|_dSdSr)rdictnamed_parameters enumerate optimizers_apply_mapping_to_param_groupsr param_groups)rr r"r%opt_idx optimizerr)rrr on_fit_startks   zBaseFinetuning.on_fit_startmodulescCsZt|tr |}t|tr"g}|D] }|t|qt|}n|}dd|DS)aHThis function is used to flatten a module or an iterable of modules into a list of its leaf modules (modules with no children) and parent modules that have parameters directly themselves. Args: modules: A given module or an iterable of modules Returns: List of modules cSs"g|] }t|r |jr|qSr)listchildren _parameters).0mrrr s"z2BaseFinetuning.flatten_modules..) isinstancer valuesrextendrflatten_modulesiterr-)r-_flatten_modulesr2_modulesrrrr7ws   zBaseFinetuning.flatten_modulesTtrain_bn requires_gradccsLt|}|D]}t|tr|sq|jddD] }|j|kr"|VqqdS)amYields the `requires_grad` parameters of a given module or list of modules. Args: modules: A given module or an iterable of modules train_bn: Whether not to train the BatchNorm module requires_grad: Whether to create a generator for trainable or non-trainable parameters. Returns: Generator FrecurseN)rr7r4r parametersr<)r-r;r<modparamrrr filter_paramss  zBaseFinetuning.filter_paramscCs@t|}|D]}t|trd|_|jddD]}d|_qqdS)zUnfreezes the parameters of the provided modules. Args: modules: A given module or an iterable of modules TFr=N)rr7r4r track_running_statsr?r<)r-modulerArrrmake_trainables  zBaseFinetuning.make_trainablerDcCs,t|trd|_|jddD]}d|_qdS)zjFreezes the parameters of the provided module. Args: module: A given module Fr=N)r4r rCr?r<)rDrArrr freeze_modules zBaseFinetuning.freeze_modulecCs<t|}|D]}t|tr|rt|qt|qdS)zFreezes the parameters of the provided modules. Args: modules: A given module or an iterable of modules train_bn: If True, leave the BatchNorm layers in training mode Returns: None N)rr7r4r rErF)r-r;r@rrrfreezes   zBaseFinetuning.freezer+paramscs\g}g}|D]tfdd|jDs|q|q|r,tdt|d|S)acThis function is used to exclude any parameter which already exists in this optimizer. Args: optimizer: Optimizer used for parameter exclusion params: Iterable of parameters used to check against the provided optimizer Returns: List of parameters not contained in this optimizer param groups c3s*|]}|dD] }t|VqqdS)rHN)torchequal)r1groupprArr s(z5BaseFinetuning.filter_on_optimizer..zThe provided params to be frozen already exist within another group of this optimizer. Those parameters will be skipped. HINT: Did you init your optimizer in `configure_optimizer` as such: z<(filter(lambda p: p.requires_grad, self.parameters()), ...) )anyr)appendrtype)r+rH out_paramsremoved_paramsrrMrfilter_on_optimizers   z"BaseFinetuning.filter_on_optimizer$@lrinitial_denom_lrcCstt||dur|jddnt|}|dur|nd}tj||dd}t||}|r8||||ddSdS)aUnfreezes a module and adds its parameters to an optimizer. Args: modules: A module or iterable of modules to unfreeze. Their parameters will be added to an optimizer as a new param group. optimizer: The provided optimizer will receive new parameters and will add them to `add_param_group` lr: Learning rate for the new param group. initial_denom_lr: If no lr is provided, the learning from the first param group will be used and divided by `initial_denom_lr`. train_bn: Whether to train the BatchNormalization layers. NrrVg?T)r;r<)rHrV)rrEr)floatrBrTadd_param_group)r-r+rVrWr; params_lrdenom_lrrHrrrunfreeze_and_add_param_groups  z+BaseFinetuning.unfreeze_and_add_param_groupstagecCs.||ddlm}t|j|rtddS)Nr)DeepSpeedStrategyzThe Finetuning callback does not support running with the DeepSpeed strategy. Choose a different strategy or disable the callback.)freeze_before_trainingpytorch_lightning.strategiesr^r4strategyNotImplementedError)rr r"r]r^rrrsetups   zBaseFinetuning.setupr)mappingcsHg}|D]}dd|D}fdd|dD|d<||q|S)NcSsi|] \}}|dkr||qS)rHr)r1kvrrr %szABaseFinetuning._apply_mapping_to_param_groups..csg|]}|qSrr)r1rLrdrrr3&szABaseFinetuning._apply_mapping_to_param_groups..rH)itemsrP)r)rdoutputg group_staterrhrr( s  z-BaseFinetuning._apply_mapping_to_param_groupsr*num_param_groupscurrent_param_groupscCsfdd|D}||jvr||||j|<dS|t|kr1|j||||d|dSdS)NcSsi|]\}}||qSrr)r1nrLrrrrg1sz)BaseFinetuning._store..)r%rr(lenr6)rr"r*rmrnrdrrr_store*s   zBaseFinetuning._storecCsHt|jD]\}}t|j}|||j||j}|||||qdS)Called when the epoch begins.N)r&r'rpr)finetune_function current_epochrq)rr r"r*r+rmrnrrron_train_epoch_start<s  z#BaseFinetuning.on_train_epoch_startrcCt)z$Override to add your unfreeze logic.rb)rr"rr+rrrrsEz BaseFinetuning.finetune_functioncCrv)z"Override to add your freeze logic.rwrr"rrrr_Irxz%BaseFinetuning.freeze_before_training)rNr r!r"r#rN)TT)T)NrUTr"r#rN)$__name__ __module__ __qualname____doc__rr r$strrrrr, staticmethodrrrr.r7boolrrBrErFrGr rTrrXr\rcr(intrqrursr_rrrrr+s ,  , ( .  0   rcseZdZdZdedddddddf d ed ed ed eed e dede de deddffdd Z e de e effddZe de e efddffdd Ze d&fdd Ze d'd d!Ze ddd"ed#eddfd$d%ZZS)(BackboneFinetuningaFinetune a backbone model based on a learning rate user-defined scheduling. When the backbone learning rate reaches the current model learning rate and ``should_align`` is set to True, it will align with it for the rest of the training. Args: unfreeze_backbone_at_epoch: Epoch at which the backbone will be unfreezed. lambda_func: Scheduling function for increasing backbone learning rate. backbone_initial_ratio_lr: Used to scale down the backbone learning rate compared to rest of model backbone_initial_lr: Optional, Initial learning rate for the backbone. By default, we will use ``current_learning / backbone_initial_ratio_lr`` should_align: Whether to align with current learning rate when backbone learning reaches it. initial_denom_lr: When unfreezing the backbone, the initial learning rate will ``current_learning_rate / initial_denom_lr``. train_bn: Whether to make Batch Normalization trainable. verbose: Display current learning rate for model and backbone rounding: Precision for displaying learning rate Example:: >>> from pytorch_lightning import Trainer >>> from pytorch_lightning.callbacks import BackboneFinetuning >>> multiplicative = lambda epoch: 1.5 >>> backbone_finetuning = BackboneFinetuning(200, multiplicative) >>> trainer = Trainer(callbacks=[backbone_finetuning]) g?NTrUF unfreeze_backbone_at_epoch lambda_funcbackbone_initial_ratio_lrbackbone_initial_lr should_alignrWr;verboseroundingrc sJt||_||_||_||_||_||_||_||_ | |_ d|_ dSN) superrrrrrrrWr;rrprevious_backbone_lr) rrrrrrrWr;rr __class__rrrms  zBackboneFinetuning.__init__cCs|j|jdS)N)rr)rrrrrrrszBackboneFinetuning.state_dictrcs|d|_t|dS)Nr)rrrrrrrrs z"BackboneFinetuning.load_state_dictr r!r"r#cs,t|drt|jtrt||Std)z Raises: MisconfigurationException: If LightningModule has no nn.Module `backbone` attribute. backbonez@The LightningModule should have a nn.Module `backbone` attribute)hasattrr4rrrr,r)rr r"rrrr,szBackboneFinetuning.on_fit_startcCs||jdSr)rGrryrrrr_sz)BackboneFinetuning.freeze_before_trainingrr+cCs||jkrC|jdd}|jdur|jn||j}||_|j|j|||j|jd|j rAt dt ||j dt ||j dSdS||jkr|jdd}||d|j}|jrb||krb|n|}||jdd<||_|j rt dt ||j dt ||j dSdSdS) rrrrVN)r;rWz Current lr: z, Backbone lr: )rr)rrrr\rr;rWrloginforoundrrr)rr"rr+ current_lrinitial_backbone_lrnext_current_backbone_lrrrrrssP      z$BackboneFinetuning.finetune_functionrzr{)r|r}r~rrrrrXrrrr r$rrrrr,r_r rs __classcell__rrrrrNsT      $r)$rloggingcollections.abcrrtypingrrrrrItorch.nnrr torch.nn.modules.batchnormr torch.optim.optimizerr typing_extensionsr pytorch_lightningpl$pytorch_lightning.callbacks.callbackr &pytorch_lightning.utilities.exceptionsr%pytorch_lightning.utilities.rank_zeror getLoggerr|rrrXrrrrrrrs&        %