o ziJ@sdZddlZddlmZmZmZmZmZmZm Z m Z ddl Z ddl m Z mZddlmZddlmZddlmZddlZddlmZdd lmZdd lmZeeZd e d e!fd dZ"GdddeZ#Gddde#Z$dS)z_ Finetuning Callback ^^^^^^^^^^^^^^^^^^^^ Freeze and unfreeze models for finetuning purposes. N)AnyCallableDict GeneratorIterableListOptionalUnion)Module ModuleDict) _BatchNorm) Optimizer)override)Callback)MisconfigurationException)rank_zero_warnepochreturncCsdS)Ng@)rrrZ/home/ubuntu/.local/lib/python3.10/site-packages/pytorch_lightning/callbacks/finetuning.pymultiplicative&src@s@eZdZdZdde eee eeffdededefddZe de eee eeffddfddZe deddfddZe d?de eee eeffdeddfddZe d ed!edefd"d#Ze  $ d@de eee eeffd ed%eed&ededdf d'd(Zed d d dd)eddfd*d+Ze d,eeeefd-edeeeeffd.d/Zd dd0e d1e d2eeeefddf d3d4Z!ed=d5d6Z"d dd7e d eddfd8d9Z#dAd:d;Z$dS)BBaseFinetuningaThis class implements the base logic for writing your own Finetuning Callback. .. warning:: This is an :ref:`experimental ` feature. Override ``freeze_before_training`` and ``finetune_function`` methods with your own logic. ``freeze_before_training``: This method is called before ``configure_optimizers`` and should be used to freeze any modules parameters. ``finetune_function``: This method is called on every train epoch start and should be used to ``unfreeze`` any parameters. Those parameters need to be added in a new ``param_group`` within the optimizer. .. note:: Make sure to filter the parameters based on ``requires_grad``. Example:: >>> from torch.optim import Adam >>> class MyModel(pl.LightningModule): ... def configure_optimizer(self): ... # Make sure to filter the parameters based on `requires_grad` ... return Adam(filter(lambda p: p.requires_grad, self.parameters())) ... >>> class FeatureExtractorFreezeUnfreeze(BaseFinetuning): ... def __init__(self, unfreeze_at_epoch=10): ... super().__init__() ... self._unfreeze_at_epoch = unfreeze_at_epoch ... ... def freeze_before_training(self, pl_module): ... # freeze any module you want ... # Here, we are freezing `feature_extractor` ... self.freeze(pl_module.feature_extractor) ... ... def finetune_function(self, pl_module, current_epoch, optimizer): ... # When `current_epoch` is 10, feature_extractor will start training. ... if current_epoch == self._unfreeze_at_epoch: ... self.unfreeze_and_add_param_group( ... modules=pl_module.feature_extractor, ... optimizer=optimizer, ... train_bn=True, ... ) rNcCsi|_d|_dSNF)_internal_optimizer_metadata _restartingselfrrr__init__Ws zBaseFinetuning.__init__cCs d|jiS)Ninternal_optimizer_metadata)rrrrr state_dict[szBaseFinetuning.state_dictrcCs&d|_d|vr|d|_dS||_dS)NTr)rrrrrrrload_state_dictas zBaseFinetuning.load_state_dicttrainer pl.Trainer pl_modulepl.LightningModulecCsL|jr$t|}t|jD]\}}||j||}||_qd|_dSdSr)rdictnamed_parameters enumerate optimizers_apply_mapping_to_param_groupsr param_groups)rr"r$r'opt_idx optimizerr+rrr on_fit_startjs   zBaseFinetuning.on_fit_startmodulescCsZt|tr |}t|tr"g}|D] }|t|qt|}n|}dd|DS)aHThis function is used to flatten a module or an iterable of modules into a list of its leaf modules (modules with no children) and parent modules that have parameters directly themselves. Args: modules: A given module or an iterable of modules Returns: List of modules cSs"g|] }t|r |jr|qSr)listchildren _parameters).0mrrr s"z2BaseFinetuning.flatten_modules..) isinstancer valuesrextendrflatten_modulesiterr/)r/_flatten_modulesr4_modulesrrrr9vs   zBaseFinetuning.flatten_modulesTtrain_bn requires_gradccsLt|}|D]}t|tr|sq|jddD] }|j|kr"|VqqdS)amYields the `requires_grad` parameters of a given module or list of modules. Args: modules: A given module or an iterable of modules train_bn: Whether not to train the BatchNorm module requires_grad: Whether to create a generator for trainable or non-trainable parameters. Returns: Generator FrecurseN)rr9r6r parametersr>)r/r=r>modparamrrr filter_paramss  zBaseFinetuning.filter_paramscCs@t|}|D]}t|trd|_|jddD]}d|_qqdS)zUnfreezes the parameters of the provided modules. Args: modules: A given module or an iterable of modules TFr?N)rr9r6r track_running_statsrAr>)r/modulerCrrrmake_trainables  zBaseFinetuning.make_trainablerFcCs,t|trd|_|jddD]}d|_qdS)zjFreezes the parameters of the provided module. Args: module: A given module Fr?N)r6r rErAr>)rFrCrrr freeze_modules zBaseFinetuning.freeze_modulecCs<t|}|D]}t|tr|rt|qt|qdS)zFreezes the parameters of the provided modules. Args: modules: A given module or an iterable of modules train_bn: If True, leave the BatchNorm layers in training mode Returns: None N)rr9r6r rGrH)r/r=rBrrrfreezes   zBaseFinetuning.freezer-paramscs\g}g}|D]tfdd|jDs|q|q|r,tdt|d|S)acThis function is used to exclude any parameter which already exists in this optimizer. Args: optimizer: Optimizer used for parameter exclusion params: Iterable of parameters used to check against the provided optimizer Returns: List of parameters not contained in this optimizer param groups c3s*|]}|dD] }t|VqqdS)rJN)torchequal)r3groupprCrr s(z5BaseFinetuning.filter_on_optimizer..zThe provided params to be frozen already exist within another group of this optimizer. Those parameters will be skipped. HINT: Did you init your optimizer in `configure_optimizer` as such: z<(filter(lambda p: p.requires_grad, self.parameters()), ...) )anyr+appendrtype)r-rJ out_paramsremoved_paramsrrOrfilter_on_optimizers   z"BaseFinetuning.filter_on_optimizer$@lrinitial_denom_lrcCstt||dur|jddnt|}|dur|nd}tj||dd}t||}|r8||||ddSdS)aUnfreezes a module and adds its parameters to an optimizer. Args: modules: A module or iterable of modules to unfreeze. Their parameters will be added to an optimizer as a new param group. optimizer: The provided optimizer will receive new parameters and will add them to `add_param_group` lr: Learning rate for the new param group. initial_denom_lr: If no lr is provided, the learning from the first param group will be used and divided by `initial_denom_lr`. train_bn: Whether to train the BatchNormalization layers. NrrXg?T)r=r>)rJrX)rrGr+floatrDrVadd_param_group)r/r-rXrYr= params_lrdenom_lrrJrrrunfreeze_and_add_param_groups  z+BaseFinetuning.unfreeze_and_add_param_groupstagecCs.||ddlm}t|j|rtddS)Nr)DeepSpeedStrategyzThe Finetuning callback does not support running with the DeepSpeed strategy. Choose a different strategy or disable the callback.)freeze_before_trainingpytorch_lightning.strategiesr`r6strategyNotImplementedError)rr"r$r_r`rrrsetups   zBaseFinetuning.setupr+mappingcsHg}|D]}dd|D}fdd|dD|d<||q|S)NcSsi|] \}}|dkr||qS)rJr)r3kvrrr $szABaseFinetuning._apply_mapping_to_param_groups..csg|]}|qSrr)r3rNrfrrr5%szABaseFinetuning._apply_mapping_to_param_groups..rJ)itemsrR)r+rfoutputg group_staterrjrr*s  z-BaseFinetuning._apply_mapping_to_param_groupsr,num_param_groupscurrent_param_groupscCsfdd|D}||jvr||||j|<dS|t|kr1|j||||d|dSdS)NcSsi|]\}}||qSrr)r3nrNrrrri0sz)BaseFinetuning._store..)r'rr*lenr8)rr$r,rorprfrrr_store)s   zBaseFinetuning._storecCsHt|jD]\}}t|j}|||j||j}|||||qdS)Called when the epoch begins.N)r(r)rrr+finetune_function current_epochrs)rr"r$r,r-rorprrron_train_epoch_start;s  z#BaseFinetuning.on_train_epoch_startrcCt)z$Override to add your unfreeze logic.rd)rr$rr-rrrruDz BaseFinetuning.finetune_functioncCrx)z"Override to add your freeze logic.ryrr$rrrraHrzz%BaseFinetuning.freeze_before_training)rNr"r#r$r%rN)TT)T)NrWTr$r%rN)%__name__ __module__ __qualname____doc__rrrstrrrr!r. staticmethodr r rrr9boolrrDrGrHrIr rVrrZr^rer&r*intrsrwrurarrrrr*s ,  , ( .  0   rcseZdZdZdedddddddf d ed ed ed eed e dede de deddffdd Z e de e effddZe de e efddffdd Ze d&fdd Ze d'd d!Ze ddd"ed#eddfd$d%ZZS)(BackboneFinetuningaFinetune a backbone model based on a learning rate user-defined scheduling. When the backbone learning rate reaches the current model learning rate and ``should_align`` is set to True, it will align with it for the rest of the training. Args: unfreeze_backbone_at_epoch: Epoch at which the backbone will be unfreezed. lambda_func: Scheduling function for increasing backbone learning rate. backbone_initial_ratio_lr: Used to scale down the backbone learning rate compared to rest of model backbone_initial_lr: Optional, Initial learning rate for the backbone. By default, we will use ``current_learning / backbone_initial_ratio_lr`` should_align: Whether to align with current learning rate when backbone learning reaches it. initial_denom_lr: When unfreezing the backbone, the initial learning rate will ``current_learning_rate / initial_denom_lr``. train_bn: Whether to make Batch Normalization trainable. verbose: Display current learning rate for model and backbone rounding: Precision for displaying learning rate Example:: >>> from pytorch_lightning import Trainer >>> from pytorch_lightning.callbacks import BackboneFinetuning >>> multiplicative = lambda epoch: 1.5 >>> backbone_finetuning = BackboneFinetuning(200, multiplicative) >>> trainer = Trainer(callbacks=[backbone_finetuning]) g?NTrWF unfreeze_backbone_at_epoch lambda_funcbackbone_initial_ratio_lrbackbone_initial_lr should_alignrYr=verboseroundingrc sJt||_||_||_||_||_||_||_||_ | |_ d|_ dSN) superrrrrrrrYr=rrprevious_backbone_lr) rrrrrrrYr=rr __class__rrrls  zBackboneFinetuning.__init__cCs|j|jdS)N)rr)rrrrrrrszBackboneFinetuning.state_dictrcs|d|_t|dS)Nr)rrr!r rrrr!s z"BackboneFinetuning.load_state_dictr"r#r$r%cs,t|drt|jtrt||Std)z Raises: MisconfigurationException: If LightningModule has no nn.Module `backbone` attribute. backbonez@The LightningModule should have a nn.Module `backbone` attribute)hasattrr6rr rr.r)rr"r$rrrr.szBackboneFinetuning.on_fit_startcCs||jdSr)rIrr{rrrrasz)BackboneFinetuning.freeze_before_trainingrr-cCs||jkrC|jdd}|jdur|jn||j}||_|j|j|||j|jd|j rAt dt ||j dt ||j dSdS||jkr|jdd}||d|j}|jrb||krb|n|}||jdd<||_|j rt dt ||j dt ||j dSdSdS) rtrrXN)r=rYz Current lr: z, Backbone lr: )rr+rrrr^rr=rYrloginforoundrrr)rr$rr- current_lrinitial_backbone_lrnext_current_backbone_lrrrrrusP      z$BackboneFinetuning.finetune_functionr|r})r~rrrrrrrZrrrrrrrrr!r.rar ru __classcell__rrrrrMsT      $r)%rloggingtypingrrrrrrrr rKtorch.nnr r torch.nn.modules.batchnormr torch.optim.optimizerr typing_extensionsrpytorch_lightningpl$pytorch_lightning.callbacks.callbackr&pytorch_lightning.utilities.exceptionsr%pytorch_lightning.utilities.rank_zeror getLoggerr~rrrZrrrrrrrs$ (       %