o }oi@sddlZddlmZmZddlmZddlmZmZddl m Z ddl m Z ddlmZddlmZddlmZGd d d e jeeeZGd d d e jeeeZdS) N)ABCabstractmethod)deepcopy)ListOptional)OptimizerLRScheduler) Optimizer)IOMixin)CallbackMethodsc@s4eZdZdZd ddZedefddZdd ZdS) LRSchedulerModulea/A module to standardize the learning rate scheduler setup and configuration. This class decouples the learning rate scheduler from the model, similar to how the LightningDataModule decouples data handling. It also acts as a Callback to hook into the training loop, which can be useful for adding custom all-reduces, logging, early stopping, etc. Next to that standard Lightning callback-event, this also supports hooking into the Megatron forward-backward function at a granular level. Example:: class MyLRSchedulerModule(LRSchedulerModule): def setup(self, model, optimizer): # Custom setup logic ... def scheduler(self, model, optimizers): # Define and return the learning rate scheduler ... Methods: setup(model, optimizer): Sets up the learning rate scheduler. scheduler(model, optimizers): Abstract method to define the learning rate scheduler. __call__(model, optimizers): Calls the setup and scheduler methods. returnNcCsdS)zSets up the learning rate scheduler. Args: model: The model for which the scheduler is being set up. optimizer: The optimizer for which the scheduler is being set up. N)selfmodel optimizerr r U/home/ubuntu/.local/lib/python3.10/site-packages/nemo/lightning/pytorch/optim/base.pyconnect5szLRSchedulerModule.connectcCtd)a7Abstract method to define the learning rate scheduler. Args: model: The model for which the scheduler is being defined. optimizers: The optimizers for which the scheduler is being defined. Returns: OptimizerLRScheduler: The learning rate scheduler. z9The scheduler method should be implemented by subclasses.NotImplementedErrorrr optimizersr r r scheduler>s zLRSchedulerModule.schedulercCs:|||||||_t|jttfs||jfS|jS)a%Calls the setup and scheduler methods. Args: model: The model for which the scheduler is being called. optimizers: The optimizers for which the scheduler is being called. Returns: OptimizerLRScheduler: The learning rate scheduler. )rr _scheduler isinstancedicttuplerr r r__call__Ks  zLRSchedulerModule.__call__r N) __name__ __module__ __qualname____doc__rrrrrr r r rr s    r c@sleZdZdZdeefddZdejddfdd Z e de e fd d Z dd d ZddejdefddZdS)OptimizerModuleaA module to standardize the optimizer setup and configuration. This class decouples the optimizer from the model, similar to how the LightningDataModule decouples data handling. It also acts as a Callback to hook into the training loop, which can be useful for adding custom all-reduces, logging, early stopping, etc. Next to that standard Lightning callback-event, this also supports hooking into the Megatron forward-backward function at a granular level. Attributes: lr_scheduler (Optional[LRSchedulerModule]): The learning rate scheduler module. Example:: class MyOptimizerModule(OptimizerModule): def __init__(self, lr_scheduler=None): super().__init__(lr_scheduler) def setup(self, model): # Custom setup logic ... def optimizers(self, model): # Define and return the optimizers ... Methods: connect(model, trainer): Connects the optimizer module to the model and trainer. setup(model): Sets up the optimizer. optimizers(model): Abstract method to define the optimizers. __call__(model, megatron_parallel): Calls the setup and optimizers methods. lr_schedulercCs ||_dS)zInitializes the OptimizerModule. Args: lr_scheduler (Optional[LRSchedulerModule]): The learning rate scheduler module. N)r$)rr$r r r__init__s zOptimizerModule.__init__rr Ncs`dfdd }t|||_|_tdr*t|dr,t|jdr.tj|j_dSdSdSdS)zConnects the optimizer module to the model and trainer. Args: model (L.LightningModule): The model to which the optimizer module is being connected. Ncs||d}|S)N)megatron_parallelr )lightning_module_selfr&optrr rcustom_configure_optimizerss z.custom_configure_optimizers__io__optimN)types MethodTypeconfigure_optimizersr,hasattrr+r)rrr*r r)rrs zOptimizerModule.connectcCr)zAbstract method to define the optimizers. Args: model: The model for which the optimizers are being defined. Returns: List[Optimizer]: The list of optimizers. z:The optimizers method should be implemented by subclasses.r)rrr r rrs zOptimizerModule.optimizerscCsR|jdur't|jdjdkr|jdjdd}nd}|jd|ddddSdS)NrlrgT) batch_sizeprog_bar) _optimizerslen param_groupslog)rtrainer pl_modulebatch batch_idxr2r r ron_train_batch_starts z$OptimizerModule.on_train_batch_startcCs|dur|n|}|jj}||vr|||jdur%|j|vr%||j|||_t|jdkr7|jdn|j}|jdurG|||}|S|jS)aBCalls the setup and optimizers methods. Args: model (L.LightningModule): The model for which the optimizers are being called. megatron_parallel: Optional parallel model. Returns: OptimizerLRScheduler: The optimizers and optionally the learning rate scheduler. Nr3r)r: callbacksappendr$rr6r7)rrr&_modelr?_optwith_schedulerr r rrs      zOptimizerModule.__call__rr-)rr r!r"rr r%LLightningModulerrrrrr>rrr r r rr#`s  r#)r.abcrrcopyrtypingrrlightning.pytorchpytorchrD!lightning.pytorch.utilities.typesr torch.optimrnemo.lightning.io.mixinr nemo.lightning.megatron_parallelr Callbackr r#r r r rs      D