o wi @sddlmZmZmZddlmZzddlmZddl m Z ddl m Z dZ Wneefy5eZ dZ YnwddlmZdd lmZdd lmZdd lmZmZGd d d eZdS))CallableListOptionalNfinalize_model_grads)OptimizerConfig)get_model_configTF) Optimizer)setup_megatron_optimizer)MegatronParallel)LRSchedulerModuleOptimizerModulec steZdZdZ    ddedeedeedeedef fd d Z dddZ de de e fddZddZZS)MegatronOptimizerModuleazA OptimizerModule for the megatron optimizers. Attributes: config (OptimizerConfig): Configuration for the optimizer. no_weight_decay_cond (Optional[Callable]): Condition for no weight decay. scale_lr_cond (Optional[Callable]): Condition for scaling learning rate. lr_mult (float): Learning rate multiplier. Example:: config = OptimizerConfig(...) lr_scheduler = MyLRSchedulerModule(...) optimizer_module = MegatronOptimizerModule(config, lr_scheduler) Methods: setup(model): Sets up the optimizer. optimizers(model): Defines the optimizers. N?config lr_schedulerno_weight_decay_cond scale_lr_condlr_multcs*tj|d||_||_||_||_dS)aInitializes the MegatronOptimizerModule. Args: config (OptimizerConfig): Configuration for the optimizer. lr_scheduler (Optional[LRSchedulerModule]): The learning rate scheduler module. no_weight_decay_cond (Optional[Callable]): Condition for no weight decay. scale_lr_cond (Optional[Callable]): Condition for scaling learning rate. lr_mult (float): Learning rate multiplier. )rN)super__init__rrrr)selfrrrrr __class__b/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/lightning/pytorch/optim/megatron.pyr:s  z MegatronOptimizerModule.__init__trainer pl.Trainer pl_modulepl.LightningModulecsfdd}|t|_dS)zWe will add the finalize_model_grads function to the model config. Args: model: The model for which the optimizer is being set up. csj|i|S)Nr)argskwargsrrrfinalize_model_grads_funcYszGMegatronOptimizerModule.on_fit_start..finalize_model_grads_funcN)rr#)rrrr#rr"r on_fit_startRs z$MegatronOptimizerModule.on_fit_startmodelreturncCs2t|ts tdt||j|j|j|jd}|gS)a0Defines the optimizers. Args: model (MegatronParallel): The model for which the optimizers are being defined. Returns: List[Optimizer]: The list of optimizers. Raises: ValueError: If the model is not an instance of MegatronParallel. z-Model must be an instance of MegatronParallel)rrr) isinstancer ValueErrorr rrrr)rr% optimizerrrr optimizers^s z"MegatronOptimizerModule.optimizerscOst|i|S)z0Return function to finalize the model gradients.r)rr r!rrrrxsz,MegatronOptimizerModule.finalize_model_grads)NNNr)rrrr)__name__ __module__ __qualname____doc__rrr rfloatrr$r rr r*r __classcell__rrrrr&s(  r)typingrrrlightning.pytorchpytorchplmegatron.core.distributedrmegatron.core.optimizerrmegatron.core.utilsrHAVE_MEGATRON_CORE ImportErrorModuleNotFoundErrorobject torch.optimr nemo.lightning._strategy_libr nemo.lightning.megatron_parallelr !nemo.lightning.pytorch.optim.baser r rrrrrs