o }oi@sddlmZmZmZddlmZddlmZddlm Z ddl m Z ddl m Z ddlmZmZddZd d d ZGd d d eZdS))CallableListOptionalN) Optimizer)ParamsT)MegatronParallel)LRSchedulerModuleOptimizerModulecCsd|vS)Nbias) param_nameparamr r X/home/ubuntu/.local/lib/python3.10/site-packages/nemo/lightning/pytorch/optim/pytorch.py_param_does_not_have_wdsrcCsgg}}|dur'|D]\}}|jsq |||r ||q ||q n ttdd|}ttt||fdksAJdddt ||f|dfDS)NcSs|jSN) requires_grad)xr r r)sz1_extract_model_params_for_optim..rz+Expected at least one optimizer with paramscSsg|] \}}||dqS))params weight_decayr ).0rwdr r r -sz3_extract_model_params_for_optim..) named_parametersrappendlistfilter parametersmaxmaplenzip)modelrno_weight_decay_condparams_with_wdparams_without_wdnamer r r r_extract_model_params_for_optims    r'c seZdZdZdeddfdeegefdee deedeede f fd d Z dddZ de efddZdejddfddZZS)PytorchOptimizerModuleaA OptimizerModule for pytorch optimizers. Attributes: optimizer_fn (Callable[[ParamsT], Optimizer]): Configuration for the optimizer. no_weight_decay_cond (Optional[Callable]): Condition for no weight decay. scale_lr_cond (Optional[Callable]): Condition for scaling learning rate. lr_mult (float): Learning rate multiplier. Example:: optimizer_fn = run.Partial( SGD, lr=lr, weight_decay=wd, ) lr_scheduler = MyLRSchedulerModule(...) optimizer_module = PytorchOptimizerModule(optimizer_fn, lr_scheduler) Methods: setup(model): Sets up the optimizer. optimizers(model): Defines the optimizers. Ng? optimizer_fn lr_schedulerr# scale_lr_condlr_multcs*tj|d||_||_||_||_dS)aInitializes the PytorchOptimizerModule. Args: optimizer_fn (Callable[[ParamsT], Optimizer]): Configuration for the optimizer. lr_scheduler (Optional[LRSchedulerModule]): The learning rate scheduler module. no_weight_decay_cond (Optional[Callable]): Condition for no weight decay. scale_lr_cond (Optional[Callable]): Condition for scaling learning rate. lr_mult (float): Learning rate multiplier. )r*N)super__init__r)r#r+r,)selfr)r*r#r+r, __class__r rr.Ks  zPytorchOptimizerModule.__init__trainer pl.Trainer pl_modulepl.LightningModulecCsdS)nooopNr )r/r2r4r r r on_fit_startcsz#PytorchOptimizerModule.on_fit_startreturncsnttr tdjjdd}t|j}|_t|t s&|g}j dur-|Sfdd|DS)a%Defines the optimizers. Args: model (nn.Module): The model for which the optimizers are being defined. Returns: List[Optimizer]: The list of optimizers. Raises: ValueError: If the model is an instance of MegatronParallel. z/Model cannot be an instance of MegatronParallelrrNcsg|] }j|qSr )r* scheduler)roptr"r/r rrsz5PytorchOptimizerModule.optimizers..) isinstancer ValueErrorr)keywordsgetr'r# _optimizersrr*)r/r"roptimr r;r optimizershs   z!PytorchOptimizerModule.optimizersr"csfdd_dS)zConnects the optimizer module to the model. Args: model (L.LightningModule): The model to which the optimizer module is being connected. cs Sr)rBr r;r rrs z0PytorchOptimizerModule.connect..N)configure_optimizers)r/r"r r;rconnectszPytorchOptimizerModule.connect)r2r3r4r5)__name__ __module__ __qualname____doc__rrrrrrfloatr.r7rrBLLightningModulerD __classcell__r r r0rr(3s(  r()rN)typingrrrlightning.pytorchpytorchplrJ torch.optimrtorch.optim.optimizerr nemo.lightning.megatron_parallelr!nemo.lightning.pytorch.optim.baserr rr'r(r r r rs