o wi=@sjddlmZddlZddlmZddlmZmZm Z ej j      d%d e d e d e dedededeededeje fddZej j     d&d e d e dedeedededeje fddZej j   d'dedededeje fdd Zej j    d(d e d e dedeededeje f d!d"Zej j  d)dededeje fd#d$ZdS)*)OptionalN)OptimizerConfig)CosineAnnealingSchedulerMegatronOptimizerModulePytorchOptimizerModule bf16-mixed?ffffff?-C6?? precision warmup_stepsconstant_steps adam_beta1 adam_beta2max_lrmin_lr clip_gradreturnc Cs^tjtd|d|dk|dk||dd|d }|dur|nd|}tjt|||d } tjt|| d S) zU Creates a distributed fused Adam optimizer with cosine annealing scheduler. adam皙?rz16-mixedh㈵>T) optimizerlr weight_decaybf16fp16rradam_epsuse_distributed_optimizerrNrrr)config lr_scheduler)runConfigrrr) r rrrrrrropt_cfgschedr'd/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/llm/recipes/optim/adam.py,distributed_fused_adam_with_cosine_annealings2r)r{Gz?Frforeachc CsDddlm}tjttj|||dd|dtjt|||pd|ddS) z[ Creates a PyTorch Adam optimizer with a cosine annealing learning rate scheduler. rAdamr g+?:0yE>rrbetasepsr+rr  optimizer_fnr") torch.optimr-r#r$rPartialr)rrrrrr+r-r'r'r("pytorch_adam_with_cosine_annealingBs$  r7Trc Cs,ddlm}tjttj|||dd|ddS)zE Creates a PyTorch Adam optimizer with a flat learning rate. rr,r.r/r0r4)r5r-r#r$rr6)rrr+r-r'r'r(pytorch_adam_with_flat_lrcs r9c CsDddlm}tjttj|||ddddtjt|||pd|dd S) z\ Creates a Transformer Engine fused Adam optimizer with cosine annealing scheduler. r FusedAdamr.r/Trrr1r2master_weightsrr r3)%transformer_engine.pytorch.optimizersr;r#r$rr6r)rrrrrr-r'r'r(te_adam_with_cosine_annealing{s$  r?c Cs,ddlm}tjttj|||dddddS)zV Creates a Transformer Engine fused Adam optimizer with a flat learning rate. rr:r.r/Tr<r8)r>r;r#r$rr6)rrr-r'r'r(te_adam_with_flat_lrs r@)rrrr r r Nr )rrrNr*F)rr*T)rrrNr*)rr*)typingrnemo_runr#megatron.core.optimizerrnemo.lightning.pytorch.optimrrrclifactorystrintfloatr$r)boolr7r9r?r@r'r'r'r(s   *