o }oi`@s ddlZddlZddlmZejjdejefddZejjdejefddZ dejefdd Z dejefd d Z dejefd d Z dejefddZ dejefddZdejefddZdejefddZdejefddZdejefddZdS)N)MegatronMixedPrecisionreturncCstjtdtjtjdddS)zCreate a MegatronMixedPrecision plugin configuration for mixed precision training using BF16. Returns: run.Config[MegatronMixedPrecision]: Configuration for BF16 mixed precision training z bf16-mixedFT precision params_dtypepipeline_dtypeautocast_enabledgrad_reduce_in_fp32)runConfigrtorchbfloat16rrj/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/recipes/precision/mixed_precision.py bf16_mixedrcCstjtdtjtjdddS)zCreate a MegatronMixedPrecision plugin configuration for mixed precision training using FP16. Returns: run.Config[MegatronMixedPrecision]: Configuration for FP16 mixed precision training z16-mixedFr)r r rr halfrrrr fp16_mixed'rrcC.t}d|_d|_d|_d|_d|_d|_|S)a9Create a MegatronMixedPrecision plugin configuration for mixed precision training using BF16 with FP8. Note: FP8 recipes are experimental and have not been tested for training convergence. Returns: run.Config[MegatronMixedPrecision]: Configuration for BF16 with FP8 mixed precision training hybriddelayedrmaxT)rfp8 fp8_recipe fp8_marginfp8_amax_history_lenfp8_amax_compute_algofp8_param_gathercfgrrrbf16_with_fp8_mixed8r!cCr)a9Create a MegatronMixedPrecision plugin configuration for mixed precision training using FP16 with FP8. Note: FP8 recipes are experimental and have not been tested for training convergence. Returns: run.Config[MegatronMixedPrecision]: Configuration for FP16 with FP8 mixed precision training rrrrrT)rrrrrrrrrrrfp16_with_fp8_mixedJr"r#cCt}d|_d|_d|_|S)zCreate a MegatronMixedPrecision plugin configuration for mixed precision training using BF16 with MXFP8. Returns: run.Config[MegatronMixedPrecision]: Configuration for BF16 with MXFP8 mixed precision training rmxfp8Trrrrrrrrbf16_with_mxfp8_mixed\ r'cCr$)zCreate a MegatronMixedPrecision plugin configuration for mixed precision training using FP16 with MXFP8. Returns: run.Config[MegatronMixedPrecision]: Configuration for FP16 with MXFP8 mixed precision training rr%Trrrrrrrrfp16_with_mxfp8_mixedir(r*cC.t}d|_d|_d|_d|_d|_d|_|S)Create a MegatronMixedPrecision plugin configuration for mixed precision training using BF16 with FP8 per-tensor current scaling. Note: The baseline current scaling recipe uses BF16 in the first and last Transformer layers. The user can choose to disable the BF16 layers or apply BF16 to more Transformer layers. Returns: run.Config[MegatronMixedPrecision]: Configuration for BF16 with FP8 per-tensor current scaling mixed precision training r tensorwiseTrrrfirst_last_layers_bf16num_layers_at_start_in_bf16num_layers_at_end_in_bf16rrrrr#bf16_with_fp8_current_scaling_mixedv r3cCr+)r,rr-Tr/rrrr.nemotron_h_bf16_with_fp8_current_scaling_mixedr4r6cCr+)aCreate a MegatronMixedPrecision plugin configuration for mixed precision training using FP16 with FP8 per-tensor current scaling. Note: The baseline current scaling recipe uses FP16 in the first and last Transformer layers. The user can choose to disable the FP16 layers or apply FP16 to more Transformer layers. Returns: run.Config[MegatronMixedPrecision]: Configuration for FP16 with FP8 per-tensor current scaling mixed precision training rr-Tr.)rrrr0r1r2rrrrr#fp16_with_fp8_current_scaling_mixedr4r7cCr$)a{Create a MegatronMixedPrecision plugin configuration for mixed precision training using BF16 with FP8 NV Subchannel scaling. This recipe uses 128x128 blockwise quantization for weight and 1x128 blockwise quantization for activation. Returns: run.Config[MegatronMixedPrecision]: Configuration for BF16 with FP8 subchannel scaling mixed precision training r blockwiseFr&rrrr&bf16_with_fp8_subchannel_scaling_mixed r9cCr$)a{Create a MegatronMixedPrecision plugin configuration for mixed precision training using FP16 with FP8 NV Subchannel scaling. This recipe uses 128x128 blockwise quantization for weight and 1x128 blockwise quantization for activation. Returns: run.Config[MegatronMixedPrecision]: Configuration for FP16 with FP8 subchannel scaling mixed precision training rr8Fr)rrrr&fp16_with_fp8_subchannel_scaling_mixedr:r;)nemo_runr r .nemo.lightning.pytorch.plugins.mixed_precisionrclifactoryr rrr!r#r'r*r3r6r7r9r;rrrrs