o wi@sddlmZddlmZmZmZmZmZddlZddl m Z ddlm Z ddl m Z ddlmZddlmZmZmZmZdd lmZerJdd lmZed Zed d dZGddde ZeededefddZdS))contextmanager) TYPE_CHECKING GeneratorLiteralOptionalTypeVarN)MixedPrecision)nn) Optimizer) to_fabric) DtypeConfigMegatronMixedPrecisionget_optim_config"update_config_with_dtype_overrides)logging)ModelParallelConfigAnyTConfigTr)boundc5@s6eZdZdZ                      d:d ed dejdejdejdededede edede de dede de dededededed ed!e d"e d#e d$e d%e d&df4d'd(Z d)e d&e fd*d+Zd)e d&e fd,d-Zd.ed&efd/d0Zd1ejd&ejfd2d3Zd4ed&efd5d6Zed&ed7fd8d9ZdS);FabricMegatronMixedPrecisionzFabric plugin for mixed precision training with Megatron models. Handles precision conversions and mixed precision training settings in the Fabric training framework. NFTr most_recent? precision)16-mixed bf16-mixed32 params_dtypepipeline_dtypeautocast_dtypeautocast_enabledgrad_reduce_in_fp32fp8 fp8_recipefirst_last_layers_bf16num_layers_at_start_in_bf16num_layers_at_end_in_bf16!reuse_grad_buf_for_mxfp8_param_ag fp8_marginfp8_amax_history_lenfp8_amax_compute_algo fp8_wgradfp8_dot_product_attentionfp8_multi_head_attention fp8_paramsfp8_param_gatherfp16_loss_scalefp16_initial_loss_scalefp16_min_loss_scalefp16_loss_scale_windowfp16_hysteresisreturncCsZ|durtd|dur||krtd|}n|durd}t|tr't|}|dvr.tjntj}t d%id|dvd|dvd |dvd |pHtjd |pM|d |pR|d |d|d|d|d| d| d| d| d| d|d|d|d|d|d|d|d|d|d|d |d!||_ |j j rd"|_ n |j j rd#|_ nd$|_ d|_dS)&Nz^fp8_params is deprecated and will be removed in a future release, use fp8_param_gather insteadzaGetting conflicting values for fp8_params and fp8_param_gather. Please only set fp8_param_gather.F)bf16rfp32)r:rfp16)r;z fp16-mixed16rr9r r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0 fp8_paramr2 loss_scaleinitial_loss_scalemin_loss_scaleloss_scale_window hysteresisrrz32-true)rwarning ValueError isinstanceintstrtorchbfloat16float32r dtype_configr;rr9scaler)selfrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7dtyperCrCZ/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/lightning/fabric/plugins.py__init__1s       z%FabricMegatronMixedPrecision.__init__datacC|S)zConvert model inputs (forward) to the floating point precision type of this plugin. Note: MegatronStrategy will take care of only doing this when: mpu.is_pipeline_first_stage() rCrNrRrCrCrP convert_inputz*FabricMegatronMixedPrecision.convert_inputcCrS)zConvert outputs to the floating point precision type expected after model's forward. Note: MegatronStrategy will take care of only doing this when: mpu.is_pipeline_first_stage() rCrTrCrCrPconvert_outputrVz+FabricMegatronMixedPrecision.convert_outputconfigcCs t|j|S)zConvert the config to the precision type this plugin handles. This is optional and depends on the precision limitations during optimization. )rrL)rNrXrCrCrPconvert_configs z+FabricMegatronMixedPrecision.convert_configmodulecCst|ds|Sddlm}ddlm}|jjs|jjrH||j}|jj|_|jj|_t|dr>t |j|s<|||j|_|St ||sH|||}|S)zConvert the module parameters to the precision type this plugin handles. This is optional and depends on the precision limitations during optimization. rZr) Float16Module)get_model_config) hasattr megatron.core.transformer.moduler[megatron.core.utilsr\rLr;r9rZrF)rNrZr[r\rXrCrCrPconvert_modules         z+FabricMegatronMixedPrecision.convert_module optimizercCs>t|D]}|j|jjksJd|j|jjksJdq|S)zConvert the optimizer parameters to the precision type this plugin handles. This is optional and depends on the precision limitations during optimization. z BF16 model/optim config mismatchz FP16 model/optim config mismatch)rr9rLr;)rNra optim_configrCrCrPconvert_optimizers z.FabricMegatronMixedPrecision.convert_optimizer)NNNccszdVWdSw)zINo explicit precision casting. Inputs are supposed to be manually casted.NrC)rNrCrCrPforward_contextsz,FabricMegatronMixedPrecision.forward_context)NNNFTNNFrrFrrrTFFNNNrrrr)__name__ __module__ __qualname____doc__rrIrOboolrHrrGfloatrQrrUrWrrYr Moduler`r rcrrrdrCrCrCrPr*s       S   rpluginr8cCs t|jdS)N)r)rr)rlrCrCrP!_convert_megatron_mixed_precisionsrm) contextlibrtypingrrrrrrI"lightning.fabric.plugins.precisionrr torch.optimr nemo.lightning.fabric.conversionr .nemo.lightning.pytorch.plugins.mixed_precisionr r rr nemo.utilsr#megatron.core.model_parallel_configrrrrregisterrmrCrCrCrPs"        !