o }oic@sddlmZddlmZmZmZmZmZddlZddl m Z ddlm Z ddl m Z ddlmZddlmZmZmZmZdd lmZerJdd lmZed Zed d dZGddde ZeededefddZdS))contextmanager) TYPE_CHECKING GeneratorLiteralOptionalTypeVarN)MixedPrecision)nn) Optimizer) to_fabric) DtypeConfigMegatronMixedPrecisionget_optim_config"update_config_with_dtype_overrides)logging)ModelParallelConfigAnyTConfigTr)boundc3@s0eZdZdZ                     d9d ed dejdejdejdededede edede de de de dedededededed e d!e d"e d#e d$e d%df2d&d'Z d(e d%e fd)d*Zd(e d%e fd+d,Zd-ed%efd.d/Zd0ejd%ejfd1d2Zd3ed%efd4d5Zed%ed6fd7d8ZdS):FabricMegatronMixedPrecisionzFabric plugin for mixed precision training with Megatron models. Handles precision conversions and mixed precision training settings in the Fabric training framework. NFTr most_recent? precision)16-mixed bf16-mixed32 params_dtypepipeline_dtypeautocast_dtypeautocast_enabledgrad_reduce_in_fp32fp8 fp8_recipefirst_last_layers_bf16num_layers_at_start_in_bf16num_layers_at_end_in_bf16 fp8_marginfp8_amax_history_lenfp8_amax_compute_algo fp8_wgradfp8_dot_product_attentionfp8_multi_head_attention fp8_paramsfp8_param_gatherfp16_loss_scalefp16_initial_loss_scalefp16_min_loss_scalefp16_loss_scale_windowfp16_hysteresisreturncCsT|durtd|dur||krtd|}n|durd}t|tr't|}|dvr.tjntj}t d$id|dvd|dvd |dvd |pHtjd |pM|d |pR|d |d|d|d|d| d| d| d| d| d|d|d|d|d|d|d|d|d|d|d ||_ |j j rd!|_ n |j j rd"|_ nd#|_ d|_dS)%Nz^fp8_params is deprecated and will be removed in a future release, use fp8_param_gather insteadzaGetting conflicting values for fp8_params and fp8_param_gather. Please only set fp8_param_gather.F)bf16rfp32)r9rfp16)r:z fp16-mixed16rr8r r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/ fp8_paramr1 loss_scaleinitial_loss_scalemin_loss_scaleloss_scale_window hysteresisrrz32-true)rwarning ValueError isinstanceintstrtorchbfloat16float32r dtype_configr:rr8scaler)selfrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6dtyperBrBQ/home/ubuntu/.local/lib/python3.10/site-packages/nemo/lightning/fabric/plugins.py__init__1s       z%FabricMegatronMixedPrecision.__init__datacC|S)zConvert model inputs (forward) to the floating point precision type of this plugin. Note: MegatronStrategy will take care of only doing this when: mpu.is_pipeline_first_stage() rBrMrQrBrBrO convert_inputz*FabricMegatronMixedPrecision.convert_inputcCrR)zConvert outputs to the floating point precision type expected after model's forward. Note: MegatronStrategy will take care of only doing this when: mpu.is_pipeline_first_stage() rBrSrBrBrOconvert_outputrUz+FabricMegatronMixedPrecision.convert_outputconfigcCs t|j|S)zConvert the config to the precision type this plugin handles. This is optional and depends on the precision limitations during optimization. )rrK)rMrWrBrBrOconvert_configs z+FabricMegatronMixedPrecision.convert_configmodulecCst|ds|Sddlm}ddlm}|jjs|jjrH||j}|jj|_|jj|_t|dr>t |j|s<|||j|_|St ||sH|||}|S)zConvert the module parameters to the precision type this plugin handles. This is optional and depends on the precision limitations during optimization. rYr) Float16Module)get_model_config) hasattr megatron.core.transformer.modulerZmegatron.core.utilsr[rKr:r8rYrE)rMrYrZr[rWrBrBrOconvert_modules         z+FabricMegatronMixedPrecision.convert_module optimizercCs>t|D]}|j|jjksJd|j|jjksJdq|S)zConvert the optimizer parameters to the precision type this plugin handles. This is optional and depends on the precision limitations during optimization. z BF16 model/optim config mismatchz FP16 model/optim config mismatch)rr8rKr:)rMr` optim_configrBrBrOconvert_optimizers z.FabricMegatronMixedPrecision.convert_optimizer)NNNccszdVWdSw)zINo explicit precision casting. Inputs are supposed to be manually casted.NrB)rMrBrBrOforward_contextsz,FabricMegatronMixedPrecision.forward_context)NNNFTNNFrrrrrTFFNNNrrrr)__name__ __module__ __qualname____doc__rrHrNboolrGrrFfloatrPrrTrVrrXr Moduler_r rbrrrcrBrBrBrOr*s       Q   rpluginr7cCs t|jdS)N)r)rr)rkrBrBrO!_convert_megatron_mixed_precisionsrl) contextlibrtypingrrrrrrH"lightning.fabric.plugins.precisionrr torch.optimr nemo.lightning.fabric.conversionr .nemo.lightning.pytorch.plugins.mixed_precisionr r rr nemo.utilsr#megatron.core.model_parallel_configrrrrregisterrlrBrBrBrOs"