o }oi+@sddlmZddlmZmZddlmZmZmZm Z ddl Z ddl m Z ddl mZddlmZddlmZed Zd efd d ZeGd ddZGddde ZddZdgZdS))contextmanager) dataclassfields) GeneratorLiteralTypeVarUnionN) Precision)Module) Optimizer)loggingAnyT optimizerccsZdd}z ddlm}t|j|r|jj}n|jg}t||EdHWdStd)zExtract optimizer configurations from a Megatron optimizer. Args: optimizer: A torch.optim.Optimizer instance Yields: Optimizer configurations cSs|jSN)config)xrb/home/ubuntu/.local/lib/python3.10/site-packages/nemo/lightning/pytorch/plugins/mixed_precision.py)sz"get_optim_config..r)ChainedOptimizerNz/Failed to extract optimizer config from module.)megatron.core.optimizerr isinstancemcore_optimizerchained_optimizersmap ValueError)rextract_configroptsrrrget_optim_config s    rc@sPeZdZUdZdZeed<dZeed<dZeed<dZ e j ed<dZ e j ed<dZ e j ed <dZeed <d Zeed <dZeed <dZeed<dZeed<dZeed<dZeed<dZeed<d Zeed<dZeed<dZeed<d Zeed<d Zeed<dZeed<dZeed<dZ eed<dZ!eed <dZ"eed!<dZ#eed"<dZ$eed#<dS)$ DtypeConfigzConfiguration class for mixed precision training settings. Contains settings for FP32/FP16/BF16 training, FP8 training. Ffp32fp16bf16N params_dtypepipeline_dtypeautocast_dtypeautocast_enabledTgrad_reduce_in_fp32fp8delayed fp8_recipefirst_last_layers_bf16r fp8_marginfp8_amax_history_len most_recentfp8_amax_compute_algo fp8_wgradfp8_dot_product_attentionfp8_multi_head_attention fp8_paramfp8_param_gatherr loss_scaleinitial_loss_scalemin_loss_scaleloss_scale_window hysteresisnum_layers_at_start_in_bf16num_layers_at_end_in_bf16)%__name__ __module__ __qualname____doc__r bool__annotations__r!r"r#torchdtyper$r%r&r'r(strr*r+r,intr.r0r1r2r3r4r5r6floatr7r8r9r:r;r<rrrrr6s8                       rc3s~eZdZdZ                   d?d eddejdejdejdededededede de dedededededede d e d!e d"e d#e d$e d%e d&df2fd'd( Z d)e d&e fd*d+Z d,ed&efd-d.Zd/ed&efd0d1Zd/ed&efd2d3Zed&ed4fd5d6Z 7 d@d,ed8ee e fd&dfd9d:Zd,ed8ee e fd&dfd;d<Zd,ed8ee e fd&dfd=d>ZZS)AMegatronMixedPrecisionzPlugin for mixed precision training with Megatron models. Handles conversion of model parameters and inputs/outputs between different precisions, and manages mixed precision training settings. NFTr)rr-r/? precision)16-mixed bf16-mixed32r#r$r%r&r'r(r*r+r,r.r0r1r2r3 fp8_paramsr5fp16_loss_scalefp16_initial_loss_scalefp16_min_loss_scalefp16_loss_scale_windowfp16_hysteresisr;r<returncs\|durtd|dur||krtd|}n|durd}t|tr't|}|dvr.tjntj}t d$id|dvd|dvd |dvd |pHtjd |pM|d |pR|d |d|d|d|d| d| d| d| d| d|d|d|d|d|d|d|d|d|d|d ||_ t |j j rd!|_dS|j jrd"|_dSd#|_dS)%Nz^fp8_params is deprecated and will be removed in a future release, use fp8_param_gather insteadzaGetting conflicting values for fp8_params and fp8_param_gather. Please only set fp8_param_gather.F)r"rOr )r rPr!)r!z fp16-mixed16rNr"r#r$r%r&r'r(r*r+r,r.r0r1r2r3r4r5r;r<r6r7r8r9r:rNrOz32-truer)r warningrrrFrErCbfloat16float32r dtype_configsuper__init__r!rMr")selfrMr#r$r%r&r'r(r*r+r,r.r0r1r2r3rQr5rRrSrTrUrVr;r<rD __class__rrr^bs          zMegatronMixedPrecision.__init__modulecCsddlm}ddlm}|jjs|jjrA||j}|jj|_|jj|_t|dr7t |j|s5|||j|_|St ||sA|||}|S)zConvert the module parameters to the precision type this plugin handles. This is optional and depends on the precision limitations during optimization. r) Float16Module)get_model_configrb) megatron.core.transformer.modulercmegatron.core.utilsrdr\r!r"rbhasattrr)r_rbrcrdrrrrconvert_modules        z%MegatronMixedPrecision.convert_modulercCs>t|D]}|j|jjksJd|j|jjksJdq|S)zConvert the optimizer parameters to the precision type this plugin handles. This is optional and depends on the precision limitations during optimization. z BF16 model/optim config mismatchz FP16 model/optim config mismatch)rr"r\r!)r_r optim_configrrrconvert_optimizers z(MegatronMixedPrecision.convert_optimizerdatacC|S)zConvert model inputs (forward) to the floating point precision type of this plugin. Note: MegatronStrategy will take care of only doing this when: parallel_state.is_pipeline_first_stage() rr_rkrrr convert_inputz$MegatronMixedPrecision.convert_inputcCrl)zConvert outputs to the floating point precision type expected after model's forward. Note: MegatronStrategy will take care of only doing this when: parallel_state.is_pipeline_last_stage() rrmrrrconvert_outputroz%MegatronMixedPrecision.convert_output)NNNccszdVWdSw)zINo explicit precision casting. Inputs are supposed to be manually casted.Nr)r_rrrforward_contextsz&MegatronMixedPrecision.forward_contextclip_valcCs|dkrtddS)aClip gradients. Raises error if clip_val > 0, otherwise it is a no-op. Args: optimizer: The optimizer to clip gradients for clip_val: The value to clip gradients to gradient_clip_algorithm: The algorithm to use for clipping Raises: ValueError: If clip_val > 0 since gradient clipping is handled by Mcore's optimizer rrzbGradient clipping is handled in Mcore's optimizer. Use the clip_grad attribute in OptimizerConfig.N)r)r_rrsgradient_clip_algorithmrrrclip_gradientss z%MegatronMixedPrecision.clip_gradientscCdS)zClip gradients by value - it is a no-op. Args: optimizer: The optimizer to clip gradients for clip_val: The value to clip gradients to Nrr_rrsrrrclip_grad_by_valueroz)MegatronMixedPrecision.clip_grad_by_valuecCrv)zClip gradients by norm - it is a no-op. Args: optimizer: The optimizer to clip gradients for clip_val: The value to clip gradients to Nrrwrrrclip_grad_by_norm roz(MegatronMixedPrecision.clip_grad_by_norm)NNNFTNr)Frr-r/TFFNNNrIrJrKrLrr)rrN)r=r>r?r@rrCrDrArErFrGr^r rhr rjr rnrprrrqrrurxry __classcell__rrr`rrH[s      Q      & rHc Cst|dr t||j|_t|D]5}t||jsqt||j}t||j}||krEt||j|tdt |j d|jd|d|q|S)zUpdate a config object with dtype settings from dtype_config. Args: dtype_config: Source of dtype settings config: Config object to update Returns: Updated config object __io__z Overwrote .z z -> ) rg"update_config_with_dtype_overridesr{rnamegetattrsetattrr debugtyper=)r\rfieldold_valnew_valrrrr}s     *r}) contextlibr dataclassesrrtypingrrrrrC#lightning.pytorch.plugins.precisionr torch.nnr torch.optimr nemo.utilsr r rrrHr}__all__rrrrs     $=