o wi|)@sddlmZmZmZmZmZmZddlmZ ddl Z ddl m Z ddl mZddlmZde jdeeeffdd Zde jdeeeffd d ZGd d d e ZGddde ZdS))AnyCallableDictListOptionalUnionN)Callback)MegatronOptimizerModule)loggingtensorreturncCs"t|tjr dt|jiSddiS)zReturns tensor's precision Precision not-a-tensor) isinstancetorchTensorstrdtyper rg/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/lightning/pytorch/callbacks/debugging.pycollect_precisions rcCs,t|tjrt|jt|jdSdddS)z"Returns tensor's shape & precision)Shaper r)rrrrshaperrrrrcollect_precision_and_shape!s  rc @seZdZdZeedfdeeej gee e e ffdeeej gee e e ffde e e e ffddZdejd ejd d fd d Zd S)ParameterDebuggera Debugging tool to help inspect parameters and gradients at any callback event. This callback handles the boilerplate needed to iterate over the model parameters and gradients, and applies user specified functions to them. These functions can be used to log attributes or apply asserts on the param and grad tensors. Attributes are logged in a table, with a row for each parameter name. Default behavior is to log the precision and shapes of each parameter and its gradient. Args: param_fn: Function to apply to model parameters. Can be used to apply assertions on the tensor, or return a mapping of labels and values to log for each parameter. grad_fn: Function to apply to model gradients. Can be used to apply assertions on the tensor, or return a mapping of labels and values to log for each gradient. log_on_hooks: PTL callback hook name or list of hook names on which to apply param_fn and grad_fn. See `PTL docs `_ for more info on callback hooks. Note that some hooks that occur before the model is constructed are invalid. Example: >>> fn = lambda x: {"Norm": str(x.norm(2).item())} >>> callback = ParameterDebugger(param_fn=fn, log_on_hooks=["on_train_start", "on_train_end"]) >>> trainer = Trainer(callbacks=[callback]) on_train_startparam_fngrad_fn log_on_hookscCs\||_||_tgd}t|tr|g}|D]}||vs$Jd||t|||jqdS)N)#teardown on_fit_endon_sanity_check_starton_sanity_check_endon_train_batch_starton_train_batch_endon_train_epoch_starton_train_epoch_endon_validation_epoch_starton_validation_epoch_endon_test_epoch_starton_test_epoch_endon_predict_epoch_starton_predict_epoch_endon_validation_batch_starton_validation_batch_endon_test_batch_starton_test_batch_endon_predict_batch_starton_predict_batch_endr on_train_endon_validation_starton_validation_end on_test_start on_test_endon_predict_starton_predict_end on_exceptionon_save_checkpointon_load_checkpointon_before_backwardon_after_backwardon_before_optimizer_stepon_before_zero_gradzTHook {} supplied to log_on_hooks is not valid or can not be used. Valid hooks are {})rrsetrrformatsetattr_apply_user_funcs)selfrrr valid_hooks hook_namerrr__init__As ( zParameterDebugger.__init__trainer pl_moduler Nc sdtjdttjffdd }ggg}}}D]@\} } || } | dddd} || t| | g|j|jg||gD]\} }}|durZ| durU||| q@|iq@qt gt g}}|D] }|durt| | qg|D] }|dur| | qwt |st |rd d l m}|}|d |td d g||g||gD]0\}}}|D](}g}|D]}|dur|||dq|dq|gkr||||qqd|_td|dSdS)z Iterate over model parameters, find gradient tensor, apply and collect outputs of param_fn and grad_fn, and log outputs in a table. paramr csBttddts |jSjD]}||jvr|j|jSqdS)zJIf using MCore optimizer, search the grad buckets for param's grad tensor.optimN)rgetattrr gradbuffersparam_to_bucket grad_data)rLbufrKrrfind_grad_tensors  z=ParameterDebugger._apply_user_funcs..find_grad_tensorzmodule.z.weightNr) PrettyTable ParameterzParam zGrad l )rrrnamed_parametersreplaceappendziprrrBupdatekeysany prettytablerW add_columngetalignr info get_string)rFrJrKargskwargsrU names_col params_output grads_output param_name param_tensor grad_tensor short_namer fnout_col param_keys grad_keysoutputrW debug_tableprefixr` output_listk col_to_logrrTrrEzsX        z#ParameterDebugger._apply_user_funcs)__name__ __module__ __qualname____doc__rrrrrrrrrrrIplTrainerLightningModulerErrrrr)s 9rc @s\eZdZdZddedefddZdd d eeeffd d Z dd d dde ded df ddZ dS)ModelTrainingStateCallbacka Callback to detect model training state corruption after validation loop. This callback monitors whether all model components maintain their training state consistently before and after validation. Designed to catch issues where some modules are left in eval() mode after PTL validation loop. Args: val_check_interval: Interval at which validation occurs. Default: 10. strict: If True, raises an exception when corruption is detected. Default: False. Fval_check_intervalstrictcCs||_||_d|_d|_dS)NF)rr_pre_validation_state_expecting_check)rFrrrrrrIs z#ModelTrainingStateCallback.__init__rJz pl.Trainerr cs|ddlm}ddlm|j}t||r"|j}t|tr|n|g}n t|tr)|n|g}tfdd|D}|t |dS)z2Get training/eval module counts from model chunks.r)MegatronParallel unwrap_modelc3s*|]}tdd|DVqdS)css|]}|jrdVqdS)N)training).0mrrr szKModelTrainingStateCallback._get_training_state...N)summodules)rchunkrrrrs(zAModelTrainingStateCallback._get_training_state..)training_modules num_chunks) nemo.lightning.megatron_parallelrnemo.utils.model_utilsrmodelrpipelinelistrlen)rFrJrrchunks model_chunkstraining_countrrr_get_training_states   z.ModelTrainingStateCallback._get_training_staterKzpl.LightningModulebatch batch_idxNc Cs|j}|jr7|jr7|jd}||d}||kr1d|d|d|d}|jr,t|t|d|_d|_|d|jd krK|||_d |_dSdS) z/Monitor training state before/after validation.rz=Model training state corruption detected! Before validation: z% training modules, after validation: z training modules (step )NFrrT) global_steprrrr RuntimeErrorr warningr) rFrJrKrrstep pre_count post_countmsgrrrr$s,     z/ModelTrainingStateCallback.on_train_batch_start)rF) r{r|r}r~intboolrIrrrrr$rrrrrs r)typingrrrrrrlightning.pytorchpytorchrrlightning.pytorch.callbacksr%nemo.lightning.pytorch.optim.megatronr nemo.utilsr rrrrrrrrrrs