o }oi@s<ddlZddlZd ddZd ddZd ddZd d d ZdS) NFcH|rtjdddtdddddfd d }|S) a A forward hook to dump all of the module input and output norms. It is called at every time after forward() has computed an output. Only float type input/output tensor norms are computed. For more details about the forward hook, check https://pytorch.org/docs/stable/generated/torch.nn.modules.module.register_module_forward_hook.html Args: name: tensor name trainer: PTL trainer rank: worker rank logger: PTL log function dump_to_file: wether dump the csv file to the disk debug_infoTexist_okzdebug_info/forward__rank.txtwFc jrg}g}t|D]<\}}t|tjrG|jtjks'|jtjks'|jtjkrGs.| d|j }| |ddd||q t|t rt|D]<\}}t|tjr|jtjksm|jtjksm|jtjkrst| d|j }| |ddd||qQn| d| |j | j s| dd|d d d|d dS) Ninputzdebug_info_forward/r_inputoutput_outputstep, Ttraining enumerate isinstancetorchTensordtypefloathalfbfloat16appenddatanormtuple global_stepwritejoinflush moduleinputsoutputsvaluesheadersni input_norm output_normfpheaderloggernameranktrainerI/home/ubuntu/.local/lib/python3.10/site-packages/nemo/utils/debug_hook.py forward_hook&@ $     $      z&get_forward_hook..forward_hookosmakedirsopen)r1r3r2r0 dump_to_filer6r4r-r5get_forward_hook  $r=cr) aE A backward hook to dump all of the module input and output grad norms. The hook will be called every time the gradients with respect to module inputs are computed. Only float type input/output grad tensor norms are computed. For more details about the backward hook, check https://pytorch.org/docs/stable/generated/torch.nn.modules.module.register_module_full_backward_hook.html Args: name: tensor name trainer: PTL trainer rank: worker rank logger: PTL log function dump_to_file: wether dump the csv file to the disk rTrzdebug_info/backward_rrrFc r ) Nr zdebug_info_backward/rr r r rrrTrr#r-r4r5 backward_hook_r7z(get_backward_hook..backward_hookr8)r1r3r2r0r<r?r4r-r5get_backward_hookMr>r@csLrtjdddtdddddfd d }|S) a A tensor hook to dump all of the tensor weight norms and grad norms at the end of each of the backward steps. For more details about the tensor hook, check https://pytorch.org/docs/stable/generated/torch.Tensor.register_hook.html Args: module: the model module name: tensor name trainer: PTL trainer rank: worker rank logger: PTL log function dump_to_file: wether dump the csv file to the disk rTrzdebug_info/tensor_rz.csvrFcsg}g}}|j}|j}ddd|ddd||||||jrks]|d|d|dd|d d d|d |S) Nzdebug_info_tensors/r _grad_norm _weight_normweightgradrrrT) get_parameterrrrrr r!r")rDr'r(rC weight_norm grad_normr<r.r/r0r$r1r2r3r4r5 tensor_hooks(        z$get_tensor_hook..tensor_hookr8)r$r1r3r2r0r<rIr4rHr5get_tensor_hooks  rJc Csd}tjr tj}|D]\}}|dkr$|t||||||q|D]\}}|dkrG|t |||||| t |||||q)dS)z Register debug hooks. It can 1. track the module forward step input/ouput norm 2. track the module backward step input/output grad norm 3. track the parameter weight norm and grad norm. rN) r distributedis_initializedget_ranknamed_parameters register_hookrJ named_modulesregister_forward_hookr=register_full_backward_hookr@)r$r3r0r<r2r1tensorlayerr4r4r5register_debug_hookss  rV)F)r9rr=r@rJrVr4r4r4r5s   9 9.