o پim+@s2dZddlZddlmmZGdddeZdS)z+PyTorch hooks for layerwise NVTX profiling.NcsPeZdZdZfddZedddZddZd d Zd d Z dddZ Z S)PytHooksa@This module contains all the code needed to enable forward hooks in a pytorch network. To register the hooks for a given network, the user needs to instantiate a PytHook object. Then call the register_hooks method. Example: my_hook = PytHook() my_hook.register_hooks(my_network_model) csti|_dS)zkInitialize module variables Returns: None: Raises: None: N)super__init__module_to_name_map)self __class__W/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/utils/nvtx_pytorch_hooks.pyr s zPytHooks.__init__NcCsb|durg}t|tst|tr|D] }t|||}q|St|tjr/t|}|||S)aDescends iterators that contains Tensors and prints the Tensor Recursive function that descends iterator type arguments until it finds a Tensor object. Args: tensor_obj: Could be a Tensor or an iterator type that contains Tensors prefix: String name to assign to the Tensor tensor_list: List to accumulate tensor dimensions Returns: List of tensor dimensions Raises: None: N) isinstancelisttupler print_tensortorchTensorsizeappend) tensor_objprefix tensor_listten tensor_dimsr r r r,s  zPytHooks.print_tensorcCsi}t|tjjst|tjjst|tjjrOi}|j|d<|j|d<|j|d<|j |d<|j |d<|j |d<|j |d<|j |d<|j|d <|j|d <|}|St|tjjsdt|tjjsdt|tjjri}|j|d<|j|d<|j|d<|j |d<|j |d<|j |d<|j |d<|j |d<|j|d <|j|d <|}|St|tjjst|tjjst|tjjrd d }i}||j|d<||j |d<||j |d<||j |d<|}|St|tjjst|tjjst|tjjri}|j|jg|d<|j |j g|d<|j |j g|d<|j|d <|j|d<|}|St|tjjs-t|tjjs-t|tjjr;i}|j|jg|d<|}|St|tjjrO|j |d<|j!|d<|St|tjj"sgt|tjj#sgt|tjj$rx|j%|d<|j&|d<|j'|d<|St|tjj(r|j)|d<|St|tjj*r|j+|d<|j)|d<|St|tjj,r|j-|d<|j.|d<|St|tjj/tjj0tjj1fr|j2|d<|S)aExtract the static parameters from LLM and VLM relevant layer types Args: module_obj(class): Module state data structure. Returns: param_info(dict): Parameter meta_data for the given op. Raises: None in_chanout_chan filter_dimstridepaddingdilation transposedoutput_paddinggroups padding_modecSs(t|tr t|St|tr||gSdSN)r r r int) parameterr r r _handle_int_or_tuples  z;PytHooks.process_layer_params.._handle_int_or_tuple ceil_modecount_include_pad output_size in_features out_features num_featuresepsilonmomentumin_placepnum_embeddings embedding_dim scale_factor)3r rnnConv1dConv2dConv3d in_channels out_channels kernel_sizerrrrrr r!ConvTranspose1dConvTranspose2dConvTranspose3d MaxPool1d MaxPool2d MaxPool3d AvgPool1d AvgPool2d AvgPool3dr&r'AdaptiveAvgPool1dAdaptiveAvgPool2dAdaptiveAvgPool3dr(Linearr)r* BatchNorm1d BatchNorm2d BatchNorm3dr+epsr-ReLUinplaceDropoutr/ Embeddingr0r1UpsampleUpsamplingNearest2dUpsamplingBilinear2dr2)r module_obj param_info conv_paramsconvtranspose_paramsr%pooling_paramsr r r process_layer_paramsIs             [             J   8     )                  zPytHooks.process_layer_paramscCs tdS)aCallback function that ends the NVTX marker Records the module name and tensor information Called after the module executes the forward method. Args: module_obj: Pointer to the module object in_tensor: Input tensor or list of tensors out_tensor: Output tensor of the resulting forward operator Returns: None: Raises: None: N)nvtx range_pop)rrR in_tensor out_tensorr r r module_fwd_hookszPytHooks.module_fwd_hookc Csi}|j|d}||d<|jdd}t|D]\}\}}|dkr%i|d<t||d|<qt|d} | r<| |d<||} | rG| |d <t d |d S) aTCreates an NVTX marker with the module name in it. This function is called before the module executes Args: module_obj: Module object data structure - used to get unique module name in_tensor: Input tensor data structure Returns: None Raises: None unknownModuleF)recurserTrainableParamsInputInputs StaticParamsz{}N) rgetnamed_parameters enumerater rrrrWrX range_pushformat) rrRrZ marker_dict module_name module_paramsidx param_name param_objin_tensor_listrSr r r module_fwd_pre_hooks    zPytHooks.module_fwd_pre_hooktopcCstjjtjjtjjtjjtjjf}|j|dD]'\}}t||r!q| |j | |j ||j vr8||j |<qtd|dS)aUser level function that activates all the hooks The user needs to call this method from the network source code The code descends all the modules in the network and registers their respective hooks. Args: network_model: Model object for the network module_prefix: (default: top) Returns: None Raises: Exception if a module instance is reused )rz!Module instance {} is not unique N)rr3IdentityrM Dropout1d Dropout2d Dropout3d named_modulesr register_forward_pre_hookrpregister_forward_hookr\r ValueErrorrh)r network_model module_prefix skip_typesnamemoduler r r register_hookss     zPytHooks.register_hooksr")rq) __name__ __module__ __qualname____doc__r staticmethodrrWr\rpr __classcell__r r rr rs  |&r)rrtorch.cuda.nvtxcudarXobjectrr r r r s