o U۷i @sdZddlmZmZddlmZmZmZmZm Z m Z m Z m Z ddl mZddlmZmZmZddlmZddlmZmZddlmZddlZd ddd d eeefd eeefd edefddZ    d,ddddeded efddZd ddddededefddZddddedefddZ d-d eeefdedeeeffd d!Z!d-d eeefdedeeeffd"d#Z"Gd$d%d%Z#Gd&d'd'Z$Gd(d)d)Z%Gd*d+d+e%Z&dS).z Shared utilities for grouped GEMM kernels. This module contains the tile scheduler classes and helper functions used by both the forward (grouped_gemm_swiglu) and backward (grouped_gemm_dswiglu) kernels. )TupleUnion)BooleanIntegerInt32minextract_mlir_valuesnew_from_mlir_values dsl_user_op const_expr)ir)scfllvmnvvm)T)Float32rNT)nanlocipabrreturnc CsR|rd}nd}ttjtt|j||dt|j||dg|dddtjjdS)zCompute the minimum of two float32 values with NaN handling. :param a: First operand :param b: Second operand :param nan: If True, propagate NaN values :return: Minimum value zmin.NaN.f32 $0, $1, $2;zmin.f32 $0, $1, $2;rrz=f,f,fTFhas_side_effectsis_align_stack asm_dialect)rr inline_asmrf32ir_value AsmDialectAD_ATT)rrrrr ptx_instrr#N/home/ubuntu/vllm_env/lib/python3.10/site-packages/cudnn/grouped_gemm/utils.pyfmin<s"r%Frmask_and_clampabsc CsTt|}|j||d}t|j||d} d} |tjt|| g| dddtjjdS)alPerform a warp-level reduction synchronization for max with abs and NaN. :param value: Value to reduce :param kind: Reduction kind (unused, kept for API compatibility) :param mask_and_clamp: Warp mask and clamp value :param abs: Whether to use absolute value :param nan: Whether to handle NaN values :return: Reduced value across warp rz&redux.sync.max.abs.NaN.f32 $0, $1, $2;z=f,f,iTFr) typerrrrrrr r!) valuekindr'r(rrr value_typevalue_irmask_irr"r#r#r$warp_redux_sync\sr/) positive_onlyrrr*r0cCs^tjt|j||d||d}tjttjj jj j ||||d}t tjt |||dS)a\Perform atomic max operation on a float32 value in global memory. This implementation works correctly for non-negative values (>= 0) using direct bitcast. :param ptr: Pointer to the memory location :param value: The float32 value to compare and potentially store (should be >= 0) :return: The old value at the memory location rresopptrrrr)rbitcastri32rr atomicrmwcutlass_mlirdialects AtomicOpKindMAXrr)r4r*r0rr value_int old_value_intr#r#r$atomic_max_float32s   r?cCs6tjttjjjjj||j ||d||d}t |S)zPerform atomic add operation on a float32 value in global memory. :param ptr: Pointer to the memory location :param value: The float32 value to add :return: The old value at the memory location rr1) rr7rrr8r9r:r;FADDrr)r4r*rr old_valuer#r#r$atomic_add_float32s    rBfastmathcCstjdtjj| |dS)zCompute the sigmoid function: 1 / (1 + exp(-a)). :param a: Input value :param fastmath: Whether to use fast math approximations :return: Sigmoid of input g?rC)cutearch rcp_approxmathexprrCr#r#r$ sigmoid_f32srKcCs|t||dS)zCompute the SiLU (Swish) activation: a * sigmoid(a). :param a: Input value :param fastmath: Whether to use fast math approximations :return: SiLU of input rD)rKrJr#r#r$silu_f32srLc@sveZdZdZdejdefddZdee j fddZ d ee j ddfd d Z e defd d Ze dejfddZdS) WorkTileInfozA class to represent information about a work tile. :ivar tile_idx: The index of the tile. :type tile_idx: cute.Coord :ivar is_valid_tile: Whether the tile is valid. :type is_valid_tile: Boolean tile_idx is_valid_tilecCs||_t||_dSN) _tile_idxr_is_valid_tile)selfrNrOr#r#r$__init__szWorkTileInfo.__init__rcCst|j}|t|j|SrP)rrNextendrOrSvaluesr#r#r$__extract_mlir_values__s z$WorkTileInfo.__extract_mlir_values__rWcCs@t|dksJt|j|dd}t|j|dg}t||S)N)lenr rQrRrM)rSrW new_tile_idxnew_is_valid_tiler#r#r$__new_from_mlir_values__s z%WorkTileInfo.__new_from_mlir_values__cC|jS)zCheck latest tile returned by the scheduler is valid or not. Any scheduling requests after all tasks completed will return an invalid tile. :return: The validity of the tile. :rtype: Boolean )rRrSr#r#r$rOs zWorkTileInfo.is_valid_tilecCr_)zgGet the index of the tile. :return: The index of the tile. :rtype: cute.Coord )rQr`r#r#r$rNszWorkTileInfo.tile_idxN)__name__ __module__ __qualname____doc__rECoordrrTlistr ValuerXr^propertyrOrNr#r#r#r$rMs rMc @sxeZdZdZe  dddddejdejded efd d Z d d Z ddZ edddde de eeeffddZdS)PersistentTileSchedulerParamsaA class to represent parameters for a persistent tile scheduler. This class is designed to manage and compute the layout of clusters and tiles in a batched gemm problem. :ivar cluster_shape_mn: Shape of the cluster in (m, n) dimensions (K dimension cta count must be 1). :type cluster_shape_mn: tuple :ivar problem_layout_ncluster_mnl: Layout of the problem in terms of number of clusters in (m, n, l) dimensions. :type problem_layout_ncluster_mnl: cute.Layout TNrproblem_shape_ntile_mnlcluster_shape_mnkraster_along_m swizzle_sizec Cs|ddkrtd|d|dkrtd|||_||_|dd|_||_||_||_tjtj |j|dd||d||d|_ |dkrt |j j |rUd|dfn|ddf}|rtj|d||d|f|df|d||df|d|df||d|_ n&tj||d|f|d|dfd||df||d|df||d|_ |dkrtj |j ||d}|j j d} |j j d} tj|||d|_tj| ||d|_tj| ||d|_dSd|_d|_d|_dS) aInitializes the PersistentTileSchedulerParams with the given parameters. :param problem_shape_ntile_mnl: The shape of the problem in terms of number of CTA (Cooperative Thread Array) in (m, n, l) dimensions. :type problem_shape_ntile_mnl: cute.Shape :param cluster_shape_mnk: The shape of the cluster in (m, n) dimensions. :type cluster_shape_mnk: cute.Shape :param swizzle_size: Swizzling size in the unit of cluster. 1 means no swizzle :type swizzle_size: int :param raster_along_m: Rasterization order of clusters. Only used when swizzle_size > 1. True means along M, false means along N. :type raster_along_m: bool :raises ValueError: If cluster_shape_k is not 1. rjzunsupported cluster_shape_k z"expect swizzle_size >= 1, but get Nrr)striderr) ValueErrorrk_cluster_shape_mnkcluster_shape_mnrn_raster_along_m_locrE make_layoutceil_divproblem_layout_ncluster_mnlround_upshapesizefast_divmod_create_divisor batch_fddcluster_shape_m_fddcluster_shape_n_fdd) rSrkrlrmrnrrproblem_shape_ncluster_mnlproblem_layout_sizecluster_count_mcluster_count_nr#r#r$rT sn     z&PersistentTileSchedulerParams.__init__c Csgg}|_|j|j|j|jfD]}t|}||7}|jt|qg}g}td|j fd|j fd|j fgD]\}\}}|durOt|} | | ||q7||7}|jt|||_ |S)Nr}r~r) _values_posrkrrrtrnrappendr[ enumerater}r~rrU_fastdivmod_indices) rSrWobj obj_valuesfastdivmod_valuesfastdivmod_indicesifdd_namefdd_obj fdd_valuesr#r#r$rXus4   z5PersistentTileSchedulerParams.__extract_mlir_values__c Csg}t|}t|j|j|j|jg|jddD]\}}|t||d|||d}qt t |d|j i}gd}t |drtt |jdkrtt|jD]%\}} || } t|| } | durs|t |krst| ||g} t|| | qN|S)NrZr)r}r~rrr)rfziprkrrrtrnrrr ritupleruhasattrr[rrgetattrsetattr) rSrWobj_list values_copyrn_items new_params fdd_namesjoriginal_indexr original_fddreconstructed_fddr#r#r$r^s.     z6PersistentTileSchedulerParams.__new_from_mlir_values__max_active_clustersrc Csxtddt|jj|jD|jjdf}tj|||d}tj|j||d}||}t||}||} g|j| RS)aComputes the grid shape based on the maximum active clusters allowed. :param max_active_clusters: The maximum number of active clusters that can run in one wave. :type max_active_clusters: Int32 :return: A tuple containing the grid shape in (m, n, persistent_clusters). - m: self.cluster_shape_m. - n: self.cluster_shape_n. - persistent_clusters: Number of persistent clusters that can run. css"|] \}}t||VqdSrP)rEr{).0xyr#r#r$ s z?PersistentTileSchedulerParams.get_grid_shape..ror)rrrxrzrsrEr{r) rSrrr num_ctas_mnlnum_ctas_in_problemnum_ctas_per_clusternum_ctas_per_wavenum_persistent_ctasnum_persistent_clustersr#r#r$get_grid_shapes  z,PersistentTileSchedulerParams.get_grid_shape)Trj)rarbrcrdr rEShapeboolintrTrXr^rrrrr#r#r#r$ris*  i$%*ric @sbeZdZdZdedededejdef ddZd e e j fd d Z d e e j d dfd dZ eeddddedeeeefdeeeeffddZeddddeded eeeeffddZdddded efddZdddded eeeeffddZedddd efddZedddd efddZed ddd!d"efd#d$Zed efd%d&ZdS)'StaticPersistentTileScheduleraA scheduler for static persistent tile execution in CUTLASS/CuTe kernels. :ivar params: Tile schedule related params, including cluster shape and problem_layout_ncluster_mnl :type params: PersistentTileSchedulerParams :ivar num_persistent_clusters: Number of persistent clusters that can be launched :type num_persistent_clusters: Int32 :ivar cta_id_in_cluster: ID of the CTA within its cluster :type cta_id_in_cluster: cute.Coord :ivar _num_tiles_executed: Counter for executed tiles :type _num_tiles_executed: Int32 :ivar _current_work_linear_idx: Current cluster index :type _current_work_linear_idx: Int32 paramsrcurrent_work_linear_idxcta_id_in_clusternum_tiles_executedcCs"||_||_||_||_||_dS)aInitializes the StaticPersistentTileScheduler with the given parameters. :param params: Tile schedule related params, including cluster shape and problem_layout_ncluster_mnl. :type params: PersistentTileSchedulerParams :param num_persistent_clusters: Number of persistent clusters that can be launched. :type num_persistent_clusters: Int32 :param current_work_linear_idx: Current cluster index. :type current_work_linear_idx: Int32 :param cta_id_in_cluster: ID of the CTA within its cluster. :type cta_id_in_cluster: cute.Coord :param num_tiles_executed: Counter for executed tiles. :type num_tiles_executed: Int32 N)rr_current_work_linear_idxr_num_tiles_executed)rSrrrrrr#r#r$rTs  z&StaticPersistentTileScheduler.__init__rcCsNt|j}|t|j|t|j|t|j|t|j|SrP)rrrUrrrrrVr#r#r$rXs z5StaticPersistentTileScheduler.__extract_mlir_values__rWcCst|dksJt|j|dg}t|j|dg}t|j|dd}t|j|dg}|dd}t|j|}t|||||SNrrjro)r[r rrrrrrrSrWnew_num_persistent_clustersnew_current_work_linear_idxnew_cta_id_in_clusternew_num_tiles_executed params_valuesrr#r#r$r^s  z6StaticPersistentTileScheduler.__new_from_mlir_values__Nr block_idxgrid_dimc Csvtj|||dtj|j||d}|\}}}t|} t||jdt||jdtdf} td} t||| | | S)a Initialize the static persistent tile scheduler. :param params: Parameters for the persistent tile scheduler. :type params: PersistentTileSchedulerParams :param block_idx: The 3d block index in the format (bidx, bidy, bidz). :type block_idx: Tuple[Integer, Integer, Integer] :param grid_dim: The 3d grid dimensions for kernel launch. :type grid_dim: Tuple[Integer, Integer, Integer] :return: A StaticPersistentTileScheduler object. :rtype: StaticPersistentTileScheduler rrrj)rEr{rsrr) rrrrrrbidxbidybidzrrrr#r#r$create%s" z$StaticPersistentTileScheduler.creatercCs|j|||dS)aCalculates the grid shape to be launched on GPU using problem shape, threadblock shape, and active cluster size. :param params: Parameters for grid shape calculation. :type params: PersistentTileSchedulerParams :param max_active_clusters: Maximum active clusters allowed. :type max_active_clusters: Int32 :return: The calculated 3d grid shape. :rtype: Tuple[Integer, Integer, Integer] r)r)rrrrr#r#r$rVsz,StaticPersistentTileScheduler.get_grid_shapecCs|tj|jj||dk}|jjdkr|j|||d}n |jjj|||d}tddt||j g|jj t dRD}t ||S)hCompute current tile coord given current_work_linear_idx and cta_id_in_cluster. :param current_work_linear_idx: The linear index of the current work. :type current_work_linear_idx: Int32 :return: An object containing information about the current tile coordinates and validity status. :rtype: WorkTileInfo rrjcss.|]\}}}t|t|t|VqdSrP)r)rrrzr#r#r$rs  zQStaticPersistentTileScheduler._get_current_work_for_linear_idx..) rEr{rrxrn%_get_cluster_work_idx_with_fastdivmodget_flat_coordrrrrsrrM)rSrrris_validcur_cluster_coordcur_tile_coordr#r#r$ _get_current_work_for_linear_idxms  z>StaticPersistentTileScheduler._get_current_work_for_linear_idxc Cs@t||jj\}}t||jj\}}t||jj\}} || |fS)aFastDivmod optimized CLUSTER coordinate calculation. CRITICAL: This should mimic problem_layout_ncluster_mnl.get_hier_coord() which returns CLUSTER coordinates, not tile coordinates! :param current_work_linear_idx: Linear index in the work space :type current_work_linear_idx: Int32 :return: Cluster coordinates (m, n, l) or None if FastDivmod not available :rtype: Tuple[Int32, Int32, Int32] or None )divmodrr}r~r) rSrrrwork_iteration work_unit_idcluster_n_batch cluster_mbatch_l cluster_nr#r#r$rs  zCStaticPersistentTileScheduler._get_cluster_work_idx_with_fastdivmodcCs|j|j||dSNr)rrrSrrr#r#r$get_current_worksz.StaticPersistentTileScheduler.get_current_workcCs|j||dSr)rrr#r#r$initial_work_tile_infosz4StaticPersistentTileScheduler.initial_work_tile_inforj) advance_countrrrcCs2|jt|t|j7_|jtd7_dS)Nrj)rrrr)rSrrrr#r#r$advance_to_next_worksz2StaticPersistentTileScheduler.advance_to_next_workcCr_rP)rr`r#r#r$rsz0StaticPersistentTileScheduler.num_tiles_executed)rarbrcrdrirrErerTrfr rgrXr^ staticmethodr rrrrrMrrrrrrrhrr#r#r#r$rs\    /  $!rc seZdZdZ ddedededejdedef fd d Z d e e j d dfd dZ ee dddddedeeeefdeeeefdefddZdddded efddZZS)$StaticPersistentRuntimeTileScheduleraA scheduler for static persistent runtime tile execution in CUTLASS/CuTe kernels. This scheduler will always launch all the SMs and the scheduler will generate the real tile info for each SM. :ivar params: Tile schedule related params, including cluster shape and problem_layout_ncluster_mnl :type params: PersistentTileSchedulerParams :ivar num_persistent_clusters: Number of persistent clusters that can be launched :type num_persistent_clusters: Int32 :ivar cta_id_in_cluster: ID of the CTA within its cluster :type cta_id_in_cluster: cute.Coord :ivar _num_tiles_executed: Counter for executed tiles :type _num_tiles_executed: Int32 :ivar _current_work_linear_idx: Current cluster index :type _current_work_linear_idx: Int32 rjrrrrr inner_modecs4t||||||dvrtd|||_dS)aFInitializes the StaticPersistentRuntimeTileScheduler with the given parameters. :param params: Tile schedule related params, including cluster shape and problem_layout_ncluster_mnl. :type params: PersistentTileSchedulerParams :param num_persistent_clusters: Number of persistent clusters that can be launched. :type num_persistent_clusters: Int32 :param current_work_linear_idx: Current cluster index. :type current_work_linear_idx: Int32 :param cta_id_in_cluster: ID of the CTA within its cluster. :type cta_id_in_cluster: cute.Coord :param num_tiles_executed: Counter for executed tiles. :type num_tiles_executed: Int32 :param inner_mode: The inner mode along which the linear index will be decomposed first. :type inner_mode: int )rrjz;inner_mode must be 0(for M mode) or 1(for N mode), but got N)superrTrqr)rSrrrrrr __class__r#r$rTs z-StaticPersistentRuntimeTileScheduler.__init__rWrcCst|dksJt|j|dg}t|j|dg}t|j|dd}t|j|dg}|dd}t|j|}t||||||jSr) r[r rrrrrrrrr#r#r$r^s  z=StaticPersistentRuntimeTileScheduler.__new_from_mlir_values__Nrrrc Csxtj|||dtj|j||d}|\}}} t| } t||jdt||jdtdf} td} t||| | | |S)aInitialize the static persistent tile scheduler. :param params: Parameters for the persistent tile scheduler. :type params: PersistentTileSchedulerParams :param block_idx: The 3d block index in the format (bidx, bidy, bidz). :type block_idx: Tuple[Integer, Integer, Integer] :param grid_dim: The 3d grid dimensions for kernel launch. :type grid_dim: Tuple[Integer, Integer, Integer] :param inner_mode: The inner mode along which the linear index will be decomposed first. :type inner_mode: int :return: A StaticPersistentRuntimeTileScheduler object. :rtype: StaticPersistentRuntimeTileScheduler rrrj)rEr{rsrr) rrrrrrrrrrrrrr#r#r$rs " z+StaticPersistentRuntimeTileScheduler.createc Cs|jjj}d}t|jdkrtj||df|ddfd}ntj|d|fd|dfd}||}|d|dtdf}d} t || S)rirj)rprT) rrxrzr rrErvget_hier_coordrrM) rSrrr ntile_shapeint_max ntile_layoutcluster_tile_coord_mnrrr#r#r$r9s    zEStaticPersistentRuntimeTileScheduler._get_current_work_for_linear_idx)rj)rarbrcrdrirrErerrTrfr rgr^rr rrrrMr __classcell__r#r#rr$rsB#  "3r)Nr&FN)F)'rdtypingrrcutlass.cutlass_dslrrrrrr r r cutlass._mlirr cutlass._mlir.dialectsr rrrcutlass.cute.typingr CuteInt32 cutlass.cuterEr8floatrr%rr/r?rBrKrLrMrirrr#r#r#r$sx(       " ) " (( -`_