o پiJ3@sdZddlmZmZddlZddlmZddlmZm Z m Z eZ e r;ddl m Z mZmZmZmZmZmZmZmZmZ    d3d ejd ejd ejd ejdejdejdejdejdejdejdejdejdejdejdejdejdejdejdejdejdeded eejd!eeefd"ejf2d#d$Zd%Zd&Z d4d ejd'ejd(ejd)ejd*ejd+ejd,ejd-ejd.ejdejdejd/ed0efd1d2ZdS)5z CUTLASS based Fused MoE kernels.)OptionalTupleN)CutlassMoEParams)is_cudais_sm90_supportedis_sm100_supported) apply_shuffle_mul_sumcutlass_fp4_group_mm"es_fp8_blockwise_scaled_grouped_mm%es_sm100_mxfp8_blockscaled_grouped_mm(es_sm100_mxfp8_blockscaled_grouped_quantfp8_blockwise_scaled_grouped_mmprepare_moe_inputscaled_fp4_experts_quant shuffle_rows silu_and_mulTFFFaw1_qw2_qw1_scalew2_scale topk_weightstopk_ids a1_strides c1_strides a2_strides c2_strides workspacea_ptrsb_ptrsout_ptrs a_scales_ptrs b_scales_ptrsexpert_offsetsproblem_sizes1problem_sizes2use_fp8_blockscale use_mxfp8output enable_esreturnc8Cs|sJd|j|jksJd|jtjksJ|jtjks J|jd|jdks.Jd|jd|jddks>Jd|jd|jdksLJd|jd|jdksZJd |jd|jdkshJd |jd|jdksvJd |jtjtjfvsJd trdd lm}|\}}|j}| d}| d}| d}| d}| d} |j }!tj | tj |!d}"tj | tj |!d}#|rW|r|sJdtsJd|ddksJd|ddksJd|jtjksJd|jtjksJd||jdd|jdf}$||jdd|jdf}%|j|$ks)Jd|$d|j|j|%ks:Jd|%d|jd}&|| }'t||'}(|'|&d|(})|)|&d|&|&}*d}+|rn|sb|rntj |dftj |!d}+t|||||"|#||||+ |r|rt||"|| |f},tj|,tjd}-tj |*|dftj|!d}.t|,||dd|+dd|-|.n||d\}/}0t|/|"|| |f}-t|0|"|| t|df}.tj || |df|!|d}1tj || |f|!|d}2tj |df|!tjd}3tj |df|!tjd}4tr|rt|1|-||.||||||dd| n3|r5|r5t|1|-||.|||dd|+ddnt|1| | ||||-||.|||||3|4||dd| tj || |f|!|d}5t|1|5|r|rtj|5tjd}6tj |*|dftj|!d}7t|5||dd|+dd|6|7n||5d\}6}7tr|rt|2|6||7|| | | ||dd| n3|r|rt|2|6||7|||dd|+ddnt|2| | ||||6||7|| | | |3|4||dd| |durtj ||f|!|d}t|2||#|||S)a2Performs Fused MoE computation using CUTLASS-like kernels with FP8 weights and activations. This function implements a Mixture of Experts (MoE) layer with a SwiGLU/SiLU activation, leveraging custom kernels likely derived from CUTLASS principles for grouped matrix multiplication (`fp8_blockwise_scaled_grouped_mm`) and data preparation (`prepare_moe_input`, `silu_and_mul`). It handles per-token routing, quantizes input activations to FP8 with per-token scales, performs the expert computations using FP8 GEMMs with pre-quantized FP8 weights (per-block scales), applies the SiLU activation, and combines the results weighted by the router scores. Args: a (torch.Tensor): Input activations. Shape: `(m, k)`, where `m` is the total number of tokens and `k` is the hidden size. Expected dtype: `torch.half` or `torch.bfloat16`. w1_q (torch.Tensor): Pre-quantized FP8 weight tensor for the first GEMM (up-projection part of SwiGLU). Expected shape: `(E, k, n*2)`, where `E` is the number of experts, `k` is the hidden size, and `n*2` is the intermediate size (`I`). Expected dtype: `torch.float8_e4m3fn`. Note: This shape implies weights are stored as (num_experts, hidden_size, intermediate_size). w2_q (torch.Tensor): Pre-quantized FP8 weight tensor for the second GEMM (down-projection). Expected shape: `(E, n, k)`, where `n` is half the intermediate size (`I // 2`). Expected dtype: `torch.float8_e4m3fn`. Note: This shape implies weights are stored as (num_experts, intermediate_size // 2, hidden_size). w1_scale (torch.Tensor): Scales corresponding to `w1_q` (per-block scales). Shape: `(E, num_blocks_n, num_blocks_k)`. Dtype: `torch.float32`. w2_scale (torch.Tensor): Scales corresponding to `w2_q` (per-block scales). Shape: `(E, num_blocks_k, num_blocks_n)`. Dtype: `torch.float32`. topk_weights (torch.Tensor): Router weights for the selected top-k experts for each token. Shape: `(m, topk)`. Dtype should ideally match `a`. topk_ids (torch.Tensor): Indices of the selected top-k experts for each token. Shape: `(m, topk)`. Dtype: `torch.int32`. a1_strides (torch.Tensor): Stride information for the first GEMM's 'a' input. Passed directly to the underlying kernel. Expected shape `(E,)`, dtype `torch.int64`. Note: Its exact usage within `fp8_blockwise_scaled_grouped_mm` needs clarification as it's passed as both a_stride and b_stride in the first call. c1_strides (torch.Tensor): Stride information for the first GEMM's 'c' output. Passed directly to the underlying kernel. Expected shape `(E,)`, dtype `torch.int64`. a2_strides (torch.Tensor): Stride information for the second GEMM's 'a' input. Passed directly to the underlying kernel. Expected shape `(E,)`, dtype `torch.int64`. Note: Its exact usage within `fp8_blockwise_scaled_grouped_mm` needs clarification as it's passed as both a_stride and b_stride in the second call. c2_strides (torch.Tensor): Stride information for the second GEMM's 'c' output. Passed directly to the underlying kernel. Expected shape `(E,)`, dtype `torch.int64`. workspace (torch.Tensor): Reusable workspace for the underlying kernel. a_ptrs (torch.Tensor): Pointers container for calculating offsets of the input activations for each expert. b_ptrs (torch.Tensor): Pointers container for calculating offsets of the input weights for each expert. out_ptrs (torch.Tensor): Pointers container for calculating offsets of the output activations for each expert. a_scales_ptrs (torch.Tensor): Pointers container for calculating offsets of the input scales for each expert. b_scales_ptrs (torch.Tensor): Pointers container for calculating offsets of the input scales for each expert. use_fp8_blockscale (bool, optional): Flag indicating usage of FP8 with block scaling. Currently, only `True` is supported. Defaults to `True`. use_mxfp8 (bool, optional): Flag indicating usage of MXFP8 (UE8M0 scales) with SM100 expert-specialization kernels. Defaults to `False`. output (torch.Tensor, optional): Output tensor. If not provided, a new tensor will be created. enable_es (tuple(bool, bool)): Flag indicating usage of expert specialization kernel for (up-projection, down-projection) Returns: torch.Tensor: The computed MoE layer output. Shape: `(m, k)`, dtype matches `a`. Raises: AssertionError: If input shapes, dtypes, or flags are inconsistent or unsupported. NotImplementedError: If CUDA is not available or `sgl_kernel` is not properly installed. z#Only support fp8 blockscale for nowtopk shape mismatchzHidden size mismatch w1zHidden size mismatch w2rzExpert number mismatchzWeights expert number mismatchz w1 scales expert number mismatchz w2 scales expert number mismatchzInvalid output dtype) sglang_per_token_group_quant_fp8dtypedevicez3MXFP8 requires expert-specialization for both GEMMszMXFP8 requires SM100 z0MXFP8 requires hidden size to be divisible by 32z6MXFP8 requires intermediate size to be divisible by 32zMXFP8 w1_scale must be uint8zMXFP8 w2_scale must be uint8zMXFP8 w1_scale must be z, got zMXFP8 w2_scale must be N)r1r2r1)shaper1torch float8_e4m3fnhalfbfloat16r)sglang.srt.layers.quantization.fp8_kernelr/sizer2emptynumelint32ruint8minrr empty_liker intrr r r rrto)8rrrrrrrrrrrrrr r!r"r#r$r%r&r'r(r)r*r/es_upes_down out_dtype num_expertsmkntopkr2a_mapc_mapexpected_w1_scale_shapeexpected_w2_scale_shapemxfp8_blockscale_align total_tokensnonzero_experts max_totalmax_blockscaleblockscale_offsetsrep_arep_a_q rep_a1_scalesa_qa1_scalec1c2 a_sf_layout w_sf_layout intermediate intemediate_qa2_scalereU/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/layers/moe/cutlass_moe.pycutlass_fused_experts_fp8s Z                                rgg@g|@ a1_gscalew1_fp4 w1_blockscale w1_alphas a2_gscalew2_fp4 w2_blockscale w2_alphasparamsapply_router_weight_on_inputc ! Cst| j| jks Jd|jtjksJd|jtjksJd|jdkr2|jdkr2|jdkr2|jdks6Jd|j\} }|j\}}}|j\}}}||krP|| jksTJd|d|kr_| j|kscJd|| jdkrq|| jdksuJd d||ksJd |jtjtj fvsJd |j}| jd }|j }tj | tj |d }tj | tj |d }t| | j| j| j||| j| j| j| j t||| j| j||d\}}t|||||||| }~~tj | ||jd df||d}t||t||| j| j|\}}t|||||||| } ~~t| || || jf} | | || j} | s1| | | |d |} | jd d|S)a MoE implementation for FP4 Inputs # Gemm 1 a: Input tensor: [m, k] (half/bfloat16) a1_gscale: Activation scale per expert: [e] (float32) w1(gate up) (not an argument to cutlass_moe_fp4): [e, 2 * n, k] w1_fp4: [e, 2 * n, k // 2], dtype: torch.uint8 (stacked fp4: E2M1) (Note: `n` is the up projection output dim, `k` is the input dim in full precision) w1_blockscale: [e, 2 * n, k // block_size] (float8_e4m3) (Block size = 16 for NVFP4) # Gemm 2 a2_gscale: Activation scale per expert: [e] w2(down projection) (not an argument to cutlass_moe_fp4): [e, k, n] w2_fp4: [e, k, n // 2], dtype: torch.uint8 (stacked E2M1) w2_blockscale: [e, k, n // block_size], dtype: float8_e4m3 Strides for activations, weights and output in logical number of elements. The activations & output stride is the number of elements to the next row. The weights stride is the number of elements to the next row per expert. For example, if the weight is [e, n, k], then the b_stride is a tensor of shape [e] with each element being k. Similarly for activations, if the shape is [m, k], then the a_stride has shape [e] with each value k. Similarly for output, if the output is [m, n], then the c_stride is a tensor of shape [e] with each element being k. Note: cutlass_fp4_group_mm is designed to accept the strides of activations and weights to be the same, so it is passed in as a single tensor. ab_strides_13: [e] dtype: int64 [Gemm 1: Activation / Weight strides] ab_strides_2: [e] dtype: int64 [Gemm 2: Activation / Weight strides] c_strides_13: [e] dtype: int64 [Gemm 1: Output Strides] c_strides_2: [e] dtype: int64 [Gemm 1: Output Strides] topk_weights: [m, topk] dtype: float8 topk_ids: [m, topk] dtype: float8 m, n, k: Unquantized weight shapes, dtype: int e: number of experts for the current rank, dtype: int assumes that topk < k < n to satisfy - up/down projection expectations. r,zweight 1 must be uint8zweight 2 must be uint8z1All Weights must be of rank 3 for cutlass_moe_fp4)zNumber of experts must matchz between weights.r.z)Hidden size mismatch between a, w1 and w2zmismatch in expected `n`zHidden size mismatch w2 and w1zInvalid input dtyper-r0) expert_mapr6)dim)r8r1r9rBndimrJ hidden_sizeintermediate_size_per_partitionr;r<r2r?r@rArr$r%r&rXrr to_gemm1_argsr to_gemm2_argsrviewrFsum)!rrhrirjrkrlrmrnrorrrprqm_ak_ae_w1nx2_w1 half_k_w1e_w2k_w2 half_n_w2rInum_topkr2rOrP rep_a_fp4rep_a_blockscaler^rbint_fp4int_blockscaler_rererfcutlass_moe_fp4Ys:             r)TFNr)F)__doc__typingrrr9(sglang.srt.layers.moe.cutlass_moe_paramsrsglang.srt.utilsrrr_is_cuda sgl_kernelrr r r r r rrrrTensorboolrgFLOAT4_E2M1_MAXFLOAT8_E4M3_MAXrrerererfs 0#       =