o پiK/@s`dZddlmZddlZddlmZeZerddlmZm Z ddlm Z ddl m Z ddl mZdd lmZmZmZmZmZmZmZmZmZ   d+d ejd ejdejdejdejdejdejdejdejdejdejdejdejdejdejdejdejdejdeejdeejd ed!ed"ejf.d#d$Z  d,d ejd ejdejdejdejdejd%ejdejdejdejdejdejdejdejdejdejdejdejdeejdeejd"ejf*d&d'Z  d,d ejd ejdejdejdejd%ejd(ejdejdejdejdejdejdejdejdejdejdejdejdeejdeejd"ejf*d)d*ZdS)-zCutlass W4A8 MoE kernel.)OptionalN) is_cuda_alike)cutlass_w4a8_moe_mmget_cutlass_w4a8_moe_mm_data) silu_and_mul)per_tensor_quant_fp8)"get_moe_expert_parallel_world_size) cutlass_w4_run_moe_ep_preproess&deepep_ll_get_cutlass_w4a8_moe_mm_datadeepep_permute_triton_kernel!deepep_post_reorder_triton_kerneldeepep_run_moe_deep_preprocesspost_reorder_for_cutlass_moepre_reorder_for_cutlass_moe-silu_and_mul_masked_post_per_tensor_quant_fwd0silu_mul_static_tensorwise_quant_for_cutlass_moeF?aw1_qw2_qw1_scalew2_scale topk_weightstopk_ids a_strides1 b_strides1 c_strides1 a_strides2 b_strides2 c_strides2 s_strides13 s_strides2expert_offsetsproblem_sizes1problem_sizes2a1_scalea2_scaleapply_router_weight_on_inputrouted_scaling_factorreturnc$Cs|j|jks Jd|jtjksJ|jtjksJ|jdd|jdks*Jd|jdd|jdks:Jd|jd|jdksHJd|jd|jdksVJd|jd|jdksdJd |jd|jdksrJd |jd|jdksJd | jd|jdksJd | jd|jdksJd |d}|d}|dd}|dd}|d}|r|dksJd|j}tdkrt|dk||}t|}tj |||f|tj d}t ||||||||| tj | tj |d}tj | tj |d}t||||||||| tj |||df|tjd} tj |||f|tjd}!t| |||||dd|||| | d| tj |||ftj |d}"t| |"||dd|||t|!|"||||dd|| | | |d| t|}#t|!|#|||||||| |#S)a This function computes a w4a8-quantized Mixture of Experts (MoE) layer using two sets of quantized weights, w1_q and w2_q, and top-k gating mechanism. The matrix multiplications are implemented with CUTLASS grouped gemm. Parameters: - a (torch.Tensor): The input tensor to the MoE layer. Shape: [M, K] - w1_q (torch.Tensor): The first set of int4-quantized expert weights. Shape: [num_experts, N * 2, K // 2] (the weights are passed transposed and int4-packed) - w2_q (torch.Tensor): The second set of int4-quantized expert weights. Shape: [num_experts, K, N // 2] (the weights are passed transposed and int4-packed) - w1_scale (torch.Tensor): The fp32 scale to dequantize w1_q. Shape: [num_experts, K // 512, N * 8] - w2_scale (torch.Tensor): The fp32 scale to dequantize w2_q. Shape: [num_experts, N // 512, K * 4] - topk_weights (torch.Tensor): The weights of each token->expert mapping. - topk_ids (torch.Tensor): The ids of each token->expert mapping. - a_strides1 (torch.Tensor): The input strides of the first grouped gemm. - b_strides1 (torch.Tensor): The weights strides of the first grouped gemm. - c_strides1 (torch.Tensor): The output strides of the first grouped gemm. - a_strides2 (torch.Tensor): The input strides of the second grouped gemm. - b_strides2 (torch.Tensor): The weights strides of the second grouped gemm. - c_strides2 (torch.Tensor): The output strides of the second grouped gemm. - s_strides13 (torch.Tensor): The input and scale strides of the first grouped gemm. - s_strides2 (torch.Tensor): The scale strides of the second grouped gemm. - a1_scale (Optional[torch.Tensor]): The optional fp32 scale to quantize a. Shape: scalar or [1, K] - a2_scale (Optional[torch.Tensor]): The optional fp32 scale to quantize the intermediate result between the gemms. Shape: scalar or [1, N] - apply_router_weight_on_input (bool): When true, the topk weights are applied directly on the inputs. This is only applicable when topk is 1. Returns: - torch.Tensor: The fp8 output tensor after applying the MoE layer. topk shape mismatchHidden size mismatch w1Hidden size mismatch w2rExpert number mismatch w1 scales expert number mismatch w2 scales expert number mismatch"A Strides 1 expert number mismatch"B Strides 1 expert number mismatch"A Strides 2 expert number mismatch"B Strides 2 expert number mismatchz;apply_router_weight_on_input is only implemented for topk=1devicedtyper9r8N)shaper9torchint8sizer8rwherer empty float8_e4m3fnrnumelint32rbfloat16rfloatr empty_liker)$rrrrrrrrrrrrrr r!r"r#r$r%r&r'r(num_local_expertsmkntopkr8src2dst gateup_inputa_mapc_mapc1c2intermediate_qoutputrUZ/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/layers/moe/cutlass_w4a8_moe.pycutlass_w4a8_moe#s@            rW topk_ids_c)Cs|j|jks Jd|jtjksJ|jtjksJ|jdd|jdks*Jd|jdd|jdks:Jd|jd|jdksHJd|jd|jdksVJd|jd|jdksdJd |jd|jdksrJd |jd|jdksJd | jd|jdksJd | jd|jdksJd |d}|d}|dd}|dd}|d}|d}|d}|dd}|dd}|d}|j}t||\}}}|}tjt ||jdf||jd}t |jdf|||| tj d||jdddtj|jtj |d}t|||d~|} t| dk|| tj} tj| tj|d}!tj| tj|d}"t| ||||!|"||| tj|||df|tjd}#tj|||f|tjd}$t|#|||||dd|||| | d| tj|||f|tjd}%t|#|%tj|%jtj |d}&t|%|&|dt|$|&||||dd|| | | |d| |jd|}'tj|'|$jdf|$jtjd}(t|'f|$|(|||||$jddd|(S)al This function computes a w4a8-quantized Mixture of Experts (MoE) layer using two sets of quantized weights, w1_q and w2_q, and top-k gating mechanism. The matrix multiplications are implemented with CUTLASS grouped gemm. Parameters: - a (torch.Tensor): The input tensor to the MoE layer. Shape: [M, K] - w1_q (torch.Tensor): The first set of int4-quantized expert weights. Shape: [num_experts, N * 2, K // 2] (the weights are passed transposed and int4-packed) - w2_q (torch.Tensor): The second set of int4-quantized expert weights. Shape: [num_experts, K, N // 2] (the weights are passed transposed and int4-packed) - w1_scale (torch.Tensor): The fp32 scale to dequantize w1_q. Shape: [num_experts, K // 512, N * 8] - w2_scale (torch.Tensor): The fp32 scale to dequantize w2_q. Shape: [num_experts, N // 512, K * 4] - topk_weights (torch.Tensor): The weights of each token->expert mapping. - a_strides1 (torch.Tensor): The input strides of the first grouped gemm. - b_strides1 (torch.Tensor): The weights strides of the first grouped gemm. - c_strides1 (torch.Tensor): The output strides of the first grouped gemm. - a_strides2 (torch.Tensor): The input strides of the second grouped gemm. - b_strides2 (torch.Tensor): The weights strides of the second grouped gemm. - c_strides2 (torch.Tensor): The output strides of the second grouped gemm. - s_strides13 (torch.Tensor): The input and scale strides of the first grouped gemm. - s_strides2 (torch.Tensor): The scale strides of the second grouped gemm. - a1_scale (Optional[torch.Tensor]): The optional fp32 scale to quantize a. Shape: scalar or [1, K] - a2_scale (Optional[torch.Tensor]): The optional fp32 scale to quantize the intermediate result between the gemms. Shape: scalar or [1, N] - apply_router_weight_on_input (bool): When true, the topk weights are applied directly on the inputs. This is only applicable when topk is 1. Returns: - torch.Tensor: The fp8 output tensor after applying the MoE layer. r*r+r,r-r.rr/r0r1r2r3r4r5r7Ni) BLOCK_SIZEr:Tr6r;)r<r9r=r>r?r8r rCrAintr toint64rBrrFr@rD contiguousrrEzerosrrr ))rrrrrrrXrrrrrrr r!r"r#r$r%r& num_expertsrIrJrKrLr8reorder_topk_idsrM_num_total_tokensgateup_input_pre_reorderrNlocal_topk_idsrOrPrQrR intermediaterS num_tokensrTrUrUrVcutlass_w4a8_moe_deepep_normals=                  rgmasked_mcCsX|jtjksJ|jtjksJ|jdd|jdks Jd|jdd|jdks0Jd|jd|jdks>Jd|jd|jdksLJd|jd|jdksZJd|jd|jdkshJd |jd|jdksvJd | jd|jdksJd | jd|jdksJd |d}|d}|dd}|dd}|d}|j}t||||||\}}tj|jtj|d }t ||| dtj|||df|tj d}tj|||f|tj d}t |||| ||dd|||| | d| tj|||f|jtjd}t ||||t |||| ||dd|| | | |d| |S)a This function computes a w4a8-quantized Mixture of Experts (MoE) layer using two sets of quantized weights, w1_q and w2_q, and top-k gating mechanism. The matrix multiplications are implemented with CUTLASS grouped gemm. Parameters: - a (torch.Tensor): The input tensor to the MoE layer. Shape: [num_local_experts, num_max_dispatch_tokens_per_rank * num_ranks, K] - w1_q (torch.Tensor): The first set of int4-quantized expert weights. Shape: [num_experts, N * 2, K // 2] (the weights are passed transposed and int4-packed) - w2_q (torch.Tensor): The second set of int4-quantized expert weights. Shape: [num_experts, K, N // 2] (the weights are passed transposed and int4-packed) - w1_scale (torch.Tensor): The fp32 scale to dequantize w1_q. Shape: [num_experts, K // 512, N * 8] - w2_scale (torch.Tensor): The fp32 scale to dequantize w2_q. Shape: [num_experts, N // 512, K * 4] - topk_weights (torch.Tensor): The weights of each token->expert mapping. - a_strides1 (torch.Tensor): The input strides of the first grouped gemm. - b_strides1 (torch.Tensor): The weights strides of the first grouped gemm. - c_strides1 (torch.Tensor): The output strides of the first grouped gemm. - a_strides2 (torch.Tensor): The input strides of the second grouped gemm. - b_strides2 (torch.Tensor): The weights strides of the second grouped gemm. - c_strides2 (torch.Tensor): The output strides of the second grouped gemm. - s_strides13 (torch.Tensor): The input and scale strides of the first grouped gemm. - s_strides2 (torch.Tensor): The scale strides of the second grouped gemm. - a1_scale (Optional[torch.Tensor]): The optional fp32 scale to quantize a. Shape: scalar or [1, K] - a2_scale (Optional[torch.Tensor]): The optional fp32 scale to quantize the intermediate result between the gemms. Shape: scalar or [1, N] - apply_router_weight_on_input (bool): When true, the topk weights are applied directly on the inputs. This is only applicable when topk is 1. Returns: - torch.Tensor: The fp8 output tensor after applying the MoE layer. r,r-r+r.rr/r0r1r2r3r4r5r:Tr7Nr6r;)r9r=r>r<r?r8r rArBrrFrErr)rrrrrrXrhrrrrrrr r!r"r#r$r%r&r_rIrJrKrLr8rNrQrRrSrUrUrVcutlass_w4a8_moe_deepep_lls=        ri)NNFr)NN)__doc__typingrr=sglang.srt.utilsr_is_cuda_alike sgl_kernelrrr&sglang.jit_kernel.per_tensor_quant_fp8rsglang.srt.distributedr$sglang.srt.layers.moe.ep_moe.kernelsr r r r r rrrrTensorboolrFrWrgrirUrUrUrVs4     ,       P      R