o (i @sddlmZddlmZddlZddlmZddlZddlm Z m Z m Z ddZ e d gd d Z e d gd d ZejddZe iede ddejdejdejdeddf ddZe ie de ddejdejdejdeddf ddZdS))flashinfer_api)gen_dsv3_router_gemm_moduleN)SimpleNamespace)register_custom_opsupported_compute_capabilitybackend_requirementc Cs|dkr td|dkrtd|dkrtd|ddkr)td|ddkr4td|ddkr?td |jd|jdkrMtd |jd|jdkr[td |jd|jdkritd d }d}d}|jd|ks}|jd|krtd|d||jd|krtd||jd|krtd||jtjkrtd|jtjkrtd|j|krtd|ddS)Nrzmat_a must be a 2D tensorzmat_b must be a 2D tensorzout must be a 2D tensorzmat_a must be row-majorzout must be row-majorrzmat_b must be column-majorz.mat_a.shape[1] must be equal to mat_b.shape[0]z,out.shape[0] must be equal to mat_a.shape[0]z,out.shape[1] must be equal to mat_b.shape[1]iz,mat_a.shape[0] (num_tokens) must be between z and z-mat_a.shape[1] (hidden_dim) must be equal to z.mat_b.shape[1] (num_experts) must be equal to zmat_a must be a bfloat16 tensorzmat_b must be a bfloat16 tensorzout must be a z tensorT)dim ValueErrorstrideshapedtypetorchbfloat16) mat_amat_boutlaunch_with_pdlexpected_num_expertsexpected_out_dtypeexpected_hidden_dim min_tokens max_tokensr\/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/flashinfer/gemm/routergemm_dsv3.py_mm_M1_16_K7168_shape_checks sP      rdcCt||||dtjdS)Nrr)rrfloat32rrrrrrr!_mm_M1_16_K7168_N256_shape_checksDr$cCr)Nr!)rrrr#rrr!_mm_M1_16_K7168_N128_shape_checksQr%r'c sttddgd ddtjdtjdtjdtddf fd d }td dgd ddtjdtjdtjdtddf fd d }t||dS)Nzflashinfer::ml3_router_gemm_opr) mutates_argsFrrrreturnc||||dSN)ml3_router_gemm_opr#modulerrmm_M1_16_K7168_N128a z8get_dsv3_router_gemm_module..mm_M1_16_K7168_N128zflashinfer::dsv3_router_gemm_opcr*r+)dsv3_router_gemm_opr#r-rrmm_M1_16_K7168_N256mr0z8get_dsv3_router_gemm_module..mm_M1_16_K7168_N256r/r2F)rbuild_and_loadrrTensorboolrr3rr-rget_dsv3_router_gemm_module]sJ r8) common_checkFrrrrr)cCt||||dS)a0Optimized GEMM for the router operation in Mistral Large 3. This function performs a highly optimized matrix multiplication specifically tailored for the expert routing GEMM in Mistral Large 3's Mixture of Experts (MoE) architecture. It computes out = mat_a @ mat_b where mat_a contains token embeddings and mat_b contains expert routing weights. The implementation is optimized for the specific problem dimensions used in Mistral Large 3: - Hidden dimension (K): 7168 - Number of experts (N): 128 - Number of tokens (M): 1-16 Args: mat_a (torch.Tensor): Input token embeddings of shape (M, K) where M is the number of tokens (1-16) and K is the hidden dimension (7168). Must be bfloat16, row-major (contiguous). mat_b (torch.Tensor): Expert routing weights of shape (K, N) where K is the hidden dimension (7168) and N is the number of experts (128). Must be bfloat16, column-major (transposed layout). out (torch.Tensor): Pre-allocated output tensor of shape (M, N) containing the routing scores. Must be bfloat16, row-major (contiguous). This tensor is mutated in-place. launch_with_pdl (bool, optional): Whether to launch the kernel using Persistent Device-side Launch. Defaults to False. Returns: None: The result is written directly to the `out` tensor. Raises: ValueError: If tensor dimensions, strides, or data types do not match the expected Mistral Large 3 router configuration. Note: This kernel is specialized for compute capability 10.0 (Blackwell architecture). The specific problem size optimization makes this significantly faster than general-purpose GEMM implementations for the router operation. N)r8r/r#rrrr/-r/cCr:)aOptimized GEMM for the router operation in DeepSeek-V3. This function performs a highly optimized matrix multiplication specifically tailored for the expert routing GEMM in DeepSeek-V3's Mixture of Experts (MoE) architecture. It computes out = mat_a @ mat_b where mat_a contains token embeddings and mat_b contains expert routing weights. The implementation is optimized for the specific problem dimensions used in DeepSeek-V3: - Hidden dimension (K): 7168 - Number of experts (N): 256 - Number of tokens (M): 1-16 Args: mat_a (torch.Tensor): Input token embeddings of shape (M, K) where M is the number of tokens (1-16) and K is the hidden dimension (7168). Must be bfloat16, row-major (contiguous). mat_b (torch.Tensor): Expert routing weights of shape (K, N) where K is the hidden dimension (7168) and N is the number of experts (256). Must be bfloat16, column-major (transposed layout). out (torch.Tensor): Pre-allocated output tensor of shape (M, N) containing the routing scores. Must be float32, row-major (contiguous). This tensor is mutated in-place. launch_with_pdl (bool, optional): Whether to launch the kernel using Persistent Device-side Launch. Defaults to False. Returns: None: The result is written directly to the `out` tensor. Raises: ValueError: If tensor dimensions, strides, or data types do not match the expected DeepSeek-V3 router configuration. Note: This kernel is specialized for compute capability 10.0 (Blackwell architecture). The specific problem size optimization makes this significantly faster than general-purpose GEMM implementations for the router operation. N)r8r2r#rrrr2r;r2r4) api_loggingrflashinfer.jitr functoolstypesrrflashinfer.utilsrrrrr$r'cacher8r6r7r/r2rrrrsR  7    ! 0