o پik@sddlmZddlZddlmZmZddlZddlmZm Z m Z m Z er*ddl m Z ed/d d Zed0ddZed1ddZed1ddZed2ddZ d3ddd4d"d#Z  d5d6d(d)Z d3d7d+d,Z d3d8d-d.ZdS)9) annotationsN) TYPE_CHECKINGOptional) cache_onceis_arch_support_pdlload_jit make_cpp_args)Modulehead_dimintdtype torch.dtypereturnr cC6t|t|}tdg|Rdgdd|dfgdS)Nqknormzelementwise/qknorm.cuhz QKNormKernel<>::run cuda_files cuda_wrappersrrr)r r argsrJ/home/ubuntu/.local/lib/python3.10/site-packages/sglang/jit_kernel/norm.py_jit_qknorm_moduler hidden_sizecCr)Nrmsnormzelementwise/rmsnorm.cuhzRMSNormKernel)r q torch.Tensorkq_weightk_weightepsfloatNonecCs0|p|d}t||j}||||||dSN)sizerr r)r2r4r5r6r7r modulerrrfused_inplace_qknormOs r>inputweightoutputOptional[torch.Tensor]cCs:|dur|n|}|d}t||j}|||||dSr:)r<rr r)r?r@rAr7rr=rrrr]s  rresidualcCst|j}|||||dS)N)r"r r)r?rCr@r7r=rrrris rcCs t|j}||||||dS)a Fused inplace QK normalization across all heads. Args: q: Query tensor of shape [batch_size, num_heads * head_dim] k: Key tensor of shape [batch_size, num_heads * head_dim] q_weight: Query weight tensor of shape [num_heads * head_dim] k_weight: Key weight tensor of shape [num_heads * head_dim] eps: Epsilon for numerical stability N)r$r r#)r2r4r5r6r7r=rrr!fused_inplace_qknorm_across_headsss rD)r r r r rr )rr r r rr )r r rr )r r r r rr%)r1)r2r3r4r3r5r3r6r3r7r8r r rr9)Nr1) r?r3r@r3rArBr7r8rr9) r?r3rCr3r@r3r7r8rr9) r2r3r4r3r5r3r6r3r7r8rr9) __future__rr)typingrrtorchsglang.jit_kernel.utilsrrrrtvm_ffi.moduler rrr"r$r0r>rrrDrrrrs8