o پi6@sddlmZddlmZmZddlZddlmZ   dRdejdejde d eejd ee d ejf d d Z  dSdejdejdejde d ee d df ddZ   dRdejdejde d eejd ee d ejf ddZ  dSdejdejdejde d ee d df ddZdejdejd dfddZdTdejd ejd ejfddZdTdejd ejd ejfddZdTdejd ejd ejfddZejjdurdTdejd ejd ejfddZeGd d!d!Zd"d#Z $  dUd%ejd&ejd'ejd(ed)ejd*e d+eed ee d dfd,d-Z $dVd%ejd&ejd'ejd(ed)ejd*e f d.d/Z 0 dWd1ejd2ejd3ejd4ejd5ejd6ejd7ejd8ed9ed dfd:d;Zdejdejfdejd?ejfd@dAZdBejdCejfdDdEZdFdGd0dHejfdIejdJedKe dLe dMe dNedOej fdPdQZ!dS)X) dataclass)ListOptionalN)is_arch_support_pdlư>inputweightepsout enable_pdlreturncC<|dur t|}|durt}tjjj||||||S)aGRoot mean square normalization. ``out[i] = (input[i] / RMS(input)) * weight[i]`` Parameters ---------- input: torch.Tensor Input tensor, shape (batch_size, hidden_size). weight: torch.Tensor Weight tensor, shape (hidden_size,). eps: float Epsilon for numerical stability. out: Optional[torch.Tensor] The output tensor, if specified, the kernel will update this tensor inplace. enable_pdl: Optional[bool] Whether to enable `programmatic dependent launch `_ If None, will be automatically enabled on Hopper architecture. Returns ------- output: torch.Tensor Normalized tensor, shape (batch_size, hidden_size). N)torch empty_likerops sgl_kernelrmsnormdefaultrrr r r rJ/home/ubuntu/.local/lib/python3.10/site-packages/sgl_kernel/elementwise.pyr  rresidualcC*|durt}tjjj|||||dS)aFused add root mean square normalization. Step 1: ``residual[i] += input[i]`` Step 2: ``input[i] = (residual[i] / RMS(residual)) * weight[i]`` Parameters ---------- input: torch.Tensor Input tensor, shape (batch_size, hidden_size). residual: torch.Tensor Residual tensor, shape (batch_size, hidden_size). weight: torch.Tensor Weight tensor, shape (hidden_size,). eps: float Epsilon for numerical stability. enable_pdl: Optional[bool] Whether to enable `programmatic dependent launch `_ If None, will be automatically enabled on Hopper architecture. N)rrrrfused_add_rmsnormrrrrr r rrrr1   rcCr )a_Gemma-style root mean square normalization. ``out[i] = (input[i] / RMS(input)) * (weight[i] + 1)`` Parameters ---------- input: torch.Tensor Input tensor, shape (batch_size, hidden_size). weight: torch.Tensor Weight tensor, shape (hidden_size,). eps: float Epsilon for numerical stability. out: Optional[torch.Tensor] The output tensor, if specified, the kernel will update this tensor inplace. enable_pdl: Optional[bool] Whether to enable `programmatic dependent launch `_ If None, will be automatically enabled on Hopper architecture. Returns ------- output: torch.Tensor Gemma Normalized tensor, shape (batch_size, hidden_size). N)rrrrr gemma_rmsnormrrrrrrVrrcCr)aGemma-style fused add root mean square normalization. Step 1: ``residual[i] += input[i]`` Step 2: ``input[i] = (residual[i] / RMS(residual)) * (weight + 1)`` Parameters ---------- input: torch.Tensor Input tensor, shape (batch_size, hidden_size). residual: torch.Tensor Residual tensor, shape (batch_size, hidden_size). weight: torch.Tensor Weight tensor, shape (hidden_size,). eps: float Epsilon for numerical stability. enable_pdl: Optional[bool] Whether to enable `programmatic dependent launch `_ If None, will be automatically enabled on Hopper architecture. N)rrrrgemma_fused_add_rmsnormrrrrrr}rroutputcCs|j|jksJ|jd|j|jdd|jddks2J|jddd|jdd|jdd|jdksOJ|jddd|jddS)N != )ndimshaperrrrr _check_shapes" r&cCz|jd|jjddkrtd|durt||ntj|jdd|jddf|j|jd}tjj j |||SNr!rz*The pointers must be multiple of 16 bytes.r"devicedtype) r$r,itemsize ValueErrorr&remptyr+rr silu_and_mulrrr rrrr0 r0cCr'r() r$r,r-r.r&rr/r+rrgelu_tanh_and_mulrr1rrrr3r2r3cCr'r() r$r,r-r.r&rr/r+rr gelu_and_mulrr1rrrr4r2r4cCs|jd|jjddkrtd|jdd|jjd|dur3|j|jks2J|jd|jnt|}tjj|||S) z Quick-GELU: y = x * sigmoid(1.702 * x) The CUDA/HIP kernel uses 128-bit (16-byte) vector loads & stores, so the last-dimension byte length must be a multiple of 16 bytes. r!r)rzThe last dimension (z) x itemsize (z!) must be a multiple of 16 bytes.Nr ) r$r,r-r.rrrr gelu_quickr1rrrr5s$ r5c@sReZdZUdZejed<ejed<ejed<eeed<eeed<ejed<dS) FusedSetKVBufferArga value : Optional[torch.Tensor] Value tensor, shape: ``(nnz, num_v_heads * head_size)``. k_buffer : Optional[torch.Tensor] Buffer for keys, shape: ``(nnz, num_k_heads * head_size)``. v_buffer : Optional[torch.Tensor] Buffer for values, shape: ``(nnz, num_v_heads * head_size)``. k_scale : Optional[float] Scale factor for keys. v_scale : Optional[float] Scale factor for values. cache_loc : Optional[torch.Tensor] Cache location tensor, used for indexing kv cache. valuek_bufferv_bufferk_scalev_scale cache_locN) __name__ __module__ __qualname____doc__rTensor__annotations__rfloatrrrrr6s      r6cCs||jdd|S)Nrr!)viewr$)x head_sizerrr_view_3d rGT positionsquerykeyrF cos_sin_cacheis_neoxfused_set_kv_buffer_argc Cs|jtjkr td|durto|du}|}dur=|jdus$Jd|jdus-Jd|jjtjks=Jd|jjtj j j t ||t ||t ||t ||||| ||durct |j|nd|durnt |j|nd|duryt |j|nd|dur|j dSd dS)a` Apply rotary embedding to keys and queries with precomputed cos/sin values. This is designed to be compatible with the SGL/vLLM implementation. The result is inplace applied to the input tensors. Parameters ---------- positions : torch.Tensor Position indices, shape: ``(nnz)``. query : torch.Tensor Query tensor, shape: ``(nnz, num_q_heads * head_size)``. key : torch.Tensor Key tensor, shape: ``(nnz, num_k_heads * head_size)``. cos_sin_cache : torch.Tensor Cosine and Sine cache tensor, shape: ``(max_seq_len, rotary_dim)``. Cosine is the first half and Sine is the second half on rotary_dim. is_neox : bool Whether to use Neox style RoPE, default: ``True``. * If ``True``, the last dimension of the query/key tensor is not interleaved, i.e., we rotate the first half dimensions ``([..., :head_dim//2])`` and the second half dimensions ``([..., head_dim//2:])``. * If ``False``, the last dimension of the query/key tensor is interleaved, i.e., we rotate the even dimensions ``([..., ::2])`` and odd dimensions ``([..., 1::2])``. fused_set_kv_buffer_arg : FusedSetKVBufferArg Fuse the set-kv-buffer operation into this kernel Note ---- The rotary dimension is determined by the cosine cache and sine cache. zcos_sin_cache should be float32Nzk_scale is not yet supportedzv_scale is not yet supportedza.cache_loc.dtype=)r,rfloat32r.rr:r;r<int64rr apply_rope_pos_ids_cos_sin_cacherrGlongr7r8r9) rIrJrKrFrLrMrNr arrr%apply_rope_with_cos_sin_cache_inplaces> *      rTcCstjjj||||||dSN)rrrrotary_embeddingr)rIrJrKrFrLrMrrrrVds  rVkvk_outv_outr:r;locmultoffsetc Cs"tjj||||||||| dSrU)rrr downcast_fp8) rXrYrZr[r:r;r\r]r^rrrr_qs r_cCstjj||dSrU)rrrcopy_to_gpu_no_cer%rrrr`rHr`k_nopek_ropecCstjj|||dSrU)rrr concat_mla_k)rXrarbrrrrcsrcrSbcCsP|j^}}tjg||jd|jdR|j|jd}tjj||||S)Nr!r*)r$rr/r+r,rrconcat_mla_absorb_q)rSrd batch_dims_r rrrres $reFgi'tdimflip_sin_to_cosdownscale_freq_shiftscale max_periodr,c Cs@tj}|jd}tj||f||jd}tjj|||||||S)a Create sinusoidal timestep embeddings. # TODO: review, output dtype always be float32. According to python code: # sglang/python/sglang/multimodal_gen/runtime/layers/visual_embedding.py Args: t: Tensor of shape [B] with timesteps dim: Embedding dimension max_period: Controls the minimum frequency of the embeddings Returns: Tensor of shape [B, dim] with embeddings r)r,r+)rrOr$r/r+rrtimestep_embedding) rhrirjrkrlrmr, batch_sizerrrrrns  rn)rNN)rNrU)TNN)T)rWr)" dataclassesrtypingrrrsgl_kernel.utilsrrArCboolrrrrr&r0r3r4versionhipr5r6rGintrTrVr_r`rcrerOr,rnrrrrsN   + ( + %    \