o پiN@sddlmZddlZddlmZddlmZmZddlZddl Z ddl m Z m Z m Z ddlmZer8ddlmZe d3d d Zed gd d  d4d5dd Zed gd!d  d4d6d"d ZeGd#d$d$Zd%d&Z '  d7d8d1d2ZdS)9) annotationsN) dataclass) TYPE_CHECKINGOptional) cache_onceis_arch_support_pdlload_jit)register_custom_op)Modulereturnr cCsfttjj}|ddsJdt|dd|dd}tddgdgt|gdS)Ndataincludezflashinfer headers are missing apply_rope_pos_ids_cos_sin_cachezelementwise/rope.cuh)rz%ApplyRopePosIdsCosSinCacheKernel::run) cuda_files cuda_wrappersextra_include_paths) pathlibPath flashinfer__file__parentresolveexistsstrr)flashinfer_dirflashinfer_include_pathrJ/home/ubuntu/.local/lib/python3.10/site-packages/sglang/jit_kernel/rope.py,_jit_apply_rope_pos_ids_cos_sin_cache_modules r.apply_rope_pos_ids_cos_sin_cache_with_kv_cache)qkq_ropek_ropek_bufferv_buffer)op_name mutates_argsFr torch.Tensorr!r"r# cos_sin_cachepos_idsvr$r% kv_cache_loc interleavebool enable_pdlNonec Cs*t} | ||||||| | |||| dS)a Apply RoPE (Rotary Positional Embedding) with position IDs and cos/sin cache. Args: q: Input Q tensor of shape [nnz, num_qo_heads, head_dim] k: Input K tensor of shape [nnz, num_kv_heads, head_dim] q_rope: Output Q tensor with RoPE applied, same shape as q k_rope: Output K tensor with RoPE applied, same shape as k cos_sin_cache: Cos/sin cache of shape [max_seq_len, rotary_dim] pos_ids: Position IDs of shape [nnz] interleave: Whether to use interleaved RoPE enable_pdl: Enable PDL (Programmable Data Layout) v: Optional V tensor for KV caching k_buffer: Optional K buffer for KV caching v_buffer: Optional V buffer for KV caching kv_cache_loc: Optional KV cache location tensor Nrr) r r!r"r#r)r*r+r$r%r,r-r/modulerrrr)s#1apply_rope_pos_ids_cos_sin_cache_without_kv_cache)r r!r"r#c Cs*t}|||||||||dddd dS)Nr1) r r!r"r#r)r*r-r/r2rrrr3^sc@sBeZdZUdZded<ded<ded<ded<ded<ded <d S) FusedSetKVBufferArga value : Optional[torch.Tensor] Value tensor, shape: ``(nnz, num_v_heads * head_size)``. k_buffer : Optional[torch.Tensor] Buffer for keys, shape: ``(nnz, num_k_heads * head_size)``. v_buffer : Optional[torch.Tensor] Buffer for values, shape: ``(nnz, num_v_heads * head_size)``. k_scale : Optional[float] Scale factor for keys. v_scale : Optional[float] Scale factor for values. cache_loc : Optional[torch.Tensor] Cache location tensor, used for indexing kv cache. r(valuer$r%zOptional[float]k_scalev_scale cache_locN)__name__ __module__ __qualname____doc____annotations__rrrrr4s  r4cCs||jdd|S)Nr)viewshape)x head_sizerrr_view_3dsrCT positionsquerykeyrBintis_neoxfused_set_kv_buffer_argOptional[FusedSetKVBufferArg]Optional[bool]c Cs|jtjkr td|durto|du}|}dur=|jdus$Jd|jdus-Jd|jjtjks=Jd|jj|du} | rpt t ||t ||t ||t |||| t |j |t |j |t |j||j| | dStt ||t ||t ||t |||| | |dS)a` Apply rotary embedding to keys and queries with precomputed cos/sin values. This is designed to be compatible with the SGL/vLLM implementation. The result is inplace applied to the input tensors. Parameters ---------- positions : torch.Tensor Position indices, shape: ``(nnz)``. query : torch.Tensor Query tensor, shape: ``(nnz, num_q_heads * head_size)``. key : torch.Tensor Key tensor, shape: ``(nnz, num_k_heads * head_size)``. cos_sin_cache : torch.Tensor Cosine and Sine cache tensor, shape: ``(max_seq_len, rotary_dim)``. Cosine is the first half and Sine is the second half on rotary_dim. is_neox : bool Whether to use Neox style RoPE, default: ``True``. * If ``True``, the last dimension of the query/key tensor is not interleaved, i.e., we rotate the first half dimensions ``([..., :head_dim//2])`` and the second half dimensions ``([..., head_dim//2:])``. * If ``False``, the last dimension of the query/key tensor is interleaved, i.e., we rotate the even dimensions ``([..., ::2])`` and odd dimensions ``([..., 1::2])``. fused_set_kv_buffer_arg : FusedSetKVBufferArg Fuse the set-kv-buffer operation into this kernel Note ---- The rotary dimension is determined by the cosine cache and sine cache. zcos_sin_cache should be float32Nzk_scale is not yet supportedzv_scale is not yet supportedza.cache_loc.dtype=)dtypetorchfloat32 ValueErrorrr6r7r8int64rrClongr5r$r%r3) rDrErFrBr)rHrIr/a save_kv_cacherrr%apply_rope_with_cos_sin_cache_inplacesD *     rT)r r )FF)r r(r!r(r"r(r#r(r)r(r*r(r+r(r$r(r%r(r,r(r-r.r/r.r r0)r r(r!r(r"r(r#r(r)r(r*r(r-r.r/r.r r0)TNN)rDr(rEr(rFr(rBrGr)r(rHr.rIrJr/rKr r0) __future__rr dataclassesrtypingrrrrMsglang.jit_kernel.utilsrrrsglang.srt.utils.custom_opr tvm_ffi.moduler rrr3r4rCrTrrrrsB     1