o پi @sddlmZddlmZddlZddlmZmZddlm Z er&ddl m Z edd d Z e d d d gd dddd Z e dd gd dd ddZ dd!ddZdS)") annotations) TYPE_CHECKINGN) cache_onceload_jit)register_custom_op)ModulereturnrcCstddgdgdS)Nrotary_embeddingzelementwise/pos_enc.cuh)r zRotaryEmbeddingKernel::run) cuda_files cuda_wrappers)rr r M/home/ubuntu/.local/lib/python3.10/site-packages/sglang/jit_kernel/pos_enc.py_jit_rotary_embedding_modules rrotary_embedding_with_keyquerykey)op_name mutates_argsT positions torch.Tensor head_sizeint cos_sin_cacheis_neoxboolNonecCst}|||||||dS)a$ Apply rotary embedding to query and key tensors. Args: positions: Position indices of shape [num_tokens] or [batch_size, seq_len] query: Query tensor of shape [num_tokens, num_heads, head_size] or [num_tokens, num_heads * head_size] key: Key tensor of shape [num_tokens, num_kv_heads, head_size] or [num_tokens, num_kv_heads * head_size] cos_sin_cache: Cosine and sine cache of shape [max_position, rot_dim] is_neox: Whether to use GPT-NeoX style rotary embedding (True) or GPT-J style (False) Nrr )rrrrrrmoduler r r rsrotary_embedding_without_keycCst}|||d|||dSNr)rrrrrrr r r r7s cCs:|durt|||||||fSt||||||||fSr)rr)rrrrrrr r r r Fs  r )rr)T)rrrrrrrrrrrrrr) rrrrrrrrrrrr) rrrrrrrrrrrr) __future__rtypingrtorchsglang.jit_kernel.utilsrrsglang.srt.utils.custom_oprtvm_ffi.modulerrrrr r r r r s.