o پi@sddlmZddlZddlmZddlZddlmZmZm Z m Z er(ddl m Z edd d Z edddZddddddZdS)) annotationsN) TYPE_CHECKING) cache_onceis_arch_support_pdlload_jit make_cpp_args)Module row_bytesintreturnrcCs4t|t}tdg|Rdgdd|dfgdS)Nkvcachezelementwise/kvcache.cuh store_cachezStoreKVCacheKernel::run) cuda_files cuda_wrappers)rrr)r argsrM/home/ubuntu/.local/lib/python3.10/site-packages/sglang/jit_kernel/kvcache.py_jit_kvcache_modules rsizeboolc Csxtt}|ddkr|d|ddSzt|WdSty;}z|d|d|WYd}~dSd}~ww) NrzUnsupported row_bytes=z/ for JIT KV-Cache kernel: must be multiple of 4FTz2Failed to load JIT KV-Cache kernel with row_bytes=z: )logging getLogger__name__warningr Exception)rloggererrrcan_use_store_caches   r)r num_splitk torch.Tensorvk_cachev_cacheindicesrNonecCsf|p |jd|}t|}|dkr'|ddkrd}n |ddkr%d}nd}|||||||dS) aStore key and value tensors into KV cache at specified indices. Args: k (torch.Tensor): Key tensor of shape (batch_size, H * D). v (torch.Tensor): Value tensor of shape (batch_size, H * D). k_cache (torch.Tensor): Key cache tensor of shape (num_pages, H * D). v_cache (torch.Tensor): Value cache tensor of shape (num_pages, H * D). indices (torch.Tensor): Indices tensor of shape (batch_size,). ririN)shape element_sizerr )r r"r#r$r%r rmodulerrrr 1s   r )r r r r)rr r r)r r!r"r!r#r!r$r!r%r!r r rr r r&) __future__rrtypingrtorchsglang.jit_kernel.utilsrrrrtvm_ffi.modulerrrr rrrrs