o ½Ù¾i©ã@s`ddlmZmZddlZzddlmZWney(ZzeZWYdZ[ndZ[wwdZe dƒZ   d"dej de de d ee d e d ee d eej ej ffd d„Z      d#dej dej dej dej de dej dej deede dej dBdej dBd e deej d eej ej ffdd„Z d$dej dej dej dede d eej ej ej ff d d!„ZdS)%é)ÚOptionalÚTupleN)Ú flashmla_opszLFailed to load sgl_kernel.flashmla_ops extension. Ensure CUDA Driver >= 12.4FÚ cache_seqlensÚnum_q_tokens_per_head_kÚ num_heads_kÚ num_heads_qÚis_fp8_kvcacheÚtopkÚreturncCs:|r|durtjjj |||¡Stjjj ||||||¡S)aÕ Arguments: cache_seqlens: (batch_size), dtype torch.int32. num_q_tokens_per_head_k: Equals to num_q_tokens_per_q_seq * num_heads_q // num_heads_k. num_heads_k: The number of k heads. num_heads_q: The number of q heads. This argument is optional when sparse attention is not enabled is_fp8_kvcache: Whether the k_cache and v_cache are in fp8 format. topk: If not None, sparse attention will be enabled, and only tokens in the `indices` array passed to `flash_mla_with_kvcache_sm90` will be attended to. Returns: tile_scheduler_metadata: (num_sm_parts, TileSchedulerMetaDataSize), dtype torch.int32. num_splits: (batch_size + 1), dtype torch.int32. N)ÚtorchÚopsÚ sgl_kernelÚ#get_mla_decoding_metadata_dense_fp8ÚdefaultÚget_mla_decoding_metadata)rrrrr r ©rúH/home/ubuntu/.local/lib/python3.10/site-packages/sgl_kernel/flash_mla.pyÚget_mla_metadatas  ý úrÚqÚk_cacheÚ block_tableÚ head_dim_vÚtile_scheduler_metadataÚ num_splitsÚ softmax_scaleÚcausalÚ descale_qÚ descale_kÚindicesc  Csº|dur |jdd}| dur|dksJdƒ‚| du| duks#Jdƒ‚| durE| ¡dkrEtjjj |||||||||| | ¡ \} }| |fStjjj |||||||||| | ¡ \} }| |fS)aà Arguments: q: (batch_size, seq_len_q, num_heads_q, head_dim). k_cache: (num_blocks, page_block_size, num_heads_k, head_dim). block_table: (batch_size, max_num_blocks_per_seq), torch.int32. cache_seqlens: (batch_size), torch.int32. head_dim_v: Head dimension of v. tile_scheduler_metadata: (num_sm_parts, TileSchedulerMetaDataSize), torch.int32, returned by get_mla_metadata. num_splits: (batch_size + 1), torch.int32, returned by get_mla_metadata. softmax_scale: float. The scale of QK^T before applying softmax. Default to 1 / sqrt(head_dim). causal: bool. Whether to apply causal attention mask. descale_q: (batch_size), torch.float32. Descaling factors for Q, used for fp8 quantization. descale_k: (batch_size), torch.float32. Descaling factors for K, used for fp8 quantization. is_fp8_kvcache: bool. Whether the k_cache and v_cache are in fp8 format. For the format of FP8 KV cache, please refer to README.md indices: (batch_size, seq_len_q, topk), torch.int32. If not None, sparse attention will be enabled, and only tokens in the `indices` array will be attended to. Invalid indices should be set to -1 or numbers >= total_seq_len_kv. For details about how to set up `indices`, please refer to README.md. Returns: out: (batch_size, seq_len_q, num_heads_q, head_dim_v). softmax_lse: (batch_size, num_heads_q, seq_len_q), torch.float32. Néÿÿÿÿgà¿Fz6causal must be `false` if sparse attention is enabled.z= s_kv sm_scale: float d_v: The dimension of value vectors. Can only be 512 Returns: (output, max_logits, lse) About the definition of output, max_logits and lse, please refer to README.md - output: [s_q, h_q, d_v], bfloat16 - max_logits: [s_q, h_q], float - lse: [s_q, h_q], float, 2-based log-sum-exp )r r rÚsparse_prefill_fwdr)rr*rr+r,ÚresultsrrrÚflash_mla_sparse_fwd€s  ÿr/)NFN)NFNNFN)r))Útypingrrr rrÚ ExceptionÚ_eÚ_flashmla_import_errorÚ ImportErrorÚ _IMPORT_ERRORÚTensorÚintÚboolrÚfloatr(r/rrrrÚsž€ÿÿ úÿþýüûú ù-óÿþýüûúùø ÷ ö õ ô ó òOûÿþýüûú