o 'i0!@sddlmZddlmZddlZddlmZddlmZzddl Z dZ Wn e y1dZ d Z Ynwda d ej jfd d ZGd ddeZddddddddejdejdejdededeedeejdeejdeejdeejdeejfddZe re je jjgde jedddddddddejdejdejdededeedeejdeejdeejdeejdeejfdd Zddddddddd!dejdejdejded"ejdedeejdeejdeejdeedeejdeejd#eejd$eejd%ejd&ejf d'd(Zeddd dddddd)dejdejdejded"ejdedeejdeejd*edeejdeejd#eejd$eejd%eejd&ejfd+d,ZdS)-)Enum)OptionalN)flashinfer_api)get_cudnn_fmha_gen_moduleTFstreamcCs"tdurtatt|jtS)N) _cudnn_handlecudnn create_handle set_stream cuda_stream)rrT/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/flashinfer/cudnn/decode.py_create_cudnn_handlesrc@sDeZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdS)UIDsrrrde234iiN)__name__ __module__ __qualname__RESERVED_INVALID_UIDQ_UIDK_UIDV_UIDACTUAL_SEQ_LENS_Q_UIDACTUAL_SEQ_LENS_KV_UIDBLOCK_TABLES_UIDBLOCK_TABLES_K_UIDBLOCK_TABLES_V_UID RAGGED_Q_UID RAGGED_O_UIDRAGGED_STATS_UIDO_UID STATS_UIDrrrrrsr) block_sizeactual_seq_lens_qactual_seq_lens_kv block_tablesbatch_offsets_qbatch_offsets_oqk_cachev_cachescalemax_sequence_kvr,r-r.r/r0r1c Csd|t|jt|jfS)Ndecode)tupleshape) r2r3r4r5r6r,r-r.r/r0r1rrr_sdpa_decode_key_fn4s r:) heur_modes)key_fnc Cs&ttj} d} d} t| [\} } |dkr.d}|jd|jd|jd}}}n&|dkrK|jd|jd|jd|jdf\}}}}n td||dks\Jd|dksfJd|jd}| j d ||||f|||||dftj j d }| dur| | }| tjj||| |}| |}| tjj| tjj| tjj|dur||jdd|jdd}| |}| tjj| |}| tjj|dur| |}| tjj|dur| |}| tjj|d |du}| jd ||||dur|nd|dur|nd|d ||||tj jd \}} | durD| | }| tjj||| tjjd  ||||g!|||||dg"tj j Wdn 1spwY||||g}|dur|#||dur|#|| |fS)Nrrrrz#q must have 3 or 4 dimensions, got z"q must have a sequence length of 1zk_cache must have 4 dimensionsr2)namedimstride data_typeFsdpaT) r>r2kv seq_len_q seq_len_kvuse_padding_mask is_inference attn_scalepaged_attention_k_tablepaged_attention_v_tablepaged_attention_max_seq_len_kvcompute_data_type)$rtorchcudacurrent_streamr graphr?r9 ValueErrortensorrABFLOAT16 tensor_likeset_uidrr'valueset_ragged_offsetrr r!reshaper%r&r"r#set_is_pass_by_valuerBFLOATr(r* set_outputset_dim set_stride set_data_typeappend) r2r3r4r5r6r,r-r.r/r0r1handleg_s_qobh_qod_qkd_vocudnn_qragged_q cudnn_k_cache cudnn_v_cachend_block_tablescudnn_k_block_tablescudnn_v_block_tablescudnn_actual_seq_lens_qcudnn_actual_seq_lens_kv padding_maskOragged_otensors_to_returnrrr_build_decode_graphLs $                  Y    rv)r-r.r/r,r0r1batch_offsets_kbatch_offsets_vworkspace_bufferrwrxoutreturnc  Cst||||||||| | dur| nd| dur| ndd \}}ttj}tjj|tjj|tj j|tj j|i}|dur@||tj j<|durJ||tj j<| durT| |tj j<| dur^| |tjj<|durn||tjj<||tjj<|j|||d|S)N) r2r3r4r5r6r-r.r/r,r0r1) workspacera)rvrrNrOrPrrrWr r!r*r"r#r'r(r%r&execute)r2r3r4r5ryr6r-r.r/r,r0r1rwrxrzrQtensorshandle_var_maprrr_batch_decode_with_kv_caches@       r)r.r/is_cuda_graph_compatibler0r1rwrxrzrc Cs|jd}|jd}|jd}| durtj||||j|jd} ts?|j|jdd}tj}||||||||||| | | | | Stj |dddf|jtj d}|jd}t |||||||||| | || d | S) a)Performs batched decode attention with paged KV cache using cuDNN. Args: q: Query tensor of shape (batch_size, num_heads_qo, head_dim), seq_len_q is the maximum sequence length of queries in the batch k_cache: Key cache tensor of shape (total_num_pages, num_heads_kv, page_size, head_dim) v_cache: Value cache tensor of shape (total_num_pages, num_heads_kv, page_size, head_dim) scale: Scaling factor for attention scores, typically 1/sqrt(head_dim) workspace_buffer: Workspace buffer for cuDNN operations. Scales with batch size. 128 MB should be sufficient for most cases max_sequence_kv: Maximum number of tokens per key/value sequence (s_kv_max) actual_seq_lens_kv: Actual sequence lengths for key/values per batch, shape (batch_size,) on CPU block_tables: Page table mapping for KV cache, shape (batch_size, num_pages_per_seq) on GPU is_cuda_graph_compatible: Whether the decode operation is compatible with CUDA graph batch_offsets: Optional batch offsets tensor of shape (batch_size,) on GPU out: Optional pre-allocated output tensor batch_offsets_q: Optional batch offsets for query tensor of shape (batch_size,) on GPU batch_offsets_o: Optional batch offsets for output tensor of shape (batch_size,) on GPU batch_offsets_k: Optional batch offsets for key tensor of shape (batch_size,) on GPU batch_offsets_v: Optional batch offsets for value tensor of shape (batch_size,) on GPU Returns: Output tensor of shape (batch_size, num_heads_qo, head_dim) Note: Currently only supports causal attention (causal must be True) All tensors must be contiguous and on the same CUDA device Query and KV heads can have different sizes (num_heads_qo >= num_heads_kv) rrrN)devicedtypeT) non_blockingr) r2r3r4r5ryr6r-r.r/r0r1r,rz) r9rNemptyrrCUDNN_AVAILABLEtorr7onesint32r)r2r3r4r5ryr6r.r/rr0r1rwrxrzbsrfrhactual_seq_lens_kv_gpurun_funcr-r,rrr cudnn_batch_decode_with_kv_cachesX .  % r)enumrtypingrrN api_loggingrutilsrr r ImportErrorr rOStreamrrTensorfloatintr:jit heur_modeA graph_cachervrboolrrrrrs:                    ;