o پi@sddlmZmZddlZ  ddejdejdejdejdeejdeejd eejejffd d Z  ddejdejdejdejdeejdeejd eejejffd d Z ddejdejdejdejdejdejdeded ejfddZ d dedededed ef ddZ dS)!)OptionalTupleNv_as_av_bs_bv_mergeds_mergedreturncC^|tj}|tj}|durt|}|durt|}tjjj||||||||fSN)totorchfloat32 empty_likeops sgl_kernel merge_statedefaultrrrrrr rH/home/ubuntu/.local/lib/python3.10/site-packages/sgl_kernel/attention.pyrs    rcCr r )r rrrrrmerge_state_v2rrrrrrs    rq_nopeq_pekv_c_and_k_pe_cacheseq_lens page_table workspacesm_scale num_kv_splitsc CsL|jdks Jd|j|jdksJd|j|jdks'Jd|j|j\}} } |j\} } } || kr;| | ks=J|j\}}}d}d}| |ksMJ| |ksSJ|||ks[Jd}| |kskJd|d | | |kr|||| f}||ddd| f<|}|||| f}||ddd| f<|}t|jd ksJ|j\}}||ksJ|d ksJd ||d|d ksJ|jtjtjfvsJd |jd|j|jkr|jksJJ|jtjksJd|jd|jtjksJd|jd||||f}tj j j ||||||||| |ddd| f S)Nz$q_nope must be a 3D tensor, but got z"q_pe must be a 3D tensor, but got z1kv_c_and_k_pe_cache must be a 3D tensor, but got i@z H must be <= z , but got rz&block num must be greater than 0, got z.q_nope.dtype needs to be fp16 or bf16 but got .z)seq_lens.dtype needs to be int32 but got z+page_table.dtype needs to be int32 but got )ndimshape new_emptylendtyperfloat16bfloat16int32rrcutlass_mla_decoder contiguous)rrrrrrr r!B_qHD_q_nopeB_q_2H_2D_q_pe_ PAGE_SIZED_ckvD_latentD_rope MAX_HEADS q_nope_padded q_pe_padded B_block_table block_numoutrrrr/0sl           "    r/ max_seq_len num_batchessm_countcCsB|dks Jd||dksJd|tjjj||||S)Nrz(max_seq_len must be greater than 0, got z(num_batches must be greater than 0, got )rrrcutlass_mla_get_workspace_sizer)rBrCrDr!rrrrEzs  rE)NN)r)rr) typingrrrTensorrrfloatintr/rErrrrs   M