o ۾iQ@sddlZddlmZddlmZmZejdejdejdejdejdejd ejf d d Zejdejdejdejdejdejd ejd ejdejd ejdejfddZ ejdejdejdejdejdejd ejd ejfddZ ejdejdejdejdejdejd ejd ejdejfddZ Gdddej j ZejZ  d,dejdejdejdejdedejdBdeejejffd d!Zejd"ejd#ejfd$d%Z &d-dejdejdejd'ejd(ejd)ejd#edejfd*d+ZdS).N) rearrange)tltritonbhdeBLOCKCBLOCKc .Cs.td} | | }| | }td}||}|||}||| }||| }|| }||}|| }|| }|| }||}|| }||||td| dddf|td|dddf}|||td| dddf|td|dddf}|||td| dddf| td| dddf}||||td| dddf| td| dddf}||} t| }!|}"td| |"| }#tj|||#dddf|kddtj}$tj| | gtjd}%t|"dD]z}&td| |&| }'|#dddf|'dddf}(|!|(})t|(dk|) t d})t |)}*tj|||'dddf|kddtj}+tj|||'dddf|kddtj},t |$|+|*}-|%t |-|,7}%|| |7}|| | 7}qtj ||%|j j||#dddf|kddS)Nrmaskotherdtypez-infr)r program_idarangeloadtofloat32zerosrangewherefloatexpdotstorer element_ty).QKVOutSrrnrrr NUM_BLOCKr offoff_bh off_block off_cblockoff_h qk_offsetv_offseto_offset block_offsetqk_block_offsetv_block_offseto_block_offset cblock_offsetq_cblock_offseto_cblock_offset Q_block_ptrK_trans_block_ptr V_block_ptr O_block_ptr S_block_ptrsiq_indexqqkvjkv_indexdiffs_indexdecayk_transvqkrH]/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/layers/lightning_attn.py_fwd_diag_kernel s       $     rJD_FBLOCKE_FBLOCK NUM_FBLOCK NUM_CBLOCKc(Cstd}td}||}|| }||}||}|||}|||}|||}|| ||}|||td|dddf|td| dddf}|||td|dddf|td| dddf}|||td| dddf|td| dddf}||| td|}td|}tj| | gtjd}|| dkr|| d| n| } t| ||| }!tt| ||}"|||"|7}t|"D]W}#d|#|!}$tj||!||dddf|$kdd}%tj||!||dddf|$kdd}&t|}'|'dddf}'|t |%|'|&7}|||7}|||7}||7}qt || |j j dS)Nrr rr r )rrrrrcdivminrrrrrrr)(r!r"K_decayKVrrr%rrr r&rKrLrMr rNr(r)r+r/k_block_offsetr1kv_block_offsetk_offsetr- kv_offsetr7r8 KV_block_ptr k_decay_ptrrAkvsplit_n left_shift num_blocksr@ left_boundrErFk_decayrHrHrI_fwd_kv_parallels               r_c CsRtd} | |} | | ||}||td| dddf|td| dddf}|| }t|}| ||}||td| dddf|td| dddf}t|tj}t| D]7}t||||}t|tj |}t|tj}t |||j j |||}|||7}qit ||dS)Nr) rrrrrrrrPrrrr)r$rR KV_HISTORYrrr%rrr r&rKrLr(r+rVrWs_ptrsr;kv_history_offsetKV_HISTORY_block_ptrkv_prer< block_size block_decaykv_currHrHrI_fwd_kv_reduces>     rhc(Cs`td}||}td}|| }|| }td}|| }|| }|| }||}||||||}|||||||}|| ||||||}||td| dddf|td|dddf}||td| dddf|td| dddf}||td|dddf|td| dddf}||}t|}td| } t|tj}!|td| }"tj||"dddf|kddtj}#t|tj || | dddf}$t|#|!|$}%tj||"dddf|kddtj}&|&|%}'tj||'|j j |"dddf|kddS)Nrr r r r) rrrrrrrrrrr)(r r#r$rRrrr%rrr r&rLr rNr(r+off_ncoff_noff_coff_en_offsetc_offsete_offsetr/q_offsetr.rVr6r9rWr:r;c_arrayrYr=r>q_decay qkv_none_diagqkv_diagr?rHrHrI_fwd_none_diag_kernel9sL    ::  (,(  rvc@seZdZeddZdS) _attentioncCsR|}|}|}|}tj}|ddkr tdd|j\}}} } |jd} tj||| | f|j|jd} d} t | | }d}| |}| |dksRJd tj d| |jd d }t | | | d d}||||f}t||||| |||| | | | ||d d }| |}| |dksJ| |}| |dksJd }| |}| |dksJd tj|||| | ftj|jd}|||f}t|||||||| | | | ||||||d|||f}t||||||| | | | |||d ||||f}t||| ||||| | | | ||||d||||||| |_| tj||dgddfS)Nrz(Flash attention currently only supportedzfor compute capability >= 80rdevice z"BLOCK must be a multiple of CBLOCK)r{r )r r&r @)r r&rKrLrMr rN)r r&rKrL)r r&rLr rNri)dim) contiguoustorchcudaget_device_capability RuntimeErrorshapeemptyrr{rrOrrreshaperJrr_rhrvsave_for_backwardr cat unsqueeze)ctxr>krFr; kv_history capabilityrrr%rror r&r rNarrayr^gridrMrKrLrYrHrHrIforwards      z_attention.forwardN)__name__ __module__ __qualname__ staticmethodrrHrHrHrIrwsrwr|r>rrFedrerreturncsJ|jd}|jd}|dkr|dddd}|dkrdnd|dks1Jd|ddfd d t|dD}|d|krK||t|} d} |d urjtj|jd|jd||ftj|j d }n| }t| dD]*} || } || d}|d | |f} |d | |f}t | ||||\}}| |} qv| |fS)a$ Apply lightning attention algorithm to compute attention efficiently. Args: q: Query tensor of shape [batch, heads, seq_len, dim] k: Key tensor of shape [batch, heads, seq_len, dim] v: Value tensor of shape [batch, heads, seq_len, dim_v] ed: Decay rate tensor of shape [heads] block_size: Size of blocks for block-sparse attention kv_history: Optional key-value history from previous computations Returns: output: Attention output kv: Updated key-value history ryr r~rz Dimension d (z) must be divisible by m ()csg|]}|qSrHrH).0r<mrHrI 4sz'lightning_attention..Nrz.) rrviewrappendlenrrrr{clonerlightning_attention_)r>rrFrrerrrarrr%outputr<r;q1k1rrYrHrrIlightning_attentions0   "     rD BLOCK_SIZEc'Cstd}td}td}t||tj}|dkr dS|}|}t||}td|}td|||}|dddf| |dddf| }|||| }|||| }|||| }|| || }||k}||k}tj||||dd}tj||||dd} tj||||dd}!| dddf|!dddf}"|dddf|dddf@}#t| }|||}$tj|$|#dd}%|"||%}"|dddftj|"}&tj|&dd}&tj |$|"|#d tj ||||&|d dS) z Kernel for linear attention decoding with KV cache. This kernel computes attention for a single token using the KV cache. rr riryNr r )axisr) rrrrint64rrrsumr)'q_ptrk_ptrv_ptr kv_cache_ptr slope_rateslot_idx output_ptrr qkv_b_stride qkv_h_stridecache_b_stridecache_h_stridecache_d0_stridecache_d1_striderpid_bpid_hpid_dslot_idbatch_idhead_idratio qk_d_offsets v_d_offsetscache_d_offsetsrqrUr- cache_offsetqk_maskv_maskr>rrFkv_outerkv_maskkv_ptr kv_cache_oldrrHrHrI_linear_attn_decode_kernelMs@    &     rr} kv_cachesrrcCs|j\}}} } |j||d| fksJ|j||d| fksJt|} ||| |f} |d} |d}|d}|d}|d}|d}t| ||||||| | | ||||||dt| d} | dS)a Perform linear attention decoding using Triton kernels. Args: q: Query tensor of shape [B, H, 1, D] k: Key tensor of shape [B, H, 1, D] v: Value tensor of shape [B, H, 1, D] kv_caches: Key-value cache tensor slope_rate: Decay rate tensor slot_idx: Slot indices for batches BLOCK_SIZE: Size of blocks for processing Returns: output: Attention output tensor r rri)rzb h n d -> b n (h d))rr empty_likestriderrsqueezer)r>rrFrrrrBH_rrrrrrrrrrHrHrIlinear_decode_forward_tritons<        r)r|N)r})reinopsrvllm.triton_utilsrrjit constexprrJr_rhrvautogradFunctionrwapplyrTensorinttuplerrrrHrHrHrIs          l   @     R : X