o پi"@sFddlmZddlZddlmZddlmZddlmZddl m Z ddl m Z ddl mZdd lmZdd lmZmZmZdd lmZ dd ejd ejdejdejdejdedejdejdeejfddZGdddejjZejj      dd ejd ejdejdejdejdedejdejdeejde de fddZ!dS) )OptionalN rearrange)chunk_gated_delta_rule_fwd_h) chunk_fwd_o)chunk_scaled_dot_kkt_fwd)chunk_local_cumsum) l2norm_fwd) solve_tril)SUPPRESS_LEVELautocast_custom_fwd input_guard)recompute_w_u_fwdqkvgbetascale initial_stateinitial_state_indices cu_seqlensc  Cst|d|d}t||||tjd} t| ||jd} t|||| ||d\} } t|| | ||||d\} } t||| | |||d}t dkrI||| d| dfSt dkrU||| | | | fSdS) N@) chunk_sizer)rrg_cumsumr output_dtype)Arr)rrrrrr)rwurrrr)rrrhrrr) rrtorchfloat32r dtyperrrr )rrrrrrrrrrrrrv_newor&Y/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/layers/attention/fla/chunk.pychunk_gated_delta_rule_fwdsH      r(c@sdeZdZeee  ddejdejdejdejdejded ejd ejd e ej d e fd dZ dS)ChunkGatedDeltaRuleFunctionNFrrrrrrrrruse_qk_l2norm_in_kernelc  CsR|} |} | rt|}t|}t||||||||| d \}} }}}}| |j|fS)N) rrrrrrrrr)r r(tor#)ctxrrrrrrrrrr*q_origk_origr%rrrr$r&r&r'forwardMs" z#ChunkGatedDeltaRuleFunction.forward)NF) __name__ __module__ __qualname__ staticmethodr r r!Tensorfloatr LongTensorboolr/r&r&r&r'r)Ks6    r)F head_firstr*c Cs|j|jkr|jksJJ|jtjksJdt|jdks&Jd| r,td|durc|jdd krBtd |jdd |durc|jdt|d krctd t|d d |jdd|durn|jdd}t |||||||||| \} } | rt | d} | d| fS)a Args: q (torch.Tensor): queries of shape `[B, T, H, K]` if `head_first=False` else `[B, H, T, K]`. k (torch.Tensor): keys of shape `[B, T, H, K]` if `head_first=False` else `[B, H, T, K]`. v (torch.Tensor): values of shape `[B, T, H, V]` if `head_first=False` else `[B, H, T, V]`. g (torch.Tensor): (forget) gating tensor (in log space!) of shape `[B, T, H]` if `head_first=False` else `[B, H, T]`. beta (torch.Tensor): betas of shape `[B, T, H]` if `head_first=False` else `[B, H, T]`. scale (Optional[int]): Scale factor for the RetNet attention scores. If not provided, it will default to `1 / sqrt(K)`. Default: `None`. initial_state (Optional[torch.Tensor]): Initial state of shape `[N, H, K, V]` for `N` input sequences. For equal-length input sequences, `N` equals the batch size `B`. Default: `None`. output_final_state (Optional[bool]): Whether to output the final state of shape `[N, H, K, V]`. Default: `False`. cu_seqlens (torch.LongTensor): Cumulative sequence lengths of shape `[N+1]` used for variable-length training, consistent with the FlashAttention API. head_first (Optional[bool]): Whether the inputs are in the head-first format, which is not supported for variable-length inputs. Default: `False`. Returns: o (torch.Tensor): Outputs of shape `[B, T, H, V]` if `head_first=False` else `[B, H, T, V]`. final_state (torch.Tensor): Final state of shape `[N, H, K, V]` if `output_final_state=True` else `None`. Examples:: >>> import torch >>> import torch.nn.functional as F >>> from einops import rearrange >>> from fla.ops.gated_delta_rule import chunk_gated_delta_rule # inputs with equal lengths >>> B, T, H, K, V = 4, 2048, 4, 512, 512 >>> q = torch.randn(B, T, H, K, dtype=torch.bfloat16, device='cuda') >>> k = F.normalize(torch.randn(B, T, H, K, dtype=torch.bfloat16, device='cuda'), p=2, dim=-1) >>> v = torch.randn(B, T, H, V, dtype=torch.bfloat16, device='cuda') >>> beta = torch.rand(B, T, H, dtype=torch.bfloat16, device='cuda').sigmoid() >>> g = F.logsigmoid(torch.rand(B, T, H, dtype=torch.bfloat16, device='cuda')) >>> h0 = torch.randn(B, H, K, V, dtype=torch.bfloat16, device='cuda') >>> o, ht = chunk_gated_delta_rule( q, k, v, g, beta, initial_state=h0, output_final_state=True ) # for variable-length inputs, the batch size `B` is expected to be 1 and `cu_seqlens` is required >>> q, k, v, beta, g = map(lambda x: rearrange(x, 'b t ... -> 1 (b t) ...'), (q, k, v, beta, g)) # for a batch with 4 sequences, `cu_seqlens` with 5 start/end positions are expected >>> cu_seqlens = q.new_tensor([0, 2048, 4096, 6144, 8192], dtype=torch.long) >>> o_var, ht_var = chunk_gated_delta_rule( q, k, v, g, beta, initial_state=h0, output_final_state=True, cu_seqlens=cu_seqlens ) zJChunkGatedDeltaRuleFunction does not support float32. Please use bfloat16.r zLbeta must be of shape [B, T, H] if head_first=False, or [B, H, T] otherwise.znhead_first is deprecated and will be removed in a future version. Please use head_first=False for now instead.cSs t|dS)Nzb h t ... -> b t h ...r)xr&r&r's z(chunk_gated_delta_rule..Nrz/The batch size is expected to be 1 rather than zQ when using `cu_seqlens`.Please flatten variable-length inputs before processing.z]The number of initial states is expected to be equal to the number of input sequences, i.e., z rather than .gzb t h ... -> b h t ...) r#r!r"lenshapeDeprecationWarningmap ValueErrorr)applyr) rrrrrrrrrr8r*r%rr&r&r'chunk_gated_delta_rulersT"M   rD)N)NNNNFF)"typingrr!einopsr-sglang.srt.layers.attention.fla.chunk_delta_hr'sglang.srt.layers.attention.fla.chunk_or4sglang.srt.layers.attention.fla.chunk_scaled_dot_kktr&sglang.srt.layers.attention.fla.cumsumr&sglang.srt.layers.attention.fla.l2normr *sglang.srt.layers.attention.fla.solve_trilr %sglang.srt.layers.attention.fla.utilsr r r 'sglang.srt.layers.attention.fla.wy_fastrr4r5r6r(autogradFunctionr)compilerdisabler7rDr&r&r&r's|           1'