o پic8@sddlmZddlZddlmZddlmZmZddlm Z m Z m Z m Z m Z ddlmZddlmZddlmZmZmZddlmZmZdd lmZdd lmZGd d d ejZGd ddeZGdddejZ GdddejZ!dS))TypeN)"sequence_model_parallel_all_gather%sequence_model_parallel_all_to_all_4D)get_ring_parallel_world_size get_sequence_parallel_world_sizeget_sp_parallel_rankget_sp_world_sizeget_ulysses_parallel_world_size) AttentionImpl)get_attn_backend)_usp_input_all_to_all_usp_output_all_to_all ring_attn)ForwardContextget_forward_context)AttentionBackendEnum)get_compute_dtypecseZdZdZ     ddedededBdedBd ed eedBd e d dffd d Z   dde j de j de j de j dBde j dBde j dBd e e j e j dBffddZZS)UlyssesAttentionz2Ulysses-style SequenceParallelism attention layer.NF num_heads head_size num_kv_heads softmax_scalecausalsupported_attention_backendsprefixreturnc st|dur|d|_n||_|dur|}t} t|| |d} | } | d||||j||dd||_||_||_||_ | |_ | |_ dSN࿩rz.impl)rrrrrr super__init__rrr get_impl_cls attn_implrrrget_enumbackenddtype) selfrrrrrrrextra_impl_argsr( attn_backendimpl_cls __class__r h/home/ubuntu/.local/lib/python3.10/site-packages/sglang/multimodal_gen/runtime/layers/attention/layer.pyr#(s4     zUlyssesAttention.__init__qkv replicated_q replicated_k replicated_vcCs|dkr|dkr|dksJd|j\}}} } t} t} t} | j}tj|||gdd}t|ddd}|j ||}|dury|durM|dusOJtj|||gdd}| | }|dddd| || d|f}tj||gdd}|j d dd\}}}|j ||||}d}|dur|dd|| df}|ddd|| f}t |dd}|j ||}t|ddd}||fS) aZForward pass for distributed attention. Args: q (torch.Tensor): Query tensor [batch_size, seq_len, num_heads, head_dim] k (torch.Tensor): Key tensor [batch_size, seq_len, num_heads, head_dim] v (torch.Tensor): Value tensor [batch_size, seq_len, num_heads, head_dim] replicated_q (Optional[torch.Tensor]): Replicated query tensor, typically for text tokens replicated_k (Optional[torch.Tensor]): Replicated key tensor replicated_v (Optional[torch.Tensor]): Replicated value tensor Returns: Tuple[torch.Tensor, Optional[torch.Tensor]]: A tuple containing: - o (torch.Tensor): Output tensor after attention for the main sequence - replicated_o (Optional[torch.Tensor]): Output tensor for replicated tokens, if provided Expected 4D tensorsrdim scatter_dim gather_dimN)r9shaperrr attn_metadatatorchcatrr%preprocess_qkvchunkforwardr contiguouspostprocess_output)r)r0r1r2r3r4r5 batch_sizeseq_lenrhead_dim local_rank world_sizeforward_contextctx_attn_metadataqkvreplicated_qkvheads_per_rankoutputreplicated_outputr r r/rFQsB,  zUlyssesAttention.forward)NNFNrNNN)__name__ __module__ __qualname____doc__intfloatboolsetrstrr#rBTensortuplerF __classcell__r r r-r/r%sT  .rc@sdeZdZdZ    d dejdejdejdejdBdejdBdejdBd ejdBd ejfd d ZdS)UlyssesAttention_VSAz-Distributed attention layer with VSA support.Nr0r1r2r3r4r5 gate_compressrc Cs|dur |dur |dusJd|dkr"|dkr"|dks&Jdt}|j} tj||||gdd} t| ddd } |j| | } | jddd\}}}}|jj ||||| d } |j | | } t| ddd } | S) aForward pass for distributed attention. Args: q (torch.Tensor): Query tensor [batch_size, seq_len, num_heads, head_dim] k (torch.Tensor): Key tensor [batch_size, seq_len, num_heads, head_dim] v (torch.Tensor): Value tensor [batch_size, seq_len, num_heads, head_dim] gate_compress (torch.Tensor): Gate compress tensor [batch_size, seq_len, num_heads, head_dim] replicated_q (Optional[torch.Tensor]): Replicated query tensor, typically for text tokens replicated_k (Optional[torch.Tensor]): Replicated key tensor replicated_v (Optional[torch.Tensor]): Replicated value tensor Returns: Tuple[torch.Tensor, Optional[torch.Tensor]]: A tuple containing: - o (torch.Tensor): Output tensor after attention for the main sequence - replicated_o (Optional[torch.Tensor]): Output tensor for replicated tokens, if provided Nz+Replicated QKV is not supported for VSA nowr6r7rr8r:r;r<)rcrA) r9rrArBrCrr%rDrErFrH) r)r0r1r2r3r4r5rcrNrOqkvgrSr r r/rFs(,  zUlyssesAttention_VSA.forward)NNNN)rVrWrXrYrBr_rFr r r r/rbs. rbcszeZdZdZ    ddedededBdedBded eedBd dffd d Z d e j de j de j d e j fddZ Z S)LocalAttentionzAttention layer.NFrrrrrrrc  st|dur|d|_n||_|dur|}t}t|||d} | } | d|||j||d||_||_||_||_ | |_ ||_ dS)Nrr)rrrrrr r!) r)rrrrrrr*r(r+r,r-r r/r#s2    zLocalAttention.__init__r0r1r2cCsP|dkr|dkr|dksJdt}|j}|jj||||d}|S)a Apply local attention between query, key and value tensors. Args: q (torch.Tensor): Query tensor of shape [batch_size, seq_len, num_heads, head_dim] k (torch.Tensor): Key tensor of shape [batch_size, seq_len, num_heads, head_dim] v (torch.Tensor): Value tensor of shape [batch_size, seq_len, num_heads, head_dim] Returns: torch.Tensor: Output tensor after local attention r6r7)rA)r9rrAr%rF)r)r0r1r2rNrOrSr r r/rFs ,zLocalAttention.forward)NNFN)rVrWrXrYrZr[r\r]rr#rBr_rFrar r r-r/res<  %recseZdZdZ      ddedededBd edBd ed eedBd e d eddffdd Z   dde j de j de j de j dBde j dBde j dBde j fddZ ZS) USPAttentiona" Ulysses Sequence Parallelism with Ring Attention. This class implements the USP algorithm, which is a combination of Ulysses-style all-to-all communication for sequence-head dimension sharding and Ring Attention for fine-grained sequence parallelism within subgroups. NFrrrrrrrr dropout_raterc st|dur|d|_n||_|dur|}t} t|| |d} | } | d||||j||dd| |_||_||_||_ | |_ | |_ ||_ ||_dSr)r"r#rrr r$r%rrrr&r'r(r dropout_p) r)rrrrrrrrhr*r(r+r,r-r r/r#(s8     zUSPAttention.__init__r0r1r2r3r4r5c Cs|dur |dur |dusJdt}|j}tdkr&|j||||} | Stdkr=t|dd}t|dd}t|dd}tdkrPt||||j|j |j d} n |j||||} tdkrdt | dd} | S)z Forward pass for USPAttention. q, k, v: [B, S_local, H, D] Note: Replicated tensors are not supported in this implementation. Nz-USPAttention does not support replicated_qkv.r;r:)rK)r% is_causalri) rrArr%rFr r rrrrir ) r)r0r1r2r3r4r5rNrOoutr r r/rFSs2        zUSPAttention.forward)NNFNrrgrU)rVrWrXrYrZr[r\r]rr^r#rBr_rFrar r r-r/rfsZ    0rf)"typingrrBtorch.nnnn:sglang.multimodal_gen.runtime.distributed.communication_oprr8sglang.multimodal_gen.runtime.distributed.parallel_staterrrrr Isglang.multimodal_gen.runtime.layers.attention.backends.attention_backendr 7sglang.multimodal_gen.runtime.layers.attention.selectorr (sglang.multimodal_gen.runtime.layers.uspr r r6sglang.multimodal_gen.runtime.managers.forward_contextrr'sglang.multimodal_gen.runtime.platformsrsglang.multimodal_gen.utilsrModulerrbrerfr r r r/s      w@C