o پi" @s ddlZddlmZddlZddlmmZddlm Z ddl m Z m Z ddl mZde _er6ddlmZeeZdejd ejfd d Zd ejd ejfd dZdd ejded ejfddZdd ejded ejfddZ  d dejdejdejdddedef ddZdS)!N) TYPE_CHECKING) _cp_options) get_sp_groupget_ulysses_parallel_world_size) torch_releaseF) AttentionImpltensorreturncCst|tjr |S|S)zu When tracing the code, the result tensor is not an AsyncCollectiveTensor, so we cannot call ``wait()``. ) isinstanceft_cAsyncCollectiveTensorwait)rr\/home/ubuntu/.local/lib/python3.10/site-packages/sglang/multimodal_gen/runtime/layers/usp.py _maybe_waits rxcCsNtj}|dus Jd|j}|}tj|dd|d}t|}||}|S)Nz)Ulysses process group is not initialized.)output_split_sizesinput_split_sizesgroup)r ulysses_groupshapeflattenr all_to_all_singlerreshape)r ulysses_pgx_shaperrr_usp_all_to_all_single$s rhead_dimc Cs:t}|dkr |S|jdksJd|j|dvs!Jd||dkr'dnd}|dkr4|dkr4|}n |d||d}|j\}}}}||dksVJd |d |d |dddd}t|}|||||d |ddddd|||d |}|dkr|dkr|Sgd } d| |<d| |<|t| S)a Perform Ulysses-style input all-to-all over the head dimension. Default layout expects heads at dim=1 and sequence at dim=2: [b, h, s_local, d] -> [b, h_local, s_global, d] If heads are at dim=2 (input is [b, s_local, h, d]), set head_dim=2, and the function returns [b, s_global, h_local, d], preserving the original head/sequence dim ordering. Args: x: A 4D tensor with layout [b, *, *, d] where '*' are sequence and heads head_dim: Which dimension index corresponds to heads (1 or 2) Returns: Tensor with the same dim order as input, with heads sharded and sequence gathered. rx must have 4 dimensions, got rhead_dim must be 1 or 2, got r"rzh (#) must be divisible by world_size ()rNNr$rndimpermute contiguousrrrtuple rr world_sizeseq_dimx_cbhsd new_orderrrr_usp_input_all_to_all1s2r7c Cs:t}|dkr |S|jdksJd|j|dvs!Jd||dkr'dnd}|dkr4|dkr4|}n |d||d}|j\}}}}||dksVJd |d |d |dddd}t|}|||||d |ddddd|d |||}|dkr|dkr|Sgd } d| |<d| |<|t| S)a Perform Ulysses-style output all-to-all over the head dimension (inverse of input). Default layout expects heads at dim=1 and sequence at dim=2: [b, h_local, s, d] -> [b, h, s_local, d] If heads are at dim=2 (input is [b, s_global, h // world_size, d]), set head_dim=2, and the function returns [b, s_local, h, d], preserving the original head/sequence dim ordering. Args: x: A 4D tensor with layout [b, *, *, d] where '*' are sequence and heads head_dim: Which dimension index corresponds to heads (1 or 2) Returns: Tensor with the same dim order as input, with heads gathered and sequence sharded. rrr r!r#r"rr$zs (r%r&r'r(r)r.rrr_usp_output_all_to_allks2r8querykeyvalue attn_implr is_causal dropout_pc sddlm}tj}|dusJdt|gd}t|gd}t|gd}fdd}tdk} t|||||||d } | rS|d d d i| ^} } n |d i| ^} } t| gd} | S) a Ring Attention implementation. This function implements Ring Attention, a strategy for distributed attention computation that reduces peak memory usage. It accepts a generic attention implementation (`attn_impl`) which is called by the underlying PyTorch distributed attention primitive. Args: query, key, value: The input tensors for attention. attn_impl: An instance of an attention implementation backend (e.g., FlashAttentionImpl) whose `forward` method will be used as the computational kernel. is_causal: Whether to apply causal masking. dropout_p: Dropout probability. r)_templated_ring_attentionNz&Ring process group is not initialized.rr"rr$csht|gd}t|gd}t|gd}j|||ddd^}}}t|gd}||g|RS)NrAT) attn_metadatareturn_softmax_lse)torchr+forward)qkvargskwargsoutput softmax_lserestr=rrattn_callable_adapters z(ring_attn..attn_callable_adapter)r")opr?r>r:r;r<rr0rr) 0torch.distributed.tensor.experimental._attentionr@r ring_grouprDr+r,rdict)r:r;r<r=r>r?r@ring_pgrOuse_segment_id attn_kwargsout_rKrrNr ring_attns8     rZ)r)Fr9)loggingtypingrrD)torch.distributed._functional_collectives distributed_functional_collectivesr rRr8sglang.multimodal_gen.runtime.distributed.parallel_staterrsglang.srt.utils.commonrenable_load_balanceIsglang.multimodal_gen.runtime.layers.attention.backends.attention_backendr getLogger__name__loggerTensorrrintr7r8boolfloatrZrrrrs<       :>