o ۷i3@s ddlmZddlmZmZddlZddlmZddl m Z er! e e Z eGdddZ eGdd d Zed d Gd d d Zed d GdddZeeeBeeeBeedfBfZeeeBeedfBZeeeeBfZdedejdeefddZdS)) dataclass) TYPE_CHECKINGLiteralN) get_loggerc@s>eZdZUdZdZedBed<dZedBed<dZe ed<dZ e ded <d Z e ed <dZ eed <dZeed <dZejed<dZejjjed<dZejjjed<dZejjjed<dZejjjed<dZeed<dZeed<ddZedeeeffddZedeeeffddZ dededejdejjjfd d!Z!dS)"ContextParallelConfiga Configuration for context parallelism. Args: ring_degree (`int`, *optional*, defaults to `1`): Number of devices to use for Ring Attention. Sequence is split across devices. Each device computes attention between its local Q and KV chunks passed sequentially around ring. Lower memory (only holds 1/N of KV at a time), overlaps compute with communication, but requires N iterations to see all tokens. Best for long sequences with limited memory/bandwidth. Number of devices to use for ring attention within a context parallel region. Must be a divisor of the total number of devices in the context parallel mesh. ulysses_degree (`int`, *optional*, defaults to `1`): Number of devices to use for Ulysses Attention. Sequence split is across devices. Each device computes local QKV, then all-gathers all KV chunks to compute full attention in one pass. Higher memory (stores all KV), requires high-bandwidth all-to-all communication, but lower latency. Best for moderate sequences with good interconnect bandwidth. convert_to_fp32 (`bool`, *optional*, defaults to `True`): Whether to convert output and LSE to float32 for ring attention numerical stability. rotate_method (`str`, *optional*, defaults to `"allgather"`): Method to use for rotating key/value states across devices in ring attention. Currently, only `"allgather"` is supported. N ring_degreeulysses_degreeTconvert_to_fp32 allgather)r alltoall rotate_methodFulysses_anything_rank _world_size_device_mesh_flattened_mesh _ring_mesh _ulysses_mesh_ring_local_rank_ulysses_local_rankcCs|jdurd|_|jdurd|_|jdkr|jdkrtd|jdks(|jdkr,td|jdkr:td|jd|jrO|jdkrFtd|jdkrQtddSdS) NzfEither ring_degree or ulysses_degree must be greater than 1 in order to use context parallel inferencezF`ring_degree` and `ulysses_degree` must be greater than or equal to 1.r z=Only rotate_method='allgather' is supported for now, but got .zIulysses_degree must be greater than 1 for ulysses_anything to be enabled.z8ulysses_anything cannot be enabled when ring_degree > 1.)rr ValueErrorr NotImplementedErrorrselfrY/home/ubuntu/vllm_env/lib/python3.10/site-packages/diffusers/models/_modeling_parallel.py __post_init__Us*      z#ContextParallelConfig.__post_init__returncCs |j|jfSN)rr rrrr mesh_shapeks z ContextParallelConfig.mesh_shapecCsdS)z$Dimension names for the device mesh.)ringulyssesrrrrrmesh_dim_namesosz$ContextParallelConfig.mesh_dim_namesrank world_sizedevicemeshcCs||_||_||_||_|j|j|kr$td|jd|jd|d|j|_|jd|_ |jd|_ |j |_ |j |_ dS)NzThe product of `ring_degree` (z) and `ulysses_degree` (z") must not exceed the world size (z).r$r%)rrrrr rr_flattenrrrget_local_rankrrrr'r(r)r*rrrsetupts    zContextParallelConfig.setup)"__name__ __module__ __qualname____doc__rint__annotations__r r boolr rrrrrtorchr)r distributed device_mesh DeviceMeshrrrrrr propertytupler#strr&r.rrrrr)s,       &rc @seZdZUdZdZedBed<dZeed<dZ eed<dZ e j ed<dZ e jjjed<ddd ed ed e j d e jjjdBfd dZdS)ParallelConfigz Configuration for applying different parallelisms. Args: context_parallel_config (`ContextParallelConfig`, *optional*): Configuration for context parallelism. Ncontext_parallel_configrrrr)r*r'r(r)r*cCs<||_||_||_||_|jdur|j||||dSdSr")rrrrr>r.r-rrrr.s zParallelConfig.setup)r/r0r1r2r>rr4rr3rrr6r)rr7r8r9r.rrrrr=s"    r=T)frozenc@s>eZdZUdZeed<dZedBed<dZeed<ddZ dS) ContextParallelInputa Configuration for splitting an input tensor across context parallel region. Args: split_dim (`int`): The dimension along which to split the tensor. expected_dims (`int`, *optional*): The expected number of dimensions of the tensor. If provided, a check will be performed to ensure that the tensor has the expected number of dimensions before splitting. split_output (`bool`, *optional*, defaults to `False`): Whether to split the output tensor of the layer along the given `split_dim` instead of the input tensor. This is useful for layers whose outputs should be split after it does some preprocessing on the inputs (ex: RoPE). split_dimN expected_dimsF split_outputcCsd|jd|jd|jdS)NzContextParallelInput(split_dim=, expected_dims=z, split_output=))rArBrCrrrr__repr__szContextParallelInput.__repr__) r/r0r1r2r3r4rBrCr5rFrrrrr@s   r@c@s2eZdZUdZeed<dZedBed<ddZdS)ContextParallelOutputa Configuration for gathering an output tensor across context parallel region. Args: gather_dim (`int`): The dimension along which to gather the tensor. expected_dims (`int`, *optional*): The expected number of dimensions of the tensor. If provided, a check will be performed to ensure that the tensor has the expected number of dimensions before gathering. gather_dimNrBcCsd|jd|jdS)Nz!ContextParallelOutput(gather_dim=rDrE)rHrBrrrrrFszContextParallelOutput.__repr__)r/r0r1r2r3r4rBrFrrrrrGs   rG.sizegroupr!csztj|d}ttj|d}d|vrdntjfddt|D}tj|tj |gtj d|ddd|D}|S)zsGather the local size from all ranks. size: int, local size return: list[int], list of size from all ranks )rJcpucsg|] }tjdtjdqS))rr)dtype)r6emptyint64).0_ gather_devicerr $sz'gather_size_by_comm..rLcSsg|]}|dqS)r)item)rPsrrrrT+s) distget_world_sizer< get_backendr6 acceleratorcurrent_acceleratorrange all_gathertensorrO)rIrJr( comm_backendsgathered_sizesrrRrgather_size_by_comms ra) dataclassesrtypingrrr6torch.distributedr7rWutilsrr/loggerrr=r@rGdictr<r3listr;ContextParallelInputTypeContextParallelOutputTypeContextParallelModelPlan ProcessGrouprarrrrs,    \  +