o ۷i@s.ddlmZddlmZddlmZddlZddlZddlmZddlm Z m Z ddl m Z m Z ddlmZdd lmZmZmZmZdd lmZeeZeGd d d Zd ededejjde fddZdejdedejfddZ dddddejde!de dBde dBde"eeejge!fddf ddZ#dS))Callable) dataclass)AnyN)nn) DeviceMeshinit_device_mesh)MixedPrecisionPolicy fully_shard) init_logger) get_fs_groupget_fully_shard_rankget_fully_shard_world_sizeget_world_group)current_omni_platformc@steZdZUdZdZeed<dZeed<dZ eed<e j Z e j ed<e jZe j ed <d Ze j d Bed <d Zeed <d S)HSDPInferenceConfigzzConfiguration for HSDP inference. This is a runtime config created from DiffusionParallelConfig's HSDP settings. Fenabledhsdp_replicate_sizehsdp_shard_size param_dtype reduce_dtypeN output_dtypeTreshard_after_forward)__name__ __module__ __qualname____doc__rbool__annotations__rintrtorchbfloat16rdtypefloat32rrrr%r%Z/home/ubuntu/vllm_env/lib/python3.10/site-packages/vllm_omni/diffusion/distributed/hsdp.pyrs    r device_typereplicate_sizeshard_pgreturncCsPtj|}||}t|||}t|||fdd}td||||S)avCreate a 2D DeviceMesh for HSDP using an existing ProcessGroup for the shard dimension. Args: device_type: The device type (e.g., "cuda", "npu") replicate_size: Number of replica groups shard_pg: The ProcessGroup for the shard dimension (from FS GroupCoordinator) Returns: A 2D DeviceMesh with dimensions ("replicate", "shard") ) replicateshard) mesh_shapemesh_dim_nameszszshard_model..rzNo modules were shardedzSharded %d modules + rootN)reversedlist named_modulesanyr rBr3rD)r;rr?r@rA hsdp_kwargs num_shardedr%rUr&rIs rI)$collections.abcr dataclassesrtypingrr!torch.distributedrrrtorch.distributed.fsdprr vllm.loggerr .vllm_omni.diffusion.distributed.parallel_stater r r rvllm_omni.platformsrrr3rstrr r/ ProcessGroupr:ModulerRrrZrIr%r%r%r&s\       + \