o ۾iK @sddlZddlZddlmZddlmZddlmZmZddl Z ddl m Z ddl mZddlmZddlmZdd l mZdd lmZdd lmZdd lmZdd lmZerfddlmZmZddlmZee Z!dZ"z7ddl#Z#ddl$m%Z%ddl&m'Z'zddl(m)Z)Wne*yddl(m+Z,e,j-Z)YnwGdddeZ.dZ/Wne*yZ0zdZ#e1e0Z/dZ.WYdZ0[0ndZ0[0wwGdddeZ2de3fddZ4ddZ5ddd e d!e1fd"d#Z6d0d%d&Z7d0d'd(Z8 d1d e d)e1dBfd*d+Z9de:fd,d-Z;de:fd.d/Z bundles z. You don't have enough GPUs available in a current node. Check `ray status` and `ray list nodes` to see if you have available GPUs in a node `{driver_node_id}` before starting an vLLM engine.aCtensor_parallel_size=%d is bigger than a reserved number of %ss (%d %ss) in a node %s. Tensor parallel workers can be spread out to 2+ nodes which can degrade the performance unless you have fast interconnect across nodes, like Infiniband. To resolve this issue, make sure you have more than %d GPUs available at each node.N)r'is_initializedrrrUitemsappendr(r)r-idr<tensor_parallel_sizeloggerwarning) rhrirjpg_databundle_to_node_idsrlnode_id_to_bundle bundle_idxr0driver_node_idr"r"r#_verify_bundless>   rycurrent_placement_groupc Cs|j}t}|}d}t|tkr?tj|g|d\}}t|dkr&n|d9}tdt t||t|tksz tj |ddWd Stj j yzt dd|D}|dkrntd |d |d td d td|dtdd w)zWait until a placement group is ready. It prints the informative log messages if the placement group is not created within time. r]ra6Waiting for creating a placement group of specs for %d seconds. specs=%s. Check `ray status` and `ray list nodes` to see if you have enough resources, and make sure the IP addresses used by ray cluster are the same as VLLM_HOST_IP environment variable specified in each node if you are running on a multi-node.css|] }|ddVqdS)GPUrN)r`).0specr"r"r# sz'_wait_until_pg_ready..z+Cannot provide a placement group requiring z GPUs (placement_group_specs=z ) within z seconds. Tensor parallel size may exceed available GPUs in your cluster. Check resources with `ray status` and `ray list nodes`. If running on K8s with limited GPUs, consider reducing --tensor-parallel-size to match available GPU resources.Nz:Cannot provide a placement group of placement_group_specs=z within z^ seconds. See `ray status` and `ray list nodes` to make sure the cluster has enough resources.) bundle_specstimereadyPG_WAIT_TIMEOUTr'waitr<rrinforVr` exceptionsGetTimeoutErrorsumre)rzplacement_group_specss pg_ready_ref wait_intervalr_total_gpu_requiredr"r"r#_wait_until_pg_readysR    rcCstj|t}d}t|tkr>tj}|durdS|d9}tdtt|t |t|tksdSdS)Nr{r|z?Waiting for removing a placement group of specs for %d seconds.) r'utilremove_placement_grouprrget_current_placement_grouprrrrVsleep)rzrrpgr"r"r#_wait_until_pg_removed's   r ray_addresscsttddlm}|r+|jdkr+ddlm}|}|j|kr+td|j|||jt r5t dn0| s=| r]zt dWn ty\tdt j||j|jd Yn wt j||jd |jsstd |jd |jrz|j}nt j}|rt d |j}d}|D]}|d} | dkrtdd| r|d7}q|j|krtddd|jd|d nht dt d} |j| krtdfddt|jD} t} t } t| }|ddkrtdd|ddd| d| d d| dd| <t jj| d d!}t||d"us/Jt ||||_d"S)#aInitialize the distributed cluster with Ray. it will connect to the Ray cluster and create a placement group for the workers, which includes the specification of the resources for each distributed worker. Args: parallel_config: The configurations for parallel execution. ray_address: The address of the Ray cluster. If None, uses the default Ray cluster address. rr r)cuda_device_count_statelesszTensor parallel size (%d) exceeds available GPUs (%d). This may result in Ray placement group allocation failures. Consider reducing tensor_parallel_size to %d or less, or ensure your Ray cluster has %d GPUs available.z8Ray is already initialized. Skipping Ray initialization.autoz_No existing RAY instance detected. A new instance will be launched with current node resources.)addressnum_gpus runtime_env)rrzcurrent platform z does not support ray.z"Using the existing placement groupz/Placement group bundle cannot have more than 1 .zThe number of required z(s exceeds the total number of available z6s in the placement group. Required number of devices: z. Total number of devices: zANo current placement group found. Creating a new placement group.z\The number of required %ss exceeds the total number of available %ss in the placement group.csg|]}diqS)g?r")r~rrjr"r# sz*initialize_ray_cluster..zCurrent node has no z" available. current_node_resource=z#. vLLM engine cannot start without z . Make sure you have at least 1 z% available in a node current_node_id=z current_ip=gMbP?znode:PACK)strategyN)!rgvllm.platformsr is_cuda world_sizevllm.utils.torch_utilsrrrrsr'rmris_rocmis_xpuinitConnectionErrorray_runtime_envr,rer.rhrrrr`cluster_resourcesrangerr(r)rrry)rirr ravailable_gpusrzrldevice_bundlesbundlebundle_devicesnum_devices_in_clusterr current_ipcurrent_node_idcurrent_node_resourcer"rr#initialize_ray_cluster9s                    rcCs@ddlm}t}t|d}|}||dksJ||S)Nr)TPUAcceleratorManagerTPU)ray._private.acceleratorsrr'rrV!get_current_node_num_accelerators)rr total_tpus tpus_per_noder"r"r#get_num_tpu_nodess  rcCsntj}tj}d}|r5t}|D]\}}||jkr0|dD] \}}||q&qt |}|S)Nrrk) r'rrrsetrnrphexaddr<)pg_table current_pg num_nodes nodes_in_pgpg_keyrrnoder"r"r# get_num_nodes_in_placement_groups   r)rzrr$)=rKr collectionsrconcurrent.futuresrtypingrrrr* vllm.configrvllm.distributedr/vllm.distributed.kv_transfer.kv_connector.utilsr vllm.loggerr r vllm.sequencer vllm.utils.network_utilsrvllm.v1.outputsrvllm.v1.worker.worker_baservllm.v1.core.sched.outputrrrrOrrrr'ray.utilrray.util.placement_grouprray._private.stater ImportErrorr_state_available_resources_per_noderrferSrZrXrdrgryrrrrVrrr"r"r"r#sv                 Z  5 A