o `۷i@sUddlZddlmZmZmZmZmZddlZddlm Z ddl m Z m Z ddl mZddlmZddlmZmZddlmZerEddlZGd d d eZiaeeefed <iaeeefed <ead aedddedeedeefddZ gdZ!e dddgee ddge e ddge e ddgededdfddZ"d d!dee fd"d#Z#d$d%d&ede$fd'd(Z%d&edefd)d*Z&d&ed+efd,d-Z'd.eded/fd0d1Z(dS)2N) TYPE_CHECKINGDictList NamedTupleOptional) ObjectRef)GLOOTensorTransportNCCLTensorTransport)CudaIpcTransport)NixlTensorTransport)TensorTransportManagerTensorTransportMetadata) PublicAPIc@s&eZdZUeeed<eeed<dS)TransportManagerInfotransport_manager_classdevicesN)__name__ __module__ __qualname__typer __annotations__rstrrr^/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/experimental/gpu_object_manager/util.pyrs rtransport_manager_infotransport_managersFalpha) stabilitytransport_namerrcCs^|}|tvrtd|dt|tstd|jdt||t|<|tvr-dadSdS)aZ Register a new tensor transport for use in Ray. Note that this needs to be called before you create the actors that will use the transport. The actors also need to be created in the same process from which you call this function. Args: transport_name: The name of the transport protocol. devices: List of PyTorch device types supported by this transport (e.g., ["cuda", "cpu"]). transport_manager_class: A class that implements TensorTransportManager. Raises: ValueError: If transport_manager_class is not a subclass of TensorTransportManager. z Transport z already registered.ztransport_manager_class z. must be a subclass of TensorTransportManager.TN) upperr ValueError issubclassr rrDEFAULT_TRANSPORTShas_custom_transports)rrrrrrregister_tensor_transport*s  r$)NIXLGLOONCCLCUDA_IPCr%cudacpur&r'r(returnr cCstt.|tvrt|WdS|tvrtd|t|t|<t|WdS1s3wYdS)aGet the tensor transport manager for the given tensor transport protocol. Args: transport_name: The tensor transport protocol to use for the GPU object. Returns: TensorTransportManager: The tensor transport manager for the given tensor transport protocol. N'Unsupported tensor transport protocol: )transport_managers_lockrrr r)rrrrget_tensor_transport_managerYs$r.actorzray.actor.ActorHandlecCs4tsdSdtttffdd}|jjdd|tS)z If there's no custom transports to register, returns None. Otherwise returns an object ref for a task on the actor that will register the custom transports. Nowner_transport_manager_infocSs>ddlm}m}|D]\}}||vr|||j|jq dS)Nr)r$r)(ray.experimental.gpu_object_manager.utilr$ritemsrr)selfr0r$rrtransport_inforrrregister_transport_on_actorszOregister_custom_tensor_transports_on_actor..register_transport_on_actor _ray_system)concurrency_group)r#rrr __ray_call__optionsremoter)r/r5rrr*register_custom_tensor_transports_on_actorus   r;devicez torch.devicetensor_transportcCs&|tvr td||jt|jvS)z*Check if the device matches the transport.r,)rr rr)r<r=rrrdevice_match_transportsr>cCs"|}|tvrtd||S)NzInvalid tensor transport: )rrr )r=rrr'normalize_and_validate_tensor_transportsr?ray_usage_funccCs(t|jstd|d|ddS)Nz*Trying to use two-sided tensor transport: z for zS. This is only supported for one-sided transports such as NIXL or the OBJECT_STORE.)rr is_one_sidedr )r=r@rrrvalidate_one_sidedsrBtensor_transport_metaz torch.TensorcCsDddl}g}|j}|jD]}|\}}|j|||d}||q |S)Nr)dtyper<)torch tensor_device tensor_metaemptyappend)rCrEtensorsr<metashaperDtensorrrr"create_empty_tensors_from_metadatas  rN)) threadingtypingrrrrrray ray._rayletr?ray.experimental.gpu_object_manager.collective_tensor_transportrr 6ray.experimental.gpu_object_manager.cuda_ipc_transportr 9ray.experimental.gpu_object_manager.nixl_tensor_transportr r?rBrNrrrrs^      &  "