o ۾i@sdZddlmZmZddlmZddlmZmZm Z ddl m Z m Z m Z ddlZddlmZddlmZe d d d Ze d d d ZeGdd d eZeGdd d eZeGdddZeGdddZGdddee eefZdS)z'Base class for weight transfer engines.)ABCabstractmethod)Callable)KW_ONLY dataclassfield)AnyGenericTypeVarN)ParallelConfig)WeightTransferConfig TInitInfoWeightTransferInitInfo)bound TUpdateInfoWeightTransferUpdateInfoc@seZdZdZdS)rz4Base class for backend-specific initialization info.N)__name__ __module__ __qualname____doc__rrY/home/ubuntu/.local/lib/python3.10/site-packages/vllm/distributed/weight_transfer/base.pyrsc@s&eZdZUdZeed<dZeed<dS)rz3Base class for backend-specific weight update info._Tis_checkpoint_formatN)rrrrr__annotations__rboolrrrrrs  c@,eZdZUdZeedZeeefe d<dS)WeightTransferInitRequestz1API-level weight transfer initialization request.default_factory init_infoN) rrrrrdictr strrrrrrrr' rc@r)WeightTransferUpdateRequestz API-level weight update request.r update_infoN) rrrrrr!r%r"rrrrrrr$.r#r$c @seZdZUdZeeed<eeed<dede ddfdd Z d e e e fdefd d Zd e e e fdefddZededdfddZededeeee ejfgdfddfddZedddZdS)WeightTransferEnginea Base class for weight transfer engines that handle transport of model weights from a trainer to inference workers. This abstraction separates weight transfer transport logic from the worker implementation, allowing different backends (NCCL, CUDA IPC[TODO], RDMA[TODO]) to be plugged in. Subclasses should define: init_info_cls: Type of backend-specific initialization info update_info_cls: Type of backend-specific update info init_info_clsupdate_info_clsconfigparallel_configreturnNcCs||_||_dS)z Initialize the weight transfer engine. Args: config: The configuration for the weight transfer engine parallel_config: The configuration for the parallel setup N)r)r*)selfr)r*rrr__init__Gs zWeightTransferEngine.__init__ init_dictc CFz |jdi|WSty"}z td|jjd||d}~ww)aH Construct typed init info from dict with validation. Args: init_dict: Dictionary containing backend-specific initialization parameters Returns: Typed backend-specific init info dataclass Raises: ValueError: If init_dict is invalid for this backend zInvalid init_info for : Nr)r' TypeError ValueError __class__r)r,r.errrparse_init_infoT z$WeightTransferEngine.parse_init_info update_dictc Cr/)aH Construct typed update info from dict with validation. Args: update_dict: Dictionary containing backend-specific update parameters Returns: Typed backend-specific update info dataclass Raises: ValueError: If update_dict is invalid for this backend zInvalid update_info for r0Nr)r(r1r2r3r)r,r7r4rrrparse_update_infohr6z&WeightTransferEngine.parse_update_infor cCt)z Initialize the weight transfer mechanism. This is called once at the beginning of training. Args: init_info: Backend-specific initialization info NotImplementedError)r,r rrrinit_transfer_engine|s z)WeightTransferEngine.init_transfer_enginer% load_weightscCr9)az Receive weights from the trainer and load them incrementally. Args: update_info: Backend-specific update info containing parameter metadata and any backend-specific data load_weights: Callable that loads weights into the model. Called incrementally for each weight to avoid OOM. r:)r,r%r=rrrreceive_weightssz$WeightTransferEngine.receive_weightscCr9)zv Shutdown the weight transfer engine. This should be called when the worker is shutting down. r:)r,rrrshutdownszWeightTransferEngine.shutdown)r+N)rrrrtyper rrr r r-r!r"rr5r8rr<rlisttupletorchTensorr>r?rrrrr&5s2       r&)rabcrrcollections.abcr dataclassesrrrtypingrr r rCvllm.config.parallelr vllm.config.weight_transferr r rrrrr$r&rrrrs&