o TiVq @s"ddlZddlZddlZddlZddlmZddlmZddlm Z m Z ddl m Z ddl mZddlmZmZddlmZdd lmZddlZdd lmZdd lmZdd lmZmZmZmZm Z erkdd l!m"Z"ertddl#m$Z$er{ddl%Z%erddl&m'Z'e rddl(m)Z)m*Z*ddl+m,Z,ddl-m.Z.ddl/m0Z0ddl1m2Z2ddl3m4Z4erddl5m6Z,e7e8Z9dej:d<GdddZ;e GdddZd$e?d%e=d!e?e?fd&d'Z@de) rdrerfrgllmcommand method_namertrurpresultr4r4r5 llm_worker5sJ        rlstncs,tt|\fddt|DS)aI Split list `lst` into `n` evenly distributed sublists. Example: ```python >>> chunk_list([1, 2, 3, 4, 5, 6], 2) [[1, 2, 3], [4, 5, 6]] >>> chunk_list([1, 2, 3, 4, 5, 6], 4) [[1, 2], [3, 4], [5], [6]] >>> chunk_list([1, 2, 3, 4, 5, 6], 8) [[1], [2], [3], [4], [5], [6], [], []] ``` cs<g|]}|t||dt|dqS)r()min).0ikrrr4r5 xs<zchunk_list..)divmodlenrange)rrr4rr5 chunk_listgsrcststdtstdtstdtstdt}ggtjD]}t\}}t t |||fd}|  | |q(t dtffdd }t|d }|d d d }|d fdd} Gdddt} Gdddt} |jd| dd| ffdd } Gdddt} |dd| ffdd }Gdddt}|d d|ffd!d" }|d#fd$d%}|d&fd'd(}tj|jjjd)dS)*Nz`FastAPI is required to run the vLLM serve script. Please install it using `pip install fastapi`.zbPydantic is required to run the vLLM serve script. Please install it using `pip install pydantic`.z`Uvicorn is required to run the vLLM serve script. Please install it using `pip install uvicorn`.zZvLLM is required to run the vLLM serve script. Please install it using `pip install vllm`.)targetrtappcst}t|jkr,D]}|}t|tr$|ddkr$||q t|jks dVD]}|jdd| rNt d|d| |q1dS)Nrnro )timeoutzProcess z< is still alive after 10 seconds, attempting to terminate...) setrrUrz isinstancedictr}addjoinis_aliveloggerwarning terminate)rready_connectionsrgmsgprocess connections processesrdr4r5lifespans$  zmain..lifespan)rz/health/cs ddiS)zM Health check endpoint to verify that the server is running. rnokr4r4r4r4r5healthszmain..healthz/get_world_size/csdjjiS)a" Retrieves the world size of the LLM engine, which is `tensor_parallel_size * data_parallel_size`. Returns: `dict`: A dictionary containing the world size. Example response: ```json {"world_size": 8} ``` r#)rTrUr4)rdr4r5get_world_sizeszmain..get_world_sizec@seZdZUeeed<dZeeeed<dZe ed<dZ e ed<dZ e ed<dZ e ed <d Ze ed <d Ze ed <dZe ed<dZeeed<eedZeed<dS)zmain..GenerateRequestpromptsNimagesr(rg?repetition_penalty temperaturetop_pr;top_kgmin_p max_tokensguided_decoding_regex)default_factorygeneration_kwargs)rJrKrLlistr+rarr rrNrrbrrrrrrrrrr4r4r4r5GenerateRequests        rc@seZdZUeeeed<dS)zmain..GenerateResponsecompletion_idsN)rJrKrLrrNrar4r4r4r5GenerateResponses rz /generate/)response_modelrequestc sT|jp dgt|j|_g}t|j|jD]\}}d|i}|dur0dttt|i|d<| |q|j durCt d|j d}nd}|j |j |j|j|j|j|j|d}||jtdi|}t|j}t|D]\} }|sxdg}||d } | d d | d qod dD} ddt| |D} tt| } dd| D} d| iS)a` Generates completions for the provided prompts. Args: request (`GenerateRequest`): - `prompts` (list of `str`): A list of prompts (text strings) for the model to generate completions. - `images` (list of `str`, *optional*, default to `None`): A list of base64 encoded images to process along with prompts. - `n` (`int`, *optional*, defaults to `1`): Number of completions to generate for each prompt. - `repetition_penalty` (`float`, *optional*, defaults to `1.0`): Repetition penalty to apply during generation. - `temperature` (`float`, *optional*, defaults to `1.0`): Temperature for sampling. Higher values lead to more random outputs. - `top_p` (`float`, *optional*, defaults to `1.0`): Top-p (nucleus) sampling parameter. It controls the diversity of the generated text. - `top_k` (`int`, *optional*, defaults to `-1`): Top-k sampling parameter. If set to `-1`, it disables top-k sampling. - `min_p` (`float`, *optional*, defaults to `0.0`): Minimum probability threshold for sampling. - `max_tokens` (`int`, *optional*, defaults to `16`): Maximum number of tokens to generate for each completion. - `guided_decoding_regex` (`str`, *optional*): A regex pattern for guided decoding. If provided, the model will only generate tokens that match this regex pattern. - `generation_kwargs` (`dict`, *optional*): Additional generation parameters to pass to the vLLM `SamplingParams`. This can include parameters like `seed`, `frequency_penalty`, etc. If it contains keys that conflict with the other parameters, they will override them. Returns: `GenerateResponse`: - `completion_ids` (list of list of `int`): A list of lists of token IDs for each generated completion. Example request: ```json {"prompts": ["Hello world", "What is AI?"]} ``` Example response: ```json {"completion_ids": [[101, 102, 103], [201, 202, 203]]} ``` Npromptimagemulti_modal_dataoutlines)backendregex)rrrrrrrguided_decodingz )rsampling_paramsrrgeneraterqrprucSg|]}|qSr4rzrrgr4r4r5r"z*main..generate..cSsg|]\}}|r|qSr4r4)routputrr4r4r5r%scSs"g|] }|jD]}t|jqqSr4)outputsr token_ids)rrrr4r4r5r)s"rr4)rrrzipropenrbase64 b64decodeappendrrrrrrrrrupdaterrrrUryrr from_iterable) rrrrrowrrrchunked_promptsrgru all_outputsrrrdr4r5rsB!     zmain..generatec@s.eZdZUeed<eed<eed<eed<dS)z%main..InitCommunicatorRequestr!r"r#r$N)rJrKrLr+rarNr4r4r4r5InitCommunicatorRequest,s  rz/init_communicator/csNjjd}d|j|j||jfd}D] }|dd|dqddiS) ae Initializes the communicator for synchronizing model weights between a client and multiple server workers. Args: request (`InitCommunicatorRequest`): - `host` (`str`): Hostname or IP address of the master node. - `port` (`int`): Port number to be used for communication. - `world_size` (`int`): Total number of participating processes in the group. - `client_device_uuid` (`str`): UUID of the device of client main process. Used to assert that devices are different from vLLM workers devices. r(r6rprtrsr|rmessagez+Request received, initializing communicator)rTrUr!r"r$ry)rr#rurgrr4r5r62s zmain..init_communicatorc@s*eZdZUeed<eed<eeed<dS)z"main..UpdateWeightsRequestr7r8r9N)rJrKrLr+rarrNr4r4r4r5UpdateWeightsRequestMs rz/update_named_param/cs@d|j|jt|jfd}D] }|dd|dqddiS)a Updates the model weights with the provided tensor. Once this endpoint is called, the client process should broadcast the updated weights to all server workers. Args: request (`UpdateWeightsRequest`): - `name` (`str`): Name of the weight tensor being updated. - `dtype` (`str`): Data type of the weight tensor (e.g., `"torch.float32"`). - `shape` (list of `int`): Shape of the weight rHrrsr|rrz*Request received, updating named parameter)r7r8tupler9ry)rrurgrr4r5rHRs z main..update_named_paramz/reset_prefix_cache/csLD] }|dddqddD}tdd|D}dd t|iS) z8 Resets the prefix cache for the model. rrreset_prefix_cache)rqrpcSrr4rrr4r4r5rqrz4main..reset_prefix_cache..css|]}|VqdSNr4)rrr4r4r5 rsz3main..reset_prefix_cache..rz1Request received, resetting prefix cache status: )ryallr+)rgrsuccessrr4r5ris z main..reset_prefix_cachez/close_communicator/cs.ddi}D] }|dd|dqddiS)zT Closes the weight update group and cleans up associated resources. rprIrsr|rrz&Request received, closing communicator)ry)rurgrr4r5rIus z main..close_communicator)r!r"r^)r ImportErrorrrrrrrUrr rstartrrrr}rpostuvicornrunr!r"r^)rdrfreparent_connectionchild_connectionrrrrrrrrrr6rrHrrIr4rr5main{sZ      T  r subparserscCs(|dur|jddtd}|Stt}|S)Nz vllm-servezRun the vLLM serve script)rPdataclass_types) add_parserrOr )rparserr4r4r5 make_parsers r__main__r)Gargparserloggingrwcollections.abcr contextlibr dataclassesrrior itertoolsrmultiprocessingrr multiprocessing.connectionr typingr r, transformersr trlr trl.import_utilsrrrrrfastapirpydanticrrPILrr_rr,vllm.distributed.device_communicators.pyncclrvllm.distributed.parallel_statervllm.distributed.utilsrvllm.sampling_paramsr vllm.utilsr3vllm_ascend.distributed.device_communicators.pyhcclr getLoggerrJrrxr rOrNrrrr_SubParsersActionrrparse_args_and_configrdr4r4r4r5sx                   Z 2