o -iT@s<UddlZddlZddlmZddlmZmZddlmZddl m Z ddl m Z m Z ddlZddlZddlmZddlmZmZdd lmZdd lmZdd lmZmZdd lmZdd lmZddlm Z m!Z!ddl"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-ddl.m/Z/ddl0m1Z1m2Z2m3Z3m4Z4ddl5m6Z6ddl7m8Z8ddl9m:Z:ddl;mZ>ddl?m@ZAe8eBZCe e,Be3Be1BZDe eEd<Gddde&ZFGdd d e&ZGGd!d"d"e&ZHd#e>fd$d%ZId&d'ZJd(ZKGd)d*d*ZLd+eMd,eMfd-d.ZNd/eMd0eOeHd,dfd1d2ZPd3eMd4eMd5eQd,dfd6d7ZRd+eMd0eOeHd8eMd,dfd9d:ZSd;eFdZTd;eFdeZdZUdZeed<eed<dZee Be Be BdBed<dS)BatchResponseData status_code request_idNr+) r<r=r>rEintrAr@r+rrrrr9r9r9r:rC^s   rCc@s:eZdZUdZeed<eed<edBed<edBed<dS)BatchRequestOutputzA The per-line object of the batch output and error files idr(Nresponseerror)r<r=r>r?r@rArCrr9r9r9r:rHos  rHparsercCs|jdddtdd|jdddtdd|jd tdd d |jd ttd dd t|}|jdtddd |jdddd|jdtddd |jdtddd |jddddd|jdddd d|S)!Nz-iz --input-fileTzThe path or url to a single input file. Currently supports local file paths, or the http protocol (http or https). If a URL is specified, the file should be available via HTTP GET.)requiredtypehelpz-oz --output-filezThe path or url to a single output file. Currently supports local file paths, or web (http or https) urls. If a URL is specified, the file should be available via HTTP PUT.z--output-tmp-dirzMThe directory to store the output file before uploading it to the output URL.)rNdefaultrOz--response-role assistantz@The role name to return if `request.add_generation_prompt=True`.z --max-log-lenz^Max number of prompt characters or prompt ID numbers being printed in log. Default: Unlimitedz--enable-metrics store_truezEnable Prometheus metrics)actionrOz--urlz0.0.0.0zLURL to the Prometheus metrics server (only needed if enable-metrics is set).z--porti@zUPort number for the Prometheus metrics server (only needed if enable-metrics is set).z--enable-prompt-tokens-detailsFz6If set to True, enable prompt_tokens_details in usage.)rSrPrOz--enable-force-include-usagezZIf set to True, include usage on every request (even when stream_options is not specified)) add_argumentr@rr add_cli_argsrGrLr9r9r:make_arg_parsersz    rWcCstdd}t|S)Nz$vLLM OpenAI-Compatible batch runner.) description)r$rW parse_argsrVr9r9r:rYs  rYz_{desc}: {percentage:3.0f}% Completed | {n_fmt}/{total_fmt} [{elapsed}<{remaining}, {rate_fmt}] c@s2eZdZddZddZddZdefdd Zd S) BatchProgressTrackercCsd|_d|_dS)Nr)_total_pbarselfr9r9r:__init__s zBatchProgressTracker.__init__cCs|jd7_dS)N)r[r]r9r9r: submittedszBatchProgressTracker.submittedcCs|jr |jdSdSN)r\updater]r9r9r: completedszBatchProgressTracker.completedreturncCs:tj p tjdk}t|jddd| td|_|jS)Nrreqz Running batch)totalunitdesc minintervaldisable bar_format)torch distributedis_initializedget_rankr r[ _BAR_FORMATr\)r^ enable_tqdmr9r9r:pbarszBatchProgressTracker.pbarN)r<r=r>r_rardr rtr9r9r9r:rZs rZ path_or_urlrec s|ds |dr\t4IdH<}||4IdH}|IdHWdIdHWdIdHS1IdHs?wYWdIdHdS1IdHsUwYdSt|dd }|WdS1sqwYdS)Nhttp://https://utf-8encoding) startswithaiohttp ClientSessiongettextopenread)rusessionrespfr9r9r: read_files& p$r output_path batch_outputscsPt|ddd}|D] }t||dq WddS1s!wYdS)z Write the responses to a local file. output_path: The path to write the responses to. batch_outputs: The list of batch outputs to write. wrxryfileN)rprintmodel_dump_json)rrror9r9r:write_local_files  "r output_url data_or_file from_filec sd}d}td|dD]}ztjtjddd4IdH}|rit|d;}|j||d4IdH}|jd krDtd |jd |WdIdHn 1IdHsTwYWdn1scwYn3|j||d4IdH}|jd krtd |jd |WdIdHn 1IdHswYWdIdHn 1IdHswYWq ty} z)||krt d || |t |IdHntd|dt | d| WYd} ~ q d} ~ wwdS)z Upload a local file to a URL. output_url: The URL to upload the file to. data_or_file: Either the data to upload or the path to the file to upload. from_file: If True, data_or_file is the path to the file to upload. rgr`i)rh)timeoutNrb)r4rDzFailed to upload file. Status: z Response: zFailed to upload data. Status: zPFailed to upload data (attempt %d). Error message: %s. Retrying in %d seconds...zFailed to upload data (attempt z). Error message: .)ranger|r} ClientTimeoutrputstatus ExceptionrloggerrKasynciosleepr@) rrr max_retriesdelayattemptrrrJer9r9r: upload_datasl    ( *( routput_tmp_dircs |ds |dr|durBtdt}|D] }t||dq|dtd|t|| dd d IdHdSt j d d|d d d*}td|j t|j |IdHtd|t||j dd IdHWddS1sywYdStd|t||IdHdS)a Write batch_outputs to a file or upload to a URL. path_or_url: The path or URL to write batch_outputs to. batch_outputs: The list of batch outputs to write. output_tmp_dir: The directory to store the output file before uploading it to the output URL. rvrwNz Writing outputs to memory bufferrrzUploading outputs to %srxF)rrtmp_batch_output_z.jsonl)r-rzdirprefixsuffixz*Writing outputs to temporary local file %sTz Writing outputs to local file %s)r{rr/rrrseekrrstripencodetempfileNamedTemporaryFilenamer)rurr output_bufferrrr9r9r: write_fileCs:     " rrequest error_msgcCs0tdt|jttjdtd|d}|S)Nvllm- vllm-batch-rErFrIr(rJrK)rHr#r(rCr BAD_REQUEST)rr batch_outputr9r9r:make_error_request_outputls   rcs t||Srb)r)rrr9r9r:make_async_error_request_output{s rserving_engine_functrackercs||jIdH}t|ttttfr(tdt|jt |dtddd}n#t|t rEtdt|jt |j j dtd|d}nt |dd}||S)Nrr)r+rFrrz'Request must not be sent in stream moder)r+ isinstancerrrrrHr#r(rCrrKcoderrd)rrrrJrr9r9r: run_requests:       rcCs@t}|jj}r||vrtd|dd|ddSdS)Nzinvalid reasoning parser: z (chose from { ,z }))r"list_registeredstructured_outputs_configreasoning_parserKeyErrorjoin)argsvalid_reasoning_parsersrr9r9r:validate_run_batch_argss r engine_clientrcs&jdur j}njg}jrtjd}nd}fdd|D}|j}|IdH}td|t ||dd}d|vrUt ||j |ddj j jjtddd nd}d |vrdt|||ddd nd} d |vort|jd ddk} d |vsy| rt|||ddnd} t} tdjg} tjIdHdD]}|}|sqt|}|jdkr|dur|jnd}|dur| t|ddq| t||| | q|jdkr| dur| j!nd}|dur| t|ddq| t||| | q|j"dr,| dur | j#nd}|dur| t|ddq| t||| | q|j"drZ| dur;| j$nd}|durL| t|ddq| t||| | q| t|d|jddq| %t&j'| IdH}Wdn 1swYt(j)|j*IdHdS)N) max_log_lencsg|] }t|jdqS))r model_path)rmodel).0rrr9r: szrun_batch..zSupported tasks: %s)rbase_model_paths lora_modulesgenerateautodefault_chat_template_kwargs)request_logger chat_templatechat_template_content_formatrenable_prompt_tokens_detailsenable_force_include_usagerembed)rrrclassify num_labelsrr`)rscore_templatezReading batch from %s... r0z/The model does not support Chat Completions APIrr1z)The model does not support Embeddings APIr2z%The model does not support Scores APIr3z%The model does not support Rerank APIzURL z was used. Supported endpoints: /v1/chat/completions, /v1/embeddings, /score, /rerank .See vllm/entrypoints/openai/api_server.py for supported score/rerank versions.)+served_model_namerenable_log_requestsrr model_configget_supported_tasksrr/rr response_rolerrrrgetattrr hf_configr rZ input_filerrsplitr'model_validate_jsonr*create_chat_completionappendrrracreate_embeddingr7 create_score do_rerankrtrgatherr output_filer)rrserved_model_namesrrrsupported_tasksopenai_serving_modelsopenai_serving_chatopenai_serving_embeddingenable_serving_rerankingopenai_serving_scoresrresponse_futures request_jsonrchat_handler_fnembed_handler_fnscore_handler_fnrerank_handler_fn responsesr9rr: run_batchs                   rc s|ddlm}ddlm}t||||jdd4IdH}t||IdHWdIdHdS1IdHs7wYdS)Nr)build_async_engine_client) UsageContextF) usage_context disable_frontend_multiprocessing)"vllm.entrypoints.openai.api_serverrvllm.usage.usage_librrOPENAI_BATCH_RUNNERr)rrrrr9r9r:mainbs  .r__main__z$vLLM batch processing API version %szargs: %szPrometheus metrics enabled)portaddrzPrometheus metrics disabled)`rrargparsercollections.abcrrhttpriortypingrrr|rnprometheus_clientr pydanticr r pydantic_core.core_schemar r vllm.engine.arg_utilsrrvllm.engine.protocolrvllm.entrypoints.loggerr0vllm.entrypoints.openai.chat_completion.protocolrr/vllm.entrypoints.openai.chat_completion.servingr'vllm.entrypoints.openai.engine.protocolrr'vllm.entrypoints.openai.models.protocolr&vllm.entrypoints.openai.models.servingr'vllm.entrypoints.pooling.embed.protocolrr&vllm.entrypoints.pooling.embed.servingr'vllm.entrypoints.pooling.score.protocolrrrr&vllm.entrypoints.pooling.score.servingr vllm.loggerr!vllm.reasoningr" vllm.utilsr#vllm.utils.argparse_utilsr$ vllm.versionr% VLLM_VERSIONr<rr&rAr'rCrHrWrYrrrZr@rlistrboolrrrrrrrrrr/enable_metricsrr*runr9r9r9r:s                    &N   5 )   &  1