o i}@sUddlZddlZddlZddlmZddlmZmZddlm Z ddl m Z m Z ddl mZmZddlmZddlZddlZddlmZdd lmZdd lmZmZmZmZdd lmZdd lmZdd l m!Z!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(m)Z)ddl*m+Z+ddl,m-Z-m.Z.m/Z/ddl0m1Z1ddl2m3Z3ddl4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:ddl;mm?Z?m@Z@ddlAmBZBddlCmDZDmEZEmFZFmGZGddlHmIZIddlJmKZKddlLmMZMddlNmOZOddlPmQZQddlRmSZSdd lTmUZVeKeWZXGd!d"d"e5ZYGd#d$d$e8ZZe(e?BeFBeDBeYBeZBZ[ee\d%<Gd&d'd'e/Z]Gd(d)d)e/Z^Gd*d+d+e/Z_d,eSfd-d.Z`d/d0Zad1ZbGd2d3d3Zcd4edd5edfd6d7Zed8edd9efe_d5dfd:d;Zgdehd5dfd?d@Zid4edd9efe_dAedd5dfdBdCZjdDedd5ekfdEdFZldGe]dHedd5e_fdIdJZmdGe]dHedd5e_fdKdLZndMedGe]dNecd5e_fdOdPZoeegefZpee\dQ< dndGe]dNecdReedgehfdSegedBfdTepdBd5ee_dBf dUdVZqdWehd5epfdXdYZrdZe$d[ed\efe1d]e&dBd^eseOd_fd5etedetedefff d`daZudbdcZvdZe$d[ed5dfdddeZwd[efdfdgZxeWdhkrHeaZyeXzdieVeXzdjeyeyj{r:eXzdkeeyj|eyj}dlneXzdme~exeydSdS)oN) Namespace) AwaitableCallable) HTTPStatus)BytesIOStringIO)Any TypeAlias)urlparse) UploadFile)start_http_server)Field TypeAdapterfield_validatormodel_validator)ValidationInfo)tqdm)AsyncEngineArgs optional_type) EngineClient) RequestLogger)ChatCompletionRequestChatCompletionResponse)OpenAIServingChat) ErrorInfo ErrorResponseOpenAIBaseModel) BaseModelPath)OpenAIServingModels)TranscriptionRequestTranscriptionResponseTranscriptionResponseVerboseTranslationRequestTranslationResponseTranslationResponseVerbose)OpenAIServingTranscriptionOpenAIServingTranslation)EmbeddingRequestEmbeddingResponse)OpenAIServingEmbedding) RerankRequestRerankResponse ScoreRequest ScoreResponse) ServingScores) init_logger)ReasoningParserManager) SupportedTask) random_uuid)FlexibleArgumentParser) __version__c@ZeZdZUdZedddZeed<edddZe dBed <e d d e d e fd dZ dS)BatchTranscriptionRequestz Batch transcription request that uses file_url instead of file. This class extends TranscriptionRequest but replaces the file field with file_url to support batch processing from audio files written in JSON format. .HEither a URL of the audio or a data URL with base64 encoded audio data.  descriptionfile_urlNTdefaultexcludefilebeforemodedatacCt|tr d|vr td|Sz4Ensure file field is not provided in batch requests.r>zLThe 'file' field is not supported in batch requests. Use 'file_url' instead. isinstancedict ValueErrorclsrBrKW/home/ubuntu/vllm_env/lib/python3.10/site-packages/vllm/entrypoints/openai/run_batch.pyvalidate_no_fileY z*BatchTranscriptionRequest.validate_no_file__name__ __module__ __qualname____doc__r r:str__annotations__r>r r classmethodrrMrKrKrKrLr6G r6c@r5)BatchTranslationRequestz Batch translation request that uses file_url instead of file. This class extends TranslationRequest but replaces the file field with file_url to support batch processing from audio files written in JSON format. .r7r8r:NTr;r>r?r@rBcCrCrDrErIrKrKrLrMwrNz(BatchTranslationRequest.validate_no_filerOrKrKrKrLrXerWrXBatchRequestInputBodyc@sTeZdZUdZeed<eed<eed<eed<edddede d e fd d Z d S) BatchRequestInputz The per-line object of the batch input file. NOTE: Currently only the `/v1/chat/completions` endpoint is supported. custom_idmethodurlbodyplainr@valueinfocCs|jd}|dkrt|S|dkrtt|S|dr%tt|S|dr/t|S|dkr8t |S|dkrAt |Stt |S)Nr]/v1/chat/completions/v1/embeddings/score/rerank/v1/audio/transcriptions/v1/audio/translations) rBrmodel_validaterr'validate_pythonendswithr,r*r6rXrY)rJr`rar]rKrKrLcheck_type_for_urls       z$BatchRequestInput.check_type_for_urlN) rPrQrRrSrTrUrYrrVrrrkrKrKrKrLrZs  rZc@sNeZdZUdZeed<eed<dZee Be Be Be Be BeBeBdBed<dS)BatchResponseData status_code request_idNr^)rPrQrRrnintrUrTr^rr(r-r+r r!r#r$rKrKrKrLrls.   rlc@s:eZdZUdZeed<eed<edBed<edBed<dS)BatchRequestOutputzA The per-line object of the batch output and error files idr[Nresponseerror)rPrQrRrSrTrUrlrrKrKrKrLrqs  rqparsercCs|jdddtdd|jdddtdd|jd tdd d |jd ttd dd t|}|jdtddd |jdddd|jdtddd |jdtddd |jddddd|jdddd d|S)!Nz-iz --input-fileTzThe path or url to a single input file. Currently supports local file paths, or the http protocol (http or https). If a URL is specified, the file should be available via HTTP GET.)requiredtypehelpz-oz --output-filezThe path or url to a single output file. Currently supports local file paths, or web (http or https) urls. If a URL is specified, the file should be available via HTTP PUT.z--output-tmp-dirzMThe directory to store the output file before uploading it to the output URL.)rwr<rxz--response-role assistantz@The role name to return if `request.add_generation_prompt=True`.z --max-log-lenz^Max number of prompt characters or prompt ID numbers being printed in log. Default: Unlimitedz--enable-metrics store_truezEnable Prometheus metrics)actionrxz--urlz0.0.0.0zLURL to the Prometheus metrics server (only needed if enable-metrics is set).z--porti@zUPort number for the Prometheus metrics server (only needed if enable-metrics is set).z--enable-prompt-tokens-detailsFz6If set to True, enable prompt_tokens_details in usage.)r{r<rxz--enable-force-include-usagezZIf set to True, include usage on every request (even when stream_options is not specified)) add_argumentrTrr add_cli_argsrprurKrKrLmake_arg_parsersz    rcCstdd}t|S)Nz$vLLM OpenAI-Compatible batch runner.r8)r3r parse_argsr~rKrKrLr,s  rz_{desc}: {percentage:3.0f}% Completed | {n_fmt}/{total_fmt} [{elapsed}<{remaining}, {rate_fmt}] c@s2eZdZddZddZddZdefdd Zd S) BatchProgressTrackercCsd|_d|_dS)Nr)_total_pbarselfrKrKrL__init__9s zBatchProgressTracker.__init__cCs|jd7_dS)N)rrrKrKrL submitted=szBatchProgressTracker.submittedcCs|jr |jdSdSN)rupdaterrKrKrL completed@szBatchProgressTracker.completedreturncCs:tj p tjdk}t|jddd| td|_|jS)Nrreqz Running batch)totalunitdesc minintervaldisable bar_format)torch distributedis_initializedget_rankrr _BAR_FORMATr)r enable_tqdmrKrKrLpbarDszBatchProgressTracker.pbarN)rPrQrRrrrrrrKrKrKrLr8s r path_or_urlrc s|ds |dr\t4IdH<}||4IdH}|IdHWdIdHWdIdHS1IdHs?wYWdIdHdS1IdHsUwYdSt|dd }|WdS1sqwYdS)Nhttp://https://utf-8encoding) startswithaiohttp ClientSessiongettextopenread)rsessionrespfrKrKrL read_fileSs& p$r output_path batch_outputscsPt|ddd}|D] }t||dq WddS1s!wYdS)z Write the responses to a local file. output_path: The path to write the responses to. batch_outputs: The list of batch outputs to write. wrrr>N)rprintmodel_dump_json)rrrorKrKrLwrite_local_file\s  "r output_url data_or_file from_filec sd}d}td|dD]}ztjtjddd4IdH}|rit|d;}|j||d4IdH}|jd krDtd |jd |WdIdHn 1IdHsTwYWdn1scwYn3|j||d4IdH}|jd krtd |jd |WdIdHn 1IdHswYWdIdHn 1IdHswYWq ty} z)||krt d || |t |IdHntd|dt | d| WYd} ~ q d} ~ wwdS)z Upload a local file to a URL. output_url: The URL to upload the file to. data_or_file: Either the data to upload or the path to the file to upload. from_file: If True, data_or_file is the path to the file to upload. rri)r)timeoutNrb)rBrmzFailed to upload file. Status: z Response: zFailed to upload data. Status: zPFailed to upload data (attempt %d). Error message: %s. Retrying in %d seconds...zFailed to upload data (attempt z). Error message: .)rangerr ClientTimeoutrputstatus ExceptionrloggerrtasynciosleeprT) rrr max_retriesdelayattemptrr>rserKrKrL upload_dataksl    ( *( routput_tmp_dircs |ds |dr|durBtdt}|D] }t||dq|dtd|t|| dd d IdHdSt j d d|d d d*}td|j t|j |IdHtd|t||j dd IdHWddS1sywYdStd|t||IdHdS)a Write batch_outputs to a file or upload to a URL. path_or_url: The path or URL to write batch_outputs to. batch_outputs: The list of batch outputs to write. output_tmp_dir: The directory to store the output file before uploading it to the output URL. rrNz Writing outputs to memory bufferrrzUploading outputs to %srF)rrtmp_batch_output_z.jsonl)rArdirprefixsuffixz*Writing outputs to temporary local file %sTz Writing outputs to local file %s)rrrarrrseekrrstripencodetempfileNamedTemporaryFilenamer)rrr output_bufferrrrKrKrL write_files:     " rr]c s8t|}|jdkr-d|vr&|dd\}}d|vrt|Std|td||jdvrt4IdHL}||4IdH,}|j d krUt d |d |j | IdHWdIdHWdIdHS1IdHsvwYWdIdHdS1IdHswYdStd |jd )z Download data from a URL or decode from a data URL. Args: url: Either an HTTP/HTTPS URL or a data URL (data:...;base64,...) Returns: Data as bytes rB,rbase64zUnsupported data URL encoding: zInvalid data URL format: )httphttpsNrmz"Failed to download data from URL: z . Status: zUnsupported URL scheme: z&. Supported schemes: http, https, data) r schemesplitr b64decoderHrrrrrr)r]parsedheaderrBrrrKrKrLdownload_bytes_from_urls2       p rrequest error_msgcCs0tdt|jttjdtd|d}|S)Nvllm- vllm-batch-rnrorrr[rsrt)rqr2r[rlr BAD_REQUEST)rr batch_outputrKrKrLmake_error_request_outputs   rcs t||Sr)r)rrrKrKrLmake_async_error_request_outputs rserving_engine_functrackerc s||jIdH}t|tttttttt fr,t dt |j t |dt ddd}n#t|trIt dt |j t |jjdt d|d}nt|dd}||S)Nrr)r^rorrz'Request must not be sent in stream moder)r^rFrr(r-r+r r!r#r$rqr2r[rlrrtcoderr)rrrrsrrKrKrL run_request sJ      r WrapperFn url_matcherhandler_getter wrapper_fncCsX||jsdS|}|durd|j}t||dS|dur"||}|t|||S)a Generic handler for endpoint requests. Args: request: The batch request input tracker: Progress tracker for the batch url_matcher: Function that takes a URL and returns True if it matches handler_getter: Function that returns the handler function or None wrapper_fn: Optional function to wrap the handler (e.g., for transcriptions) Returns: Awaitable[BatchRequestOutput] if the request was handled, None if URL didn't match Nz!Model does not support endpoint: r)r]rrr)rrrrr handler_fnrrKrKrLhandle_endpoint_request;s    ris_translationcsdtffdd }|S)a Factory function to create a wrapper for transcription/translation handlers. The wrapper converts BatchTranscriptionRequest or BatchTranslationRequest to TranscriptionRequest or TranslationRequest and calls the appropriate handler. Args: is_translation: If True, process as translation; otherwise process as transcription Returns: A function that takes a handler and returns a wrapped handler rcs0dttBdttBtBtBtBffdd }|S)Nbatch_request_bodyrc sz9t|jIdH}tt|dd}|jdhd}||d<r-t|}||IdHWSt|}||IdHWStyf}z rFdnd}t t d|d t |d t j jd d WYd}~Sd}~ww) Nz audio.bin)r>filenamer:)r=r> translation transcriptionzFailed to process z: BadRequestError)messagerwr)rt)rr:r r model_dumpr"rhrrrrrTrrr`)r audio_data mock_file request_dicttranslation_requesttranscription_requestr operation)rrrKrLtranscription_wrapperos:  zJmake_transcription_wrapper..wrapper..transcription_wrapper)r6rXr r!r#r$r)rrr)rrLwrapperns.z+make_transcription_wrapper..wrapper)r)rrrKrrLmake_transcription_wrapper`s1r engine_clientargsbase_model_pathsrequest_loggersupported_tasks.c s||j}t||dd}d|vr%t|||j|dd|jj|j|jt|ddd ndd|vr4t |||dddndd |voBt|j d d d k}d|vsI|rQt |||dd ndd|vr`t ||||jdndd|vrot ||||jdndddfddddddfddddddfddddddfddddddfddtdddddfddtd ddd!}|S)"a Build the endpoint registry with all serving objects and handler configurations. Args: engine_client: The engine client args: Command line arguments base_model_paths: List of base model paths request_logger: Optional request logger supported_tasks: Tuple of supported tasks Returns: Dictionary mapping endpoint keys to their configurations N)rr lora_modulesgenerateautodefault_chat_template_kwargs)r  chat_templatechat_template_content_formatreasoning_parserenable_prompt_tokens_detailsenable_force_include_usagerembed)r rrclassify num_labelsrr)r score_templater)r rcS|dkS)NrbrKr]rKrKrLz)build_endpoint_registry..cdurjSdSr)create_chat_completionrK)openai_serving_chatrKrLrrrrcSr)NrcrKrrKrKrLrrcrr)create_embeddingrK)openai_serving_embeddingrKrLrrcS |dS)NrdrjrrKrKrLr crr) create_scorerKopenai_serving_scoresrKrLrrcSr#)Nrer$rrKrKrLr"r%crr) do_rerankrKr'rKrLr#rcSr)NrfrKrrKrKrLr+rcrr)create_transcriptionrK)openai_serving_transcriptionrKrLr,rFrcSr)NrgrKrrKrKrLr4rcrr)create_translationrK)openai_serving_translationrKrLr5rT) completions embeddingsscorereranktranscriptions translations) model_configrr response_rolestructured_outputs_configrrrgetattrr) hf_configr.r%r&r) rrrr r r4openai_serving_modelsenable_serving_rerankingendpoint_registryrK)rr"r(r+r-rLbuild_endpoint_registrys        9r<cCs@t}|jj}r||vrtd|dd|ddSdS)Nzinvalid reasoning parser: z (chose from { rz }))r0list_registeredr6rKeyErrorjoin)rvalid_reasoning_parsersrrKrKrLvalidate_run_batch_argsAs rAcsjdur j}njg}jrtjd}nd}fdd|D}|IdH}td|t||||d}t }tdj g}t j IdH dD]G} | } | s[qRt| } | j dd } d} | |vr|| } t| || d | d | d d } | dur|| qR|t| d| jddqR|tj|IdH}Wdn1swYtj|jIdHdS)N) max_log_lencsg|] }t|jdqS))r model_path)rmodel).0rrrKrL Zszrun_batch..zSupported tasks: %s)rrrr r zReading batch from %s... /rrrr zURL z was used. Supported endpoints: /v1/chat/completions, /v1/embeddings, /v1/audio/transcriptions, /v1/audio/translations, /score, /rerank. See vllm/entrypoints/openai/api_server.py for supported score/rerank versions.r)served_model_namerDenable_log_requestsrrBget_supported_tasksrrar<r input_filerrrrZmodel_validate_jsonr]rappendrrrgatherr output_filer)rrserved_model_namesr rr r;rresponse_futures request_jsonr endpoint_keyresultendpoint_config responsesrKrFrL run_batchLsd       rZc s|ddlm}ddlm}t||||jdd4IdH}t||IdHWdIdHdS1IdHs7wYdS)Nr)build_async_engine_client) UsageContextF) usage_context disable_frontend_multiprocessing)"vllm.entrypoints.openai.api_serverr[vllm.usage.usage_libr\rAOPENAI_BATCH_RUNNERrZ)rr[r\rrKrKrLmains  .rb__main__z$vLLM batch processing API version %szargs: %szPrometheus metrics enabled)portaddrzPrometheus metrics disabledr)rrrargparsercollections.abcrrrriorrtypingrr urllib.parser rrfastapir prometheus_clientr pydanticr rrrpydantic_core.core_schemarrvllm.engine.arg_utilsrrvllm.engine.protocolrvllm.entrypoints.loggerr0vllm.entrypoints.openai.chat_completion.protocolrr/vllm.entrypoints.openai.chat_completion.servingr'vllm.entrypoints.openai.engine.protocolrrr'vllm.entrypoints.openai.models.protocolr&vllm.entrypoints.openai.models.servingr/vllm.entrypoints.openai.speech_to_text.protocolrr r!r"r#r$.vllm.entrypoints.openai.speech_to_text.servingr%r&'vllm.entrypoints.pooling.embed.protocolr'r(&vllm.entrypoints.pooling.embed.servingr)'vllm.entrypoints.pooling.score.protocolr*r+r,r-&vllm.entrypoints.pooling.score.servingr. vllm.loggerr/vllm.reasoningr0 vllm.tasksr1 vllm.utilsr2vllm.utils.argparse_utilsr3 vllm.versionr4 VLLM_VERSIONrPrr6rXrYrUrZrlrqrrrrrTrlistrboolrrbytesrrrrrrrtuplerGr<rArZrbrraenable_metricsrdr]runrKrKrKrLs*                        *N   5 )+   /   %B     M