o ۷i0@s ddlZddlZddlZddlZddlmZddlZddlZddlZddl Z ddl m Z ddl m Z ddlmZddlmZddlmZmZmZddlZddlmZddlmZmZmZmZmZmZm Z ddl!m"Z"m#Z#dd l$m%Z%dd l&m'Z'm(Z(dd l)m*Z*dd l+m,Z,dd l-m.Z.ddl/m0Z0ddl1m2Z2ddl3m4Z4ddl5m6Z6ddl7m8Z8ddl9m:Z:m;Z;mZ?ddl=m@ZAzddlBmCZCWneDyddlEmCZCYnwddlFmGZGmHZHddlImJZJddlKmLZLmMZMmNZNmOZOddlPmQZQddlRmSZSddlTmUZUddlVmWZWddlXmYZYddlZm[Z[m\Z\dd l]m^Z^dd!l_m`Z`dd"lambZbdd#lcmdZddd$lemfZfdd%lgmhZhdd&limjZjdd'lkmlZlmmZmmnZndd(lompZpdd)lqmrZrdd*lsmtZtdd+lumvZvdd,lwmxZxdd-lymzZzm{Z{dd.l|m}Z}dd/l~mZmZmZdd0lmZmZdd1lmZdd2lmZdd3lmZdd4lmZmZmZdd5lmZdd6lmZepeZeZeZd7e d8efd9d:ZGd;d<ded?eedBd8dfd@dAZdBZdd>ed?eedBfdCdDZGdEdFdFZddGdHZdddIdJZedddKd7e dLedBdMeeefdBd8e e0fdNdOZedPdQd7e dLed8e e0fdRdSZdTe0dUe*d7e d8dfdVdWZdXed8edBfdYdZZdXed8edBfd[d\ZdXed8edBfd]d^Zejd_ee^gejjd`daiiiejjdbeLiejjdbeLiejjdbeLiidceneldXeGddefdedfZeedgdhhejdgee^gejjd`diiiiejjdbeLiejjdbeLiejjdbeLiidceneldXe}ddefdjdkZejdlejjdbeiejjdbeLiejjdbeLiejjdbeLiidmddefdndoZeedpedpdded8e"fdqdrZeedsedsdded8e"fdtduZejdvee^gejjdbeiejjdbeLiejjdbeLiejjdbeLiidcdXedded8efdwdxZejdyejjdbeiejjdbeLiejjdbeLiejjdbeLiidmededdzd{ededd|d{ed}eded~ededededdededededededededfddedee dBdee dBdeedBdeedBdedbedededededBdedBdeeedddfdedBdedBdedBdedBdedBdedBdedBdedBd8ef,ddZddefddZddZdeeeffddZdTexeBdedeefddZdededed8dfddZded8eefddZdeed8ee%j%fddZdedBdedBd8efddZ dde%j%deded8efddZdedBdeded8dfddZeddddedXedBfddZeddddedXedBfddZdddXeddededBd8efdd„ZdedBd8efddńZejdejjdbeiejjdbeLiejjdbeLiejjdbeLiidmed}eddǍeddǍeddǍeddǍeddǍeddǍeddǍeddǍeddǍeddǍeddǍeddǍeddǍeddǍeddǍeddǍeddǍeddǍeddǍeddǍfddedede dBdbedBdedBdedBdedBdedBdedBdedBdedBdedBdedBdedBdedBdedBdedBdedBdedBdedBdedBdedBd8ef.dd҄ZdS)N) Namespace) AsyncIterator)asynccontextmanager) HTTPStatus) AnnotatedAnycast) APIRouterDependsFileForm HTTPExceptionRequest UploadFile) JSONResponseStreamingResponse)Image) BaseModelField)State)Route)SamplingParams) EngineClient)AnthropicServingMessages)load_chat_template) serve_http) RequestLogger)DemoToolServer MCPToolServer ToolServer) build_app) setup_server)base)ChatCompletionRequestChatCompletionResponse)OpenAIServingCompletion) ErrorResponse ModelCard ModelListModelPermission) BaseModelPath)OpenAIServingModels)metrics_header)OpenAIServingResponses)get_uvicorn_log_config)OpenAIServingTranscriptionOpenAIServingTranslation)validate_json_request)ServingClassification)OpenAIServingEmbedding)OpenAIServingPooling) ServingScores) ServingTokens)OpenAIServingTokenization)load_aware_callprocess_lora_moduleswith_cancellation) init_logger POOLING_TASKS)ToolParserManager) decorate_logs) AsyncOmni)encode_image_base64 parse_size)OpenAICreateSpeechRequest) ImageDataImageGenerationRequestImageGenerationResponse)VideoGenerationRequestVideoGenerationResponse)OmniOpenAIServingChat)OmniOpenAIServingSpeech)OmniOpenAIServingVideo)OmniDiffusionSamplingParamsOmniSamplingParamsOmniTextPrompt) LoRARequest)stable_lora_int_idargsreturncCs@t|dd}|durt|dd}|durdStjd}|duS)Nprofiler_configprofilerTVLLM_TORCH_PROFILER_DIR)getattrosenvironget)rQrSrT env_valuer[]/home/ubuntu/vllm_env/lib/python3.10/site-packages/vllm_omni/entrypoints/openai/api_server.py!_should_enable_profiler_endpointsps   r]c@s.eZdZUdZedddZeedBed<dS)ProfileRequestz&Request model for profiling endpoints.Nz;List of stage IDs to profile. If None, profiles all stages.)default descriptionstages) __name__ __module__ __qualname____doc__rralistint__annotations__r[r[r[r\r^~s r^routerpathmethodscCsv|r dd|Dnd}t|jD](}t|dd|krq|dur2ddt|ddp*tD}||@s2q|j|qdS)NcSh|]}|qSr[upper.0methodr[r[r\ z,_remove_route_from_router..rjcSrlr[rmror[r[r\rrrsrk)rfroutesrVsetremove)rirjrk methods_setroute route_methodsr[r[r\_remove_route_from_routersrzzendpoint-load-metrics-formatcCsdg}|jD]}t|tr$|j|kr$|dust|dr$|j|@r$||q|D]}|j|q'dS)zRemove a route from the app by path and optionally by methods. OMNI: used to override upstream /v1/chat/completions with omni behavior. Nrk)rt isinstancerrjhasattrrkappendrv)apprjrkroutes_to_removerxr[r[r\_remove_route_from_apps  rc@s4eZdZdZdeeddfddZdefddZdS) _DiffusionServingModelsaZMinimal OpenAIServingModels implementation for diffusion-only servers. vLLM's /v1/models route expects `app.state.openai_serving_models` to expose `show_available_models()`. In pure diffusion mode we don't initialize the full OpenAIServingModels (it depends on LLM-specific processors), so we provide a lightweight fallback. base_model_pathsrRNcCs ||_dSN)_base_model_paths)selfrr[r[r\__init__ z _DiffusionServingModels.__init__cstdd|jDdS)NcSs"g|] }t|j|jtgdqS))idroot permission)r'name model_pathr))rp base_modelr[r[r\ szA_DiffusionServingModels.show_available_models..)data)r(r)rr[r[r\show_available_modelss z-_DiffusionServingModels.show_available_models) rbrcrdrerfr*rr(rr[r[r[r\rsrcs\ddl}|jddtd|jddtdtdt|\}}t|||fi|IdHdS)zRun a single-worker API server. Unified entry point that automatically handles both LLM and Diffusion models through AsyncOmni, which manages multi-stage pipelines. rNignorez.*Pydantic.*serialization.*messagecategoryz(.*PydanticSerializationUnexpectedValue.* APIServer)warningsfilterwarnings UserWarningr?setup_openai_serveromni_run_server_worker)rQuvicorn_kwargswarnings_modulelisten_addresssockr[r[r\omni_run_servers rc s|jrt|jdkrt|j|jr't|jdkr'ddlm}||jt|}|dur3||d<t ||d4IdH}t |drMt | IdH}nd}|sSd}t ||} t| d d ht| d d h| tt|| j|IdHt|rtd | t|IdH} | du} | rtd|n td| jj|t| f||j|j|j|j|j t!j"|j#|j$|j%|j&|j'|j(d |IdH} WdIdHn 1IdHswYz | IdHW|)dS|)w)zRun a single API server worker.r)ReasoningParserManagerN log_config) client_configget_supported_tasks)generate/v1/chat/completionsPOST /v1/modelsGETzOProfiler endpoints are enabled. This should ONLY be used for local development!z4Starting vLLM API server (pure diffusion mode) on %sz!Starting vLLM API server %d on %s) renable_ssl_refreshhostport log_level access_logtimeout_keep_alive ssl_keyfile ssl_certfile ssl_ca_certs ssl_cert_reqsh11_max_incomplete_event_sizeh11_max_header_count)*tool_parser_pluginlenr>import_tool_parserreasoning_parser_pluginvllm.reasoningrimport_reasoning_parserr.build_async_omnir|tuplerbuild_openai_apprinclude_routerriomni_init_app_statestater]loggerwarningprofiler_routerget_vllm_configinfoparallel_config_api_process_rankrrrruvicorn_log_leveldisable_uvicorn_access_logenvsVLLM_HTTP_TIMEOUT_KEEP_ALIVErrrrrrclose) rrrQrrrr engine_clientsupported_tasksr~ vllm_configis_pure_diffusion shutdown_taskr[r[r\rs|        (> r) disable_frontend_multiprocessingrrrc Cstddkr!tdtdtdgttdt ||d4IdH}|VWdIdHdS1IdHs@wYdS)aVBuild an AsyncOmni instance from command-line arguments. Creates an async context manager that yields an AsyncOmni instance configured from the provided arguments. Handles forkserver setup if needed and ensures proper cleanup on exit. Args: args: Parsed command-line arguments containing model and configuration disable_frontend_multiprocessing: Optional flag to disable frontend multiprocessing (deprecated in V1) client_config: Optional client configuration dictionary Yields: EngineClient instance (AsyncOmni) ready for use VLLM_WORKER_MULTIPROC_METHOD forkserverz!Setup forkserver with pre-importszvllm.v1.engine.async_llmzForkserver setup complete!rN) rWgetenvrdebugmultiprocessingset_start_methodset_forkserver_preloadrensure_running"build_async_omni_from_stage_config)rQrr async_omnir[r[r\r.s    .rFrcCsn|rtdd}z$t|}|ddtdd|ji|}|VW|r-|dSdS|r6|ww)aCreate AsyncOmni from stage configuration. Creates an AsyncOmni instance either in-process or using multiprocess RPC. Loads stage configurations from the model or from a specified path. Args: args: Parsed command-line arguments containing model and stage configs disable_frontend_multiprocessing: Flag to disable frontend multiprocessing (deprecated in V1) client_config: Optional client configuration dictionary Yields: EngineClient instance (AsyncOmni) ready for use Note: Stage configurations are loaded from args.stage_configs_path if provided, otherwise from the model's default configuration. z:V1 is enabled, but got --disable-frontend-multiprocessing.Nmodelr[)rrvarscopypopr@rshutdown)rQrrkwargsr[r[r\rVs     rrrcs|IdH}d}t|dr.|jr.|j}t|dkr.|ddd}|dkr.d }td jdur7j}njg}j rEt j d }nd}fd d |D} ||_ j |_|_t|drc|jnd|_|r|rn|dnj} d|_||_t| |_d|_tj|| d|_t|dr|jnd} tj|| | d|_tdd|_d|_td| dS|dur|IdH}|durtd||_dh} t|drt|IdH} td| t j!} j"dkrt#}t$|t#sJ|%IdHnj"rt&}|'j"IdHnd}|dur|j(dur|j(j)ni}t*j+|}t|drD|j,dusDt|drD|j-dusDt|drD|j.dur|durzrddl/m0}ddl1m2}|3IdH}|durt|drn|j,dury||d|_,tdt|dr|j.dur|j.|_.tdt|dr|j-durt|dr|j.n|j.}|j4}||||_-tdntd Wnt5y}z td!|WYd}~n d}~wwtd"t6|| |d#|_|j7IdHd| vr t8||j|| j9j:j;j<|j=j>j?j@jAjBd$nd|_Cd| vrj?j@jAjHjBd%nd|_|jdurM|jIIdHd| vrbtJ||j|j:j?j@jBd&nd|_KtLd'd(| DrtM||j| || j9jFjBd)nd|_Nd*| vrtO||j|| j9jFjBd+nd|_Pd,| vrtQ||j|| j9jFjBd+nd|_Rd*| vsd-| vrtS||j|| jBd.nd|_TtU||j|| j9jFjBd+|_d/| vrtV||j|jBj@d0nd|_Wd/| vrtX||j|jBj@d0nd|_Yd| vrtZ||jjD|| j9j:j;j<j=j>j?j@d1 nd|_[d| vr9t\||j|j:jBj?jAj]d2nd|_^t_||j|d3|_`t||rN|dnd|jd4|_j|_d|_dS)5aInitialize the FastAPI application state for omni API server. Sets up the application state with model information, request logger, and other server configuration needed for handling API requests. Automatically detects pure diffusion mode (single diffusion stage) and handles it appropriately. Args: engine_client: Engine client instance (AsyncOmni) state: FastAPI application state object to initialize args: Parsed command-line arguments NF stage_configsr stage_typellm diffusionTz5Detected pure diffusion mode (single diffusion stage)) max_log_lencsg|] }t|jdqS))rr)r*r)rprrQr[r\rsz'omni_init_app_state..)diffusion_engine model_name)rrrenable_server_load_trackingz3Pure diffusion API server initialized for model: %sz9vllm_config is None, some features may not work correctlyrrzSupported tasks: %sdemoinput_processor io_processor model_config)get_io_processor)OmniInputProcessor)rz)Initialized input_processor for AsyncOmniz&Initialized model_config for AsyncOmniz&Initialized io_processor for AsyncOmnizNCannot initialize processors: tokenizer is None. OpenAIServingModels may fail.zPFailed to initialize processors for AsyncOmni: %s. OpenAIServingModels may fail.zPCannot initialize processors: vllm_config is None. OpenAIServingModels may fail.)rr lora_modules) request_logger chat_templatechat_template_content_formatreturn_tokens_as_token_idsenable_auto_tools tool_parser tool_serverreasoning_parserenable_prompt_tokens_detailsenable_force_include_usageenable_log_outputslog_error_stack)rrrdefault_chat_template_kwargstrust_request_chat_templaterr#exclude_tools_when_tool_choice_nonerrrrrenable_log_deltasr)rrrrrcss|]}|tvVqdSrr<)rptaskr[r[r\ usz&omni_init_app_state..)rrrrrrembed)rrrrrclassifyscore)rscore_templater transcription)rrr) rrrrrrrrr)rrrrrforce_no_detokenize)r)rr)arr|rrrYrrserved_model_namerenable_log_requestsrrrdisable_log_stats log_statsrQrrropenai_serving_modelsopenai_serving_tokenizationrI for_diffusionopenai_serving_chatrKopenai_serving_videorVrserver_load_metricsrrurrrrrr{init_and_validateradd_tool_server lora_configdefault_mm_lorasr9rrrrvllm.plugins.io_processorsr vllm_omni.engine.input_processorr get_tokenizerio_processor_plugin Exceptionr+init_static_lorasr-rrenable_auto_tool_choicetool_call_parserstructured_outputs_configrrrrropenai_serving_responses response_rolerrrrwarmupr%openai_serving_completionanyr4openai_serving_poolingr3openai_serving_embeddingr2openai_serving_classificationr5openai_serving_scoresr7r/openai_serving_transcriptionr0openai_serving_translationranthropic_serving_messagesr6 tokens_onlyserving_tokensrJopenai_serving_speech)rrrQrrrrserved_model_namesrrrdiffusion_stage_configsrresolved_chat_templaterrrrr tokenizerrrer[rr\rs:                            )               rrequestcC |jjjSr)r~rrr9r[r[r\ Omnivideorr<cCr:r)r~rrr;r[r[r\Omnichatrr=cCr:r)r~rr3r;r[r[r\ Omnispeechrr>rcontenttext/event-streamr) dependencies responses raw_requestc s@|jtd}t|}|dur*t|jjdd}|dur$ttj j dd|j ddSz | ||IdH}Wnt yR}ztd|ttjj t|d|d}~wwt|trit||jre|jjdSddSt|trd dl}d dl}||jd td |jd d td z|jdddd} t| t|dWWdSt y z|jddd} | | } t| t|dWYWdSt y|'|jd td t|jdddt|dWdYYWdS1swYYnwYnwWdn 1swYt!|ddS)Nrz/The model does not support Chat Completions API status_codedetailrzChat completion failed: %sr?rFrr)rz .*Pydantic.*rjsonTnone)modeserialize_as_anyr)r?headers)rrN)rMrr@)r? media_type)"rOrY)ENDPOINT_LOAD_METRICS_FORMAT_HEADER_LABELr=rVr~rr r NOT_FOUNDvaluecreate_error_responsecreate_chat_completionr r exceptionINTERNAL_SERVER_ERRORstrr{r&r model_dumperrorcoder$rKrcatch_warningsrrr,model_dump_jsonloadsr) r9rCmetrics_header_formathandler base_server generatorr8json_libr response_dict response_jsonr[r[r\rUs~              rUz/v1/audio/speechrzaudio/*c st|}|dur#t|jjdd}|durttjjdd|jddSz | ||IdHWSt yD}z ttj jt |d|d}~ww)Nr%The model does not support Speech APIrErH) r>rVr~rr rrRrSrT create_speechr rWrX)r9rCr`rar8r[r[r\rgCs   rgz/v1/audio/voices)rBcsDt|}|durt|jddS|jrt|jng}td|idS)z9List available TTS voices/speakers from the loaded model.NrfrHvoicesr?)r>r"rTsupported_speakerssortedr)rCr`speakersr[r[r\ list_voices_s  rmz/healthcst|jjdd}|dur't|dr|jrtddidStddd tjjd St|jjd d}|durA| IdHtddidStdd d tjjd S) zHealth check endpoint that works for both LLM and diffusion modes. Returns 200 OK if the server is healthy. For LLM mode: delegates to engine_client health check For diffusion mode: checks if diffusion_engine is running rN is_runningstatushealthyri unhealthyzDiffusion engine is not running)roreasonrJrzNo engine initialized) rVr~rr|rnrrSERVICE_UNAVAILABLErS check_health)rCrrr[r[r\healthzs" rurcs|t|jjdd}|durtd|dddgdgdd St|jjd d}|dur6|IdH}t|d Stdgdd S) zShow available models endpoint that works for both LLM and diffusion modes. Returns model information in OpenAI-compatible format. diffusion_model_nameNrfrrz vllm-omni)robjectcreatedowned_byr)rwrrir)rVr~rrrrY)rCrvrmodelsr[r[r\rs*rz/v1/images/generationsc s\t|\}}}|jdur|j|krtd|jd|dzd|ji}|jdur/|j|d<t|jd}t|j \}}t |d|t |d |d \} } |j r^t |j \} } | d | } nd } t |d | t |d| t |d|j t |d|jt |d|jt |d|jdur|jntddt |d|jdtj} td|jd| t||||| dIdH} | durttjjddt| }tdt|ddd|D}tt t!!|d WStyt"y }zt#d!|ttj$jt%|dd}~wt&y-}zt'd"|ttjjd"t%|dd}~ww)#aGenerate images from text prompts using diffusion models. OpenAI DALL-E compatible endpoint for text-to-image generation. Only supports multi-stage omni mode with diffusion stages. Args: request: Image generation request with prompt and parameters raw_request: Raw FastAPI request for accessing app state Returns: ImageGenerationResponse with generated images as base64 PNG Raises: HTTPException: For validation errors, missing engine, or generation failures N#Model mismatch: request specifies '' but server is running ''. Using server model.promptnegative_prompt)num_outputs_per_prompt lora_request lora_scaleNNxz model defaultwidthheightnum_inference_stepsguidance_scaletrue_cfg_scaleseedrgenerator_deviceimg_gen_ Generating image(s) r gen_params stage_typesr~ request_id.No output generated from multi-stage pipeline.rESuccessfully generated image(s)cSsg|] }tt|ddqS)Nb64_jsonrevised_prompt)rDrArpimgr[r[r\r%z#generate_images..)rxrValidation error: zImage generation failed: )(_get_engine_and_modelrrrr~rrLn_parse_lora_requestlora_update_if_not_nonesizerBrrrrrandomrandintruuiduuid4hexr_generate_with_async_omnir rrWrS_extract_images_from_resultrrFrgtime ValueErrorrZ BAD_REQUESTrXr rV)r9rCrrrr~rrrrrsize_strrresultimages image_datar8r[r[r\generate_imagess            rz/v1/images/editszimage[])aliaszurl[].rautorpngdimage image_arrayurl url_arrayr~rrresponse_format output_format backgroundoutput_compression)geleuserrrrrrrrc+ s`t|\}}}|dur||krtd|d|dt| | dkr-ttjjddz9d|i}|dur;||d <g}|p@|}|pD|}|rL|||rS|||s[td d dt |IdH}i|d <||d d <t }t |j j dd}t |dd}ddt|Dd}t||t|t|d|t|} t| \}!}"t|d|!t|d|"t |dd}#d\}$}%|dkr|dj\}$}%nt|\}$}%|#dur|$|%|#krttjjd|$d|%d|#dd|$d|%}&t|d|$t|d|%t|d|t|d |t|d!|t|d"|dur|ntdd#t|d$|d%tt}'td&|d'|&t|||||'d(IdH}(t|(}td)t|d*fd+d|D})ttt|)|&d,WStypt y}*zt!d-|*ttjjt|*dd}*~*wt"y}*zt#d.|*ttj$jd.t|*dd}*~*ww)/z0 OpenAI-compatible image edit endpoint. Nr{r|r}rz1Only response_format 'b64_json' is supported now.rEr~riz"Field 'image' or 'url' is requiredmulti_modal_datarrQdefault_sampling_paramscSsg|] \}}|dkr|qSrr[)rpitr[r[r\rrzedit_images..rrrrmax_generated_image_sizerrzRequested image size rz% exceeds the maximum allowed size of z pixels.rrrrrrrr img_edit_rrrrrcs"g|] }tt|dddqS))formatrNr)rD%_encode_image_base64_with_compressionrrrr[r\rs)rxrrrrzImage edit failed: )%rrr_choose_output_formatr rrrSextend_load_input_imagesrLrVr~r enumerate#apply_stage_default_sampling_paramsrXr_get_lora_from_json_strrlowerrrBrrrgrrrrrrFrrZr rVrW)+rCrrrrr~rrrrrrrrrrrrrrrrrrinput_images_listrurls pil_imagesrapp_state_argsdefault_sample_paramdiffusion_stage_id lora_dictrrrrrrrrrr8r[rr\ edit_images9s&                  "     rc CsLt|jjdd}|dust|dsttjjddt|jjdd}|s+ttjjddd}g}|D]J}d}t|t rA| dd }n/t|d rM| dd }n#t|drV|j }nz d|vr_|dnd }Wn t t fyod }Ynw|d krvd }||q1|sttjjd dt|jjdd}|rt|dr|jr|jdj}nd}|||fS)Nr stage_listzOMulti-stage engine not initialized. Start server with a multi-stage omni model.rErzDStage configs not found. Start server with a multi-stage omni model.FrrrYrTz1No diffusion stage found in multi-stage pipeline.rrrunknown)rVr~rr|r rrsrSr{dictrYr TypeErrorKeyErrorr}rr) rCrrhas_diffusion_stagerstagerserving_modelsrr[r[r\rsP     rcCsR|durdSzt|}Wntjytdddwt|ts'tddd|S)NrIzInvalid LoRA JSON stringrEzLoRA must be a JSON object)rKr^JSONDecodeErrorr r{r) lora_bodyrr[r[r\rs   rrcCs|durvt|tsttjjdd|dp|dp|d}|dp3|dp3|dp3|d }|d }|durB|d }|d }|durP|d }|dur\|r\tt|}|r`|shttjjddt t|t |t||fSdS)Nz'Invalid lora field: expected an object.rEr lora_nameadapter local_pathrj lora_pathlora_local_pathscalerint_id lora_int_idz5Invalid lora object: both name and path are required.r) r{rr rrrSrYrPrXrOrg)rrrrrr[r[r\rs:        rrrc s tt|}d}t|dd}t|trtt|dd}t|ts&dd|D}nt|}t|t|krA|dd|Ddt|}g}t|D]\}} | dkrU||qG||} || qG|jd d|i|2z3dHW} | }qi6n|jd d|gi|IdH}|durt t j j dd |S) Nrdefault_sampling_params_listcS g|] }|dkr tntqSrrLrrpstr[r[r\rMsz-_generate_with_async_omni..cSrrrrr[r[r\rUs rsampling_params_listrrEr[) rr@rVr{rfrrr}rr rrWrS) rrrrrrdefault_params_listridxr base_paramsoutputr[r[r\r?sV         rrwkeyvalcCs|dur t|||dSdSr)setattr)rwrrr[r[r\rssrrcCshg}t|dr|jr|j}|St|dr2|j}t|tr'|dr'|d}|St|dr2|jr2|j}|S)Nrrequest_output)r|rrr{rrY)rrrr[r[r\rxs rinputsc st|tr |g}g}|D]}t|trI|drIz|dd\}}t|}tt |}| |Wq t yH}zt d|d}~wwt|tr|drt jdd4IdH>}z||IdH} | tt | j}| |Wnt y}z t d |d |d}~wwWdIdHn 1IdHswYq t|d rz|IdH} tt | }| |Wq t y}zt d |d}~wwt d ||st d|S)z) convert to PIL.Image.Image list z data:image,rzInvalid base64 image: Nhttp<)timeoutz"Failed to download image from URL z: filezFailed to open uploaded file: zUnsupported input: zNo valid input images found)r{rX startswithsplitbase64 b64decoderopenioBytesIOr}r rhttpx AsyncClientrYraise_for_statusr?r|read) rrinp_b64_data image_bytesrr8clientrespimg_datar[r[r\rsT  ( rcCs0|pd}|dvr |S|pddkrdSdS)NrD>jpgrjpegwebpr transparentrr)r)rrfmtr[r[r\rs rrcCsvt}i}|dvr||d<n|dkr!tdtdd|d|d<|j|fd|i||dt| d S) aEncode PIL Image to base64 PNG string. Args: image: PIL Image object format: Output image format (e.g., "PNG", "JPEG", "WEBP") output_compression: Compression level (0-100%), 100 for best quality Returns: Base64-encoded image as string )rrrqualityrr compress_levelrzutf-8) rrmaxminsaveseekr b64encoder decode)rrrbuffer save_kwargsr[r[r\rs   rdefault_params_jsonsampling_params stage_keycCsV|dur%t|}||vr'||}|D]\}}t||r$t|||qdSdSdS)a Update a stage's sampling parameters with vLLM-Omni defaults. Args: default_params_json: JSON string of stage-keyed default parameters sampling_params: The sampling parameters object to update stage_key: The stage ID/key in the pipeline N)rKr^itemsr|r)r$r%r&default_params_dictstage_defaults param_name param_valuer[r[r\rs   rz/start_profilec z*|r|jnd}td|r|nd|jjj}|j|dIdH}tdt|dWStyJ}zt d|t t j j dt|d d}~ww) a$Start profiling for the engine. Args: request: Optional request body with stages to profile. - stages: List of stage IDs to profile. If None, profiles all stages. Example: POST /start_profile {"stages": [0, 1]} # Profile only stages 0 and 1 Nz Starting profiler for stages: %sallrazProfiler started.rizFailed to start profiler: %szFailed to start profiler: rE)rarrr~rr start_profilerr rVr rrWrSrXrCr9rarrr8r[r[r\r/     r/z /stop_profilec r,) a4Stop profiling for the engine. Args: request: Optional request body with stages to stop profiling. - stages: List of stage IDs to stop profiling. If None, stops all stages. Example: POST /stop_profile {"stages": [0, 1]} # Stop profiling only stages 0 and 1 Nz Stopping profiler for stages: %sr-r.zProfiler stopped.rizFailed to stop profiler: %szFailed to stop profiler: rE)rarrr~rr stop_profilerr rVr rrWrSrXr0r[r[r\r2 r1r2input_reference_bytesr4c st|}|durttjjddtdt|jz |j |||dIdHWSty.t yL}zt d|ttj jdt |dd}~ww)Nz)Video generation handler not initialized.rEzVideo generation handler: %sr3zVideo generation failed: %szVideo generation failed: )r<r rrsrSrrtyperbgenerate_videosr rVrWrX)r9rCr4r`r8r[r[r\_run_video_generation%s(   r7rSc CsN|dus|dkr dSzt|WStjy&}z ttjjdd|d}~ww)NrDzInvalid JSON in form field.rE)rKr^rr rrrS)rSexcr[r[r\_parse_form_json>s r9z /v1/videos)r_input_referencesecondsrr num_framesfpsguidance_scale_2boundary_ratio flow_shiftcs|dur |IdHnd}id|d|d|d|d|d|ddd |d | d | d | d | d| d|d|d|d||||t|d}dd|D}tdi|}t|||dIdHS)z9OpenAI-style video create endpoint (multipart form-data).Nr~rr;rrrr:rrrr<r=rrr>r?r@)rrrrcSsi|] \}}|dur||qSrr[)rpkvr[r[r\ sz create_video..r3r[)r r9r'rGr7)rCr~r:rrr;rrrrrr<r=rrr>r?r@rrrrr4 request_datar9r[r[r\ create_videoJsZ"     rEr)rRN)rr)rrrKrmultiprocessing.forkserverrrWrrrargparsercollections.abcr contextlibrrrtypingrrrr vllm.envsrfastapir r r r r rrfastapi.responsesrrPILrpydanticrrstarlette.datastructuresrstarlette.routingrvllmrvllm.engine.protocolr"vllm.entrypoints.anthropic.servingrvllm.entrypoints.chat_utilsrvllm.entrypoints.launcherrvllm.entrypoints.loggerr vllm.entrypoints.mcp.tool_serverrrr"vllm.entrypoints.openai.api_serverr rr!r+vllm.entrypoints.serve.instrumentator.basicr"ModuleNotFoundError(vllm.entrypoints.openai.basic.api_router0vllm.entrypoints.openai.chat_completion.protocolr#r$*vllm.entrypoints.openai.completion.servingr%'vllm.entrypoints.openai.engine.protocolr&r'r(r)'vllm.entrypoints.openai.models.protocolr*&vllm.entrypoints.openai.models.servingr+$vllm.entrypoints.openai.orca_metricsr,)vllm.entrypoints.openai.responses.servingr-$vllm.entrypoints.openai.server_utilsr..vllm.entrypoints.openai.speech_to_text.servingr/r0vllm.entrypoints.openai.utilsr1)vllm.entrypoints.pooling.classify.servingr2&vllm.entrypoints.pooling.embed.servingr3(vllm.entrypoints.pooling.pooling.servingr4&vllm.entrypoints.pooling.score.servingr5%vllm.entrypoints.serve.disagg.servingr6'vllm.entrypoints.serve.tokenize.servingr7vllm.entrypoints.utilsr8r9r: vllm.loggerr; vllm.tasksr=vllm.tool_parsersr>vllm.utils.system_utilsr? vllm_omni.entrypoints.async_omnir@,vllm_omni.entrypoints.openai.image_api_utilsrArB+vllm_omni.entrypoints.openai.protocol.audiorC,vllm_omni.entrypoints.openai.protocol.imagesrDrErF,vllm_omni.entrypoints.openai.protocol.videosrGrH)vllm_omni.entrypoints.openai.serving_chatrI+vllm_omni.entrypoints.openai.serving_speechrJ*vllm_omni.entrypoints.openai.serving_videorKvllm_omni.inputs.datarLrMrNvllm_omni.lora.requestrOvllm_omni.lora.utilsrPrbrrirboolr]r^rXrurzrQrrrrrrrrr<r=r>postOKrSrrRrWrUrgrYrmrurrsrrfrgfloatrrrrrr)rrrrrrr/r2bytesr7r9rEr[r[r[r\s      $                                         S ' . h    =          &     j                8! 4  2