o ٷiJC@sfdZddlZddlZddlZddlZddlmZddlZddl Z ddl Z ddl m Z ddl mZmZddlmZddlmZddlmZdd lmZdd lmZdd lmZmZdd lmZdd lm Z ddl!m"Z"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)ee*Z+dZ,dZ-Gddde Z.dej/de0e1e2effddZ3dej/ddfddZ4de0e fddZ5dS)z Omni serve command for vLLM-Omni. Supports both multi-stage LLM models (e.g., Qwen2.5-Omni) and diffusion models (e.g., Qwen-Image) through the same CLI interface. N)Any) CLISubcommand)make_arg_parservalidate_parsed_serve_args)VLLM_SUBCMD_PARSER_EPILOG) init_logger)FlexibleArgumentParser)make_zmq_socket)get_engine_client_zmq_addr)get_connectors_config_for_stageload_omni_transfer_config) resolve_omni_kv_config_for_stage)log_logo)OmniBaseomni_snapshot_download) OmniStage)omni_run_server)inject_omni_kv_configaxLaunch a local OpenAI-compatible API server to serve Omni models via HTTP. Supports both multi-stage LLM models and diffusion models. The server automatically detects the model type: - LLM models: Served via /v1/chat/completions endpoint - Diffusion models: Served via /v1/images/generations endpoint Examples: # Start an Omni LLM server vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 # Start a diffusion model server vllm serve Qwen/Qwen-Image --omni --port 8091 Search by using: `--help=` to explore options by section (e.g., --help=OmniConfig) Use `--help=all` to show all available flags at once. c@sTeZdZdZdZedejddfddZdejddfdd Z d ej de fd d Z dS) OmniServeCommandz(The `serve` subcommand for the vLLM CLI.serveargsreturnNcCs\tjdsdtjd<tt|dr|jdur|j|_|jr%t|dSt t |dS)NVLLM_DISABLE_LOG_LOGO1 model_tag) osenvirongetrhasattrrmodelheadless run_headlessuvlooprunr)rr%S/home/ubuntu/.local/lib/python3.10/site-packages/vllm_omni/entrypoints/cli/serve.pycmdCs   zOmniServeCommand.cmdcCsr|jdur|jdus|jdurtdddlm}t|ddp$t|dd}|r3||r3td|dSt |dS)NzO--stage-id requires both --omni-master-address and --omni-master-port to be setr)is_diffusion_modelrr zDetected diffusion model: %s) stage_idomni_master_addressomni_master_port ValueError"vllm_omni.diffusion.utils.hf_utilsr(getattrloggerinfor)selfrr(r r%r%r&validateRs    zOmniServeCommand.validate subparserscCs|j|jtdd}t|}tj|jd|_|jddd}|jddd d |jd t dd d |jdt ddd |jdt ddd |jdt ddd |jdt ddd |jdddd |jdt ddd |jdt ddd |jd t d!d!d"gd#d$|jd%t dd&d |jd'd(t d)d*|jd+d,t d-d*|jd.t dd/d |jd0d1d2t dd3d4|jd5d6t dd7d4|jd8t j dd9d |jd:d;ddt d?d@d |jdAt dBdCd |jdDt dEdFd |jdGt ddHd |jdIddJd |jdKddLd |jdMddNd |jdOddPd |jdQddRd |jdSt ddTd |jdUt ddVd |jdWt dBdBdXgdYd$|jdZt d[d*|jd\t d]d*|jd^t dd_d |S)`Nz'vllm serve [model_tag] --omni [options]) descriptionusage)subcmd OmniConfigz=Configuration for vLLM-Omni multi-stage and diffusion models.)titler4z--omni store_truez:Enable vLLM-Omni mode for multi-modal and diffusion models)actionhelpz--stage-configs-pathzbPath to the stage configs file. If not specified, the stage configs will be loaded from the model.)typedefaultr;z --stage-idz-Select and launch a single stage by stage_id.z--stage-init-timeouti,zEThe timeout for initializing a single stage in seconds (default: 300)z--init-timeoutiXz(The timeout for initializing the stages.z--shm-threshold-bytesiz)The threshold for the shared memory size.z --log-statszEnable logging the stats.z --log-filezThe path to the log file.z--batch-timeout zThe timeout for the batch.z--worker-backend multi_processrayz%The backend to use for stage workers.)r<r=choicesr;z --ray-addressz-The address of the Ray cluster to connect to.z--omni-master-addressz-omaz9Hostname or IP address of the Omni orchestrator (master).)r<r;z--omni-master-portz-ompz'Port of the Omni orchestrator (master).z --num-gpusz4Number of GPUs to use for diffusion model inference.z--uspz--ulysses-degreeulysses_degreezwUlysses Sequence Parallelism degree for diffusion models. Equivalent to setting DiffusionParallelConfig.ulysses_degree.)destr<r=r;z--ring ring_degreezqRing Sequence Parallelism degree for diffusion models. Equivalent to setting DiffusionParallelConfig.ring_degree.z--quantization-configzoJSON string for diffusion quantization_config. Example: '{"method":"gguf","gguf_model":"/path/to/model.gguf"}'.z --use-hsdpuse_hsdpzEnable HSDP (Hybrid Sharded Data Parallel) for diffusion models. Shards model weights across GPUs to reduce per-GPU memory usage.)rCr:r;z--hsdp-shard-sizezPNumber of GPUs to shard weights across. -1 = auto (world_size / replicate_size).z--hsdp-replicate-sizezHNumber of replica groups for HSDP. Each group holds a full sharded copy.z--cache-backendnonezECache backend for diffusion models, options: 'tea_cache', 'cache_dit'z--cache-configzDJSON string of cache configuration (e.g., '{"rel_l1_thresh": 0.2}').z--enable-cache-dit-summaryz@Enable cache-dit summary logging after diffusion forward passes.z--vae-use-slicingzNEnable VAE slicing for memory optimization (useful for mitigating OOM issues).z--vae-use-tilingzMEnable VAE tiling for memory optimization (useful for mitigating OOM issues).z--enable-cpu-offloadz+Enable CPU offloading for diffusion models.z--enable-layerwise-offloadz7Enable layerwise (blockwise) offloading on DiT modules.z--boundary-ratiozOBoundary split ratio for low/high DiT in video models (e.g., 0.875 for Wan2.2).z --flow-shiftzJScheduler flow_shift for video models (e.g., 5.0 for 720p, 12.0 for 480p).z--cfg-parallel-sizezNumber of devices for CFG parallel computation for diffusion models. Equivalent to setting DiffusionParallelConfig.cfg_parallel_size.z--default-sampling-paramszJson str for Default sampling parameters, Structure: {"": {: value, ...}, ...} e.g., '{"0": {"num_inference_steps":50, "guidance_scale":1}}'. Currently only supports diffusion models.z--max-generated-image-sizez0The max size of generate image (height * width).z--tts-max-instructions-lengthzWMaximum length for TTS voice style instructions (overrides stage config, default: 500).) add_parsername DESCRIPTIONrrformatepilogadd_argument_group add_argumentstrintjsonloadsfloat)r1r3 serve_parseromni_config_groupr%r%r&subparser_init_s    zOmniServeCommand.subparser_init) __name__ __module__ __qualname____doc__rK staticmethodargparse Namespacer'r2_SubParsersActionrrXr%r%r%r&r>s rrrcCstt}|t|SN)r__new__#_create_default_diffusion_stage_cfgvars)r omni_baser%r%r&rcOs rcc s|jdur|jdkrtd|jdkrtdt|j}tt}t|}||d<| ||\}}|j }|durLt |dkrDtdt |ddd}d}|D]}t |dd|kr^|}nqP|durktd |d t ||jd } t| |} |j} |j} t} td | | d }t| |tjd dd}d|d}|tj||jtddstdtd|d|z tj|}Wn4tj y}ztd|d|d||d}~wt!y}ztd|d|d||d}~ww|ds |d}td|d||d|d}}t"#d|d |d!|Wdn 1s+wYWdn 1s;wYd fd"d#}t$$t$j%|t$$t$j&|t'||j(d$}|)||zt*| |\}}}|rvt+||||Wnt!y}z t",d%||WYd}~nd}~wwt-j./d&}d'|d(|durd)n|t-j.d&<z+|j0|d*t1|jt1|j2| dd*d+|j3dur|j34W|5dSW|5dS|5w),NrGz.api_server_count can't be set in headless moder?z3headless mode requires worker_backend=multi_processr z?--stage-id is required in headless mode for multi-stage configsrr)zNo stage matches stage_id=.)default_shm_thresholdF) local_onlyhostporti)bindlinger handshake)r<r)i`)timeoutzHandshake timeout (z minutes) for stage-z at z"Handshake decode failed for stage-z: z.Unexpected error decoding handshake for stage-okerrorzHandshake failed for stage- in_endpoint out_endpointz[Headless] Stage-z& received endpoints via handshake: in=z, out=csrdSdt)NT) SystemExit)signumframeshutdown_requestedr%r&signal_handlersz$run_headless..signal_handler)stage_init_timeoutzC[Headless] Failed to inject omni connector config into stage-%s: %sVLLM_LOGGING_PREFIXz[Stage-z] T)is_asyncshm_threshold_bytes batch_timeoutconnectors_configworker_backendignore_runtime_config)6api_server_countr,rrr rrbrdcopy_resolve_stage_configsr)lenr.r r}r r*r+zmqContextr r REQsendmsgspecmsgpackencodepollHANDSHAKE_TIMEOUT_MINS RuntimeErrordecoderecv DecodeError Exceptionr/r0signalSIGTERMSIGINTrry attach_queuesr rdebugrrrinit_stage_workerrRr~_procjoinstop_stage_worker)rr re args_dict config_path stage_configssingle_stage_id stage_configcfgtransfer_configrr*r+zmq_ctxhandshake_endpointhandshake_socket handshake_msgresponseexc error_msgrqrrrxstage omni_conn_cfg omni_fromomni_toeold_envr%rvr&r"Ts         '   $  r"cCstgSra)rr%r%r%r&cmd_initsr)6r\r^rSrrtypingrmsgspec.msgpackrr#rvllm.entrypoints.cli.typesr vllm.entrypoints.openai.cli_argsrrvllm.entrypoints.utilsr vllm.loggerrvllm.utils.argparse_utilsrvllm.utils.network_utilsr vllm.v1.utilsr %vllm_omni.distributed.omni_connectorsr r :vllm_omni.distributed.omni_connectors.utils.initializationr vllm_omni.entrypoints.cli.logorvllm_omni.entrypoints.omnirr vllm_omni.entrypoints.omni_stager'vllm_omni.entrypoints.openai.api_serverrvllm_omni.entrypoints.utilsrrYr/rrLrr_listdictrQrcr"rr%r%r%r&s@             s