o -iE@s0dZddlZddlZddlZddlZddlZddlmZddlZddl Z ddl m Z ddl m Z mZmZddlmZddlmZddlmZmZdd lmZdd lmZdd lmZmZdd lmZdd l m!Z!ddl"m#Z#ddl$m%Z&ee'Z(Gdddej)Z)dej*fddZ+ddZ,e'dkre,dSdS)a" vLLM gRPC Server Starts a gRPC server for vLLM using the VllmEngine protocol. Usage: python -m vllm.entrypoints.grpc_server --model Example: python -m vllm.entrypoints.grpc_server --model meta-llama/Llama-2-7b-hf --host 0.0.0.0 --port 50051 N)AsyncGenerator) reflection)SamplingParams TextPrompt TokensPrompt)AsyncEngineArgs)log_version_and_model)vllm_engine_pb2vllm_engine_pb2_grpc) init_logger) RequestOutput)RequestOutputKindStructuredOutputsParams) UsageContext)FlexibleArgumentParser)AsyncLLM) __version__c@s.eZdZdZdedefddZdejde j j de ej d ffd d Zdejde j j dejfd d Zdejde j j dejfddZdejde j j dejfddZdejde j j dejfddZdejde j j dejfddZe d dejde defddZ!ede"dej fddZ#ede"dej fddZ$d S)!VllmEngineServicera( gRPC servicer implementing the VllmEngine service. Handles 6 RPCs: - Generate: Streaming text generation - Embed: Embeddings (TODO) - HealthCheck: Health probe - Abort: Cancel requests out-of-band - GetModelInfo: Model metadata - GetServerInfo: Server state async_llm start_timecCs||_||_tddS)z Initialize the servicer. Args: async_llm: The AsyncLLM instance start_time: The server start time, in seconds since epoch zVllmEngineServicer initializedN)rrloggerinfo)selfrrrY/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/vllm/entrypoints/grpc_server.py__init__;szVllmEngineServicer.__init__requestcontextreturnNc Cs<|j}td|zN|ddkr%dt|jji}|jjr$|jj|d<nd|ji}|j |j |j d}|j j |||d2z3dHW}|j rK||V|jrT||Vq<6WdStyx}z|tjjt|IdHWYd}~dSd}~wty}ztd ||tjjt|IdHWYd}~dSd}~ww) z Handle streaming generation requests. Args: request: The GenerateRequest protobuf context: gRPC context Yields: GenerateResponse protobuf messages (streaming) zGenerate request %s received.input tokenizedprompt_token_idsprompt)stream)r"sampling_params request_idNz Error in Generate for request %s)r%rdebug WhichOneoflistr input_ids original_texttext_sampling_params_from_protor$r#rgenerate_chunk_responsefinished_complete_response ValueErrorabortgrpc StatusCodeINVALID_ARGUMENTstr Exception exceptionINTERNAL)rrrr%r"r$outputerrrGenerateGsB      ( (zVllmEngineServicer.Generatecs&td|tjjdIdHdS)z Handle embedding requests. TODO: Implement in Phase 4 Args: request: The EmbedRequest protobuf context: gRPC context Returns: EmbedResponse protobuf zEmbed RPC not yet implementedN)rwarningr2r3r4 UNIMPLEMENTED)rrrrrrEmbed~s  zVllmEngineServicer.Embedcs4|jj }|r dnd}td||tj||dS)z Handle health check requests. Args: request: The HealthCheckRequest protobuf context: gRPC context Returns: HealthCheckResponse protobuf HealthzEngine is not alivez+HealthCheck request: healthy=%s, message=%s)healthymessage)rerroredrr&r HealthCheckResponse)rrr is_healthyrBrrr HealthChecks   zVllmEngineServicer.HealthCheckcs.|j}td||j|IdHtS)z Out-of-band abort requests. Args: request: The AbortRequest protobuf context: gRPC context Returns: AbortResponse protobuf zAbort requests: %sN) request_idsrr&rr2r AbortResponse)rrrrGrrrAborts  zVllmEngineServicer.Abortcs.|jj}tj|j|jdk|j||jdS)z Handle model info requests. Args: request: The GetModelInfoRequest protobuf context: gRPC context Returns: GetModelInfoResponse protobuf r-) model_path is_generationmax_context_length vocab_sizesupports_vision) r model_configr GetModelInfoResponsemodel runner_type max_model_lenget_vocab_sizeis_multimodal_model)rrrrOrrr GetModelInfoszVllmEngineServicer.GetModelInfocs0|jj}tj|dtt|jddS)z Handle server info requests. Args: request: The GetServerInfoRequest protobuf context: gRPC context Returns: GetServerInfoResponse protobuf Fz vllm-grpc)active_requests is_pausedlast_receive_timestampuptime_seconds server_type)routput_processorget_num_unfinished_requestsr GetServerInfoResponsetimer)rrr num_requestsrrr GetServerInfos  z VllmEngineServicer.GetServerInfoTparamsr#cCs$|jrt|jnd}|jrt|jnd}d}|d}|ra|dkr(t|jd}n9|dkr3t|jd}n.|dkr>t|jd}n#|d krIt|jd }n|d krTt|j d }n |d kratt|j j d}t d+id| drm|jndd|jdkrx|jndd|jd|jd|jd|jd|jdkr|jndd| dr|jndd|jd|d|d|jd|jd|jd|jd kr|jnd!d"| d"r|jndd#| d#r|jndd$| d$r|jndd%|jd&|jrt |jndd'| d'r|j!ndd(|d)t"|d*|r t#j$St#j%S),a Convert protobuf SamplingParams to vLLM SamplingParams. Args: params: Protobuf SamplingParams message stream: Whether streaming is enabled Returns: vLLM SamplingParams with detokenize=False and structured_outputs N constraint json_schema)jsonregex)rfgrammar)rgstructural_tag)rh json_object)richoice)rj temperatureg?top_pgtop_kmin_pfrequency_penaltypresence_penaltyrepetition_penalty max_tokens min_tokensstopstop_token_idsskip_special_tokensspaces_between_special_tokens ignore_eosnrlogprobsprompt_logprobsseedinclude_stop_str_in_output logit_biastruncate_prompt_tokensstructured_outputs detokenize output_kindr)&rtr(rur'rrdrfrgrhrirjchoicesrHasFieldrkrlrmrnrorprqrrrsrvrwrxryr{r|r}r~rdictrboolr DELTA FINAL_ONLY)rbr#rtrurconstraint_fieldrrrr,s        !z.VllmEngineServicer._sampling_params_from_protor:cCsj|jr|jdnd}|durtjtjgdddddStjtj|j|jr)t|jndt|j|jddS)a5 Build a streaming chunk response from vLLM output. When output_kind=DELTA, vLLM returns only new tokens automatically. Args: output: vLLM RequestOutput (with delta tokens when output_kind=DELTA) Returns: GenerateResponse with chunk field set rN) token_ids prompt_tokenscompletion_tokens cached_tokens)chunk)outputsr GenerateResponseGenerateStreamChunkrr!lennum_cached_tokensr: completionrrrr.Bs(   z"VllmEngineServicer._chunk_responsecCst|jr|jdnd}|durtjtjgddddddStjtj|j|jp&d|jr.t|jndt|j|jddS)z Build a final completion response from vLLM output. Args: output: vLLM RequestOutput (finished=True) Returns: GenerateResponse with complete field set rNerror) output_ids finish_reasonrrr)completert) rr rGenerateCompleterrr!rrrrrrr0js,  z%VllmEngineServicer._complete_response)T)%__name__ __module__ __qualname____doc__rfloatrr GenerateRequestr3aioServicerContextrrr< EmbedRequest EmbedResponser?HealthCheckRequestrDrF AbortRequestrHrIGetModelInfoRequestrPrVGetServerInfoRequestr^ra staticmethodrrr,r r.r0rrrrr.st    7      N'rargsc sttt|jtd|t}t|}|jt j d}t j |t j |j |jd}t||}tjjddgd}t||tjjdjtjf}t|||jd|j}|||Id Htd |td t !} t "fd d } t#j$t#j%fD]} | &| | qz`z 'Id HWnt(ytdYn%wWtd|j)ddId Htd|*tdtdd SWtd|j)ddId Htd|*tdtdd Std|j)ddId Htd|*tdtdw)zW Main serving function. Args: args: Parsed command line arguments zvLLM gRPC server args: %s) usage_context) vllm_configrenable_log_requestsdisable_log_stats)zgrpc.max_send_message_length)zgrpc.max_receive_message_lengthr)options VllmEngine:NzvLLM gRPC server started on %sz"Server is ready to accept requestscstddS)NzReceived shutdown signal)rrsetr stop_eventrrsignal_handlers  z"serve_grpc..signal_handlerzInterrupted by userz!Shutting down vLLM gRPC server...g@)gracezgRPC server stoppedzAsyncLLM engine stoppedzShutdown complete)+rr VLLM_VERSIONrQrr_r from_cli_argscreate_engine_configrOPENAI_API_SERVERrfrom_vllm_configrdisable_log_stats_serverrr3rserverr add_VllmEngineServicer_to_serverr DESCRIPTORservices_by_name full_namer SERVICE_NAMEenable_server_reflectionhostportadd_insecure_portstartasyncioget_running_loopEventsignalSIGTERMSIGINTadd_signal_handlerwaitKeyboardInterruptrtshutdown) rr engine_argsrrservicerr service_namesaddresslooprsigrrr serve_grpcsx                     rc Cstdd}|jdtddd|jdtdd d|jd d d d t|}|}z tt |WdSt yP}zt d|t dWYd}~dSd}~ww)zMain entry point.zvLLM gRPC Server) descriptionz--hostz0.0.0.0zHost to bind gRPC server to)typedefaulthelpz--portizPort to bind gRPC server toz--disable-log-stats-server store_truez$Disable stats logging on server side)actionrzServer failed: %srzN)r add_argumentr6intr add_cli_args parse_argsuvlooprunrr7rr8sysexit)parserrr;rrrmains:  r__main__)-rargparserrrr_collections.abcrr3rgrpc_reflection.v1alpharvllmrrrvllm.engine.arg_utilsrvllm.entrypoints.utilsr vllm.grpcr r vllm.loggerr vllm.outputsr vllm.sampling_paramsr rvllm.usage.usage_librvllm.utils.argparse_utilsrvllm.v1.engine.async_llmr vllm.versionrrrrr Namespacerrrrrrs<          iX&