o `۷i,-@sUdZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z ddl Z ddlmZddlmZddlmZmZdd lmZdd lmZmZmZdd lmZdd lmZdd lm Z ddl!m"Z"ddl#m$Z$e%eZ&da'de(d<da)ee(d<e$eGdddZ* dQde+de+de efddZ,ddZ-ddZ.ddd ed!e d"ed#e"d$e/d%e e e0e0gdff d&d'Z1dRde+defd(d)Z2edd*Gd+d,d,Z3ej4d-dd.Z5ej4d/dd.Z6d0d1Z7d2d3Z8 4 4 4 4 4dSd5e0d6e0d7e0d8e0d9e0f d:d;Z9de e3fd?d@Z;ee<Z=e e0e e0ej>ffe(dA<dBe0de e0ej>ffdCdDZ?dBe0dEe0fdFdGZ@dBe0dEe0fdHdIZAee<ZBe e0e e0effe(dJ<dKdLZCdMdNZDdOdPZEdS)Tz This file stores global state for a Serve application. Deployment replicas can use this state to access metadata or the Serve controller. N) defaultdict) dataclass)CallableDictListOptional) RayActorError)ServeControllerClient) DeploymentID ReplicaID)DeploymentConfig)SERVE_CONTROLLER_NAMESERVE_LOGGER_NAMESERVE_NAMESPACE) ReplicaResult)RayServeException)RayServegRPCContext) ReplicaRank) DeveloperAPIReplicaContext_INTERNAL_REPLICA_CONTEXT_global_clientc@seZdZUdZeed<eed<eed<eed<e ed<dZ e ee gdfed<e d efd d Ze d efd d Ze d efddZdS)raStores runtime context info for replicas. Fields: - app_name: name of the application the replica is a part of. - deployment: name of the deployment the replica is a part of. - replica_tag: unique ID for the replica. - servable_object: instance of the user class/function this replica is running. - rank: the rank of the replica. - world_size: the number of replicas in the deployment. replica_idservable_object_deployment_configrank world_sizeN_handle_registration_callbackreturncC |jjjSN)r deployment_idapp_nameselfr%G/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/serve/context.pyr"8 zReplicaContext.app_namecCrr )rr!namer#r%r%r& deployment<r'zReplicaContext.deploymentcCs|jjSr )r unique_idr#r%r%r& replica_tag@zReplicaContext.replica_tag)__name__ __module__ __qualname____doc__r __annotations__rr rintrrr propertystrr"r)r+r%r%r%r&r#s  FT_health_check_controllerraise_if_no_controller_runningrcCsZztdur|rttjjtWSWt |Sty,tdt dYt |Sw)aGets the global client, which stores the controller's handle. Args: _health_check_controller: If True, run a health check on the cached controller if it exists. If the check fails, try reconnecting to the controller. raise_if_no_controller_running: Whether to raise an exception if there is no currently running Serve controller. Returns: ServeControllerClient to the running Serve controller. If there is no running controller and raise_if_no_controller_running is set to False, returns None. Raises: RayServeException: If there is no running Serve controller actor and raise_if_no_controller_running is set to True. Nz-The cached controller has died. Reconnecting.) rrayget _controller check_aliveremoterloggerinfo_set_global_client_connect)r5r6r%r%r&_get_global_clientEs  r@cCs|adSr )r)clientr%r%r&r>gr,r>cCstSr )rr%r%r%r&_get_internal_replica_contextlsrB)handle_registration_callbackrrrrrrCcCst||||||dadS)N)rrrrrr)rr)rrrrrrCr%r%r&_set_internal_replica_contextps  rDcCshdtjjj_tstjtdz tjt td}Wnt y)|r&t dYdSwt |}t ||S)aOConnect to an existing Serve application on this Ray cluster. If called from within a replica, this will connect to the same Serve app that the replica is running in. Returns: ServeControllerClient that encapsulates a Ray actor handle to the existing Serve application's Serve Controller. None if there is no running Serve controller actor and raise_if_no_controller_running is set to False. Raises: RayServeException: If there is no running Serve controller actor and raise_if_no_controller_running is set to True. F) namespacez7There is no Serve instance running on this Ray cluster.N)r7_privateworker global_worker_filter_logs_by_jobis_initializedinitr get_actorr ValueErrorrr r>)r6 controllerrAr%r%r&r?s"   r?)frozenc@seZdZUdZeed<dZeed<dZeed<dZeed<dZ eed<dZ e e ed<d Z eed <d Zeed <dZe eed <dS) _RequestContextroute request_id_internal_request_idr"multiplexed_model_idN grpc_contextFis_http_requestcancel_on_parent_request_cancel_ray_trace_ctx)r-r.r/rRr4r1rSrTr"rUrVrrrWboolrXrYdictr%r%r%r&rPs       rPz'Serve internal request context variable)defaultz0Serve internal batching request context variablecCs tdur tttS)zWGet the current request context. Returns: The current request context N)_serve_request_contextr8setrPr%r%r%r&_get_serve_request_contexts  r_cCstdur tgtS)z7Get the list of request contexts for the current batch.N)_serve_batch_request_contextr8r^r%r%r%r& _get_serve_batch_request_contexts  rarQrRrSrTr"rUc Cs@t}tt|p |j|p|j|p|j|p|j|p|jddS)z]Set the request context. If the value is not set, the current context value will be used.)rRrSrTr"rUN) r_r]r^rPrRrSrTr"rU)rRrSrTr"rUcurrent_request_contextr%r%r&_set_request_contexts rccCsttdS)zUnset the request context.N)r]r^rPr%r%r%r&_unset_request_contextsrdrequest_contextscCst|dS)z5Add the request context to the batch request context.N)r`r^)rer%r%r&_set_batch_request_contextsrf_requests_pending_assignmentparent_request_idcC|tvrt|SiSr rgrhr%r%r& _get_requests_pending_assignmentrl response_idcC|t||<dSr rj)rhrntaskr%r%r&_add_request_pending_assignment"srqcC4|t|vr t||=tt|dkrt|=dSdSNr)rglenrhrnr%r%r&"_remove_request_pending_assignment(   rv_in_flight_requestscCrir rxrkr%r%r&_get_in_flight_requests?rmrzcCror ry)rhrnreplica_resultr%r%r&_add_in_flight_requestFsr|cCrrrs)rxrtrur%r%r&_remove_in_flight_requestJrwr})FT)T)rQrQrQrQrQ)Fr0asyncio contextvarslogging collectionsr dataclassesrtypingrrrrr7ray.exceptionsrray.serve._private.clientr ray.serve._private.commonr r ray.serve._private.configr ray.serve._private.constantsr rr!ray.serve._private.replica_resultrray.serve.exceptionsrray.serve.grpc_utilrray.serve.schemarray.util.annotationsr getLoggerr<rr1rrrZr@r>rBr2r4rDr?rP ContextVarr]r`r_rarcrdrfr[rgTaskrlrqrvrxrzr|r}r%r%r%r&s             ! "  6  "