o `۷ii@sddlZddlZddlmZmZmZmZmZmZm Z m Z ddl m Z ddl mZddlmZddlmZddlmZmZmZmZmZmZmZddlmZdd lmZdd lm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&dd l'm(Z(m)Z)dd l*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0dd l1m,Z2m3Z4m5Z5m6Z7m8Z9m:Z;m0Z<ddl=m>Z>de5de?fddZ@dedefddZAGdddeZ3de eBe(fdee ee,e(ffddZCGdddZ:de deDe/fd e deEe.fde.fd!d"ZFdS)#N)AnyCallableDictListOptionalSetTupleUnion)FieldDescriptor)Message) cloudpickle)ray_option_utils) BaseModelFieldNonNegativeFloatNonNegativeInt PositiveFloat PositiveInt validator) pickle_dumps)resources_from_ray_options)DEFAULT_CONSTRUCTOR_RETRY_COUNT#DEFAULT_GRACEFUL_SHUTDOWN_TIMEOUT_S%DEFAULT_GRACEFUL_SHUTDOWN_WAIT_LOOP_SDEFAULT_HEALTH_CHECK_PERIOD_SDEFAULT_HEALTH_CHECK_TIMEOUT_SDEFAULT_MAX_ONGOING_REQUESTSMAX_REPLICAS_PER_NODE_MAX_VALUE)DEFAULTDeploymentOptionUpdateType)AggregationFunctionAutoscalingConfigDeploymentMode HTTPOptions ProxyLocationRequestRouterConfig)r!DeploymentConfigDeploymentLanguage EncodingType LoggingConfig ReplicaConfigr%)validate_placement_groupdeployment_languageis_cross_languagecCs(|tjkr |s dS|tjkr|rdSdS)zGFrom Serve client API's perspective, decide whether pickling is needed.TF)r'PYTHONJAVA)r,r-r0O/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/serve/_private/config.py _needs_pickle3s r2protoreturncCsi}|D]6\}}|jtjkr)|jtjkr!dd|D||j<qt|||j<q|jtjkr7t|||j<q|||j<q|j j D]}|j|vrW|jtjkrW|j sW|j ||j<qA|S)zRecursively convert a protobuf into a Python dictionary. This is an alternative to protobuf's `MessageToDict`. Unlike `MessageToDict`, this function doesn't add an extra base64 encoding to bytes when constructing a json response. cSsg|]}t|qSr0)_proto_to_dict).0vr0r0r1 Pz"_proto_to_dict..) ListFieldslabelr LABEL_REPEATEDtype TYPE_MESSAGEnamelistr5 DESCRIPTORfieldscontaining_oneof default_value)r3datafieldvaluer0r0r1r5@s&        r5c@seZdZUdZedejdZee e d<ee ej dZ ee d<edejdZee d<edej dZee d <eeejdZee d <eeej dZee d <eeejdZee d <eeejdZee d <edej dZeee d<ee ej dZ!e e d<dZ"e#e d<e$j%Z&ee d<edej'dZ(ee)e d<edej dZ*ee+e d<ee,ejdZ-ee d<e.Z/e0e)e d<GdddZ1e2d ddddZ3e2dddddZ4e2dddd d!Z5d"d#Z6d$d%Z7d&d'Z8d(d)Z9e:d*e;fd+d,Ze:d0d1Z?dS)2r&aInternal datastructure wrapping config options for a deployment. Args: num_replicas: The number of processes to start up that handles requests to this deployment. Defaults to 1. max_ongoing_requests: The maximum number of queries that is sent to a replica of this deployment without receiving a response. Defaults to 5. max_queued_requests: Maximum number of requests to this deployment that will be queued at each *caller* (proxy or DeploymentHandle). Once this limit is reached, subsequent requests will raise a BackPressureError (for handles) or return an HTTP 503 status code (for HTTP requests). Defaults to -1 (no limit). user_config: Arguments to pass to the reconfigure method of the deployment. The reconfigure method is called if user_config is not None. Must be JSON-serializable. graceful_shutdown_wait_loop_s: Duration that deployment replicas wait until there is no more work to be done before shutting down. graceful_shutdown_timeout_s: Controller waits for this duration to forcefully kill the replica for shutdown. health_check_period_s: Frequency at which the controller health checks replicas. health_check_timeout_s: Timeout that the controller waits for a response from the replica's health check before marking it unhealthy. autoscaling_config: Autoscaling configuration. logging_config: Configuration for deployment logs. user_configured_option_names: The names of options manually configured by the user. request_router_config: Configuration for deployment request router. max_constructor_retry_count: Maximum number of times to retry the deployment constructor. Defaults to 20. )default update_type num_replicasmax_ongoing_requestsmax_queued_requestsN user_configgraceful_shutdown_timeout_sgraceful_shutdown_wait_loop_shealth_check_period_shealth_check_timeout_sautoscaling_config)default_factoryrJrequest_router_configFr-r,versionlogging_configmax_constructor_retry_countuser_configured_option_namesc@seZdZdZdZdS)zDeploymentConfig.ConfigTN)__name__ __module__ __qualname__validate_assignmentarbitrary_types_allowedr0r0r0r1Configsr`T)alwaysc CsXt|tr|S|dur*zt|W|Sty)}z tdt|dd}~ww|S)Nz&user_config is not JSON-serializable: .) isinstancebytesjsondumps TypeError ValueErrorstr)clsr7er0r0r1user_config_json_serializables  z.DeploymentConfig.user_config_json_serializablecCsL|dur|St|tstdt|dddlm}|di|}|S)NzGot invalid type 'z,' for logging_config. Expected a dictionary.r)r)r0)rcdictrgr=ray.serve.schemar))rjr7r)r0r0r1logging_config_valids  z%DeploymentConfig.logging_config_validcCs.t|ts td|dkr|dkrtd|S)Nz'max_queued_requests must be an integer.rHrMz@max_queued_requests must be -1 (no limit) or a positive integer.)rcintrgrh)rjr7r0r0r1validate_max_queued_requestss z-DeploymentConfig.validate_max_queued_requestscCst|j|jSN)r2r,r-selfr0r0r1 needs_pickleszDeploymentConfig.needs_picklecCsL|}|ddur|rt|d|d<|dr3|jjj|ddd<td i|d|d<|drt|dd}|dura|sLd|dd<n|rZt||dd<nt d||j j |dd <t d i|d|d<|d rd |d vrt |d d |d d <td i|d |d <t|d |d <td i|S)NrOrTpolicy_serialized_policy_defrVrequest_router_kwargszRNon-empty request_router_kwargs not supportedfor cross-language deployments. Got: _serialized_request_router_clsrXencodingrZr0)rmgetrur rfrTrvrwAutoscalingConfigProtorhrVrzRequestRouterConfigProtoEncodingTypeProtoValueLoggingConfigProtor@DeploymentConfigProto)rtrE router_kwargsr0r0r1to_protosX       zDeploymentConfig.to_protocC |SrrrSerializeToStringrsr0r0r1to_proto_bytes, zDeploymentConfig.to_proto_bytescCs|Srr)rmrsr0r0r1to_dict/szDeploymentConfig.to_dictr3cCs"t|}d|vr |dntj}d|vr|dnd}t||}d|vr=|ddkr9|r3t|j|d<n |j|d<nd|d<d|vryd|dvrn|dd}|dkrh|r_t|jj|dd<n|jj|dd<ni|dd<t di|d|d<d|vr|d d sd|dd <|d d sd|dd <|d d sd|dd <|d d sd|dd <|d d sd|dd <|d dst j |dd<t di|d|d<d|vr|ddkrd|d<d|vrt|d|d<d|vr d|dvr t|dd|dd<|di|S)Nr,r-FrOryrVrxrTupscale_smoothing_factordownscale_smoothing_factorupscaling_factordownscaling_factortarget_ongoing_requestsaggregation_functionrWrZrXr{r0)r5r'r.r2r loadsrOrVrxr%r|r MEANr!setrName)rjr3rEr,r-rurxr0r0r1 from_proto3s               zDeploymentConfig.from_proto proto_bytescCst|}||Srr)r FromStringr)rjrr3r0r0r1from_proto_bytes|s  z!DeploymentConfig.from_proto_bytesc Ks|}t|}|D]\}}||vr'td|d|dt|dqdd|D}|D] \}}|||q5|S)zCreates a default DeploymentConfig and overrides it with kwargs. Ignores any kwargs set to DEFAULT.VALUE. Raises: TypeError: when a keyword that's not an argument to the class is passed in. z&Got invalid Deployment config option "z" (with value zO) as keyword argument. All Deployment config options must come from this list: rbcSs i|] \}}|tjkr||qSr0)rVALUE)r6keyvalr0r0r1 s z1DeploymentConfig.from_default..)rrmkeysitemsrgr@ __setattr__)rjkwargsconfigvalid_config_optionsrrr0r0r1 from_defaults  zDeploymentConfig.from_default)@r[r\r]__doc__rr LightWeightrKrr__annotations__rNeedsActorReconfigurerLrrNrprOrrNeedsReconfigurerPrrrQrrRrrrSrTr!r%rVr-boolr'r.r, HeavyWeightrWrirXrmrrYrrZrr`rrlrorqrurrr classmethodrrrdrrr0r0r0r1r&fs #       0Hr&rLrTcCsh|tjdfvrt}||fStjdd}t|tr|n|jdd}||tdi|}||fS)aFReturn modified `max_ongoing_requests` and `autoscaling_config` for when num_replicas="auto". If `autoscaling_config` is unspecified, returns the modified value AutoscalingConfig.default(). If it is specified, the specified fields in `autoscaling_config` override that of AutoscalingConfig.default(). NT exclude_unsetr0)rrr!rIrmrcupdate)rLrTdefault_configr0r0r1handle_num_replicas_autos    rc@sbeZdZdZ      d5dededededed eeeee fd eed eeeeefd eeeee fd ee de fddZ ddZddZ     d6ded eeeee fd eed eeeeefd eeeee fd ee f ddZe         d7deeefdeee deee e fdeed eeeee fd eed eeeeefd eeeee fd ee deefddZddZd8dd Zd8d!d"Zedeeeffd#d$Zedeeee effd%d&Zedeee fd'd(Zed9d)ede fd*d+Zed9d,ede fd-d.Z d/d0Z!d1d2Z"d3d4Z#dS):r*a>Internal datastructure wrapping config options for a deployment's replicas. Provides five main properties (see property docstrings for more info): deployment_def: the code, or a reference to the code, that this replica should run. init_args: the deployment_def's init_args. init_kwargs: the deployment_def's init_kwargs. ray_actor_options: the Ray actor options to pass into the replica's actor. resource_dict: contains info on this replica's actor's resource needs. Offers a serialized equivalent (e.g. serialized_deployment_def) for deployment_def, init_args, and init_kwargs. Deserializes these properties when they're first accessed, if they were not passed in directly through create(). Use the classmethod create() to make a ReplicaConfig with the deserialized properties. Note: overwriting or setting any property after the ReplicaConfig has been constructed is currently undefined behavior. The config's fields should not be modified externally after it is created. NTdeployment_def_nameserialized_deployment_defserialized_init_argsserialized_init_kwargsray_actor_optionsplacement_group_bundlesplacement_group_strategy%placement_group_bundle_label_selector!placement_group_fallback_strategymax_replicas_per_noderuc Cst||_||_||_||_d|_d|_d|_||_||_||_ ||_ | |_ | |_ | |t|j|_| |_dS)zConstruct a ReplicaConfig with serialized properties. All parameters are required. See classmethod create() for defaults. N)rrrr_deployment_def _init_args _init_kwargsrrrrrr _normalize_bundle_label_selector _validater resource_dictru) rtrrrrrrrrrrrur0r0r1__init__s$  zReplicaConfig.__init__csd|jr*|jr,t|jdkr.t|jdkr0|jdfddtt|jD|_dSdSdSdSdS)zsIf a single selector is provided for multiple bundles, it is broadcasted uniformly to all bundles. rHrcsg|]}qSr0)copy)r6_single_selectorr0r1r8r9zBReplicaConfig._normalize_bundle_label_selector..N)rrlenrangersr0rr1rs    z.ReplicaConfig._normalize_bundle_label_selectorcCs<||||jdur|jdurtddSdS)NzVSetting max_replicas_per_node is not allowed when placement_group_bundles is provided.)_validate_ray_actor_options!_validate_placement_group_options_validate_max_replicas_per_noderrrhrsr0r0r1rs  zReplicaConfig._validatecCsD||_||_||_||_||_||_||t|j|_ dSrr) rrrrrrrrrr)rtrrrrrrr0r0r1r+s zReplicaConfig.updatedeployment_def init_args init_kwargsc Cs&t|s t|ts td|dust|ttfstd|dus)t|ts)tdt|r:|r4t d|r:t dt|t tfsKtdt |d|durQd }|durWi}|dur]i}| durlt|tri|} n|j } || t |d t|t |d t |d |||||| d } || _|| _|| _| S)z4Create a ReplicaConfig from deserialized parameters.z8@serve.deployment must be called on a class or function.Nzinit_args must be a tuple.zinit_kwargs must be a dict.z1init_args not supported for function deployments.z3init_kwargs not supported for function deployments.Got invalid type "zQ" for deployment_def. Expected deployment_def to be a class, function, or string.r0z#Could not serialize the deployment z,Could not serialize the deployment init argsz.Could not serialize the deployment init kwargs) rrrrrrrrrr)callablercrirgtupler@rminspect isfunctionrhrr=r[rreprrrr) rjrrrrrrrrrrrr0r0r1createDs`   zReplicaConfig.createcCst|jtstdt|jdhd}|jD]}||vr(td|d|qtj|jdd|jddur@d |jd<dSdS) Nrz/" for ray_actor_options. Expected a dictionary.>memorynum_cpusnum_gpus resources runtime_envlabel_selectoraccelerator_typefallback_strategyz Specifying 'z8' in ray_actor_options is not allowed. Allowed options: T) in_optionsrrH) rcrrmrgr=rhr validate_actor_optionsr|)rtallowed_ray_actor_optionsoptionr0r0r1rs"  z)ReplicaConfig._validate_ray_actor_optionsr4cCsf|jdurdSt|jtstdt|jdtd|jdks%|jtkr1td|jdtddS)NzGet invalid type 'zM' for max_replicas_per_node. Expected None or an integer in the range of [1, z].rHzInvalid max_replicas_per_node z:. Valid values are None or an integer in the range of [1, )rrcrprgr=rrhrsr0r0r1rs     z-ReplicaConfig._validate_max_replicas_per_nodec Cs|jdur|jdurtd|jdurI|jdurtdt|jts-tdt|jdt|jD]\}}t|t sHtd|dt|dq2|j durW|jdurWtd|jdurt |j|jpcdd |j d d }|jd }| d d }|j dd }||krt|d|d|d| dd }|j dd }||krt|d|d|d|j di} | D]#\} } | | d } | | krt|d| d| d| d| d qdSdS)Nz[If `placement_group_strategy` is provided, `placement_group_bundles` must also be provided.zdIf `placement_group_fallback_strategy` is provided, `placement_group_bundles` must also be provided.zGplacement_group_fallback_strategy must be a list of dictionaries. Got: rbz1placement_group_fallback_strategy entry at index z must be a dictionary. Got: zhIf `placement_group_bundle_label_selector` is provided, `placement_group_bundles` must also be provided.PACKdetached)bundlesstrategylifetimebundle_label_selectorzWhen using `placement_group_bundles`, the replica actor will be placed in the first bundle, so the resource requirements for the actor must be a subset of the first bundle.rCPUrz `num_cpus` for the actor is z, but the bundle only has z `CPU` specified.GPUrz `num_gpus` for the actor is z `GPU` specified.rz `z` requirement for the actor is z ` specified.)rrrhrrcr@rgr= enumeratermrr+r|rr) rtirresource_error_prefix first_bundle bundle_cpureplica_actor_num_cpus bundle_gpureplica_actor_num_gpusreplica_actor_resourcesactor_resource actor_value bundle_valuer0r0r1rs              %z/ReplicaConfig._validate_placement_group_optionscCs:|jdur|jrt|j|_|jS|jjdd|_|jS)aDThe code, or a reference to the code, that this replica runs. For Python replicas, this can be one of the following: - Function (Callable) - Class (Callable) - Import path (str) For Java replicas, this can be one of the following: - Class path (str) Nzutf-8)r{)rrur rrdecodersr0r0r1rs zReplicaConfig.deployment_defcCs2|jdur|jrt|j|_|jS|j|_|jS)zThe init_args for a Python class. This property is only meaningful if deployment_def is a Python class. Otherwise, it is None. N)rrur rrrsr0r0r1r)s zReplicaConfig.init_argscCs|jdur t|j|_|jS)zThe init_kwargs for a Python class. This property is only meaningful if deployment_def is a Python class. Otherwise, it is None. N)rr rrrsr0r0r1r8s zReplicaConfig.init_kwargsr3c Cst|j|j|jdkr |jnd|jdkr|jndt|j|jr%t|jnd|j dkr.|j nd|j r8t|j nd|j rBt|j nd|j rL|j |d Sd|d S)Nryr) rrrrrrrrrrru) r*rrrrrerrrrrrr)rjr3rur0r0r1rEs0   zReplicaConfig.from_protorcCst|}|||Srr)ReplicaConfigProtorr)rjrrur3r0r0r1ris  zReplicaConfig.from_proto_bytesc Cs|jdur t|jnd}|jdurt|jnd}|jdur%t|jnd}|jdur/|jnd}t|j|j|j |j t|j ||j |||d S)Nrr) rrrrrrrrrr) rrerfrrrrrrrrrr)rtrrrrr0r0r1rns4       zReplicaConfig.to_protocCrrrrrsr0r0r1rrzReplicaConfig.to_proto_bytescCs"|j|j|j|j|j|j|jdS)Nrrrrrrrrrsr0r0r1rszReplicaConfig.to_dict)NNNNNT)NNNNN) NNNNNNNNN)r4N)T)$r[r\r]rrirdrrrfloatrrprrrrrmrrr rrrrrrpropertyrrrrrrrrrr0r0r0r1r*s    4      K ! O  #$ r*proxy_location http_optionscCs|dur d}t}n/t|trd|v}tdi|}nt|tr.d}tdi|jdd}n tdt|jd|durE|sCtj|_|St ||_|S) aPrepare `HTTPOptions` with a resolved `location` based on `proxy_location` and `http_options`. Precedence: - If `proxy_location` is provided, it overrides any `location` in `http_options`. - Else if `http_options` specifies a `location` explicitly (HTTPOptions(...) or dict with 'location'), keep it. - Else (no `proxy_location` and no explicit `location`) set `location` to `DeploymentMode.EveryNode`. A bare `HTTPOptions()` counts as an explicit default (`HeadOnly`). Args: proxy_location: Optional ProxyLocation (or its string representation). http_options: Optional HTTPOptions instance or dict. If None, a new HTTPOptions() is created. Returns: HTTPOptions: New instance with resolved location. Note: 1. Default ProxyLocation (when unspecified) resolves to DeploymentMode.EveryNode. 2. Default HTTPOptions() location is DeploymentMode.HeadOnly. 3. `HTTPOptions` is used in `imperative` mode (Python API) cluster set-up. `Declarative` mode (CLI / REST) uses `HTTPOptionsSchema`. Raises: ValueError: If http_options is not None, dict, or HTTPOptions. NFlocationTrz#Unexpected type for http_options: ``r0) r#rcrmrhr=r[r" EveryNoderr$_to_deployment_mode)rrlocation_set_explicitlyr0r0r1prepare_imperative_http_optionss$   r)Grretypingrrrrrrrr google.protobuf.descriptorr google.protobuf.messager rayr ray._commonr ray._common.pydantic_compatrrrrrrrray._common.serializationrray._common.utilsrray.serve._private.constantsrrrrrrrray.serve._private.utilsrrray.serve.configr r!r"r#r$r%ray.serve.generated.serve_pb2r}r&rr'r(rr)rr*rr~ray.util.placement_groupr+rr2r5rprrirmrr0r0r0r1sH(    $ $ $  &=   d