o $ij@sddlZddlZddlmZmZmZmZmZmZm Z m Z ddl m Z ddl mZddlmZddlmZddlmZmZmZmZmZmZmZddlmZdd lmZdd lm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&dd l'm(Z(m)Z)dd l*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0dd l1m,Z2m3Z4m5Z5m6Z7m8Z9m:Z;m0Z<ddl=m>Z>de5de?fddZ@dedefddZAGdddeZ3de eBe(fdee ee,e(ffddZCGdddZ:de deDe/fd e deEe.fde.fd!d"ZFdS)#N)AnyCallableDictListOptionalSetTupleUnion)FieldDescriptor)Message) cloudpickle)ray_option_utils) BaseModelFieldNonNegativeFloatNonNegativeInt PositiveFloat PositiveInt validator) pickle_dumps)resources_from_ray_options)DEFAULT_CONSTRUCTOR_RETRY_COUNT#DEFAULT_GRACEFUL_SHUTDOWN_TIMEOUT_S%DEFAULT_GRACEFUL_SHUTDOWN_WAIT_LOOP_SDEFAULT_HEALTH_CHECK_PERIOD_SDEFAULT_HEALTH_CHECK_TIMEOUT_SDEFAULT_MAX_ONGOING_REQUESTSMAX_REPLICAS_PER_NODE_MAX_VALUE)DEFAULTDeploymentOptionUpdateType)AggregationFunctionAutoscalingConfigDeploymentMode HTTPOptions ProxyLocationRequestRouterConfig)r!DeploymentConfigDeploymentLanguage EncodingType LoggingConfig ReplicaConfigr%)validate_placement_groupdeployment_languageis_cross_languagecCs(|tjkr |s dS|tjkr|rdSdS)zGFrom Serve client API's perspective, decide whether pickling is needed.TF)r'PYTHONJAVA)r,r-r0V/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/ray/serve/_private/config.py _needs_pickle3s r2protoreturncCsi}|D]6\}}|jtjkr)|jtjkr!dd|D||j<qt|||j<q|jtjkr7t|||j<q|||j<q|j j D]}|j|vrW|jtjkrW|j sW|j ||j<qA|S)zRecursively convert a protobuf into a Python dictionary. This is an alternative to protobuf's `MessageToDict`. Unlike `MessageToDict`, this function doesn't add an extra base64 encoding to bytes when constructing a json response. cSsg|]}t|qSr0)_proto_to_dict).0vr0r0r1 Psz"_proto_to_dict..) ListFieldslabelr LABEL_REPEATEDtype TYPE_MESSAGEnamelistr5 DESCRIPTORfieldscontaining_oneof default_value)r3datafieldvaluer0r0r1r5@s&        r5c@seZdZUdZedejdZee e d<ee ej dZ ee d<edejdZee d<edej dZee d <eeejdZee d <eeej dZee d <eeejdZee d <eeejdZee d <edej dZeee d<ee ej dZ!e e d<dZ"e#e d<e$j%Z&ee d<edej'dZ(ee)e d<edej dZ*ee+e d<ee,ejdZ-ee d<e.Z/e0e)e d<GdddZ1e2d ddddZ3e2dddddZ4e2dddd d!Z5d"d#Z6d$d%Z7d&d'Z8e9d(e:fd)d*Z;e9d+edS)0r&aInternal datastructure wrapping config options for a deployment. Args: num_replicas: The number of processes to start up that handles requests to this deployment. Defaults to 1. max_ongoing_requests: The maximum number of queries that is sent to a replica of this deployment without receiving a response. Defaults to 5. max_queued_requests: Maximum number of requests to this deployment that will be queued at each *caller* (proxy or DeploymentHandle). Once this limit is reached, subsequent requests will raise a BackPressureError (for handles) or return an HTTP 503 status code (for HTTP requests). Defaults to -1 (no limit). user_config: Arguments to pass to the reconfigure method of the deployment. The reconfigure method is called if user_config is not None. Must be JSON-serializable. graceful_shutdown_wait_loop_s: Duration that deployment replicas wait until there is no more work to be done before shutting down. graceful_shutdown_timeout_s: Controller waits for this duration to forcefully kill the replica for shutdown. health_check_period_s: Frequency at which the controller health checks replicas. health_check_timeout_s: Timeout that the controller waits for a response from the replica's health check before marking it unhealthy. autoscaling_config: Autoscaling configuration. logging_config: Configuration for deployment logs. user_configured_option_names: The names of options manually configured by the user. request_router_config: Configuration for deployment request router. max_constructor_retry_count: Maximum number of times to retry the deployment constructor. Defaults to 20. )default update_type num_replicasmax_ongoing_requestsmax_queued_requestsN user_configgraceful_shutdown_timeout_sgraceful_shutdown_wait_loop_shealth_check_period_shealth_check_timeout_sautoscaling_config)default_factoryrIrequest_router_configFr-r,versionlogging_configmax_constructor_retry_countuser_configured_option_namesc@seZdZdZdZdS)zDeploymentConfig.ConfigTN)__name__ __module__ __qualname__validate_assignmentarbitrary_types_allowedr0r0r0r1Configsr_T)alwaysc CsXt|tr|S|dur*zt|W|Sty)}z tdt|dd}~ww|S)Nz&user_config is not JSON-serializable: .) isinstancebytesjsondumps TypeError ValueErrorstr)clsr7er0r0r1user_config_json_serializables  z.DeploymentConfig.user_config_json_serializablecCsL|dur|St|tstdt|dddlm}|di|}|S)NzGot invalid type 'z,' for logging_config. Expected a dictionary.r)r)r0)rbdictrfr<ray.serve.schemar))rir7r)r0r0r1logging_config_valids  z%DeploymentConfig.logging_config_validcCs.t|ts td|dkr|dkrtd|S)Nz'max_queued_requests must be an integer.rGrLz@max_queued_requests must be -1 (no limit) or a positive integer.)rbintrfrg)rir7r0r0r1validate_max_queued_requestss z-DeploymentConfig.validate_max_queued_requestscCst|j|jSN)r2r,r-selfr0r0r1 needs_pickleszDeploymentConfig.needs_picklecCsL|}|ddur|rt|d|d<|dr3|jjj|ddd<td i|d|d<|drt|dd}|dura|sLd|dd<n|rZt||dd<nt d||j j |dd <t d i|d|d<|d rd |d vrt |d d |d d <td i|d |d <t|d |d <td i|S)NrNrSpolicy_serialized_policy_defrUrequest_router_kwargszRNon-empty request_router_kwargs not supportedfor cross-language deployments. Got: _serialized_request_router_clsrWencodingrYr0)rlgetrtr rerSrurvAutoscalingConfigProtorgrUryRequestRouterConfigProtoEncodingTypeProtoValueLoggingConfigProtor?DeploymentConfigProto)rsrD router_kwargsr0r0r1to_protosX       zDeploymentConfig.to_protocC |SrqrSerializeToStringrrr0r0r1to_proto_bytes, zDeploymentConfig.to_proto_bytesr3cCs"t|}d|vr |dntj}d|vr|dnd}t||}d|vr=|ddkr9|r3t|j|d<n |j|d<nd|d<d|vryd|dvrn|dd}|dkrh|r_t|jj|dd<n|jj|dd<ni|dd<t di|d|d<d|vr|d d sd|dd <|d d sd|dd <|d d sd|dd <|d d sd|dd <|d d sd|dd <|d dst j |dd<t di|d|d<d|vr|ddkrd|d<d|vrt|d|d<d|vr d|dvr t|dd|dd<|di|S)Nr,r-FrNrxrUrwrSupscale_smoothing_factordownscale_smoothing_factorupscaling_factordownscaling_factortarget_ongoing_requestsaggregation_functionrVrYrWrzr0)r5r'r.r2r loadsrNrUrwr%r{r MEANr!setr~Name)rir3rDr,r-rtrwr0r0r1 from_proto/s               zDeploymentConfig.from_proto proto_bytescCst|}||Srq)r FromStringr)rirr3r0r0r1from_proto_bytesxs  z!DeploymentConfig.from_proto_bytesc Ks|}t|}|D]\}}||vr'td|d|dt|dqdd|D}|D] \}}|||q5|S)zCreates a default DeploymentConfig and overrides it with kwargs. Ignores any kwargs set to DEFAULT.VALUE. Raises: TypeError: when a keyword that's not an argument to the class is passed in. z&Got invalid Deployment config option "z" (with value zO) as keyword argument. All Deployment config options must come from this list: racSs i|] \}}|tjkr||qSr0)rVALUE)r6keyvalr0r0r1 s z1DeploymentConfig.from_default..)rrlkeysitemsrfr? __setattr__)rikwargsconfigvalid_config_optionsrrr0r0r1 from_default}s  zDeploymentConfig.from_default)?rZr[r\__doc__rr LightWeightrJrr__annotations__rNeedsActorReconfigurerKrrMrorNrrNeedsReconfigurerOrrrPrrQrrrRrSr!r%rUr-boolr'r.r, HeavyWeightrVrhrWrlrrXrrYrr_rrkrnrprtrr classmethodrrrcrrr0r0r0r1r&fs #       0Hr&rKrScCsh|tjdfvrt}||fStjdd}t|tr|n|jdd}||tdi|}||fS)aFReturn modified `max_ongoing_requests` and `autoscaling_config` for when num_replicas="auto". If `autoscaling_config` is unspecified, returns the modified value AutoscalingConfig.default(). If it is specified, the specified fields in `autoscaling_config` override that of AutoscalingConfig.default(). NT exclude_unsetr0)rrr!rHrlrbupdate)rKrSdefault_configr0r0r1handle_num_replicas_autos    rc@seZdZdZ    d/dededededed eeeee fd eed ee d e fd dZ ddZ   d0ded eeeee fd eed ee fddZe       d1deeefdeeedeeeefdeed eeeee fd eed ee deefddZddZd2ddZd2ddZedeeeffdd Zedeeeeeffd!d"Zedeeefd#d$Zed3d%ed e fd&d'Zed3d(ed e fd)d*Zd+d,Z d-d.Z!dS)4r*a>Internal datastructure wrapping config options for a deployment's replicas. Provides five main properties (see property docstrings for more info): deployment_def: the code, or a reference to the code, that this replica should run. init_args: the deployment_def's init_args. init_kwargs: the deployment_def's init_kwargs. ray_actor_options: the Ray actor options to pass into the replica's actor. resource_dict: contains info on this replica's actor's resource needs. Offers a serialized equivalent (e.g. serialized_deployment_def) for deployment_def, init_args, and init_kwargs. Deserializes these properties when they're first accessed, if they were not passed in directly through create(). Use the classmethod create() to make a ReplicaConfig with the deserialized properties. Note: overwriting or setting any property after the ReplicaConfig has been constructed is currently undefined behavior. The config's fields should not be modified externally after it is created. NTdeployment_def_nameserialized_deployment_defserialized_init_argsserialized_init_kwargsray_actor_optionsplacement_group_bundlesplacement_group_strategymax_replicas_per_nodertc Cs`||_||_||_||_d|_d|_d|_||_||_||_ ||_ | t |j|_ | |_dS)zConstruct a ReplicaConfig with serialized properties. All parameters are required. See classmethod create() for defaults. N)rrrr_deployment_def _init_args _init_kwargsrrrr _validater resource_dictrt) rsrrrrrrrrrtr0r0r1__init__s  zReplicaConfig.__init__cCs<||||jdur|jdurtddSdS)NzVSetting max_replicas_per_node is not allowed when placement_group_bundles is provided.)_validate_ray_actor_options!_validate_placement_group_options_validate_max_replicas_per_noderrrgrrr0r0r1rs  zReplicaConfig._validatecCs0||_||_||_||_|t|j|_dSrq)rrrrrrr)rsrrrrr0r0r1rs zReplicaConfig.updatedeployment_def init_args init_kwargsc Cs t|s t|ts td|dust|ttfstd|dus)t|ts)tdt|r:|r4t d|r:t dt|t tfsKtdt |d|durQd }|durWi}|dur]i}|durlt|tri|}n|j }||t |d t|t |d t |d ||||} || _|| _|| _| S) z4Create a ReplicaConfig from deserialized parameters.z8@serve.deployment must be called on a class or function.Nzinit_args must be a tuple.zinit_kwargs must be a dict.z1init_args not supported for function deployments.z3init_kwargs not supported for function deployments.Got invalid type "zQ" for deployment_def. Expected deployment_def to be a class, function, or string.r0z#Could not serialize the deployment z,Could not serialize the deployment init argsz.Could not serialize the deployment init kwargs)callablerbrhrftupler?rlinspect isfunctionrgrr<rZrreprrrr) rirrrrrrrrrr0r0r1create#sT   zReplicaConfig.createcCst|jtstdt|jdhd}|jD]}||vr(td|d|qtj|jdd|jddur@d |jd<dSdS) Nrz/" for ray_actor_options. Expected a dictionary.>memorynum_cpusnum_gpus resources runtime_envaccelerator_typez Specifying 'z8' in ray_actor_options is not allowed. Allowed options: T) in_optionsrrG) rbrrlrfr<rgr validate_actor_optionsr{)rsallowed_ray_actor_optionsoptionr0r0r1rgs"  z)ReplicaConfig._validate_ray_actor_optionsr4cCsf|jdurdSt|jtstdt|jdtd|jdks%|jtkr1td|jdtddS)NzGet invalid type 'zM' for max_replicas_per_node. Expected None or an integer in the range of [1, z].rGzInvalid max_replicas_per_node z:. Valid values are None or an integer in the range of [1, )rrbrorfr<rrgrrr0r0r1rs     z-ReplicaConfig._validate_max_replicas_per_nodec Cs$|jdur|jdurtd|jdurt|j|jpdddd}|jd}|dd}|jdd}||krCt|d |d |d |d d}|jd d}||krat|d|d |d|jdi}|D]#\}} ||d} | | krt|d|d| d | d|d qldSdS)Nz[If `placement_group_strategy` is provided, `placement_group_bundles` must also be provided.PACKdetached)bundlesstrategylifetimezWhen using `placement_group_bundles`, the replica actor will be placed in the first bundle, so the resource requirements for the actor must be a subset of the first bundle.rCPUrz `num_cpus` for the actor is z, but the bundle only has z `CPU` specified.GPUrz `num_gpus` for the actor is z `GPU` specified.rz `z` requirement for the actor is z ` specified.)rrrgr+r{rr) rsresource_error_prefix first_bundle bundle_cpureplica_actor_num_cpus bundle_gpureplica_actor_num_gpusreplica_actor_resourcesactor_resource actor_value bundle_valuer0r0r1rsd        $z/ReplicaConfig._validate_placement_group_optionscCs:|jdur|jrt|j|_|jS|jjdd|_|jS)aDThe code, or a reference to the code, that this replica runs. For Python replicas, this can be one of the following: - Function (Callable) - Class (Callable) - Import path (str) For Java replicas, this can be one of the following: - Class path (str) Nzutf-8)rz)rrtr rrdecoderrr0r0r1rs zReplicaConfig.deployment_defcCs2|jdur|jrt|j|_|jS|j|_|jS)zThe init_args for a Python class. This property is only meaningful if deployment_def is a Python class. Otherwise, it is None. N)rrtr rrrrr0r0r1rs zReplicaConfig.init_argscCs|jdur t|j|_|jS)zThe init_kwargs for a Python class. This property is only meaningful if deployment_def is a Python class. Otherwise, it is None. N)rr rrrrr0r0r1rs zReplicaConfig.init_kwargsr3c Csvt|j|j|jdkr |jnd|jdkr|jndt|j|jr%t|jnd|j dkr.|j nd|j r7|j | Sd| S)Nrxr) r*rrrrrdrrrrr)rir3rtr0r0r1rs$    zReplicaConfig.from_protorcCst|}|||Srq)ReplicaConfigProtorr)rirrtr3r0r0r1rs  zReplicaConfig.from_proto_bytesc CsTt|j|j|j|jt|j|jdurt|jnd|j |j dur&|j dSddS)Nrr)rrrrrrrr) rrrrrrdrerrrrrrr0r0r1rs     zReplicaConfig.to_protocCrrqrrrr0r0r1r'rzReplicaConfig.to_proto_bytes)NNNT)NNN)NNNNNNN)r4N)T)"rZr[r\rrhrcrrrfloatrorrrrlrrr rrrrrrrpropertyrrrrrrrrr0r0r0r1r*s  -    C  5   r*proxy_location http_optionscCs|dur d}t}n/t|trd|v}tdi|}nt|tr.d}tdi|jdd}n tdt|jd|durE|sCtj|_|St ||_|S) aPrepare `HTTPOptions` with a resolved `location` based on `proxy_location` and `http_options`. Precedence: - If `proxy_location` is provided, it overrides any `location` in `http_options`. - Else if `http_options` specifies a `location` explicitly (HTTPOptions(...) or dict with 'location'), keep it. - Else (no `proxy_location` and no explicit `location`) set `location` to `DeploymentMode.EveryNode`. A bare `HTTPOptions()` counts as an explicit default (`HeadOnly`). Args: proxy_location: Optional ProxyLocation (or its string representation). http_options: Optional HTTPOptions instance or dict. If None, a new HTTPOptions() is created. Returns: HTTPOptions: New instance with resolved location. Note: 1. Default ProxyLocation (when unspecified) resolves to DeploymentMode.EveryNode. 2. Default HTTPOptions() location is DeploymentMode.HeadOnly. 3. `HTTPOptions` is used in `imperative` mode (Python API) cluster set-up. `Declarative` mode (CLI / REST) uses `HTTPOptionsSchema`. Raises: ValueError: If http_options is not None, dict, or HTTPOptions. NFlocationTrz#Unexpected type for http_options: ``r0) r#rbrlrgr<rZr" EveryNoderr$_to_deployment_mode)rrlocation_set_explicitlyr0r0r1prepare_imperative_http_options+s$   r)Grrdtypingrrrrrrrr google.protobuf.descriptorr google.protobuf.messager rayr ray._commonr ray._common.pydantic_compatrrrrrrrray._common.serializationrray._common.utilsrray.serve._private.constantsrrrrrrrray.serve._private.utilsrrray.serve.configr r!r"r#r$r%ray.serve.generated.serve_pb2r|r&rr'r(r~r)rr*rr}ray.util.placement_groupr+rr2r5rorrhrlrr0r0r0r1sF(    $ $ $  &9   p