o ;iO@sXddlZddlZddlZddlZddlZddlZddlZddlmZddl m Z m Z m Z m Z ddlmZddlmZddlmZddlmZddlmZdd lmZd d lmZd d lmZd d lmZ d dl!m"Z"m#Z#d dl$m%Z%d dl&m'Z'd dl(m)Z)dZ*GdddZ+e"e+e,dZ-e#j.    d=de/de ej0de e1de/de/de2de+fdd Z3Gd!d"d"Z4e"e4Z5e#j.   # # $ % d>d&e1d'e1d(e1d)e1d*e6d+e e/d,e e/d-e6d.e6d/e/d0e/d1e/d2e e/de4fd3d4Z7e#j.d&e1d'e1de8e9e1e ffd5d6Z: d?gdddd7de e/d8e8e1de/de e/de2f d9d:Z;e"e;e,dZ.check_process_is_runningF localhostr,r-)TN皙?zWaited too long for port z to start accepting connections)time monotonicrr0asynciowait_foropen_connectionrclose wait_closedCancelledErrorOSError TimeoutErrorsleep)r(rr- start_timer4error_writerr)r3r*is_port_connection_healthy5s.  " z(_FlashManager.is_port_connection_healthyc s|jIdH|_t|jj}|j}|jpd}z |||IdHWntt t j fyA| IdH|jj tIdHwt ||||_t ||_dS)Ni)r" __aenter__tunnelrurlhostnamer_wait_for_port_successr0KeyboardInterruptr:r? _deregister __aexit__sysexc_info create_task_run_heartbeatheartbeat_task_drain_container drain_task)r( parsed_urlhostrr)r)r*_startQs  z_FlashManager._startcs*t|jjjtdddIdHdS)Nr r-retry)r:shieldrstubFlashContainerDeregisterr FlashContainerDeregisterRequestr(r)r)r*rNasz_FlashManager._deregisterc s z2|jtkr3td|jd|jjd|IdH|jr0|jj t j |jdIdHWdSWn1t jyDtdYdStye}ztd|t d IdHWYd}~nd}~wwz t d IdHWnt jytdYdSwq) zt Background task that checks if we've encountered too many failures and drains the container if so. Tz[Modal Flash] Draining task  on z due to too many failures.N)r'[Modal Flash] Shutting down...z([Modal Flash] Error draining container: )r$ _MAX_FAILURESrwarningr'rIrJstoprr] ContainerStopr ContainerStopRequestr:r?r0rDrBr(er)r)r*rUjs6   z_FlashManager._drain_containerrXc s*t}t||jkrz9|j|jdIdH\}}|rF|jjjtj dd||ddddIdH}t d|j d|j j d|jWd SWn'tjyUt d tyn}zt d |WYd}~nd}~wwz td IdHWntjyt d wt||jkstd ) Nr3rpriorityweightrXrrZz Listening at z over z for task_id TzIWaited too long for port to start accepting connections. Shutting down...z7Error waiting for port to start accepting connections: rc)r8r9rrGrrr]FlashContainerRegisterr FlashContainerRegisterRequestrinforJrIr'r:r?rer0rDrBrA)r(rXrrCport_check_resprEresprjr)r)r*rLsD $   z$_FlashManager._wait_for_port_successc s8 zK|j|jdIdH\}}|r)|jjjtjdd||ddddIdH}d|_n#t d|j d |j j d |d |j|jd 7_| IdHWn0tjydtd | IdHYdSty}}zt d|WYd}~nd}~wwz td IdHWntjy| IdHYdSwq)NTr3rrkrlrZrz&[Modal Flash] Deregistering container raz due to error: z, num_heartbeat_failures: rcrbz [Modal Flash] Heartbeat failed: )rGrrr]ror rpr$rrDr'rIrJrNr:r?rer0rB)r(rXrrrport_check_errorrsrjr)r)r*rSsR   z_FlashManager._run_heartbeatcCs|jjSN)rIrJr`r)r)r*get_container_urlsz_FlashManager.get_container_urlc sz*|jr*|jz tj|jddIdHWntjtjfy)tdYnwWntyE}zt d|WYd}~nd}~wwd|_ td|j j ddS)Nrkr6z4[Modal Flash] Heartbeat task did not stop within 5s.z[Modal Flash] Error stopping: Tz2[Modal Flash] No longer accepting new requests on .) rTcancelr:r;rAr?rrer0rDr#rIrJrir)r)r*rfs  z_FlashManager.stopcsX|js |IdHt|jIdHtd|jjd|j j t IdHdS)Nz [Modal Flash] Closing tunnel on rw) r#rfr:rBrrrerIrJr"rOrPrQr`r)r)r*r=s z_FlashManager.closeNNrrF)r,)__name__ __module__ __qualname__rintr subprocessPopenstrboolr+floattupler0rGrYrNrUrLrSrvrfr=r)r)r)r*rsJ   # r) target_modulerFrrrrrrr.c s8tIdH}t|||||||d}|IdH|S)a Forward a port to the Modal Flash service, exposing that port as a stable web endpoint. This is a highly experimental method that can break or be removed at any time without warning. Do not use this method unless explicitly instructed to do so by Modal support. N)rrrrr)rfrom_envrrY)rrrrrrrmanagerr)r)r* flash_forwards rc@sFeZdZdZdededededededeed eed eed ed ed ededefddZ ddZ ddZ dedefddZ dedededededef ddZ deeeffd d!Zd"edeeeeeffd#d$Zd%d&Zd'efd(d)Z * + , , ,d2ded-eeeefd ededeed eed eedefd.d/Zd0d1Zd,S)3_FlashPrometheusAutoscalerirapp_namecls_namemetrics_endpoint target_metrictarget_metric_valuemin_containersmax_containersbuffer_containersscale_up_tolerancescale_down_tolerance%scale_up_stabilization_window_seconds'scale_down_stabilization_window_secondsautoscaling_interval_secondscCsddl}| |jkrtd|j| |jkrtd|j|dkr&td||_||_||_||_||_||_||_ ||_ | |_ | |_ | |_ | |_| |_||_t||}|j|_||_||_tj|d|ddd|_d|_dS) NrzDscale_up_stabilization_window_seconds must be less than or equal to zFscale_down_stabilization_window_seconds must be less than or equal to z*target_metric_value must be greater than 0-z-autoscaling-decisionsT)create_if_missing)aiohttp_max_window_secondsrrrrrrrrrrrrrrrr from_name_class_service_functionfncls ClientSession http_clientr autoscaling_decisions_dictautoscaler_thread)r(rrrrrrrrrrrrrrr FlashClassr)r)r*r+ sD       z#_FlashPrometheusAutoscaler.__init__cs,|jj|jdIdHt||_dS)Nr)rhydraterr:rR_run_autoscaler_looprr`r)r)r*startCsz _FlashPrometheusAutoscaler.startc s. ztjddIdH}jdgIdH}t|ts*td|d}t|ts9td|g}|D]'}t|trVt |dksVt|dt rVt|dtsbtd|g}nq;fdd |D}j |d IdH}| |fj ||jjjjjd }td |d tdjd|IdHjd|IdH|IdHtjkrtjtIdHWnBtjytdjIdHYdSty}ztd|tttjIdHWYd}~nd}~wwq)NTcurrent_replicasrautoscaling_decisionsz5[Modal Flash] Invalid item in autoscaling decisions: r rccs&g|]\}}|jkr||fqSr))r).0 timestampdecisionautoscaling_timer(r)r* _s zC_FlashPrometheusAutoscaler._run_autoscaler_loop..)r)rrrrrz2[Modal Flash] Scaling to actual_target_containers=z+ containers. Autoscaling decision made in z seconds.z)[Modal Flash] Shutting down autoscaler...z#[Modal Flash] Error in autoscaler: )r8rget isinstancer}rrelistrlenr_compute_target_containersappend_make_scaling_decisionrrrrrput_set_target_slotsrr:rBr?rr=r0rD traceback format_exc)r(rritemcurrent_target_containersactual_target_containersrjr)rr*rGs~          z/_FlashPrometheusAutoscaler._run_autoscaler_looprr.cs|IdH}t||kr td|dt|dt|}|dkr&dS||IdH\}}|j|||t||jd}td|S)zF Gets metrics from container to autoscale up or down. Nz[Modal Flash] Current replicas z' is less than the number of containers z,. Setting current_replicas = num_containers.rrc)n_current_replicas sum_metricn_containers_with_metricsn_total_containersr)_get_all_containersrrrq_get_scaling_info_calculate_desired_replicasrmax)r(r containersrrdesired_replicasr)r)r*rs(  z5_FlashPrometheusAutoscaler._compute_target_containersrrrrcCs|jpd}||}||}t||d} |dd|j||| } |||| } | |} | |} | }| d|jkrDt|| }n| d|jkrRt|| }td|d|jd|d|d|d |d |d | d | d | d||S)zK Calculate the desired number of replicas to autoscale to. rrcg?z [Modal Flash] Current replicas: z, target metric: ztarget metric value: z , current sum of metric values: z%, number of containers with metrics: z", number of containers unhealthy: z<, number of containers missing metric (includes unhealthy): z$, number of provisioned containers: z, scale up ratio: z, scale down ratio: z, desired replicas: ) rrrmathceilrrrer)r(rrrrrrn_containers_missing_metricn_containers_unhealthynum_provisioned_containersscale_up_target_metric_valuescale_down_target_metric_valuescale_up_ratioscale_down_ratiorr)r)r*rsR    z6_FlashPrometheusAutoscaler._calculate_desired_replicascs|d}d}tjfdd|DIdH}|D]#}|dus*j|vs*t|jdkr+q||jdj7}|d7}q||fS)z6Get metrics using container exposed metrics endpoints.rc s.g|]}d|jd|jdjqS)zhttps://:/) _get_metricsrXrr)r containerr`r)r*rs z@_FlashPrometheusAutoscaler._get_scaling_info..Nrc)r:gatherrrvalue)r(rrrcontainer_metrics_listcontainer_metricsr)r`r*rs     z,_FlashPrometheusAutoscaler._get_scaling_inforJc sDddlm}m}z|jj|ddIdH}|Wn0tjy-t d|YdSt yJ}zt d|d|WYd}~dSd}~wwz | IdH}Wn0tjygt d|YdSt y}zt d |d|WYd}~dSd}~wwt t }||D]}|jD] } || j| g7<qq|S) Nr)Sampletext_string_to_metric_familiesr6z+[Modal Flash] Timeout getting metrics from z)[Modal Flash] Error getting metrics from z: z0[Modal Flash] Timeout reading metrics body from z.[Modal Flash] Error reading metrics body from )prometheus_client.parserrrrrraise_for_statusr:rArrer0textrrsamplesname) r(rJrrresponserj text_bodymetricsfamilysampler)r)r*rs:   z'_FlashPrometheusAutoscaler._get_metricscs,tj|jjd}|jj|IdH}|jS)N function_id)r FlashContainerListRequestr object_idrr]FlashContainerListr)r(reqrsr)r)r*rsz._FlashPrometheusAutoscaler._get_all_containers target_slotscs,tj|jj|d}|jj|IdHdS)N)rr)r !FlashSetTargetSlotsMetricsRequestrrrr]FlashSetTargetSlotsMetrics)r(rrr)r)r*rsz,_FlashPrometheusAutoscaler._set_target_slotsr,Nrc s|s|S|jddd|d\}} | |kr1||fdd|D} | r)t| n| } t|| } n!| |krP||fdd|D} | rHt| n| } t|| } n|} |dur[t|| } |durdt|| } |durl| |7} | S) a] Return the target number of containers following (simplified) Kubernetes HPA stabilization-window semantics. Args: current_replicas: Current number of running Pods/containers. autoscaling_decisions: List of (timestamp, desired_replicas) pairs, where timestamp is a UNIX epoch float (seconds). The list *must* contain at least one entry and should already include the most-recent measurement. scale_up_stabilization_window_seconds: 0 disables the up-window. scale_down_stabilization_window_seconds: 0 disables the down-window. min_containers / max_containers: Clamp the final decision to this range. Returns: The target number of containers. cSs|dS)Nrr))recr)r)r*CszC_FlashPrometheusAutoscaler._make_scaling_decision..)keycg|] \}}|kr|qSr)r)rtsdesired window_startr)r*rJzE_FlashPrometheusAutoscaler._make_scaling_decision..crr)r)rrr)r*rQrN)sortminr) r(rrrrrrrnow_tslatest_desireddesired_candidates candidate new_replicasr)rr*r"s,     z1_FlashPrometheusAutoscaler._make_scaling_decisioncs|j|jIdHdSru)rrxr`r)r)r*rfds z_FlashPrometheusAutoscaler.stop)rrNNN)rzr{r|rrrrrr}r+rrrrrrdictrrrrrrrfr)r)r)r*rs      9C :"   Brr7rrrrrrrrrrrrrrc sjzddl} Wn tytdwtIdH}t||||||||| ||| | | d}|IdH|S)aH Autoscale a Flash service based on containers' Prometheus metrics. The package `prometheus_client` is required to use this method. This is a highly experimental method that can break or be removed at any time without warning. Do not use this method unless explicitly instructed to do so by Modal support. rNz?The package `prometheus_client` is required to use this method.)rrrrrrrrrrrrrr)prometheus_client ImportErrorrrrr)rrrrrrrrrrrrrrr autoscalerr)r)r*flash_prometheus_autoscalerls2#  rcsbtIdH}t||j}|dusJ|j|dIdHtj|jd}|j |IdH}|j S)a Return a list of flash containers for a deployed Flash service. This is a highly experimental method that can break or be removed at any time without warning. Do not use this method unless explicitly instructed to do so by Modal support. Nrr) rrr rrrr rrr]rr)rrrrrrsr)r)r*flash_get_containerss r) proxy_regionsrrrrcs||durtdt||||ddlmm}|tj|||pd|p"d|dddttdt ffd ffd d }|S) akDecorator for Flash-enabled HTTP servers on Modal classes. Args: port: The local port to forward to the HTTP server. proxy_regions: The regions to proxy the HTTP server to. startup_timeout: The maximum time to wait for the HTTP server to start. exit_grace_period: The time to wait for the HTTP server to exit gracefully. NzePositional arguments are not allowed. Did you forget parentheses? Suggestion: `@modal.http_server()`.r)_PartialFunction_PartialFunctionParams)rrrrr) http_configobj.r.cs8tj}t|r||}n||}|d|S)Nz `http_server`)r HTTP_WEB_INTERFACErstackvalidate_obj_compatibility)r flagspfr paramsr)r*wrappers    z_http_server..wrapper) rrmodal._partial_functionr r r HTTPConfigrrr)rrrrrr rr)rr* _http_servers"(  rc@sFeZdZUdZeeed<dejfddZ ddZ dd Z d d Z d S) _FlashContainerEntryai A class that manages the lifecycle of Flash manager for Flash containers. It is intentional that stop() runs before exit handlers and close(). This ensures the container is deregistered first, preventing new requests from being routed to it while exit handlers execute and the exit grace period elapses, before finally closing the tunnel. flash_managerr cCs||_d|_dSru)r r)r(r r)r)r*r+s z_FlashContainerEntry.__init__c Cs|jtkrJz tj}|dkrWdSWn tyYnwzt|jj|jj|jj |jj d|_ WdSt yI}z t d|td}~wwdS)Nr)rrrz[Modal Flash] Startup failed: )r r rrrankrrrrrrrr0rrer)r(rrjr)r)r*enters, z_FlashContainerEntry.entercC|jr |jdSdSru)rrfr`r)r)r*rfz_FlashContainerEntry.stopcCrru)rr=r`r)r)r*r=rz_FlashContainerEntry.closeN) rzr{r|__doc__r FlashManager__annotations__r rr+rrfr=r)r)r)r*rs   rry)NNr7r7rrrNru)>r:rr%r~rPr8r collectionsrtypingrrrr urllib.parsermodal._clustered_functionsrrr modal.clsr modal.dictr modal_protor _runtime.container_io_managerr_serverr_tunnelrr!_utils.async_utilsrrrrconfigr exceptionrrdrrzrcreate_blockingr}rrrrrFlashPrometheusAutoscalerrrrrrr http_serverrr)r)r)r*s              J d     <$ 0