o iH @sUdZddlZddlmZddlmZmZmZddlm Z e e Z Gddde eZ ie jde jde jd e jd e jd e jd e jd e jd e jd e jd e jd e jd e jde jd e jd e jd e jdZee e fe!d< d/de de de deee efdee eff ddZ"de de fddZ#da$de%de&fddZ'de%de ddfddZ(de&fd d!Z)d0d"d#Z*  $d1d%e d&ed'e defd(d)Z+d%e d*e dee fd+d,Z,Gd-d.d.Z-dS)2aD Error handling and GPU fault recovery for Veena3 TTS. Features: - Standardized error responses - GPU fault detection and container draining - Graceful degradation for missing configs Usage: from veena3modal.api.error_handlers import ( handle_gpu_fault, create_error_response, ErrorCode, ) N)Enum)DictAnyOptional) get_loggerc@sTeZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZdZdZdZdS) ErrorCodez!Standard error codes for TTS API.INVALID_API_KEYEXPIRED_API_KEYINSUFFICIENT_CREDITSRATE_LIMIT_EXCEEDEDVALIDATION_ERROR TEXT_TOO_LONGINVALID_SPEAKERINVALID_FORMATMODEL_NOT_LOADED GPU_FAULTGPU_OOMGENERATION_FAILEDGENERATION_TIMEOUTSTREAMING_ERRORSERVICE_UNAVAILABLEINTERNAL_ERRORFORMAT_NOT_IMPLEMENTEDN)__name__ __module__ __qualname____doc__rr r r r r rrrrrrrrrrrrr9/home/ubuntu/veenaModal/veena3modal/api/error_handlers.pyrs&riiiiiiiERROR_STATUS_CODEScodemessage request_iddetailsreturncCs&d|j||di}|r||dd<|S)a$ Create a standardized error response body. Args: code: Error code enum message: Human-readable error message request_id: Request identifier for tracking details: Optional additional details Returns: Error response dictionary error)r!r"r#r$)value)r!r"r#r$responserrrcreate_error_responseOs r)cCs t|dS)z'Get HTTP status code for an error code.r)r get)r!rrrget_error_statusos r+F exceptioncs*t|gd}tfdd|DS)z Check if an exception indicates a GPU fault. Args: exception: The caught exception Returns: True if this is a GPU-related fault )z cuda errorz cudnn errorz out of memoryoomzillegal memory accesszdevice-side assertz nccl errorgpuc3s|]}|vVqdSNr).0 indicator error_msgrr szis_gpu_fault..)strlowerany)r,gpu_fault_indicatorsrr2r is_gpu_faultxs  r9c Cstjd|t|t|jdddaz&ddl}t|dr3t|jdr6|j tj d d |idWdSWdSWdSt yBYdSt y`}ztj d |t|d dWYd}~dSd}~ww) aF Handle a GPU fault by logging and signaling container drain. In Modal, this should call `modal.experimental.stop_fetching_inputs()` to prevent the container from accepting more requests while it drains. Args: exception: The GPU fault exception request_id: Request ID for logging gpu_fault_detected)r#r,exception_typeextraTrN experimentalstop_fetching_inputscontainer_drain_initiatedr#container_drain_failed)r#r&)loggerr&r5typer_gpu_fault_detectedmodalhasattrr>r?info ImportError Exceptionwarning)r,r#rEerrrhandle_gpu_faults8    rLcCstS)z9Check if a GPU fault has been detected in this container.rDrrrr has_gpu_faultsrNcCsdadS)z#Reset GPU fault flag (for testing).FNrMrrrrreset_gpu_fault_flagsrOenv_vardefault required_forcCsBtj|}|dur|rtjd||t|dd|dur|S|S)a[ Get an optional configuration value with graceful degradation. Logs a warning if the config is missing but doesn't crash. Args: env_var: Environment variable name default: Default value if not set required_for: Feature name (for warning message) Returns: Config value or default Noptional_config_missing)rQfeature using_defaultr<)osenvironr*rBrJr5)rQrRrSr'rrrget_optional_configs   rY feature_namecCs,tj|}|durtjd||dd|S)a_ Check if a required config is available. Unlike get_optional_config, this doesn't provide a default. Returns None if missing, allowing the caller to handle gracefully. Args: env_var: Environment variable name feature_name: Feature that requires this config Returns: Config value or None Nrequired_config_missing)rQrUr<)rWrXr*rBrJ)rQrZr'rrrcheck_required_configs r\c@sjeZdZdZedefddZedefddZedefddZedefd d Z edefd d Z d S) FeatureFlagszx Feature flags for graceful degradation. Checks environment to determine which features are available. r%cCs0tjd}tjdptjd}t|o|S)z Check if Supabase is configured. SUPABASE_URLSUPABASE_SERVICE_KEY SUPABASE_KEY)rWrXr*bool)urlkeyrrris_supabase_enableds  z FeatureFlags.is_supabase_enabledcCsttjdS)zCheck if Redis is configured. REDIS_URL)rarWrXr*rrrris_redis_enabledszFeatureFlags.is_redis_enabledcCstjdd}|dkS)z#Check if authentication is enabled.AUTH_BYPASS_MODEfalsetruerWrXr*r6)bypassrrris_auth_enabled zFeatureFlags.is_auth_enabledcCstjdd}|dkS)z"Check if rate limiting is enabled.RATE_LIMIT_ENABLEDrirj)enabledrrris_rate_limiting_enabled&rmz%FeatureFlags.is_rate_limiting_enabledcCstjdddkS)z(Check if Prometheus metrics are enabled.PROMETHEUS_ENABLEDrirjrrrris_metrics_enabled,szFeatureFlags.is_metrics_enabledN) rrrr staticmethodrardrfrlrprrrrrrr] sr]r/)r%N)NrP).rrWenumrtypingrrrveena3modal.shared.loggingrrrBr5rrr r r r r rrrrrrrrrrrr int__annotations__r)r+rDrIrar9rLrNrOrYr\r]rrrrs           .   !