o $i$@sddlZddlZddlZddlZddlZddlmZddlZddl m Z e e Z ddZddZGdd d eZGd d d eZd eed efddZd eed eefddZGdddejZdS)N)Optional)_ERROR_REPORT_TIMEOUTcCs t|SN)npisnanvaluer S/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/ray/air/_internal/util.pyis_nans r cCst|pt|Sr)r risinfrr r r is_nan_or_infsr c@seZdZdZdS)StartTracebackzMThese exceptions (and their tracebacks) can be skipped with `skip_exceptions`N)__name__ __module__ __qualname____doc__r r r r rsrcs.eZdZdeddffdd ZddZZS)StartTracebackWithWorkerRank worker_rankreturnNcst||_dSr)super__init__r)selfr __class__r r rs  z%StartTracebackWithWorkerRank.__init__cCs|j|jffSr)rr)rr r r __reduce__"sz'StartTracebackWithWorkerRank.__reduce__)rrrintrr __classcell__r r rr rsrexcrcCsbtttjdd}|r|St|trt|jSt | |j }t |dd}|r/t||_|S)a Skip all contained `StartTracebacks` to reduce traceback output. Returns a shallow copy of the exception with all `StartTracebacks` removed. If the RAY_AIR_FULL_TRACEBACKS environment variable is set, the original exception (not a copy) is returned. RAY_AIR_FULL_TRACEBACKS0 __cause__N) boolrosenvironget isinstancerskip_exceptionsr!copywith_traceback __traceback__getattr)rshould_not_shortennew_exccauser r r r'&s    r'cCs|sdSt|ddS)Nr!)r+)rr r r exception_causeBs r/cs@eZdZdZddZdefddZddZd fd d ZZ S) RunnerThreadz(Supervisor thread that runs your script.cOs*tjj|g|Ri|||_d|_dSr) threadingThreadr _error_queue_ret)r error_queueargskwargsr r r rLs zRunnerThread.__init__ecCs:z |jj|dtdWdStjytdYdSw)NT)blocktimeoutzRunner Thread was unable to report error to main function runner thread. This means a previous error was not processed. This should never happen.)r3putrqueueFullloggercriticalrr8r r r _propagate_exceptionQs z!RunnerThread._propagate_exceptionc Csz|j|ji|j|_WdStytdYdStyF}z|jdkr/tdn | |WYd}~dSWYd}~dSd}~wt y]}z | |WYd}~dSd}~ww)NzfThread runner raised StopIteration. Interpreting it as a signal to terminate the thread without error.rzuThread runner raised SystemExit with error code 0. Interpreting it as a signal to terminate the thread without error.) _target_args_kwargsr4 StopIterationr>debug SystemExitcoderA BaseExceptionr@r r r run`s$     zRunnerThread.runNcstt|||jSr)rr0joinr4)rr:rr r rK{szRunnerThread.joinr) rrrrrrIrArJrKrr r rr r0Is r0)r(loggingr#r<r1typingrnumpyrray.air.constantsr getLoggerrr>r r Exceptionrrr'r/r2r0r r r r s