o پi0( @sBddlZddlZddlZddlZddlZddlZddlmZddlm Z m Z m Z m Z ddl mZddlmZddlmZejejddeeZeGd d d Zgd Zgd Zd edeeeffddZ   d&de dede ede fddZ!defddZ"    d'de e ee efde d ed!ed"e#d#e#f d$d%Z$dS)(N) dataclass)CallableListOptionalUnion) cuda_coredump)kill_process_tree) CIRegistryz %(message)s)levelformatc@s"eZdZUeed<dZeed<dS)TestFilename<estimated_timeN)__name__ __module__ __qualname__str__annotations__rfloatrrK/home/ubuntu/.local/lib/python3.10/site-packages/sglang/test/ci/ci_utils.pyr s r ) z!AssertionError:.*not greater thanzAssertionError:.*not less thanzAssertionError:.*not equal tozAssertionError:.*!=.*expectedaccuracyscorelatency throughputtimeout) SyntaxError ImportErrorModuleNotFoundError NameError TypeErrorAttributeError RuntimeErrorzCUDA out of memoryOOMzSegmentation faultz core dumpedConnectionRefusedErrorFileNotFoundErroroutputreturncCsltD]}t||tjrdd|fSqtD]}t||tjr+dd|fSqtd|r4dSdS)z Determine if a test failure is retriable based on output patterns. Returns: tuple: (is_retriable, reason) Fznon-retriable error: Tzretriable pattern: AssertionError)Tz#AssertionError (assuming retriable))Fzunknown failure type)NON_RETRIABLE_PATTERNSresearch IGNORECASERETRIABLE_PATTERNS)r'patternrrris_retriable_failure8s r0rfuncargskwargsrcsVgfdd}tj|d}||j|d|r"ts'tdS)zRun a function with timeout.csipidS)N)appendrr2r1r3 ret_valuerr _target_funcZsz&run_with_timeout.._target_func)target)rr) threadingThreadstartjoinis_alive TimeoutErrorr#)r1r2r3rr7trr5rrun_with_timeoutQs  r@contentcCsNtjd}|r%t|d}||WddS1swYdSdS)z2Write content to GitHub Step Summary if available.GITHUB_STEP_SUMMARYaN)osenvirongetopenwrite)rA summary_filefrrrwrite_github_step_summaryis   "rKFrfilestimeout_per_filecontinue_on_error enable_retry max_attemptsretry_wait_secondsc sNt}|r tt}d}g} g} g} tD]\} t| tr,| j| j } n| j | j } dgd:fdd }d}d}d}||rM|ndkr+|dkrft d|d|d | d d}zt|| fd |i|d }|d krd}|rt d|d| d | | |df| | Wn|r||krd}t|\}}|rt d| d|t d|dt||d7}WqHt d| d|dt d| d|d |r| | |df| | d|fWn6ty*tjtdt d| d|d|r| | |df| | d |d!fYnw|s5d}|s5nqt|}|rF|sFt|rTt d"|d#d!n t d$|d#d!t d d%t d&t| dtd'|r| rt d(t| d)t d%| rt d*| D] }t d+|q| rt d,| D]\}}t d+|d-|d.q| rt d/| D]\}}}t d+|d-|d0|d.qt d%d | r d1d2| D}d3d2| D}d4t| d5}|r|d6d7|d 7}|r|d8d7|d 7}t||r%d Sd9S);a Run a list of test files. Args: files: List of TestFile objects to run timeout_per_file: Timeout in seconds for each test file continue_on_error: If True, continue running remaining tests even if one fails. If False, stop at first failure (default behavior for PR tests). enable_retry: If True, retry failed tests that appear to be accuracy/performance assertion failures (not code errors). max_attempts: Maximum number of attempts per file including initial run (default: 2). retry_wait_seconds: Seconds to wait between retries (default: 60). TNFc stjt|}tddtdd|dt}|rIt j d|gt j t j ddd gj D]}t||q5nt j d|gddd t|}td dtdd |d |ddd jS)Nz . . Begin (/z ): python3 z . . python3Tignore)stdoutstderrtexterrors)rWrXz . . End (z ): filename=z , elapsed=z.0fz, estimated_time=)rDpathr<getcwdloggerinfolentime perf_counter subprocessPopenPIPESTDOUTrWrstripr4wait returncode)filenamecapture_output full_pathfile_ticlineelapsedrrMi output_linesprocessrr run_one_files6     ,z(run_unittest_files..run_one_filerTz [CI Retry] Attempt rSz for  rj)r2r3rru ✓ PASSED on retry (attempt z): passedz [CI Retry] z failed with z[CI Retry] Waiting zs before retry... z - not retrying u ✗ FAILED: z returned exit code failedz exit code u ✗ TIMEOUT: z after z seconds rztimeout after szSuccess. Time elapsed: z.2fzFail. Time elapsed: z<============================================================zTest Summary: z passedz Retries: z test(s) were retriedu ✓ PASSED:z u ✗ FAILED:z ()u ↻ RETRIED:z attempts, cSsg|] \}}}|dkr|qSrur.0r?_rrrr 'z&run_unittest_files..cSsg|] \}}}|dkr|qSr{rr|rrrr(ru**↻ Retried z test(s):** u- ✓ Passed on retry: z, u- ✗ Still failed: )F)r is_enabledcleanup_dump_dirr`ra enumerate isinstancer riest_timer rr]r^r@r4r<r0sleepr>rpidreportr_rK)rMrNrOrPrQrRcoredump_enabledticsuccess passed_tests failed_tests retried_testsfilerirsattempt file_passed was_retriedret_coder' is_retriablereason elapsed_totaltestattemptsresultpassed_on_retryfailed_after_retrysummaryrrorrun_unittest_filesqs %                  "r)rNN)FFrLr)%loggingrDr+rbr9r` dataclassesrtypingrrrrsglang.srt.debug_utilsrsglang.srt.utils.commonrsglang.test.ci.ci_registerr basicConfigINFO getLoggerrr]r r.r*rtupleboolr0dictrr@rKintrrrrrs`