o -wiRG@sdZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z m Z mZmZmZmZddlZddlZddlmZddlZddlmZddlmZd d lmZd Zd Zd Zd Z dZ!e"Z#ddZ$deee%ee%fde&ddfddZ'de%de&fddZ(dede%de&fddZ)de%de%de%ddfdd Z*de%d!e%ddfd"d#Z+dede&fd$d%Z,d&e e%e fd'e e%e fd(ee%ddfd)d*Z-d+d,d-d,d(ee%ddfd.d/Z. 0dJd1e%d2ee%d3e&dd,fd4d5Z/d6d,d7e%d1e%d8e%d9ee%d:e&dee&ed,ee%ffd;d<Z0de&fd=d>Z1dede%dee&ee%ffd?d@Z2de&fdAdBZ3deddfdCdDZ4dede&fdEdFZ5dGe de fdHdIZ6dS)KzUtilities for wandb verify.N)partial)Path)AnyCallableDictListOptionalTupleUnion)gql)Artifact)runid)Apiverify u✅u❌cCs td|S)N-) ID_PREFIX)namerT/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/wandb/sdk/verify/verify.pynice_ids rfailed_test_or_testswarningreturncs|rdndt|trttttj|dddSt|tr;t|dkr;tttdfdd|DdStt dS) NyellowredTfgboldr csg|] }tj|ddqS)Tr)clickstyle).0fcolorrr 0sz!print_results..) isinstancestrprintRED_Xr!r"listlenjoin CHECKMARK)rrrr%r print_results"s  r0hostcCs|dvr tdddSdS)N)z api.wandb.aizhttp://api.wandb.aizhttps://api.wandb.aiz,Cannot run wandb verify against api.wandb.aiFT)r0)r1rrr check_host7s r2apicCsjtdddddd}d}|jdurdtj|dd d }n |j}|s,d |d }t|d |duS)NzChecking if logged inH.endz)https://docs.wandb.ai/ref/cli/wandb-loginzBNot logged in. Please log in using `wandb login`. See the docs: {}Tblue) underlinerzPCould not get viewer with default API key. Please relogin using `WANDB_BASE_URL=z% wandb login --relogin` and try againF) r*ljustapi_keyformatr!r"r3viewerr0)r3r1 login_doc_url fail_stringresrrrcheck_logged_in>s    rBurltest_url_stringfailure_outputcCs4t|ddddd}|ds|}t|ddS)Nr4r5r6r7httpsT)r*r; startswithr0)rCrDrEr@rrrcheck_secure_requestsVs  rHorigincCsTtdddddd}tj||ddd}|jd dur#d |}t|d dS) Nz)Checking CORs configuration of the bucketr4r5r6r7GET)OriginzAccess-Control-Request-Method)headerszAccess-Control-Allow-OriginzbYour object store does not have a valid CORs configuration, you must allow GET and PUT to Origin: T)r*r;requestsoptionsrLgetr0)rCrIr@res_getrrrcheck_cors_configuration_s rQc Cstdddddg}d}d}dd d }gd }||||d }d }t|d}|d|tjtdd|tdf} | j } | j } d} z%t ddD] } | j dd| i| dqJddd}| j d|i| ddWnt yyd} |dYnwz | dtdiWnt y|dYnw| |Wdn1swYt}|| dtd| }|dur|d t|ddS|D]\}}|j||kr|d!nq| r|jd"dd#d$ks|jd%dks|jd"d&d#dks|jd"d'd#dkr|d(| r&|jdd$kr&|d)ztt|j|}tt|jdd*}Wnt yN|d+t|dYdSw|}|dkr]|d,t|dt|d-kS).Nz6Checking logged metrics, saving and downloading a filer4r5r6r7z A test configz config string) config_val config_string)rone2)epochs stringTestdictTestlistTestz"./test with_special-characters.txtwtest check_runT)idreinitconfigproject lossg?)step)val1val2dictFz5Failed to log values to run. Contact W&B for support.zHT%3ML z!Linkz0Failed to log to media. Contact W&B for support./z:Failed to access run through API. Contact W&B for support.zCRead config values don't match run config. Contact W&B for support.keys previousValue皙?lastStepz dict.val1z dict.val2zCHistory metrics don't match logged values. Check database encoding.zdRead summary values don't match expected value. Check database encoding, or contact W&B for support.replacez]Unable to download file. Check SQS configuration, topic configuration and bucket permissions.zTContents of downloaded file do not match uploaded contents. Contact W&B for support.r)r*r;openwriteclosewandbinitr PROJECT_NAMEr_entityrangelog ExceptionappendHtmlsaverrunr0itemsrarO history_keyssummaryretry_fnrfiledownloadreadr-)r3failed_test_stringsn_epochs string_test dict_test list_testrafilepathr$r~run_idrwloggedilog_dict public_apiprev_runkeyvalue read_filecontentsrrrr^os             r^downloaded_manifestcomputed_manifest fails_listcCslz&|D]}||d||dksJ||d||dks#JqWdSty5|dYdSw)NdigestsizezGArtifact manifest does not appear as expected. Contact W&B for support.)rkAssertionErrorr{)rrrrrrrverify_manifests   r downloadedr computedcCs|j|jkr |ddSdS)NzEArtifact digest does not appear as expected. Contact W&B for support.)rr{)rrrrrr verify_digests rFr verify_dirsingularcCs$tjd|d}tdd}|d|||jWdn1s&wY|r/|S|dur5d}|d }|dWdn1sJwYtj |sZt |t|d d }|d Wdn1srwY| |t |d }|d |||S) Nartsy)typerverify_int_test.txtr\ztest 1z./ verify_a.txtztest 2z /verify_1.txt1 verify_3.txt3)rtr rqrrrsadd_filernew_fileospathexistsmakedirsadd_dirr write_text add_referenceresolveas_uri)rrrartr$file3rrrartifact_with_path_or_pathss.          rartifactalias download_dirradd_extra_filec Cstjtddtddid^}|r4tdd}|d|||jWdn1s/wYz |j ||d Wn&t yc}z| d |d d|fWYd}~WdSd}~wwWdn1snwYtjtd tdd id^} z | |d|} Wn&t y}z| d|d d|fWYd}~WdSd}~wwz| j |dWnt y| dd d|fYWdSwWdn1swYd| |fS)N log_artifactTr]z artifact logr_r`rbra verify_2.txtr\rW)aliaseszUnable to log artifact. F use_artifactz artifact use)r_rbra:zUnable to use artifact. rootz6Unable to download artifact. Check bucket permissions.)rtrurrvrqrrrsrrrrzr{rr) rrrrrr log_art_runr$e use_art_runused_artrrrlog_use_download_artifact sb      rc Csftdddddg}d}d}td}t|d d }t|||||d \}}}|r,|dur3t|d d Sz|j|d WntyI|d Ynwd}d}td}t|dd d }t|||||d \}}}|rk|durrt|d d St t |hdkr|dt j ddd} | |t|| || jd} |jd} t| | |t|d t|dkS)Nz-Checking artifact save and download workflowsr4r5r6r7z./verify_sing_art sing_art1z sing-artysT)rFrzEArtifact does not contain expected checksum. Contact W&B for support.z ./verify_artart1zmy-artysz./verify_art_dir> verify_1.txtrrrrz=Artifact directory is missing files. Contact W&B for support.rdataset)rrr)r*r;rrrr0r ValueErrorr{setrlistdirrtr rrmanifestto_manifest_jsonrr-) r sing_art_dirrr singular_art cont_testdownload_artifact multi_art_dirrrrrrrrcheck_artifacts;sR             rc Cshtdddddg}d}t|d}|d|tjtd d td d id  }| |Wdn1s9wYt }| |j dtd|j }|dura|dt|ddSztt|j|}|j} tt|jd d}Wnty|dt|dYdSw|} z| dksJWnty|dYnwt|dt|dk| fS)NzChecking signed URL uploadr4r5r6r7zgql_test_file.txtr\test2 graphql_putTr]zput to graphqlrrjzJUnable to access previous run through public API. Contact W&B for support.F)FNroz}Unable to read file successfully saved through a put request. Check SQS configurations, bucket permissions and topic configs.zMRead file contents do not match saved file contents. Contact W&B for support.r)r*r;rqrrrsrtrurrvr}rr~rwr_r{r0rrrrCrrzrrr-) r3r1rgql_fpr$r~rrrrCrrrrcheck_graphql_putssX        rc Cstddddddtd}t}g}td}t}|j}z|j ||t d|d d d Wn2t yb}z&t |t jrO|jdurO|jjd krO|d n |d|dWYd}~nd}~wwt|dt|dkS)Nz5Checking ability to send large payloads through proxyr4r5r6r7aia6 query Project($entity: String!, $name: String!, $runName: String!, $desc: String!){ project(entityName: $entity, name: $name) { run(name: $runName, desc: $desc) { name summaryMetrics } } } )rwrrunNamedesc<)variable_valuestimeoutizYFailed to send a large payload. Check nginx.ingress.kubernetes.io/proxy-body-size is "0".z+Failed to send a large payload with error: Fr)r*r;intgetpassgetuserr rtr _base_client _get_resultrvrzr(rM HTTPErrorresponse status_coder{r0r-)descyusernamerqueryrclientrrrrcheck_large_postsH         rcCstddddd|\}}d}d}|didd}|did d }d d lm}|tj||kr=d |}n|tj||krNd|d}d}t||dS)Nz,Checking wandb package version is up to dater4r5r6r7FcliVersionInfomax_cli_versionmin_cli_versionz0.0.1r)parsezCwandb version out of date, please run pip install --upgrade wandb==zwandb version is not supported by your local installation. This could cause some issues. If you're having problems try: please run `pip install --upgrade wandb==`T) r*r;viewer_server_inforOpackaging.versionrrt __version__r0)r3_ server_infor@rrrrrrrcheck_wandb_versions$    rc sdtdddddg}dddd d gd id gd id dd}z"tttj|t|j d}Wdn1s=wYWn t yc}z| d|t |dWYd}~dSd}~ww|sr| dt |ddSzddfdd}tj ||ddWn t y}z| d|t |dWYd}~dSd}~wwt |dt|dkS)Nz+Checking sweep creation and agent executionr4r5r6r7randomminimizescore)goalrvalues)g{Gz?g?rm)rcrSrxy verify_sweep)methodmetric parametersr)sweeprbrwzFailed to create sweep: Fz&Sweep creation returned an invalid ID.cSs|jd|j}|S)Nrr)rarrrr objective szcheck_sweeps..objectivecsJtjtd}|j}|d|iWddS1swYdS)N)rbr)rtrurvrary)r~rrrrmains "zcheck_sweeps..mainr)functioncountzFailed to run sweep agent: r)r*r; contextlibredirect_stdoutioStringIOrtrrvdefault_entityrzr{r0agentr-)r3r sweep_configsweep_idrrrrr check_sweepssL          rfncCs^t}d}d}|tkst|tkr-|d7}z|}W|Sty,tdYqw|S)Nrrc)time MIN_RETRYSGET_RUN_MAX_TIMErzsleep)rini_timerArrrrrs r)NF)7__doc__r rr rr functoolsrpathlibrtypingrrrrrr r r!rM wandb_gqlr rtwandb.sdk.artifacts.artifactr wandb.sdk.libr apis.internalrrvrrr/r+ generate_idrrr)boolr0r2rBrHrQr^rrrrrrrrrrrrrrs  $      f       /"8.32