o i < @sdZddlZddlmmZddlZddlZddl Z ddl Z ddl Z ddl Z ddl mZddlmZe jdeeejje jddddlmZeeejjddd lmZmZdd lmZdd l m!Z!dd l"m#Z#dd l$m%Z%m&Z&ddl'm(Z(ddl)m*Z*e j+e j,dde -dZ.eejdZ/eejdZ0dddddddddddd d! Z1d"Z2d#Z3d$Z4d%ed&e5fd'd(Z6d)ed*e5ee7ffd+d,Z8e3fd-e5ee7fd.ed/e9d*e5fd0d1Z:e4fd-e5ee7fd.ed2e9d*e5fd3d4Z;d5d6Ze<dSdS)8uD 12-language canary test: determinism + concurrency on both AI Studio and OpenRouter. Phase 1: Download 1 video per language, extract 5 segments each Phase 2: Determinism test — 3 runs per provider, compare exact matches Phase 3: 500-concurrency stress test on AI Studio Phase 4: 500-concurrency stress test on OpenRouter N) defaultdict)Path MOCK_MODEfalse) load_dotenvz.env) EnvConfig LANGUAGE_MAP)polish_all_segments)AIStudioProvider)OpenRouterProvider)TranscriptionRequest RequestStatus) CacheManager)R2Clientz1%(asctime)s [%(levelname)s] %(name)s: %(message)s)levelformatcanarycanary_results canary_data efw1HydOOfI D7iL59a39Bs nM2KMwb86IU X7tqVj0IuOI BFU9eSKO_t4z o-zn8bAjkBM dlo3pvcyqFY mqyv2attoI0z IYKHdR3-w_U l5eQd1E_Huo 8ozSnoys7JE 5MDwLvubhR8) asbnenguhiknmlmrorpatateinamedatacCsntjdddt|d}t|d}tj||ddtdWdn1s(wYtd|dS) NTparentsexist_okz.jsonwF)indent ensure_asciidefaultz Saved -> ) RESULTS_DIRmkdiropenjsondumpstrloggerinfo)r-r.pathfrA1/home/ubuntu/transcripts/preflight/canary_test.py save_result9s  rCconfigreturnc st|}i}tD]\}}t|}|jdddztd|d|d|||}|||}|j sCt d|d|Wq t |j dt d}d d |Ddt } | sft d|d Wq g} | D]#} | j j} | j jr| j jd | j j} | t| | j|| j jd qj| ||<td|dt| d|Wq ty} ztd|d|d| WYd} ~ q d} ~ ww|S)zFDownload tars, extract, polish, return {lang: [TranscriptionRequest]}.Tr/[z] Downloading z...z] No segments in Nr,cSsg|]}|jjs|qSrA) trim_meta discarded).0prArArB Uz(download_and_extract..z] All segments discarded_split segment_id audio_base64 language_code original_filez ] Ready: z segments from z] Failed to prepare : )r CANARY_VIDEOSitemsWORK_DIRr8r=r> download_tar extract_tar segment_pathswarningr SEGMENTS_PER_LANGrGrR was_split split_indexappendr base64_audiolen Exceptionerror)rDr2 all_requests lang_codevideo_idlang_dirtar_path extractedpolishedvalidrequestssegseg_iderArArBdownload_and_extractAsH   $(rprequests_by_lang provider_namenum_runsc stddtd|d|dtdtt}g}t|D]}td|dd|d t}i}t|D]r\} } td | d t | d | | Id H} | D]U} | j t j kr| jr| d| j} || | j| jdd| jdd| j| jj| jj| jj| jjd|| <q]| j t j krtd | d| jd| j d| jq]qAt|}||dt |t|d|dtd|ddt |d|dd||dkrtdId Hq%d}d}tdd}g}|D]p\} }| dd}||d d7<t ||kr|d7}qd!d"|Dtfd#d$DrA|d7}||d%d7<q|d7}|| d d&td'd$t t!ddDt"t dt dd(q||}|dkr~||d)nd}|||||t|dt#||d d*d+}td,|d-td.|d|d/|dd0t|D]\}}td.|d|d%d|d q|rtd1|d d2D]}td3|d4d5|d6q||d7S)8zIRun same segments N times, compare transcription outputs for exact match. F======================================================================zDETERMINISM TEST:  (z runs)z --- Run /z ---z [z ] Sending z segments...N: transcriptiondetected_language)rzr| latency_ms cache_hit input_tokens cached_tokens output_tokensz] rSz - r3)run segments_oktime_sresultsz Run z done: z OK in .1fsrcSs dddS)Nr)total deterministicrArArArArBs z&run_determinism_test..rcSsg|]}|ddqS)rzr{)getrIrrArArBrKrLz(run_determinism_test..c3s|] }|dkVqdS)rNrA)rIttextsrArB z'run_determinism_test..rcss$|] \}\}}||kr|VqdSNrA)rIiabrArArBrs")segmentrfirst_diff_chard)providerrstotal_segmentsrnon_deterministicdeterminism_pct by_languagediffsz === DETERMINISM RESULTS () === z segments deterministic (%)z Non-deterministic segments:r+ rz: first diff at char r)summary run_details)$r=r>rlistrangetime monotonicsortedrUr` send_batchstatusr SUCCESStranscription_datarOr^rr} token_usager~rrrrb error_messageroundasynciosleepsplitallnext enumeratezipmindict)rqrrrrsall_runs run_summariesrun_idx run_start run_resultsrerl responsesrespkeyrun_timerr lang_statsrrunslangrpctrstatsdrArrBrun_determinism_testps     ( (       ( "rtarget_concurrencyc s@tddtd|d|dtdg}t|D] \}}||q#t||krc|t|d}g}t|D]} |D]} |t| j d| | j | j | j dqEqA|d |}td t|d t } ||Id H} t | } d d | D}dd | D}dd | D}dd | D}dd |D}tdd|D}tdd|D}tdd|D}tdd|D}id|d|dt|dt|dt|dt|dt|dt| dd | d!krtt|| dnd!d"|r tt|t|dnd!d#|rtt|t|ddnd!d$|r4tt|tt|d%dnd!d&|rAtt|dnd!d'|d(|d)|d*|d+t|tt|dd,i}g}|d d-D]}||j |jd d.d/qf|d d0D]}||j d1|jd d.d2q~td3|d4td5t|d6t|d7t|d8t|d9t|d: td;| d<d=|d d>td?|d"d@|d#dA|d$dBtdC|dD|dE|tdF|d6t|d|d+dGd<dH|rtdI|d d,D] }tdJ|q||dKS)LzDFire target_concurrency requests simultaneously, measure throughput.rtruzCONCURRENCY TEST: rvz concurrent)rw_reprNNzSending z requests concurrently...cSg|] }|jtjkr|qSrA)rr rrrArArBrKz(run_concurrency_test..cSrrA)rr ERRORrrArArBrKrcSrrA)rr RATE_LIMITEDrrArArBrKrcSrrA)rr TIMEOUTrrArArBrKrcSsg|] }|jdkr|jqS)r)r}rrArArBrKrcs|]}|jjVqdSr)rrrrArArBrz'run_concurrency_test..csrr)rrrrArArBrrcsrr)rrrrArArBrrcss|] }|jjrdVqdS)rwN)rr~rrArArBr rrr actual_sent successeserrors rate_limitedtimeouts wall_time_sr3throughput_rpsravg_latency_msp50_latency_msp95_latency_msgffffff?max_latency_mstotal_input_tokenstotal_output_tokenstotal_cached_tokens cache_hitscache_hit_rater, )rrbr+429)rrrbz === CONCURRENCY RESULTS (rrrxz OK, z errors, z 429s, z timeoutsz Wall time: rzs, Throughput: z req/sz Latency: avg=zms, p50=zms, p95=msz Tokens: input=z , output=z , cached=z Cache hits: rrz Sample errors:r)rr)r=r>rrUextendr`rr^r rOrPrQrRrrrsumrintmaxr)rqrrrrrdrerl multiplierexpandedrreqstartr wall_timerrrr latencies total_input total_output total_cachedrr error_detailsrrorArArBrun_concurrency_tests          $ (   ">*,  rc sft}tjdddtdtdtdt|IdH}tdd|D}td|dt|d t d d d | D|d |sRt ddStdt |j }|IdH}t|j |d}t||dtIdH}t d|tdt|jd}t||dtIdH}t d|t||dtIdH} t d| t||dtIdH} t d| t|dd | D|d|dd| d| ddd} t d| tdtdtdtd t|td!|dd"d#td$|dd"d#td%| dd&d'| dd(d)td*| dd&d'| dd(d)dS)+NTr/ruz7PHASE 1: Downloading canary videos for all 12 languagescss|]}t|VqdSrr`rrArArBr?rzmain..z Prepared z segments across z languagesphase1_preparationcSi|] \}}|t|qSrArrIkvrArArB Bzmain..) languagesrz No segments available. Aborting.z' Setting up AI Studio with V2 cache...)api_keycached_content_nameaistudiophase2a_determinism_aistudioz Setting up OpenRouter...)r openrouterphase2b_determinism_openrouterphase3_concurrency_aistudiophase4_concurrency_openroutercSrrArrrArArBrerr)rr)languages_testedsegments_per_language determinism concurrencycombined_summaryzG ======================================================================zCANARY TEST COMPLETEz Languages: z AI Studio determinism: r%z OpenRouter determinism: z AI Studio concurrency: rrxrz OKz OpenRouter concurrency: )rr7r8r=r>rprvaluesr`rCrUrbr gemini_key ensure_cacher rDETERMINISM_RUNSr openrouter_api_keyrCONCURRENCY_TARGETrkeys) rDrqrcm cache_namer det_aistudiordet_openrouter conc_aistudioconc_openroutercombinedrArArBmain5s`                (,r__main__)?__doc__builtins @py_builtins_pytest.assertion.rewrite assertionrewrite @pytest_arrr:loggingossysr collectionsrpathlibrr?insertr<__file__resolveparentenviron setdefaultdotenvr src.configrrsrc.audio_polishr src.providers.aistudior src.providers.openrouterr src.providers.baser r src.cache_managerr src.r2_clientr basicConfigINFO getLoggerr=r7rVrTr[rrrrCrrprrrr__name__rrArArArBs"         3  q  X F