o 3NPi+J @sdZddlZddlZddlZddlZddlmZmZddlm Z m Z m Z m Z ddl mZedZeGdddZeGd d d ZGd d d ZeZddedededefddZdS)a Compute Resource Detection & Monitoring - Adaptive Pipeline Configuration Auto-detects: nproc, vCPUs, GPU vRAM, CPU cores Monitors: CPU/GPU utilization at each stage Adapts: Worker counts, batch sizes, parallelism based on available resources Key principle: Use ALL available compute at every stage. - CPU-bound: Maximize workers up to core count - GPU-bound: Keep GPU saturated, prepare CPU work in parallel - Memory-bound: Adapt batch sizes to available RAM/VRAM N) dataclassfield)DictOptionalCallableAny)contextmanagerzFastPipelineV5.Computec@seZdZUdZdZeed<dZeed<dZeed<dZ e ed<dZ e ed<d Z e ed <d Zeed <dZe ed <dZe ed<d Zeed<dZe ed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dS)ComputeResourcesz2Detected compute resources for the current system. cpu_countcpu_count_logicalnproc ram_total_gbram_available_gbF gpu_availablegpu_namegpu_vram_total_gbgpu_vram_free_gbgpu_compute_capabilityg?max_utilizationoptimal_vad_workersoptimal_embedding_batch@optimal_music_batchoptimal_download_workersoptimal_cpu_workersoptimal_chunk_workersN)__name__ __module__ __qualname____doc__r int__annotations__r r rfloatrrboolrstrrrrrrrrrrr r*r*Q/home/ubuntu/.cursor/worktrees/maya3data__SSH__216.81.248.184_/zxg/src/compute.pyr s&                 r c@seZdZUdZeed<dZeed<dZeed<dZ eed<dZ eed<dZ eed<dZ eed <dZ eed <dZeed <d Zeed <dZeed<dS) StageMetricsz$Metrics for a single pipeline stage. stage_namer start_timeend_timedurationcpu_percent_avgcpu_percent_maxgpu_percent_avggpu_percent_maxgpu_memory_used_gbritems_processed throughputN)r!r"r#r$r)r&r.r'r/r0r1r2r3r4r5r6r%r7r*r*r*r+r,:s          r,cseZdZdZdZfddZddZddZd d Zd e e e ffd d Z de fddZd%ddZed&de defddZddZddZddZdefdd Zd e fd!d"Zd#d$ZZS)'ComputeMonitorz Singleton compute resource monitor. Features: 1. Auto-detect system resources on init 2. Compute optimal worker counts 3. Track utilization during stages 4. Suggest optimizations Ncs&|jdurt||_d|j_|jS)NF) _instancesuper__new__ _initialized)cls __class__r*r+r;Vs zComputeMonitor.__new__cCsH|jrdSd|_t|_i|_d|_d|_d|_g|_g|_| dS)NTF) r<r resources stage_metrics _monitoring_monitor_thread_current_stage _cpu_samples _gpu_samples_detect_resourcesselfr*r*r+__init__\s zComputeMonitor.__init__c CstdtdtdzWddl}||j_zddl}|jddp(|jj|j_Wnty;|jj|j_Ynwzddl}|j dgddd d }t |j |j_ Wntyd|jj|j_ YnwWn'ty}ztd |d d |j_d |j_d |j_ WYd}~nd}~wwtd|jjd|jjdtd|jj dz*ddl}|}|jd|j_|jd|j_td|jjdd|jjddWntytdYnwzxddl}|jrTd|j_|jd|j_|jdjd}|jdd} |jdd} ||j_|| |j_ |jd} | j!d| j"|j_#td|jjtd|jj dd|jjddtd|jj#ntdWntyu}ztd|WYd}~nd}~ww|$td td!|jj%d"d#d$td%|jj&td&|jj'td'|jj(td(|jj)td)|jj*td*|jj+tddS)+z'Detect all available compute resources.F======================================================================u8🖥️ DETECTING COMPUTE RESOURCES (Adaptive Pipeline)rNF)logicalr T)capture_outputtexttimeoutzCPU detection failed: z, using defaultsrz CPU: z physical / z logical coresz nproc: z usable processors@z RAM: .1fz / z GB availablez.psutil not available, memory detection skipped.z GPU: z VRAM: z GB freez Compute: SM z& GPU: None available (CPU-only mode)zGPU detection failed: F----------------------------------------------------------------------u'🎯 OPTIMAL SETTINGS (auto-computed @ d.0fz% cap):z VAD workers: z Embedding batch: z Music batch: z Download workers: z CPU workers: z Diarization workers: ),loggerinfomultiprocessingr r@r psutil ImportError subprocessrunr%stdoutstripr Exceptionwarningvirtual_memorytotalr availablertorchcuda is_availablerget_device_namerget_device_properties total_memorymemory_allocatedmemory_reservedrrmajorminorr_compute_optimal_settingsrrrrrrr ) rIrYrZr\resultememrerc allocatedreservedpropsr*r*r+rGls       (    $  z ComputeMonitor._detect_resourcescCs:|j}|j}tt|j|j|}tdt|d|_t|j|dd}t|j||_td|j|_|j rI|j |}tdtt|dd|_ nd|_ |j rb|j |}tdtt|dd |_ nd|_ td tdt|j|d|_ tdt|j|d|_|j r|j|}d }td t|||_d Sd |_d S)z Compute optimal worker counts based on detected resources. All settings respect max_utilization cap (default 80%) to leave headroom for system stability and unexpected spikes. rrii,r( g @r N)r@rr%maxr r minrrrrrrrrrr )rIrcap base_workers max_by_ram usable_vramvram_per_diarizationr*r*r+ros,     z(ComputeMonitor._compute_optimal_settingsreturncCs$|j}|j|j|j|j|j|jdS)z3Return optimal configuration dict for Config class.) vad_workers max_workersdownload_workers chunk_workersembedding_batch_sizemusic_batch_size)r@rrrr rr)rIr}r*r*r+get_optimal_configsz!ComputeMonitor.get_optimal_configr~cCs<tdtd|}||j_|td|ddddS)z Set max utilization cap and recompute optimal settings. Args: cap: Utilization cap between 0.1 and 1.0 (default 0.8 = 80%) 皙??u$🎚️ Utilization cap updated to rUrVz%, settings recomputedN)r{r|r@rrorWrX)rIr~r*r*r+set_max_utilizationsz"ComputeMonitor.set_max_utilizationcCs4|}|D]\}}t||rt|||qdS)a  Apply optimal settings to a Config object. This updates the config's parallelism and batch size settings based on auto-detected system resources. Args: config: Config dataclass instance to update N)ritemshasattrsetattr)rIconfigoptimalkeyvaluer*r*r+apply_to_configs    zComputeMonitor.apply_to_configrr-rc cs t||d}t|_||_g|_g|_|zx|VW|t|_|j|j|_ |jrBt |jt |j|_ t |j|_|jrtdd|jD}dd|jD}|r_t |t |nd|_|rht |nd|_|rqt |nd|_|dkr|j dkr||j |_||j|<d|_||dS|t|_|j|j|_ |jrt |jt |j|_ t |j|_|jrdd|jD}dd|jD}|rt |t |nd|_|rt |nd|_|rt |nd|_|dkr|j dkr||j |_||j|<d|_||w)z Context manager to monitor a pipeline stage. Usage: with COMPUTE.monitor_stage("VAD", items=100) as metrics: # do work print(metrics.duration, metrics.cpu_percent_avg) )r-r6cSg|]}|dqSrr*.0sr*r*r+ Fz0ComputeMonitor.monitor_stage..cSr)r r*rr*r*r+rGrrN)r,timer.rDrErF_start_monitoring_stop_monitoringr/r0sumlenr1r{r2r3r4r5r7rA_log_stage_summary)rIr-rmetrics gpu_utilsgpu_memsr*r*r+ monitor_stage#sV          zComputeMonitor.monitor_stagecCs&d|_tj|jdd|_|jdS)z(Start background utilization monitoring.T)targetdaemonN)rB threadingThread _monitor_looprCstartrHr*r*r+rUsz ComputeMonitor._start_monitoringcCs"d|_|jr|jjdddSdS)zStop background monitoring.Fr)rPN)rBrCjoinrHr*r*r+r[szComputeMonitor._stop_monitoringc CsBzddl}d}Wn tyd}Ynwz ddl}|j}Wn ty*d}Ynw|jrz]|r=|jdd}|j||rz)ddl }| | d}| |}| |} |j|j| jdf|Wntyddl}|jd} |jd| fYnwtdWntytdYnw|js.dSdS) z3Background thread that samples CPU/GPU utilization.rNTFr)intervalrQg?)rZr[rerfrgrB cpu_percentrEappendpynvmlnvmlInitnvmlDeviceGetHandleByIndexnvmlDeviceGetUtilizationRatesnvmlDeviceGetMemoryInforFgpuused nvmlShutdownr`rkrsleep) rIrZ has_psutilre has_torchcpurhandleutilrrr*r*r+rasH          zComputeMonitor._monitor_looprcCs|jdkrdn|jdkrdnd}|jdkrdn|jdkrdnd}td|jd|jdd |jd d |d |jd d |d |jdd|jdkrW|jdkrWtd|j|jdkrm|jdkro|jdkrqtddSdSdSdS)z"Log summary for a completed stage.Fu✅ HIGHu ⚠️ LOWu➖ OKu📊 : rR s | CPU: rVz% (z ) | GPU: z ) | VRAM: GBrMu< 💡 HINT: CPU underutilized - consider more workers for ru@ 💡 HINT: GPU underutilized - could run CPU work in parallelN)r1r3rWrXr-r0r5)rIr cpu_status gpu_statusr*r*r+rs$    z!ComputeMonitor._log_stage_summaryc Cs"gd}d}d}d}|jD]2\}}|d|dd|jdd|jdd |jdd ||j7}||j|j7}||j|j7}q|dkr||}||}|d |dd dd|dd|dd |dd ||d } | dkrvd} n | dkr}d} nd} |d| |dd|S)zReturn full pipeline summary.)zG ======================================================================u 📊 COMPUTE UTILIZATION SUMMARYrKrz 20srz6.1frz5.1fz % | GPU: %rTTOTALrz<u🟢 EXCELLENTrvu 🟡 GOODu🔴 NEEDS OPTIMIZATIONz Efficiency: rK )rArrr0r1r3r) rIlines total_time cpu_weighted gpu_weightednamemavg_cpuavg_gpu efficiencyratingr*r*r+summarys.2  ,   zComputeMonitor.summarycCsdz'ddl}|jr%|jdd}|jdd}|jj||j_WdSWdSty1YdSw)z5Refresh GPU memory stats (call after clearing cache).rNrQ) rerfrgrkrlr@rrr`)rIrersrtr*r*r+refresh_gpu_memorys  z!ComputeMonitor.refresh_gpu_memory)rNr)r!r"r#r$r9r;rJrGrorr)rrr'rrrr%rrrrr,rrr __classcell__r*r*r>r+r8Js$ U; 1)%r8segment_lengths base_batch max_memory_mbrcCsL|s|St|t|}|d}|dkr|St||}tdt||dS)z Compute adaptive batch size based on segment lengths. Shorter segments = larger batches possible Longer segments = smaller batches to avoid OOM g?rr rz)rrr%r{r|)rrr avg_lengthavg_memory_per_seg_mbrr*r*r+get_adaptive_batch_sizes r)r)r$osrloggingr dataclassesrrtypingrrrr contextlibr getLoggerrWr r,r8COMPUTElistr%rr*r*r*r+s&