o پi@s2ddlZdaddZddZd ddZd d ZdS) NcCstS)z Cache Management Operation(CMO). Launch a new stream to prefetch the weight of matmul when running other AIV or communication kernels, aiming to overlap the memory access time.  cmo_streamrrW/home/ubuntu/.local/lib/python3.10/site-packages/sglang/srt/hardware_backend/npu/cmo.pyget_cmo_streamsrcCs|adSNr)streamrrrset_cmo_streamsr ʚ;cCsddl}t}|durtj}t||tjtj|*t |t r5|D] }| |||q*n| |||WddSWddS1sOwYdS)u PREFETCH_MAX_SIZE: maximum size (bytes) for each prefetch operation. This affects the time spent in prefetch: time ≈ PREFETCH_MAX_SIZE / system_bandwidth rN) torch_npurtorchnpuStreamr wait_streamcurrent_streamr isinstancelist npu_prefetch)handlecachePREFETCH_MAX_SIZEr rweightrrrprepare_weight_caches.  "rcCs*t}|durtj}||dSdSr)rr r rr)r cur_streamrrrwait_cmo_stream2s  r)r )r rrr rrrrrrs