o Xi@shddlmZddlZddlZdddZGdd d Zd d ZGd d d Z d d!ddZd"d#ddZ dS)$) annotationsNpidintreturncCs ddl}||}|j}|S)z Returns the physical memory used by a process. Args: pid: Process id, current one is `os.getpid()`. Returns: Physical memory. It relies on the module *psutil*. rN)psutilProcess memory_inforss)rrprocessmemr P/home/ubuntu/.local/lib/python3.10/site-packages/onnxscript/tools/memory_peak.pyget_memory_rss s  rc@sJeZdZddZdddd Zdd d ZdddZddZedddZ dS)MonitorcCs"d|_d|_d|_d|_d|_dS)Nr)max_peakaverage n_measuresbeginendselfr r r __init__s  zMonitor.__init__unitrrdict[str, float]cCsBt|}t|j||jd|j||j||j||j|dS)Ng?)peakmeannrr)floatdictrrrrr)rrfunitr r r to_dict%szMonitor.to_dictstrcCs&|jjd|jd|jd|jdS)Nz(peak=z , average=z, n=)) __class____name__rrrrr r r __repr__/s zMonitor.__repr__r rcCsD|jdkr||_t||j|_|j|7_||_|jd7_dS)Nrr)rrmaxrrr)rr r r r update5s zMonitor.updatecCs@||j||j||j||j||jdSN)sendrrrrr)rconnr r r r*=s    z Monitor.sendcCs<|}||_||_||_||_||_|Sr))recvrrrrr)clsr+mr r r r,Ds     z Monitor.recvN)r)rrrr)rr")r r)rr) r% __module__ __qualname__rr!r&r(r* classmethodr,r r r r rs  rcs|d|}|}|}ddl}||}|rLddlm}mmm}m }||} fddt | Dfdd} ddt | D} ng} t } |d | j } | | |rst| | D] \}}||qi|j|d r|}|d krnqW| j }| ||rt| | D] \}}||q| ||t| | D]}||q|r||dS) Nr)nvmlDeviceGetCountnvmlDeviceGetHandleByIndexnvmlDeviceGetMemoryInfonvmlInit nvmlShutdowncsg|]}|qSr r .0i)r4r r ksz'_process_memory_spy..csfddDS)Ncsg|]}|jqSr )used)r9h)r5r r r;nsz9_process_memory_spy..gpu_used..r r )handlesr5r r gpu_usedmsz%_process_memory_spy..gpu_usedcSsg|]}tqSr )rr8r r r r;psT)timeout)r*r,rrpynvmlr3r4r5r6r7rangerrr r(zippolllenclose)r+rr@cudarr r3r6r7n_gpusr?gpuscpur rgcoderr )r>r4r5r _process_memory_spyOsN            rOc@s0eZdZdZddd d Zdd dZdddZdS) MemorySpyz Information about the spy. It class method `start`. Method `stop` can be called to end the measure. Args: pid: process id of the process to spy on delay: spy on every delay seconds cuda: enable cuda monitoring {Gz?FrrdelayrrHboolcCs||_||_||_|dSr))rrRrHstart)rrrRrHr r r rs zMemorySpy.__init__rcCst\|_|_tjt|jfd|_|j|j}|dkr(t d|d|j |j |j |j |j |j r>dnd|j}|dkrRt d|d|S)z+Starts another process and tells it to spy.)targetargsr2z0The child processing is supposed to send -2 not .rrz6The child processing is supposed to send -2 again not )multiprocessingPipe parent_conn child_connrrO child_processrTr, RuntimeErrorr*rrRrH)rdatar r r rTs"    zMemorySpy.startdict[str, list[Monitor]]cCsx|jdt|jg}|j}g}t|D] }|t|jq|j|jt |d}|j r:||d<|S)zStops spying on.rA)rKrJ) rZr*rr,rCappendrGr\joinrrH)rrKrIrJ_resr r r stops      zMemorySpy.stopN)rQF)rrrRrrHrS)rrP)rr_)r%r/r0__doc__rrTrdr r r r rPs  rPrQF int | NonerRrrHrScCs|durt}t|||S)a&Starts the memory spy. The function starts another process spying on the one sent as an argument. Example:: .. code-block:: python from onnxscript.tools.memory_peak import start_spying_on, flatten p = start_spying_on() # ... # code to measure # ... stat = p.stop() print(stat) print(flatten(stat)) Args: pid: process id to spy or the the current one. delay: delay between two measures. cuda: True or False to get memory for cuda devices N)osgetpidrP)rrRrHr r r start_spying_ons ripsr_prefixr"rcs~|ddjdd}d|vr0t|dD]\}}|jddD]\}}||d|d|<q qr=fdd |D}|S) z9Flattens a dictionary produced by :meth:`MemorySpy.stop`.rKri)rrJgpurbcsi|] \}}||qSr r )r9kvrlr r szflatten..)r! enumerateitems)rkrlobsr:rMrnror rpr flattensru)rrrr)NrQF)rrfrRrrHrSrrP)rj)rkr_rlr"rr) __future__rrXrgrrrOrPrirur r r r s  2F9