o ;iQ/@sUddlZddlZddlmZmZddlmZddlmZddl m Z ddl m Z m Z ddlmZmZedZeed <d Zeed <d Zeed <eeZeed<GdddeZGdddeZeGdddZGdddZdS)N)ThreadPoolExecutor as_completed) dataclass)Enum)Path)ListOptional)configloggercuda_checkpoint_pathCUDA_CHECKPOINT_PATHgf@CUDA_CHECKPOINT_TIMEOUT"CUDA_CHECKPOINT_TOGGLE_NUM_RETRIESCUDA_CHECKPOINT_TOGGLE_TIMEOUTc@s eZdZdZdZdZdZdZdS)CudaCheckpointStatezyState representation from the CUDA API [1]. [1] https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__TYPES.htmlrunninglocked checkpointedfailedN)__name__ __module__ __qualname____doc__RUNNINGLOCKED CHECKPOINTEDFAILEDrrV/home/ubuntu/.local/lib/python3.10/site-packages/modal/_runtime/gpu_memory_snapshot.pyrs rc@seZdZdZdS)CudaCheckpointExceptionz0Exception raised for CUDA checkpoint operations.N)rrrrrrrrr *sr c @sleZdZUdZeed<eed<ddededdfd d Z dded e d edefddZ dddZ dddZ dS)CudaCheckpointProcesszContains a reference to a PID with active CUDA session. This also provides methods for checkpointing and restoring GPU memory.pidstateF target_stateskip_first_refreshreturnNc Cs:td|jd|jt}d}t}d}|j|||o |dk dr|d7}z|t dd|d}t |d}WnAt y}z5|d7}||kr\t d|jd |d |td|jd |d |d |t d|WYd}~nd}~ww|j|||o|dk ds$td|jd|jddS)zToggle CUDA checkpoint state for current process, moving GPU memory to the CPU and back depending on the current process state when called. PID: z# Toggling CUDA checkpoint state to r)refreshg?g?z Failed to toggle state after z retries: z Retry /z after error: g?Nz Target state z reached) r debugr"valuetime monotonicr_should_continue_toggle_execute_toggle_commandminsleepr ) selfr$r% start_time retry_count max_retriesattempts sleep_timeerrrtoggle8s8 $ zCudaCheckpointProcess.toggleTr5r(c Cs~|r||j|kr dS|jtjkrtd|jd|jt|}|tkr=td|jd|dd|j d|jdS) zMCheck if toggle operation should continue based on current state and timeout.Fr'z CUDA process state is z Timeout after z.2fzs waiting for state z. Current state: T) refresh_stater#rrr r"r.r/rr-)r4r$r5r(elapsedrrrr0Xs   z-CudaCheckpointProcess._should_continue_togglec Csztjtddt|jgdddtd}td|jdWdStjy>}zd|jd|j }t|t |d}~wtj yUd|jd }t|t |w) z+Execute the cuda-checkpoint toggle command.z--toggle--pidTcheckcapture_outputtexttimeoutr'z+ Successfully toggled CUDA checkpoint statez) Failed to toggle CUDA checkpoint state: Nz Toggle command timed out) subprocessrunr strr"r r r,CalledProcessErrorstderrr TimeoutExpired)r4_r: error_msgrrrr1ns&  z-CudaCheckpointProcess._execute_toggle_commandc Csz tjtddt|jgdddtd}|j}t ||_ WdStj y@}zd|jd|j }t |t|d}~wtjyWd|jd}t |t|w) z=Refreshes the current CUDA checkpoint state for this process. --get-stater>Tr?r'z& Failed to get CUDA checkpoint state: Nz Get state command timed out)rDrEr rFr"r stdoutstriplowerrr#rGrHr r,r rI)r4result state_strr:rKrrrr<s(  z#CudaCheckpointProcess.refresh_state)F)Tr&N) rrrrint__annotations__rboolr;floatr0r1r<rrrrr!0s" ! r!c@s|eZdZdZddZdeefddZdede efdd Z dd d Z dd dZ defddZ deeeeffddZd S)CudaCheckpointSessionzGManages the checkpointing state of processes with active CUDA sessions.cCsJ||_|jrtdt|jddd|jDdStddS)NzFound z PID(s) with CUDA sessions: cSsg|]}|jqSrr").0crrr sz2CudaCheckpointSession.__init__..zNo CUDA sessions found.)_get_cuda_pidscuda_processesr r,lenr4rrr__init__s  zCudaCheckpointSession.__init__r&c sg}td}|stddd|D}ttdt|dGfdd|D}t|D]0}||}z |}|rB| |Wq0t y`}zt d |d |WYd }~q0d }~wwWd n1skwY|j d d d|S)zWIterates over all PIDs and identifies the ones that have running CUDA sessions.z/proczPOS does not have /proc path rendering it incompatible with GPU memory snapshots.cSsg|] }|jr|qSr)nameisdigitrYentryrrrr[sz8CudaCheckpointSession._get_cuda_pids..2) max_workerscs(i|]}jt|jt|jqSr)submit_check_cuda_sessionrSrarcexecutorr4rr sz8CudaCheckpointSession._get_cuda_pids..Error checking PID : NcSs|jSNrX)xrrrsz6CudaCheckpointSession._get_cuda_pids..)key)rexistsr iterdirrr2r^rrPappend Exceptionr r,sort) r4 cuda_pidsproc_dirpid_dirs future_to_pidfuturer" cuda_processr:rrirr\s4   "z$CudaCheckpointSession._get_cuda_pidsr"c Csz)tjtddt|gddtd}|jdkr'|j}t |}t ||dWSWd Stj y4Yd Stj yFt d|Yd Styc}zt d|d |WYd }~d Sd }~ww) z+Check if a specific PID has a CUDA session.rLr>T)rArBrCr)r"r#z$Timeout checking CUDA state for PID rlrmN)rDrEr rFr returncoderMrNrOrr!rGrIr r,ru)r4r"rPrQr#r:rrrrhs0   z)CudaCheckpointSession._check_cuda_sessionNc sV|js tddS|jD]}||jtjkr+td|jdtjj d|jj q t }dt ddfdd t Ofd d |jD}g}t|D]}z|WqNtym}z ||WYd}~qNd}~ww|rtd t|d ddd|DWdn1swYt |}tdt|jd|dddS)z8Checkpoint all CUDA processes, moving GPU memory to CPU.z No CUDA processes to checkpoint.NzPID z: CUDA session not in z state. Current state: procr&cSs|tjdSrn)r;rrr~rrrcheckpoint_implsz9CudaCheckpointSession.checkpoint..checkpoint_implcsg|]}|qSrrgrYr~rrjrrr[z4CudaCheckpointSession.checkpoint..zFailed to checkpoint processes: ; cs|]}t|VqdSrnrFrYr:rrr z3CudaCheckpointSession.checkpoint..zCheckpointing z CUDA sessions took => .3fs)r]r r,r<r#rrr r"r-r. perf_counterr!rrrPrurtr^join)r4r~startfutures exceptionsr{r:r=rrr checkpoints@     " $z CudaCheckpointSession.checkpointc s|js tddSt}dtddfddtOfdd|jD}g}t|D]}z|Wq,t yK}z | |WYd}~q,d}~ww|rbt d t |d d d d |DWdn1slwYt|}tdt |jd|dddS)z?Restore all CUDA processes, moving memory back from CPU to GPU.zNo CUDA sessions to restore.Nr~r&cSs|jtjdddS)NT)r%)r;rrrrrrrestore_processsz6CudaCheckpointSession.restore..restore_processcsg|]}|qSrrrrjrrrr[rz1CudaCheckpointSession.restore..zFailed to restore rrcsrrnrrrrrr$rz0CudaCheckpointSession.restore..z Restoring z CUDA session(s) took => rr)r]r r,r.rr!rrrPrurtr r^r)r4rrrr{r:r=rrrrestore s.   " $zCudaCheckpointSession.restorecCs t|jS)z9Get the number of CUDA processes managed by this session.)r^r]r_rrrget_process_count*s z'CudaCheckpointSession.get_process_countcCs.g}|jD]}|||j|jfq|S)z,Get current states of all managed processes.)r]r<rtr"r#)r4statesr~rrrget_process_states.s  z(CudaCheckpointSession.get_process_statesrR)rrrrr`rr!r\rSrrhrrrtuplerrrrrrrWs "  )rW)rDr.concurrent.futuresrr dataclassesrenumrpathlibrtypingrr modal.configr r getr rFrTr rVrrSrrrur r!rWrrrrs"       i