o 2wi @s|dZddlmZddlZddlZddlmZmZddl m Z ddl m Z ddl mZdd lmZeed Z d e fd d ZdS)z Launch command. )partialN) SubmitRulesupdate_from_args) DecoratedMain)Shepherd) simple_log)reliable_rmtreezLaunch:mainc CsDt|td}|}t||t}t||||j}td|||jrG|j dur6tddS| r@tddS|j dS|j rptd|j durY| |j | td|jjrmt|jjd|_ ||||| |js|jr d}d}d}zjz, |jr|durtd d d d |jg}| d rtd|d}ntdqWntyd}tdYnwW|r|rtd||jr|s|j durtd|j j|j dSdSdSdS|r|rtd||jr|s|j durtd|j j|j wwwwdS)N)logzFetched sheep z!Could not cancel non existing jobzJob is not runningzCanceling current job...zDeleting XP folder...FTtailz-n200z-fforcez"Remote process finished with statezKeyboardInterrupt received...z"attach is set, killing remote job )rr get_slurm_configrrget_sheep_from_argvargvupdatecanceljobis_doneclear cancel_lazycommitxpfolderexistsr maybe_submit_lazyr attachspPopenstatetimesleepKeyboardInterruptkilljob_id) argsr shepherdslurmrulessheepdone tail_processwaitr0H/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/dora/launch.py launch_actions                    r2)__doc__ functoolsr subprocessr r#confrrr rsheprr rutilsr r2r0r0r0r1s