o rri& @sddlmZddlmZddlZddlmZddlZddlZ ddl Z ddl m Z ddl mZddlmZdd lmZmZmZdd lmZeed Zdd e jededefddZdefddZdS)) ExitStack)partialN)rmtree)git_save)start_ddp_workers) DecoratedMain)disable_logging simple_logred)ShepherdzLaunch:FargvmainclearcCst|t}||}|jdurE||sE|j}ttd|jd|jd|r?td| |j| t dnttd|ri|j jrktdz t|j jWdStyhtd YdSwdSdS) z{This will check if an existing job is running and warn, unless --clear is passed, in which case we must cancel it. NzFound existing slurm job z with status .zCancelling the existing job.z4PLEASE ABORT NOW UNLESS YOU ARE SURE OF WHAT YOU DO.zRemoving existing XP folder.z>Failed to properly remove folder, but things should be okay...)r logget_sheep_from_argvjobupdateis_doner job_idstate cancel_lazycommittimesleepxpfolderexistsrOSError)r rrshepherdsheeprr#s