o „o™iõ"ã@s²ddlZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z ddlmZddlmZe e¡ZGdd „d e ƒZd efd d „Zd efd d„ZdS)éN)ÚOptional)Úoverride)ÚClusterEnvironment)Ú _IS_WINDOWS)Úrank_zero_warn)ÚPossibleUserWarningcszeZdZdZd.dedeejddf‡fdd„ Ze e defd d „ƒƒZ e e de fd d „ƒƒZ e e defd d„ƒƒZee defdd„ƒƒZedee fdd„ƒZedeefdd„ƒZe defdd„ƒZe deddfdd„ƒZe defdd„ƒZe deddfdd„ƒZe defdd „ƒZe defd!d"„ƒZe d#ed$eddfd%d&„ƒZed'e de fd(d)„ƒZed/d*d+„ƒZed/d,d-„ƒZ‡ZS)0ÚSLURMEnvironmenta'Cluster environment for training on a cluster managed by SLURM. You can configure the `main_address` and `main_port` properties via the env variables `MASTER_ADDR` and `MASTER_PORT`, respectively. Args: auto_requeue: Whether automatic job resubmission is enabled or not. How and under which conditions a job gets rescheduled gets determined by the owner of this plugin. requeue_signal: The signal that SLURM will send to indicate that the job should be requeued. Defaults to SIGUSR1 on Unix. TNÚ auto_requeueÚrequeue_signalÚreturncs<tƒ ¡||_|durtstj}||_| ¡| ¡dS)N) ÚsuperÚ__init__r rÚsignalÚSIGUSR1r Ú_validate_srun_usedÚ_validate_srun_variables)Úselfr r ©Ú __class__©ú_/home/ubuntu/.local/lib/python3.10/site-packages/lightning/fabric/plugins/environments/slurm.pyr /s   zSLURMEnvironment.__init__cCsdS)NTr©rrrrÚcreates_processes_externally8sz-SLURMEnvironment.creates_processes_externallycCsPtj d¡}|durtj dd¡}| |¡}|tjd<t dtjd›¡|S)NÚ MASTER_ADDRÚSLURM_NODELISTz 127.0.0.1z MASTER_ADDR: )ÚosÚenvironÚgetÚresolve_root_node_addressÚlogÚdebug)rÚ root_nodeÚnodelistrrrÚ main_address=s   zSLURMEnvironment.main_addresscCsttj d¡}|dur|dd…}t|ƒd}nd}dtjvr&ttjdƒ}nt|ƒtjd<t dtjd›¡|S)NÚ SLURM_JOB_IDéüÿÿÿi˜:in2Ú MASTER_PORTz MASTER_PORT: )rrrÚintÚstrrr )rÚjob_idÚ default_portrrrÚ main_portIs   zSLURMEnvironment.main_portcCst ¡tƒS)aÒReturns ``True`` if the current process was launched on a SLURM cluster. It is possible to use the SLURM scheduler to request resources and then launch processes manually using a different environment. For this, the user can set the job name in SLURM to 'bash' or 'interactive' (srun --job- name=interactive). This will then avoid the detection of ``SLURMEnvironment`` and another environment can be detected automatically. )rrÚ _is_srun_usedrrrrÚdetectes zSLURMEnvironment.detectcCs tj d¡S)NÚSLURM_JOB_NAME)rrrrrrrÚjob_namess zSLURMEnvironment.job_namecCsBtƒrdStj d¡}|durdSzt|ƒWSty YdSw)Nr$©Ú_is_slurm_interactive_moderrrr'Ú ValueError)r)rrrr)ws   ÿzSLURMEnvironment.job_idcCóttjdƒS©NÚ SLURM_NTASKS©r'rrrrrrÚ world_size…ózSLURMEnvironment.world_sizeÚsizecCót d¡dS)Nz[SLURMEnvironment.set_world_size was called, but setting world size is not allowed. Ignored.©rr )rr9rrrÚset_world_size‰r8zSLURMEnvironment.set_world_sizecCr3)NÚ SLURM_PROCIDr6rrrrÚ global_rankr8zSLURMEnvironment.global_rankÚrankcCr:)Nz]SLURMEnvironment.set_global_rank was called, but setting global rank is not allowed. Ignored.r;)rr?rrrÚset_global_rank‘r8z SLURMEnvironment.set_global_rankcCr3)NÚ SLURM_LOCALIDr6rrrrÚ local_rank•r8zSLURMEnvironment.local_rankcCr3)NÚ SLURM_NODEIDr6rrrrÚ node_rank™r8zSLURMEnvironment.node_rankÚ num_devicesÚ num_nodescCsŠtƒrdStj d¡}|dur#t|ƒ|kr#td|›d|›d|›dƒ‚tj d¡}|durAt|ƒ|krCtd|›d|›d |›dƒ‚dSdS) NÚSLURM_NTASKS_PER_NODEzYou set `devices=zX` in Lightning, but the number of tasks per node configured in SLURM `--ntasks-per-node=z%` does not match. HINT: Set `devices=z`.Ú SLURM_NNODESzYou set `num_nodes=zE` in Lightning, but the number of nodes configured in SLURM `--nodes=z'` does not match. HINT: Set `num_nodes=r0)rrErFÚntasks_per_nodeÚnnodesrrrÚvalidate_settingss* ÿÿÿ ÿÿÿÿz"SLURMEnvironment.validate_settingsÚnodescCs4t dd|¡}t dd|¡}| d¡d d¡dS)a¢The node selection format in SLURM supports several formats. This function selects the first host name from - a space-separated list of host names, e.g., 'host0 host1 host3' yields 'host0' as the root - a comma-separated list of host names, e.g., 'host0,host1,host3' yields 'host0' as the root - the range notation with brackets, e.g., 'host[5-9]' yields 'host5' as the root z\[(.*?)[,-].*\]z\1z \[(.*?)\]ú rú,)ÚreÚsubÚsplit)rLrrrr®s z*SLURMEnvironment.resolve_root_node_addresscCsftrdSt d¡du}|r/tƒs1d dtj tj ¡gtj ¢¡dd…}t d|›dt ddSdSdS)aUChecks if the `srun` command is available and used. Parallel jobs (multi-GPU, multi-node) in SLURM are launched by prepending `srun` in front of the Python command. Not doing so will result in processes hanging, which is a frequent user error. Lightning will emit a warning if `srun` is found but not used. NÚsrunrMé@z§The `srun` command is available on your system but is not used. HINT: If your intention is to run Lightning on SLURM, prepend your python command with `srun` like so: z ...)Úcategory) rÚshutilÚwhichr,ÚjoinrÚpathÚbasenameÚsysÚ executableÚargvrr)Ú srun_existsÚhintrrrr½s  &ÿ ýþz$SLURMEnvironment._validate_srun_usedcCsBttj dd¡ƒ}|dkrdtjvrtd|›d|›dƒ‚dSdS) a2Checks for conflicting or incorrectly set variables set through `srun` and raises a useful error message. Right now, we only check for the most common user errors. See `the srun docs `_ for a complete list of supported srun variables. r5Ú1érGzYou set `--ntasks=z^` in your SLURM bash script, but this variable is not supported. HINT: Use `--ntasks-per-node=z ` instead.N)r'rrrÚ RuntimeError)ÚntasksrrrrÒs ÿÿÿz)SLURMEnvironment._validate_srun_variables)TN)r N)Ú__name__Ú __module__Ú __qualname__Ú__doc__ÚboolrrÚSignalsr Úpropertyrrr(r#r'r+Ú staticmethodr-r/r)r7r<r>r@rBrDrKrrrÚ __classcell__rrrrr!sN"      rr cCsdtjvotƒ Sr4)rrr1rrrrr,ãsr,cCs t ¡dvS)N)ÚbashÚ interactive)rr/rrrrr1çs r1)ÚloggingrrOrUrrZÚtypingrÚtyping_extensionsrÚ9lightning.fabric.plugins.environments.cluster_environmentrÚ"lightning.fabric.utilities.importsrÚ$lightning.fabric.utilities.rank_zerorÚ#lightning.fabric.utilities.warningsrÚ getLoggerrcrrrgr,r1rrrrÚs"       C