o
z“©i!
ã @ sZ d dl Z d dlZd dlZd dlmZ d dlmZ d dlm Z e
e ¡Z G dd„ deƒZ
dS )é N)Úoverride)ÚClusterEnvironment)Úrank_zero_warnc @ sö e Zd ZdZeedefdd„ƒƒZeedefdd„ƒƒZ eede
fdd„ƒƒZ e edefd d
„ƒƒZ
ede
fd d „ƒZed
e
ddfdd„ƒZede
fdd„ƒZede
ddfdd„ƒZede
fdd„ƒZede
fdd„ƒZede
de
ddfd d „ƒZdS ) ÚTorchElasticEnvironmentzgEnvironment for fault-tolerant and elastic training with `torchelastic `_Úreturnc C s dS )NT© ©Úselfr r úf/home/ubuntu/.local/lib/python3.10/site-packages/lightning_fabric/plugins/environments/torchelastic.pyÚ creates_processes_externally s z4TorchElasticEnvironment.creates_processes_externallyc C s< dt jvrtdƒ dt jd< t dt jd › ¡ t jd S )NÚ MASTER_ADDRzAMASTER_ADDR environment variable is not defined. Set as localhostz 127.0.0.1z
MASTER_ADDR: )ÚosÚenvironr ÚlogÚdebugr r r r
Ú main_address# s
z$TorchElasticEnvironment.main_addressc C s@ dt jvrtdƒ dt jd< t dt jd › ¡ tt jd ƒS )NÚ MASTER_PORTz=MASTER_PORT environment variable is not defined. Set as 12910Ú12910z
MASTER_PORT: )r
r r r r Úintr r r r
Ú main_port, s
z!TorchElasticEnvironment.main_portc C s t j ¡ o t j ¡ S )zTReturns ``True`` if the current process was launched using the torchelastic command.)ÚtorchÚ distributedÚ is_availableÚis_torchelastic_launchedr r r r
Údetect6 s z TorchElasticEnvironment.detectc C ó t tjd ƒS )NÚ
WORLD_SIZE©r r
r r r r r
Ú
world_size= ó z"TorchElasticEnvironment.world_sizeÚsizeNc C ó t d¡ d S )NzbTorchElasticEnvironment.set_world_size was called, but setting world size is not allowed. Ignored.©r r )r r r r r
Úset_world_sizeA r z&TorchElasticEnvironment.set_world_sizec C r )NÚRANKr r r r r
Ú global_rankE r z#TorchElasticEnvironment.global_rankÚrankc C r! )NzdTorchElasticEnvironment.set_global_rank was called, but setting global rank is not allowed. Ignored.r" )r r& r r r
Úset_global_rankI s ÿz'TorchElasticEnvironment.set_global_rankc C r )NÚ
LOCAL_RANKr r r r r
Ú
local_rankO r z"TorchElasticEnvironment.local_rankc C s t tj dd¡ƒS )NÚ
GROUP_RANKr )r r
r Úgetr r r r
Ú node_rankS s z!TorchElasticEnvironment.node_rankÚ num_devicesÚ num_nodesc C s@ || | ¡ kr td|› d|› d|› d|› d| ¡ › d ƒ‚d S )NzYou set `devices=z` and `num_nodes=z!` in Lightning, but the product (z * z!) does not match the world size (z).)r Ú
ValueError)r r- r. r r r
Úvalidate_settingsW s ÿÿÿÿÿz)TorchElasticEnvironment.validate_settings)Ú__name__Ú
__module__Ú __qualname__Ú__doc__Úpropertyr Úboolr Ústrr r r Ú staticmethodr r r# r% r' r) r, r0 r r r r
r s8 r )Úloggingr
Útorch.distributedr Útyping_extensionsr Ú9lightning_fabric.plugins.environments.cluster_environmentr Ú$lightning_fabric.utilities.rank_zeror Ú getLoggerr1 r r r r r r
Ú s