o rri@sddlmZddlZddlZddlZddlZddlZddlZddl m Z e e Z eddZddZd d Zdd d ZddZddZddZdS)) namedtupleN)get_xp DistribSpecz5rank world_size local_rank node_rank num_nodes sourcec Cst}|jdkr dSdtjvr8dtjvsJdtjd}tjd|gdddj}|d }|tjd<d tjvrjt }|j }zt }Wn t yRYnw||j7}t|}|d d }t|tjd <d tjvrt|jtjd <t|jtjd<t|jtjd<dSdS)aCalling this function will set the distributed environement including the master addr, master port etc. You shouldn't call this if you call `dora.distrib.init`, but it can be useful if you need to let some other framework handle the distributed initialization. rN MASTER_ADDRSLURM_JOB_NODELISTzcase not handledzscontrol show hostnamesT)capture_outputcheckr MASTER_PORTi Ni` WORLD_SIZERANK LOCAL_RANK)get_distrib_spec world_sizeosenvironsprunsplitstdoutdecodersigsubmititJobEnvironment RuntimeErrorjob_idrandomRandomrandintstrrank local_rank) specnodelistnodes master_nodexpseedenvrng master_portr+@/home/ubuntu/.local/lib/python3.10/site-packages/dora/distrib.pyset_distrib_envs<            r-cCsdtjvr)ttjd}ttjd}dtjvr ttjd}n|}d}d}d}n-zt}WntyDd}d}d}d}d}d}Ynw|j}|j}|j}|j }|j }d}t ||||||S) zReturn information on the distributed setup, i.e. world size, rank etc. This can be used even before distributed training is initialized, which is useful for PytorchLightning for instance. r r r rrr(emptyr) rrintrrr global_rank num_tasksr!node num_nodesr)r rr! node_rankr3sourcer(r+r+r,r>s4    rncclcCstjrdSt}|jdkrtddSt}tj r&tj |j n|dks,J|j j r:dtj|j}ntd}tjj|||j|jdtd|j|j|j |j|j j rmtjtd kro|jdSdSdS) z Initialize DDP. Nrzworld_size is 1, skipping init.r6zfile://zenv://)backend init_methodrr z*Distributed init: %d/%d (local %d) from %sr)torch distributedis_initializedrrloggerinforcuda is_available set_devicer!dorause_rendezvousrpathabspathrendezvous_filer-init_process_groupr r5barrierunlink)r7r"r&r8r+r+r,initas<       rIcCs tdkSNr)r r+r+r+r, is_masters rKcCtjr tjSdSrJ)r9r:r;get_rankr+r+r+r,r   r cCrL)Nr)r9r:r;get_world_sizer+r+r+r,rrNr)r6) collectionsrloggingrr subprocessrrr9r&r getLogger__name__r<rr-rrIrKr rr+r+r+r,s$   % #$