o iZ6@s2ddlZddlZddlZddlmZddlZddlZejGdddZddZ de fdd Z de fd d Z d ee deefd dZddZd de deefddZd de defddZd de deefddZd de dee fddZd!deefddZd de deefddZd de deefddZdS)"N)Optionalc@seZdZUdZeed<dZeed<dZeed<dZ e e ed<dZ e e ed <dZ e e ed <d Ze ed <dZe eed <dZe e ed<dZe eed<dZeed<ddZddZdS)DistributedOptionF distributednccl dist_backendenv://dist_init_methodNdist_world_size dist_rank local_rankrngpudist_master_addrdist_master_port dist_launcherTmultiprocessing_distributedcCsn|jr|jdkr t|j|jdurtdt|jdur tdt|j |j|_ t |j |j|_ t |j |j|_ |j duri|jdkrJtd|jdtjvritjd}|j t|dkritd|j d ||j dur|j dur|j |j krtd |j d |j |jdkrt|j|j|_t|j|_|jdur|jdurd |jd |j|_dSdSdSdSdS)NrzO--dist_master_addr or MASTER_ADDR must be set if --dist_init_method == 'env://'zM--dist_master_port or MASTER_PORT must be set if --dist_init_port == 'env://'z!Assuming 1GPU in this case: ngpu=CUDA_VISIBLE_DEVICES,z LOCAL_RANK=z/ is bigger than the number of visible devices: zRANK >= WORLD_SIZE: z >= ztcp://:)rrget_master_addrr r RuntimeErrorget_master_portrget_rankr get_world_sizer get_local_rankr r osenvironlensplit)selfcvdr!S/home/ubuntu/.local/lib/python3.10/site-packages/espnet2/train/distributed_utils.py init_optionssZ              ,zDistributedOption.init_optionscCsp|jr2tjddtjddtjj|j|j|j|j d|j dur4|j dkr6tj |j dSdSdSdS)N NCCL_DEBUGINFONCCL_BLOCKING_WAIT1)backend init_method world_sizerankr)rrr setdefaulttorchinit_process_grouprrr r r r cuda set_device)rr!r!r"init_torch_distributedVs z(DistributedOption.init_torch_distributed)__name__ __module__ __qualname__rbool__annotations__rstrrr rintr r r r rrrr#r1r!r!r!r"r s      :rcCs|jr@t|j|j}|dkrd|_n |jdkrd|_nd|_|jdkr&d|_|jdkr.d|_|dkr?t|j|jdur?t dn5t |j|jdkrMd|_nd|_|jre|jdkret |j|jduret d|jrut|j|jdurut d|jr|jdkrt st d dSdSdS) NrTFrzH--dist_rank or RANK must be set if --multiprocessing_distributed == truezP--local_rank or LOCAL_RANK must be set if --multiprocessing_distributed == falsezI--dist_rank or RANK must be set if --multiprocessing_distributed == falseslurmz3Launch by 'srun' command if --dist_launcher='slurm') r get_num_nodesr rrr r get_node_rankr rrris_in_slurm_step)args num_nodesr!r!r"resolve_distributed_modeps@     r?returncCsdtjvo dtjvS)N SLURM_PROCID SLURM_NTASKS)rrr!r!r!r"is_in_slurm_jobsrCcCsto dtjvo dtjvS)NSLURM_STEP_NUM_NODESSLURM_STEP_NODELIST)rCrrr!r!r!r"r<s r<xcCs|dur|St|SN)r8)rFr!r!r" _int_or_nonesrHcCsJttjtj}|d|dWdS1swYdS)zFind free port using bind(). There are some interval between finding this port and using it and the other process might catch the port by that time. Thus it is not guaranteed that the port is really empty. )rrN)socketAF_INET SOCK_STREAMbind getsockname)sockr!r!r" free_ports  $rPlaunchercCsr|dur)|dkrtstdtjd}n|dkrtd|dur)td|d|dur1t|SttjdS) Nr9/This process seems not to be launched by 'srun'rAmpi;launcher=mpi is used for 'multiprocessing-distributed' mode launcher='' is not supportedRANK)r<rrrr8rHgetpriorrQr!r!r"rs rcCsx|dur+|dkrtstdttjd}n|dkrtd|dur+td|d|dur3t|Sttjdd S) Nr9rRrBrSrTrUrV WORLD_SIZEr')r<rr8rrrXrYr!r!r"rsrcCs|dur+|dkrtstdttjd}n|dkrtd|dur+td|d|dur3t|Sdtjvr?ttjdSd tjvratjd d }t|d kr_dtjvr_ttjd SdSdS) Nr9rR SLURM_LOCALIDrSrTrUrV LOCAL_RANKrrr)r<rr8rrrrpop)rZrQr r!r!r"rs*  rcCsd|dur$|dkr$tstdtjd}|dddddd}|dur,t|Stjd S) Nr9rRrErr-[rI MASTER_ADDR)r<rrrrreplacer7rX)rZrQnodelistr!r!r"rs   rcCs|dur|SttjdS)N MASTER_PORT)rHrrrX)rZr!r!r"r+srcCs|dur|S|dkr&tstdtjdtjdkrtdttjdS|dkr7d d lm}|j}|S|durCtd |d t tj d S)zGet Node Rank. Use for "multiprocessing distributed" mode. The initial RANK equals to the Node id in this case and the real Rank is set as (nGPU * NodeID) + LOCAL_RANK in torch.distributed. Nr9rRrDrB@Run with --ntasks_per_node=1 if mutliprocessing_distributed=true SLURM_NODEIDrSrMPIrUrVrW) r<rrrr8mpi4pyrh COMM_WORLDGet_rankrHrXrZrQrhcommr!r!r"r;2s" r;cCs|dur|S|dkr&tstdtjdtjdkrtdttjdS|dkr7dd lm}|j}|S|durCtd |d ttj d d S)zGet the number of nodes. Use for "multiprocessing distributed" mode. RANK equals to the Node id in this case and the real Rank is set as (nGPU * NodeID) + LOCAL_RANK in torch.distributed. Nr9rRrDrBrerSrrgrUrVr[r) r<rrrr8rirhrjGet_sizerXrlr!r!r"r:Ss" r:)NNrG) dataclassesrrJtypingrr-torch.distributed dataclassrr?r5rCr<r7r8rHrPrrrrrr;r:r!r!r!r"s( e9 +!