o wOi @sddlZddlZddlmZddlmZddlmZeZ dZ dZ dZ dZ dZ dd ededededef ddZddZddZdejfddZdS)N)closing) get_loggerzAddress already in usezconnect() timed out.zSocket Timeoutz_tcp_store/num_membersz_tcp_store/last_memberX is_server server_addr server_port world_sizetimeoutc Csh|dkr|dkrtd|d||dkrtd|d|dkr%|nd} |dkr/|}nt}td|d |d |d |d |d ztj||||tj|dd}t||td|WSt y} zDt | t kr}|s}t d|d || t | t kr||krtd|d|d|d|d7}n td|d|d| WYd} ~ nd} ~ wwq()NrrzCserver_port must be specified when world_size > 1, got server_port=z , world_size=z sever_port: z, specified, ignoring retriesTzCreating c10d store on :z world_size : z is_server : z timeout(sec):  )seconds) host_nameportr is_masterr zSuccessfully created c10d storez*timed out waiting for tcp store's server: zport: z already in use, attempt: [/]zon z, port: z already in use) ValueErrorloginfo get_free_portdistTCPStoredatetime timedelta_check_full_rank RuntimeErrorstr_CONNECT_TIMEOUT TimeoutError_ADDRESS_IN_USEwarningIOError) rr r r r retriesattemptrstoreer)R/home/ubuntu/.local/lib/python3.10/site-packages/torchelastic/utils/distributed.pycreate_c10d_storesl      r+c Csj|td}||kr|tdz|tWdSty4}zt|tkr/td|d|d}~ww)Nrz ztimed out waiting for all z members to join) add_MEMBER_CHECKINset_LAST_MEMBER_CHECKINgetrr_SOCKET_TIMEOUTr!)r'r idxr(r)r)r*r\s    rcCs>t}t||dWdS1swYdS)Nr)get_socket_with_portr getsockname)sockr)r)r*rls  $rreturnc Cstjddtjtjd}|D]<}|\}}}}}t|||}z|d|d|WStyI}z|tj d|dWYd}~q d}~wwt d) a Returns a free port on localhost that is "reserved" by binding a temporary socket on it. Close the socket before passing the port to the entity that requires it. Usage example :: sock = _get_socket_with_port() with closing(sock): port = sock.getsockname()[1] sock.close() # there is still a race-condition that some other process # may grab this port before func() runs func(port) localhostN)hostrfamilytype)r7rrzSocket creation attempt failed.)exc_infozFailed to create a socket) socket getaddrinfo AF_UNSPEC SOCK_STREAMbindlistenOSErrorcloserrr)addrsaddrr9r:proto_sr(r)r)r*r3rs     r3)rrrr)rr< contextlibrtorch.distributed distributedrtorchelastic.utils.loggingrrr"r r1r-r/boolrintfloatr+rrr3r)r)r)r*s:    B