o $iA@sddlZddlZddlZddlZddlZddlmZddlZddlm Z ddl m Z ddl m Z ddlmZddlmZeeZedd eGd d d eZd d ZddZddZddZddZGddde ZdS)N) dataclass)BaseWorkerGroup)get_address_and_port)Backend) TorchConfig) PublicAPIalpha) stabilityc@s*eZdZUdZdZeed<eddZdS)TorchXLAConfigz Configuration for torch XLA setup. See https://pytorch.org/xla/release/1.13/index.html for more info. Currently, only "neuron_cores" accelerator (AwsNeuronXLABackend) is supported with xrt runtime. Fneuron_parallel_compilecCstS)N)_TorchAwsNeuronXLABackendselfrW/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/ray/train/torch/xla/config.py backend_clsszTorchXLAConfig.backend_clsN) __name__ __module__ __qualname____doc__r bool__annotations__propertyrrrrrr s  r cCsddl}|gddS)Nr)pkillz-fxrt_run_server) subprocesscall)rrrr_kill_xrt_server#srcCstj}t|tjd<t|tjd<t|tjd<t| tjd<t| tjd<t| |tjd<t|tjd<t|tjd<t| tjd <d tjd <d tjd <d tjd<d tjd<d tjd<dS)N LOCAL_RANKRANKLOCAL_WORLD_SIZE WORLD_SIZE GROUP_RANKGROUP_WORLD_SIZE ROLE_RANKROLE_WORLD_RANKROLE_WORLD_SIZEefa FI_PROVIDER1FI_EFA_USE_DEVICE_RDMAFI_EFA_FORK_SAFEXLA_TRANSFER_SEED_ASYNCNCCL_ASYNC_ERROR_HANDLING) raytrain get_contextstrget_local_rankosenvironget_world_rankget_local_world_sizeget_world_size get_node_rank)contextrrr_set_xla_env_vars)s"      r:cCsNzddlm}ddlmm}ddl}|dWdSty&tdw)Nrxlaz5torch_xla must be installed to use torch_xla backend.)torch.distributed distributedtorch_xla.core.xla_modelcore xla_model!torch_xla.distributed.xla_backendinit_process_group ImportError)distxm torch_xlarrr_setup_xla_torch_process_groupCs  rGcCs"dtjd<dtjd<dtjd<dS)Nr)NEURON_PARALLEL_COMPILENEURON_EXTRACT_GRAPHS_ONLYNEURON_FALL_BACK_TO_NULL_NEFF)r3r4rrrr%_set_neuron_parallel_compile_env_varsPs  rKcCszddlm}ddlm}Wn tytdwtjddkret ddtjd d d }tj |r=t |tj|d d d}tjd}r[td|}r[|d}||||dSdS)Nr)CacheUrl)parallel_compilezBlibneuronxla must be installed to use Neuron parallel compilation.r0z0Compiling extracted graphs on local rank0 workerz/tmp/USERzno-userz/parallel_compile_workdir/T)exist_okNEURON_CC_FLAGSz--cache_dir[= ](\S+))libneuronxla.neuron_cc_cacherL$libneuronxla.neuron_parallel_compilerMrCr3r4getloggerinfopathexistsshutilrmtreemakedirsresearchgroup get_cache_url)rLrMparallel_compile_workdirexplicit_cache_dirneuron_cc_flagssrrr _neuron_compile_extracted_graphsWs0      rec@sXeZdZUeeZeed<dede fddZ dede fddZ dede fdd Z d S) r unique_run_id worker_groupbackend_configcsX|t|dt\}}fdd}|j|||d|jr*td|tdSdS)z+Logic ran right before training is started.rcs(|tjd<t|tjd<jtjd<dS)N MASTER_ADDR MASTER_PORTTORCHELASTIC_RUN_ID)r3r4r1rfaddrportr rr set_env_varss z8_TorchAwsNeuronXLABackend.on_start..set_env_varsrlz1Extracting graphs for Neuron parallel compilationN)executerexecute_singlerr rVrWrK)rrgrh master_addr master_portrorr ron_start{s   z"_TorchAwsNeuronXLABackend.on_startcCs|t|tdS)z Configure the environment variables for the worker group. And initialize the xla distributed process group. TODO: Current setup only supports homogenous cluster with neuron_cores accelerator and xrt runtime. N)rpr:rGrrgrhrrron_training_starts z+_TorchAwsNeuronXLABackend.on_training_startcCs"|t|jr|tdSdS)z Logic ran right after training is finished. This is a sanity cleanup to kill xrt server, and to optionally run neuron parallel graph compilation N)rprr rerurrr on_shutdowns z%_TorchAwsNeuronXLABackend.on_shutdownN) rrrr1uuiduuid4rfrrr rtrvrwrrrrr xs   r )loggingr3r]rZrx dataclassesrr.%ray.train._internal.base_worker_grouprray.train._internal.utilsrray.train.backendrray.train.torchrray.utilr getLoggerrrVr rr:rGrKrer rrrrs,        !