o zim@sddlZddlZddlZddlmZmZmZmZddlm Z ddl m Z ddl Z ddlmZddlmZmZmZddlmZddlmZdd lmZeeZe d ZGd d d eZdS) N)AnyCallableListOptional)RequirementCache)override)ClusterEnvironment)_basic_subprocess_cmd_hydra_subprocess_cmd_launch_process_observer)_set_num_threads_if_needed) _Launcher)_SIGNUMz hydra-corec seZdZdZdedededdffdd Zeede fd d Z edd d e d e de dde de f ddZededdfddZdddZdddZZS)_SubprocessScriptLauncheraA process launcher that invokes the current script as many times as desired in a single node. This launcher needs to be invoked on each node. In its default behavior, the main process in each node then spawns N-1 child processes via :func:`subprocess.Popen`, where N is the number of devices (e.g. GPU) per node. It is very similar to how :mod:`torch.distributed.run` launches processes. For example, if the script gets invoked with the command .. code-block:: bash python train.py --devices 4 The launcher will create three additional subprocesses that get called like so: .. code-block:: bash LOCAL_RANK=1 python train.py --devices 4 LOCAL_RANK=2 python train.py --devices 4 LOCAL_RANK=3 python train.py --devices 4 It is implied that the main process which launched the others has ``LOCAL_RANK=0``. Beside the local rank, the following other environment variables also get set, but unlike the local rank, these get determined by the cluster environment: 1. `MASTER_ADDR`: The IP address of the main node. 2. `MASTER_PORT`: The port number of the main node through which all processes communicate. 3. `NODE_RANK`: The index of the node the current process is running on. Ranges from 0 to ``num_nodes - 1``. 4. `WORLD_SIZE`: The total number of processes across all nodes, i.e., ``num_processes * num_nodes``. Arguments: cluster_environment: A cluster environment that provides access to world size, node rank, etc. num_processes: The number of processes to launch in the current node. num_nodes: The total number of nodes that participate in this process group. cluster_environment num_processes num_nodesreturnNcs&t||_||_||_g|_dS)N)super__init__rrrprocs)selfrrr __class__l/home/ubuntu/.local/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.pyrKs  z"_SubprocessScriptLauncher.__init__cCsdS)NFrrrrris_interactive_compatibleRsz3_SubprocessScriptLauncher.is_interactive_compatible)trainerfunctionargsrz pl.TrainerkwargscOsH|jj|j|jd|jjs|t|jt|jd||i|S)a0Creates new processes, then calls the given function. Arguments: function: A callback function to execute after all processes have been created. It is up to the implementation of this function to synchronize the processes, e.g., with barriers. *args: Optional positional arguments to be passed to the given function. trainer: Optional reference to the :class:`~pytorch_lightning.trainer.trainer.Trainer`. **kwargs: Optional keyword arguments to be passed to the given function. ) num_devicesr)r) rvalidate_settingsrrcreates_processes_externally_call_children_scriptsr rr )rrrr r!rrrlaunchWs    z _SubprocessScriptLauncher.launchsignumc Cs<|jD]}tdtd|jd|||qdS)NzProcess z is terminating z with )rlogdebugosgetpidpid send_signal)rr'procrrrkillks " z_SubprocessScriptLauncher.killcCs|g|_|jjtjd<t|jjtjd<t|jtjd<t|j tjd<|j |j tjd<t d|j D]E}tj }||d<tjddurVd|vrV|d=d}d}trfd d lm}|}|rot|\}}nt}tj|||d }|j|q;dS) N MASTER_ADDR MASTER_PORT NODE_RANK LOCAL_RANK WORLD_SIZEPL_GLOBAL_SEEDFr) HydraConfig)envcwd)_check_can_spawn_childrenrr main_addressr*environstr main_port node_rank local_rankrrrangecopyget_HYDRA_AVAILABLEhydra.core.hydra_configr7 initializedr r subprocessPopenappend)rr@env_copy hydra_in_user9r7command new_processrrrr%rs.   z0_SubprocessScriptLauncher._call_children_scriptscCs0t|jdkr td|jdkrtddS)Nrz/The launcher can only create subprocesses once.aLightning attempted to launch new distributed processes with `local_rank > 0`. This should not happen. Possible reasons: 1) LOCAL_RANK environment variable was incorrectly modified by the user, 2) `ClusterEnvironment.creates_processes_externally` incorrectly implemented.)lenr RuntimeErrorrr@rrrrr:sz3_SubprocessScriptLauncher._check_can_spawn_children)rN)__name__ __module__ __qualname____doc__rintrpropertyrboolrrrrr&rr/r%r: __classcell__rrrrr%s%* %r)loggingr*rGtypingrrrr lightning_utilities.core.importsrtyping_extensionsrpytorch_lightningpllightning_fabric.pluginsr7lightning_fabric.strategies.launchers.subprocess_scriptr r r &lightning_fabric.utilities.distributedr /pytorch_lightning.strategies.launchers.launcherr 5pytorch_lightning.trainer.connectors.signal_connectorr getLoggerrPr(rDrrrrrs