o oi$@sddlZddlZddlZddlZddlZddlmZddlmZm Z m Z ddl Z ddl m Z ddlmZddlmZmZmZddlmZmZddlmZdd lmZdd lmZdd lmZdd lm Z e!e"Z#e d Z$e dZ%dZ&de e'fddZ(e$rAddl)Z)dQddZ*e)+dQddZ,e,j-dddide)j.de)j/ddde)j0de)1e&ddd e)j0d!e)1e(dd"d e)j0d#e'd$d%d e)j0d&d'e2d(d)d e)j0d*d+e2dd,d e)j0d-d.e'd/d0d e)j0d1d2e2d3d4d e)j0d5e)1eeeedd6d e)j.d7d8e)j3d9d:eddfd;d<Z4e,j-d=ddide)j.d>e)j/ddde)j0d?e)j/dddd@d d>e'dAe e'ddfdBdCZ5dDeddfdEdFZ6dGe'dHe'de2fdIdJZ7dDed7e e'ddfdKdLZ8dRdDed7e e e'ddfdMdNZ9e"dOkre$se#:dPe;d(e4dSdS)SN) Namespace)AnyListOptional)RequirementCache)get_args)CPUAcceleratorCUDAAcceleratorMPSAccelerator)_PRECISION_INPUT_STR_PRECISION_INPUT_STR_ALIAS)STRATEGY_REGISTRY)_process_cli_args)_parse_gpu_ids)_suggested_max_num_threads)_load_distributed_checkpointclick lightning_sdk)cpugpucudampstpureturncst}dfdd|DS)zReturns strategy choices from the registry, with the ones removed that are incompatible to be launched from the CLI or ones that require further configuration by the user.z).*(spawn|fork|notebook|xla|tpu|offload).*csg|] }t|s|qS)rematch).0strategyexcludedrR/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/lightning_fabric/cli.py /sz-_get_supported_strategies..)r available_strategies)r#rrr!_get_supported_strategies*sr$cCsftjdd}t|dkr"|ddkr"|ddkr"tdtdStr1ttjdd g|dSdS) zLegacy CLI handler for fabric. Raises deprecation warning and runs through fabric cli if necessary, else runs the entrypoint directly Nrrunmodelzm`lightning run model` is deprecated and will be removed in future versions. Please call `fabric run` instead.z-mzlightning_sdk.cli.entrypoint) sysargvlenprint_main_LIGHTNING_SDK_AVAILABLE subprocessr' executable)hparamsrrr! _legacy_main5s$r2cCsdSNrrrrr!r-Hsr-r'ignore_unknown_optionsT)context_settingsscript)exists)typez --acceleratorz#The hardware accelerator to run on.)r8defaulthelpz --strategyz0Strategy for how to run across multiple devices.z --devices1zNumber of devices to run on (``int``), which devices to run on (``list`` or ``str``), or ``'auto'``. The value applies per node.z --num-nodesz --num_nodesr%z5Number of machines (nodes) for distributed execution.z --node-rankz --node_rankzrThe index of the machine (node) this command gets started on. Must be a number in the range 0, ..., num_nodes - 1.z--main-addressz--main_addressz 127.0.0.1zTThe hostname or IP address of the main machine (usually the one with node_rank = 0).z --main-portz --main_portirz-The main port to connect to the main machine.z --precisionzDouble precision (``64-true`` or ``64``), full precision (``32-true`` or ``64``), half precision (``16-mixed`` or ``16``) or bfloat16 precision (``bf16-mixed`` or ``bf16``) script_args)nargsr8kwargscKs*t|dg}ttdi||ddS)aRun a Lightning Fabric script. SCRIPT is the path to the Python script with the code to run. The script must contain a Fabric object. SCRIPT_ARGS are the remaining arguments that you can pass to the script itself and are expected to be parsed there. r<argsr<Nr)listpopmainr)r?r<rrr!_runLsQrE consolidatecheckpoint_folderz --output_filezPath to the file where the converted checkpoint should be saved. The file should not already exist. If no path is provided, the file will be saved next to the input checkpoint folder with the same name and a '.consolidated' suffix. output_filecCs0t||d}t|}t|j}t||jdS)zConvert a distributed/sharded checkpoint into a single file that can be loaded with `torch.load()`. Only supports FSDP sharded checkpoints at the moment. )rGrHN)rrrrGtorchsaverH)rGrHrAconfig checkpointrrr! _consolidates  rMrAcCsdtjd<|jdurt|jtjd<|jdurt|jtjd<t|jtjd<t|jtjd<|jdur>t|jtjd<dSdS) zwSet the environment variables for the new processes. The Fabric connector will parse the arguments set here. r; LT_CLI_USEDNLT_ACCELERATOR LT_STRATEGY LT_DEVICES LT_NUM_NODES LT_PRECISION)osenviron acceleratorstrrdevices num_nodes precision)rArrr!_set_env_variabless    r[rVrXcCsn|dkr t|ddd}n!|dkrt|}n|dkr t|}n |dkr(tdt|S|dur5t|Sd S) zhParse the `devices` argument to determine how many processes need to be launched on the current machine.rT) include_cuda include_mpsrrrz=Launching processes for TPU through the CLI is not supported.Nr)rr parse_devicesr ValueErrorrr+)rVrXparsed_devicesrrr!_get_num_processess   racCsddlmm}|jdkrdnt|j|j}d|d|jd|jd|j d |j |j g}| |t jd tt||dS) zYThis will invoke `torchrun` programmatically to launch the given script in new processes.rNdpr%z--nproc_per_node=z --nnodes=z --node_rank=z--master_addr=z--master_port=OMP_NUM_THREADS)torch.distributed.run distributedr'rrarVrXrY node_rank main_address main_portr6extendrTrU setdefaultrWrrD)rAr<torchrun num_processes torchrun_argsrrr!_torchrun_launchs     rncCst|t||p gdSr3)r[rnr@rrr!rDsrD__main__zoTo use the Lightning Fabric CLI, you must have `click` installed. Install it by running `pip install -U click`.)rNr3)<loggingrTrr/r)argparsertypingrrrrI lightning_utilities.core.importsrtyping_extensionsrlightning_fabric.acceleratorsrr r ,lightning_fabric.plugins.precision.precisionr r lightning_fabric.strategiesr 1lightning_fabric.utilities.consolidate_checkpointr(lightning_fabric.utilities.device_parserr&lightning_fabric.utilities.distributedrlightning_fabric.utilities.loadr getLogger__name___log_CLICK_AVAILABLEr._SUPPORTED_ACCELERATORSrWr$rr2groupr-commandargumentPathoptionChoiceint UNPROCESSEDrErMr[rarnrDerror SystemExitrrrr!s                 (