o oin@sddlZddlmZddlmZmZmZmZmZm Z ddl Z ddl m Z ddl mZddlmZddlmZddlmZdd lmZdd lmZmZmZmZmZmZmZmZdd l m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&dd l'm(Z(dd l)m*Z*ddl+m,Z,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9ddl:m;Z;ddlZ>ddl?m@Z@ddlAmBZBmCZCddlDmEZEddlFmGZGeee!efZHGdddZIdee,dee.fddZJdeKfddZLdS)N)Counter)AnyDictListOptionalUnioncast)get_args)ACCELERATOR_REGISTRY) Accelerator)CUDAAccelerator)MPSAccelerator)XLAAccelerator)BitsandbytesPrecision CheckpointIODeepSpeedPrecision HalfPrecisionMixedPrecision PrecisionTransformerEnginePrecision XLAPrecision)ClusterEnvironmentLightningEnvironmentLSFEnvironmentMPIEnvironmentSLURMEnvironmentTorchElasticEnvironment)DoublePrecision) FSDPPrecision)_PRECISION_INPUT_PRECISION_INPUT_INT_PRECISION_INPUT_STR_PRECISION_INPUT_STR_ALIAS%_PRECISION_INPUT_STR_ALIAS_CONVERSION)STRATEGY_REGISTRYDeepSpeedStrategyParallelStrategySingleDeviceStrategySingleDeviceXLAStrategyStrategyXLAFSDPStrategy XLAStrategy)_DDP_FORK_ALIASES) _FSDP_ALIASES FSDPStrategy)ModelParallelStrategy)rank_zero_inforank_zero_warn)_determine_root_gpu_device)_IS_INTERACTIVEc@seZdZdZ      d+deeefdeeefdeee ee fde d e e d e ee ee fd dfd d Z deeefdeeefd e e d e ee ee fd df ddZdeee ee fde d dfddZed efddZed efddZd,ddZd,ddZd efddZd eeeffddZd,ddZd,d d!Zd efd"d#Zd,d$d%Zed&ed'ed(ed efd)d*ZdS)- _ConnectoradThe Connector parses several Fabric arguments and instantiates the Strategy including its owned components. A. accelerator flag could be: 1. accelerator class 2. accelerator str 3. accelerator auto B. strategy flag could be: 1. strategy class 2. strategy str registered with STRATEGY_REGISTRY 3. strategy str in _strategy_type enum which listed in each strategy as backend (registed these too, and _strategy_type could be deprecated) C. plugins flag could be: 1. precision class (should be removed, and precision flag should allow user pass classes) 2. checkpoint_io class 3. cluster_environment class priorities which to take when: A. Class > str B. Strategy > Accelerator/precision/plugins autoN acceleratorstrategydevices num_nodes precisionpluginsreturncCs(|jd|dd}|jd|dd}|jd|dd}t|jd|dd}|jd|dd}t|_t|_d|_d|_ d |_ d|_ d|_ g|_ d|_|j||||d |j||d |j dkrd||_ n |j d krn||_ |||_|jdkr||_||||_|dS) Nr7r5)defaultr8r9r:r6r;32-true)r8r7r;r<)r9r:gpu)_argument_from_envintr$available_strategies_registered_strategiesr available_accelerators_registered_accelerators_strategy_flag_accelerator_flag_precision_input_precision_instance_cluster_environment_flag_parallel_devices checkpoint_io!_check_config_and_set_final_flags(_check_device_config_and_set_final_flags_choose_auto_accelerator_choose_gpu_accelerator_backend*_set_parallel_devices_and_init_accelerator$_choose_and_init_cluster_environmentcluster_environment_choose_strategy_check_strategy_and_fallback_init_strategy_check_and_init_precisionr;_lazy_init_strategy)selfr7r8r9r:r;r<r[N/home/ubuntu/.local/lib/python3.10/site-packages/lightning/fabric/connector.py__init__bsB            z_Connector.__init__cCs|durt|ts |gn|}t|tr|}||_|dkr0||jvr0t|ts0td|d||jvrM|dvrMt|t sMtd|dd |jd t|toUd |v}t|to^d |v}t|togd |v}t|t pr|pr|pr|}t o|d vpt|t } | r|rtd|d||_t|} |r t} |D]?} t| tr| |_| tjd7<qt| tr| |_| tjd7<qt| tr| |_| tjd7<qtd| ddd| D} | rtdd | d| tjr | dur td| d|jd| durdn| |_t|jtr|jjr.|jdkr)td|jj|_|jjr@|jr;td|jj|_|jjrR|jrMtd|jj|_t|jddri|jrbtdt|jd|_t |jd r|jj!r|jj!d!j"d"kr|jr|jd#vrtd$|jj#jd%|jd&d"|_|jj!d!j"d'kr|jr|jd(vrtd)|jj#jd%|jd&d'|_|jj!|_$dSdSdSdS)*aThis method checks: 1. strategy: whether the strategy name is valid, and sets the internal flags if it is. 2. accelerator: if the value of the accelerator argument is a type of accelerator (instance or string), set self._accelerator_flag accordingly. 3. precision: The final value of the precision flag may be determined either by the precision argument or by a plugin instance. 4. plugins: The list of plugins may contain a Precision plugin, CheckpointIO, ClusterEnvironment and others. Additionally, other flags such as `precision` can populate the list with the corresponding plugin instances. Nr5z1You selected an invalid strategy name: `strategy=z`. It must be either a string or an instance of `lightning.fabric.strategies.Strategy`. Example choices: auto, ddp, ddp_spawn, deepspeed, dp, ... Find a complete list of options in our documentation at https://lightning.ai)r5r@z7You selected an invalid accelerator name: `accelerator=z`. Available names are: auto, , .ddpdp deepspeed)mpsr5r@NzYou set `strategy=z` but strategies from the DDP family are not supported on the MPS accelerator. Either explicitly set `accelerator='cpu'` or change the strategy.r6zFound invalid type for plugin z>. Expected one of: Precision, CheckpointIO, ClusterEnviroment.cSsg|] \}}|dkr|qS)r6r[).0kvr[r[r\ sz@_Connector._check_config_and_set_final_flags..zReceived multiple values for z> flags in `plugins`. Expected one value for each type at most.zReceived both `precision=z` and `plugins=z`. Choose one.r?zLaccelerator set through both strategy class and accelerator flag, choose onezAprecision set through both strategy class and plugins, choose onezEcheckpoint_io set through both strategy class and plugins, choose onerTzKcluster_environment set through both strategy class and plugins, choose oneparallel_devicesrcpu)r5riz!CPU parallel_devices set through z class, but accelerator set to z, please choose one device typecuda)r5rjr@z!GPU parallel_devices set through )% isinstanceliststrlowerrGrDr) ValueErrorrFr joinr&r is_availablerH"_convert_precision_to_unified_argsrrrJ__name__rrMrrK TypeErroritemsgetrI _accelerator _precision_checkpoint_iogetattrhasattrrhtype __class__rL)rZr8r7r;r< is_ddp_str is_dp_stris_deepspeed_stris_parallel_strategyis_mps_acceleratorprecision_inputplugins_flags_typespluginduplicated_plugin_keyr[r[r\rNs                z,_Connector._check_config_and_set_final_flagscCsvt|tr |dkrtd|d||_||_|jgddfvr9t|jtr*|jjjn|j}td|jd|ddS) Nr6z0`num_nodes` must be a positive integer, but got r_r0z`Fabric(devices=z$)` value is not a valid input using z accelerator.) rkrBro_num_nodes_flag _devices_flagrHr r} __qualname__)rZr9r:accelerator_namer[r[r\rO*s   z3_Connector._check_device_config_and_set_final_flagscCs(trdStr dStrdSdS)zTChoose the accelerator type (str) based on availability when ``accelerator='auto'``.tpurcrjri)rrqr r r[r[r[r\rP<sz#_Connector._choose_auto_acceleratorcCs trdStr dStd)NrcrjzNo supported gpu backend found!)r rqr RuntimeErrorr[r[r[r\rQGs z*_Connector._choose_gpu_accelerator_backendcCst|jtr |j|_n|jdusJt|j|_|jj}|s5dd|jD}t d|j d|d| | |j |_ |jsL||j |_dSdS)NcSs g|] }t|dr|qS)r7)r rq)rdacc_strr[r[r\rgXs zI_Connector._set_parallel_devices_and_init_accelerator..`z` can not run on your system since the accelerator is not available. The following accelerator(s) is available and can be passed into `accelerator` argument of `Fabric`: r_)rkrHr r7r rvr}rqrFrr _set_devices_flag_if_auto_passed parse_devicesrrLget_parallel_devices)rZaccelerator_clsavailable_acceleratorr[r[r\rROs&   z5_Connector._set_parallel_devices_and_init_acceleratorcCsh|jdkrdStr,t|jtr,|jdkr,d|_td|jd|jddS|j|_dS)Nr5r6zFabric will use only 1 of zm GPUs because it is running inside an interactive / notebook environment. You may try to set `Fabric(devices=z)` but please note that multi-GPU inside interactive / notebook environments is considered experimental and unstable. Your mileage may vary.)rr3rkr7r auto_device_countr0rZr[r[r\rjs  z+_Connector._set_devices_flag_if_auto_passedcCs<t|jtr |jSttttfD] }|r|SqtS)N) rkrKrrrrrdetectr)rZenv_typer[r[r\rS|s  z/_Connector._choose_and_init_cluster_environmentcCs|jdks t|jtr|jrt|jdkrdSt|jddS|jdkr&dSt|jdkrMt|jttfs@t|jt rF|jdvrFt |j}nd}t |dSt|jdkrXt rXd SdS) Nrr6xlar)devicer`)rjr@rcriddp_fork) rHrkrrLlenr(rr r rmr2r'r3)rZrr[r[r\rUs"     z_Connector._choose_strategycCst|jtrdn|j}|dkr|jdkrd}|dkr(|jdkr(t|dd}|tvr;d tjvr;t d |d |t vsFt |jt urO|jd vrOt d |rV||_dSdS)zChecks edge cases when the strategy selection was a string input, and we need to fall back to a different choice depending on other parameters or the environment.fsdprxla_fsdprariz: is not supported on CPUs, hence setting `strategy='ddp'`.r`forkzYou selected `Fabric(strategy='zn')` but process forking is not supported on this platform. We recommed `Fabric(strategy='ddp_spawn')` instead.)rjr@zYou selected the FSDP strategy but FSDP is only available on GPU. Set `Fabric(accelerator='gpu', ...)` to continue or select a different strategy.N) rkrGr)rHr1r,torchmultiprocessingget_all_start_methodsror-r|r.)rZ strategy_flagr[r[r\rVs$   z'_Connector._check_strategy_and_fallbackcCs>t|jttfs Jt|jtrt|j|_dS|j|_dS)zNInstantiate the Strategy given depending on the setting of ``_strategy_flag``.N)rkrGrmr)r$rvr8rr[r[r\rWs  z_Connector._init_strategycCs~t|jtrt|jtrt|jtstd|jSt|jtt t fr't |j St|jt r2t|j St|jtr>t|j dSd}t|jtrZ|j |vrZtd|j dd|d|j dvrdt|j S|j d krltS|j d krttS|j d krttjd S|j d krttjd S|j dkr|jdkrtdd|_ |j dvrt|j dkrdnd|jdkrdnd}t|j |dStd)Nz,Bitsandbytes is only supported on CUDA GPUs.)r;)r? bf16-mixed bf16-true16-truezDThe `ModelParallelStrategy` does not support `Fabric(..., precision=z()`. Choose a different precision among: r^r_)rrr?z64-trueztransformer-engine) weights_dtypeztransformer-engine-float1616-mixedrizYou passed `Fabric(accelerator='cpu', precision='16-mixed')` but AMP with fp16 is not supported on CPU. Using `precision='bf16-mixed'` instead.r)rrz,Using 16-bit Automatic Mixed Precision (AMP)z.Using bfloat16 Automatic Mixed Precision (AMP)rj)r;rzNo precision set)rkrJrrr7r rr8r(r+r*rrIr%rr.rr/rorprrrrbfloat16float16rHr1r0r)rZmp_precision_supportedrr[r[r\rXsV                 z$_Connector._check_and_init_precisioncCs|j|j_|jr |j|j_|jr|j|j_t|jdr+|jjdur&|j|j_|jj|_t|jdr@|jjr;|jj|_n|j|j_t|jdrK|j|j_ t|jdrV|j |j t ro|jj ro|jj jsotd|jdt|jtrt|jtttfstd|jjjd dSdS) zFLazily set missing attributes on the previously instantiated strategy.rTNrhr:_set_world_ranksz`Fabric(strategy=a)` is not compatible with an interactive environment. Run your code as a script, or choose one of the compatible strategies: `Fabric(strategy='dp'|'ddp_notebook')`. In case you are spawning processes yourself, make sure to include the Fabric creation inside the worker function.zsThe `XLAAccelerator` can only be used with a `SingleDeviceXLAStrategy`, `XLAStrategy`, or `XLAFSDPStrategy`. Found r_)r7r8r;rMr{rTrhrLr _num_nodesr_configure_launcherr3launcheris_interactive_compatiblerrGrkrr(r+r*ror}rsrr[r[r\rYs>                 z_Connector._lazy_init_strategynamecurrentr>c Csptjd|}|dur|S|dur6|t|kr6t|t|kr6tr6td|d|d|d|d |S)NLT_zYour code has `Fabric(=z+, ...)` but it conflicts with the value `--zZ` set through the CLI. Remove it either from the CLI or from the Lightning Fabric object.)osenvironrvupperrm _is_using_cliro)rrr> env_valuer[r[r\rAs*z_Connector._argument_from_env)r5r5r5r6NN)r=N)rs __module__r__doc__rrmr r)rrBrr _PLUGIN_INPUTr]rNrO staticmethodrPrQrRrrrSrUrVrWrrXrYrrAr[r[r[r\r4Isd   A   $      0) r4r;r=cCs|durdStttttt}||vr"tdt|d|t|}|ttvrGt|dddvrCtd|dt|dt|}t t|S)Nz Precision z' is invalid. Allowed precision values: )3264z `precision=za` is supported for historical reasons but its usage is discouraged. Please set your precision to z instead!) r r!r r"roreprrmr1r#r)r;supported_precisionr[r[r\rr-s   rrcCstttjddS)N LT_CLI_USEDr)boolrBrrrvr[r[r[r\rCsr)Mr collectionsrtypingrrrrrrrtyping_extensionsr lightning.fabric.acceleratorsr )lightning.fabric.accelerators.acceleratorr "lightning.fabric.accelerators.cudar !lightning.fabric.accelerators.mpsr !lightning.fabric.accelerators.xlarlightning.fabric.pluginsrrrrrrrr%lightning.fabric.plugins.environmentsrrrrrr)lightning.fabric.plugins.precision.doubler'lightning.fabric.plugins.precision.fsdpr,lightning.fabric.plugins.precision.precisionrr r!r"r#lightning.fabric.strategiesr$r%r&r'r(r)r*r+lightning.fabric.strategies.ddpr, lightning.fabric.strategies.fsdpr-r.*lightning.fabric.strategies.model_parallelr/lightning.fabric.utilitiesr0r1(lightning.fabric.utilities.device_parserr2"lightning.fabric.utilities.importsr3rr4rrrrr[r[r[r\s:        (  (    g