o ziG @sddlZddlmZmZddlmZddlmZmZm Z m Z m Z m Z m Z mZmZmZddlZddlmZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lm Z dd l!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*m+Z+m,Z,m-Z-edZ.edZ/e0e1Z2GdddeZ3GdddeZ4GdddeZ5de e6de e6de7ddfddZ8d e6d!e e6ee6ege7ffd"e9d#e e6efddf d$d%Z:dS)&N)ABCabstractmethod) ExitStack) AnyCallableContextManagerDictIterableListOptionalTupleTypeVarUnion)Tensor)Module) Optimizer) DataLoader) Accelerator) CheckpointIO)TorchCheckpointIO) Precision) _Launcher)_StrategyRegistry)move_data_to_device) _EmptyInit)_PATH OptimizableReduceOp _Stateful TBroadcastTReducec@seZdZdZ   dsdeedeedeeddfddZe e de j fd d Z e e defd d Ze deefd dZe deefddZejdeddfddZe defddZejdeddfddZe defddZejdeeddfddZdtddZdtddZdedefddZdefd d!Zdud"eedefd#d$Zd%ed&eede eeeffd'd(Z!d%edefd)d*Z"d+edefd,d-Z#e d%eddfd.d/Z$dud0e%d1ee j de%fd2d3Z&d4e'd%eed5e%d6e%ddf d7d8Z(d+e)d6e%de%fd9d:Z*e dvd4e'dd?Z+e  @dwd4e,e'e%fdjd+edde,e?e2fdee,e?e2fdfede j'f dgdhZ@d%e j>jd+edie,e?e2fddfdjdkZAeBdleCddfdmdnZDde.fdodpZEdRe6e.e,eee%ffdTe6e.e7e.e%geffde6e.e%ffdqdrZFdS)}StrategyzcBase class for all strategies that change the behaviour of the training, validation and test- loop.N accelerator checkpoint_io precisionreturncCs(||_||_d|_||_d|_d|_dSN) _accelerator_checkpoint_io _precisionr$ _launcher_backward_sync_control)selfr"r#r$r-X/home/ubuntu/.local/lib/python3.10/site-packages/lightning_fabric/strategies/strategy.py__init__,s  zStrategy.__init__cCdS)zReturns the root device.Nr-r,r-r-r. root_device:zStrategy.root_devicecCr0)zcWhether the current process is the rank zero process not only on the local node, but for all nodes.Nr-r1r-r-r.is_global_zero?r3zStrategy.is_global_zerocC|jSr&)r*r1r-r-r.launcherDzStrategy.launchercCr5r&r'r1r-r-r.r"Hr7zStrategy.acceleratorcC ||_dSr&r8)r,r"r-r-r.r"L cCs|jdur t|_|jSr&)r(rr1r-r-r.r#Ps zStrategy.checkpoint_ioiocCr9r&)r()r,r;r-r-r.r#Vr:cCs|jdur|jStSr&)r)rr1r-r-r.r$ZszStrategy.precisioncCr9r&)r))r,r$r-r-r.r$^r:cCr0)z&Attach the launcher based on Strategy.Nr-r1r-r-r._configure_launcherbr3zStrategy._configure_launchercCs |jdusJ|j|jdS)zSetup any processes or distributed connections. This must be called by the framework at the beginning of every process, before any distributed communication takes place. N)r" setup_devicer2r1r-r-r.setup_environmenteszStrategy.setup_environment dataloadercC|S)zWraps the dataloader if necessary. Args: dataloader: iterable. Ideally of type: :class:`torch.utils.data.DataLoader` r-)r,r?r-r-r.process_dataloaderoszStrategy.process_dataloadercCs*|j}t}||j|||S)z1Controls how tensors get created (device, dtype).)r$tensor_init_contextr enter_contextr2)r,precision_init_ctxstackr-r-r.rBxs   zStrategy.tensor_init_context empty_initcCs>|j}t}||j|tt|d|||S)aA context manager wrapping the model instantiation. Here, the strategy can control how the parameters of the model get created (device, dtype) and or apply other patches to the model. Args: empty_init: Whether to initialize the model with empty weights (uninitialized memory). If ``None``, the strategy will decide. Some strategies may not support all options. )enabled)r$module_init_contextrrCr2rbool)r,rFprecision_module_ctxrEr-r-r.rHs   zStrategy.module_init_contextmodule optimizerscs$|}fdd|D}||fS)zSet up a model and multiple optimizers together. The returned objects are expected to be in the same order they were passed in. The default implementation will call :meth:`setup_module` and :meth:`setup_optimizer` on the inputs. csg|]}|qSr-)setup_optimizer).0 optimizerr1r-r. sz8Strategy.setup_module_and_optimizers..) setup_module)r,rKrLr-r1r.setup_module_and_optimizerss z$Strategy.setup_module_and_optimizerscCr@)zDPerforms setup for the model, e.g., by wrapping it by another class.r-r,rKr-r-r.rQzStrategy.setup_modulerOcCr@)zHPerforms setup for the optimizer, e.g., by wrapping it by another class.r-r,rOr-r-r.rMrTzStrategy.setup_optimizercCr0)z&Moves the model to the correct device.Nr-rSr-r-r.module_to_devicer3zStrategy.module_to_devicebatchdevicecCs|p|j}t||S)a&Moves the batch to the correct device. The returned batch is of the same type as the input batch, just having all tensors on the correct device. Args: batch: The batch of samples to move to the correct device device: The target device )r2r)r,rWrXr-r-r.batch_to_devices zStrategy.batch_to_devicetensorargskwargscOs<|j|||jj||g|Ri||j||dS)z0Forwards backward-calls to the precision plugin.N)r$ pre_backwardbackward post_backward)r,rZrKr[r\r-r-r.r^szStrategy.backwardcKs|jj|fi|S)zPerforms the actual optimizer step. Args: optimizer: the optimizer performing the step **kwargs: Any extra arguments to ``optimizer.step`` )r$optimizer_step)r,rOr\r-r-r.r`s zStrategy.optimizer_stepFgroup sync_gradscCr0)aPerform an all_gather on all processes. Args: tensor: the tensor to all_gather group: the process group to gather results from sync_grads: flag that allows users to synchronize gradients for all_gather op Nr-)r,rZrarbr-r-r. all_gatherr3zStrategy.all_gathermean reduce_opcCr0)a,Reduces the given tensor (e.g. across GPUs/processes). Args: tensor: the tensor to sync and reduce group: the process group to reduce reduce_op: the reduction operation. Defaults to 'mean'. Can also be a string 'sum' or ReduceOp. Nr-)r,rZrarer-r-r. all_reducer3zStrategy.all_reducenamecCr0)zSynchronizes all processes which blocks processes until the whole group enters this function. Args: name: an optional name to pass into barrier. Nr-)r,rgr-r-r.barrierr3zStrategy.barrierrobjsrccCr0)zBroadcasts an object to all processes. Args: obj: the object to broadcast src: source rank Nr-)r,rirjr-r-r. broadcastr3zStrategy.broadcastTdecisionallcCr@)z/Reduce a boolean decision across all processes.r-)r,rlrmr-r-r.reduce_boolean_decisionrTz Strategy.reduce_boolean_decisionpathstatestorage_optionsfiltercCs2|j||pid}|jr|jj|||ddSdS)aSave model, optimizer, and other state as a checkpoint file. Args: path: A path to where the file(s) should be saved state: A dictionary with contents to be saved. If the dict contains modules or optimizers, their state-dict will be retrieved and converted automatically. storage_options: Additional options for the ``CheckpointIO`` plugin filter: An optional dictionary containing filter callables that return a boolean indicating whether the given item should be saved (``True``) or filtered out (``False``). Each filter key should match a state key, where its filter will be applied to the ``state_dict`` generated. )rr) checkpointrorqN)"_convert_stateful_objects_in_stater4r#save_checkpoint)r,rorprqrrr-r-r.ruszStrategy.save_checkpointcCs|S)zReturns model state.) state_dictrSr-r-r.get_module_state_dictszStrategy.get_module_state_dictrvstrictcCs|j||ddS)z%Loads the given state into the model.rxN)load_state_dict)r,rKrvrxr-r-r.load_module_state_dictszStrategy.load_module_state_dictcCs,t|dr||jr|SiS|S)zReturns state of an optimizer. Allows for syncing/collating optimizer state from processes in custom plugins. consolidate_state_dict)hasattrr|r4rvrUr-r-r.get_optimizer_state!s zStrategy.get_optimizer_statecCstj|j|}|s|St|tr|j|||diSt|tr*| |iSt | | |d| D]/\}}||vrDq;t|trct|trZ|j||||dq;| ||q;||||<q;|S)aLoad the contents from a checkpoint and restore the state of the given objects. Args: path: A path to where the file is located state: Can be one of: - A dictionary of objects whose state will be restored in-place from the checkpoint path. - ``None`` or the empty dict: The loaded checkpoint will be returned in full. - A :class:`~torch.nn.Module` instance, if the checkpoint file contains a raw module state dict. - A :class:`~torch.optim.Optimizer` instance, if the checkpoint file contains a raw optimizer state. strict: Whether to enforce that the keys in `state` match the keys in the checkpoint. Returns: The remaining items that were not restored into the given state dictionary. If no state dictionary is given, the full checkpoint will be returned. )rKrvrxry)torchcuda empty_cacher#load_checkpoint isinstancerr{rrz!_validate_keys_for_strict_loadingkeyscopyitemsrpop)r,rorprxrsrgrir-r-r.r0s(       zStrategy.load_checkpointcCs0|j|jdus J|j|jdS)zThis method is called to teardown the training process. It is the right place to release memory and free other resources. N)r$teardownr"r#r1r-r-r.rbs  zStrategy.teardown@max_norm norm_typeerror_if_nonfinitecCs.|j||j|}tjjj||||dS)zClip gradients by norm.)rrr)r$unscale_gradients main_paramsrnnutilsclip_grad_norm_)r,rKrOrrr parametersr-r-r.clip_gradients_normms zStrategy.clip_gradients_normclip_valcCs*|j||j|}tjjj||dS)zClip gradients by value.) clip_value)r$rrrrrclip_grad_value_)r,rKrOrrr-r-r.clip_gradients_value|s  zStrategy.clip_gradients_valuestrategy_registrycCsdSr&r-)clsrr-r-r.register_strategiesrTzStrategy.register_strategiescCsdt|jdS)NzThe `z` does not support setting up the module and optimizer(s) independently. Please call `setup_module_and_optimizers(model, [optimizer, ...])` to jointly set them up.)type__name__r1r-r-r._err_msg_joint_setup_requiredsz&Strategy._err_msg_joint_setup_requiredcCspi}|D]/\}}t|tr|j|d}nt|tr"|j|d}n t|tr,|}n|}t||||q|S)N)rK)rO) rrrrwrr~rrv _apply_filter)r,rprrconverted_statekeyri convertedr-r-r.rts    z+Strategy._convert_stateful_objects_in_state)NNN)r%Nr&)NF)Nrd)r)T)NN)NT)rT)Gr __module__ __qualname____doc__r rrrr/propertyrrrXr2rIr4rr6r"setterr#r$r<r>rrArrBrHrr rr rRrQrMrVrrYrr^rr`rcrrstrrfrhrintrkrnrrrrurwr{r~rrrfloatrr classmethodrrrrtr-r-r-r.r!)s       " "      " "  2   & r!c@s*eZdZdZedededefddZdS)_BackwardSyncControlaInterface for any :class:`Strategy` that wants to offer a functionality to enable or disable gradient synchronization during/after back-propagation. The most common use-case is gradient accumulation. If a :class:`Strategy` implements this interface, the user can implement their gradient accumulation loop very efficiently by disabling redundant gradient synchronization. rKrGr%cCr0)zBlocks the synchronization of gradients during the backward pass. This is a context manager. It is only effective if it wraps a call to `.backward()`. Nr-)r,rKrGr-r-r.no_backward_syncr3z%_BackwardSyncControl.no_backward_syncN) rrrrrrrIrrr-r-r-r.rsrc@s"eZdZdZedefddZdS)_ShardedzkMixin-interface for any :class:`Strategy` that wants to expose functionality for sharding model parameters.r%cCr0)aA context manager that goes over the instantiation of an :class:`torch.nn.Module` and handles sharding of parameters on creation. By sharding layers directly on instantiation, one can reduce peak memory usage and initialization time. Nr-r1r-r-r.module_sharded_contextr3z_Sharded.module_sharded_contextN)rrrrrrrr-r-r-r.rsrrequested_keyscheckpoint_keysrxr%cs6fdd|D}|r|rtd|dddSdS)Ncsg|]}|vr|qSr-r-)rNkrr-r.rPsz5_validate_keys_for_strict_loading..z$The requested state contains a key 'rz^' that does not exist in the loaded checkpoint. To disable strict loading, set `strict=False`.)KeyError)rrrx invalid_keysr-rr.rs rrrr source_dict target_dictcCs^||vr)t|tr)||}|D]\}}|||r&||i||||<qdS|||<dSr&)rdictr setdefault)rrrrr filter_fnrvr-r-r.rs    r);loggingabcrr contextlibrtypingrrrrr r r r r rrrtorch.nnr torch.optimrtorch.utils.datarlightning_fabric.acceleratorsr)lightning_fabric.plugins.io.checkpoint_ior$lightning_fabric.plugins.io.torch_ior"lightning_fabric.plugins.precisionr.lightning_fabric.strategies.launchers.launcherr$lightning_fabric.strategies.registryr%lightning_fabric.utilities.apply_funcrlightning_fabric.utilities.initr lightning_fabric.utilities.typesrrrrrr getLoggerrlogr!rrrrIrobjectrr-r-r-r.s\  0             w