o zi@sddlmZmZddlmZddlmZmZmZm Z m Z ddl Z ddl m Z ddl mZddlZddlmZmZddlmZmZdd lmZdd lmZdd lmZGd d d eeZdS))ABCabstractmethod)contextmanager)AnyDict GeneratorListOptionalN)Tensor)override) CheckpointIOClusterEnvironment)ReduceOp_all_gather_ddp_if_available) LayerSync) Precision)Strategyc seZdZdZ     d,deddeeejdeedee dee f fd d Z e e ed ejfd d Ze d efddZe d efddZe d efddZe d efddZe ed efddZe d eeejfddZejdeeejd dfddZe d eeeffddZed-dedeed ed efd!d"Zed.d$ed%ed efd&d'Ze d e!fd(d)Z"ed/fd*d+ Z#Z$S)0ParallelStrategyz:Strategy for training with multiple processes in parallel.N acceleratorzpl.accelerators.Acceleratorparallel_devicescluster_environment checkpoint_ioprecision_plugincs(tj|||d||_||_d|_dS)N)rrr)super__init__rr _layer_sync)selfrrrrr __class__Y/home/ubuntu/.local/lib/python3.10/site-packages/pytorch_lightning/strategies/parallel.pyr!s zParallelStrategy.__init__returncCsdS)zReturn the root device.Nrrrrr root_device.szParallelStrategy.root_devicecC|jdur |jSdSNr)r global_rankr"rrr r&4zParallelStrategy.global_rankcCr$r%)r local_rankr"rrr r(8r'zParallelStrategy.local_rankcCr$r%)r node_rankr"rrr r)<r'zParallelStrategy.node_rankcCr$)N)r world_sizer"rrr r+@r'zParallelStrategy.world_sizecCs |jdkSr%)r&r"rrr is_global_zeroDs zParallelStrategy.is_global_zerocCs|jSN_parallel_devicesr"rrr rIsz!ParallelStrategy.parallel_devicescCs ||_dSr-r.)rrrrr rMs cCs |jdur t|jnd|jdS)Nr) num_replicasrank)rlenr&r"rrr distributed_sampler_kwargsQsz+ParallelStrategy.distributed_sampler_kwargsFtensorgroup sync_gradscCst|||dS)z&Perform a all_gather on all processes.)r5r6)r)rr4r5r6rrr all_gatherXszParallelStrategy.all_gatherTdecisionallcCsFtjt||jd}|j|tjd}|rt||jk}|St|}|S)aReduces a boolean decision over distributed processes. By default is analagous to ``all`` from the standard library, returning ``True`` only if all input decisions evaluate to ``True``. If ``all`` is set to ``False``, it behaves like ``any`` instead. Args: decision: A single input decision. all: Whether to logically emulate ``all`` or ``any``. Defaults to True. Returns: bool: The reduced boolean decision. )device) reduce_op) torchr4intr#reducerSUMboolr+)rr8r9rrr reduce_boolean_decision]sz(ParallelStrategy.reduce_boolean_decisionccsTt|jtjjjr%|j dVWddS1swYdSdVdS)zBlocks ddp sync gradients behaviour on backwards pass. This is useful for skipping sync when accumulating gradients, reducing communication overhead Returns: context manager with sync behaviour off N) isinstancemodelpl utilitiestypesDistributedDataParallelno_syncr"rrr block_backward_syncss  " z$ParallelStrategy.block_backward_synccs&|jdusJ|jtdSr-)rteardownrr"rrr rJs zParallelStrategy.teardown)NNNNN)NF)T)r!N)%__name__ __module__ __qualname____doc__r rr<r:r r rrpropertyrr r#r=r&r(r)r+r@r,rsetterrstrrr3r r7rArrrIrJ __classcell__rrrr rs^  " r)abcrr contextlibrtypingrrrrr r<r typing_extensionsr pytorch_lightningrDlightning_fabric.pluginsr r &lightning_fabric.utilities.distributedrrpytorch_lightning.pluginsr#pytorch_lightning.plugins.precisionr%pytorch_lightning.strategies.strategyrrrrrr s