o 8wi@sddlmZmZddlmZddlmZddlmZm Z ddl Z ddl m Z ddl m Z ddlZddlmZmZdd lmZmZdd lmZdd lmZdd lmZGd ddeeZdS))ABCabstractmethod) Generator)contextmanager)AnyOptionalN)Tensor)override) CheckpointIOClusterEnvironment)ReduceOp_all_gather_ddp_if_available) LayerSync) Precision)Strategyc seZdZdZ     d,deddeeejdeedee dee f fd d Z e e ed ejfd d Ze d efddZe d efddZe d efddZe d efddZe ed efddZe d eeejfddZejdeeejd dfddZe d eeeffddZed-dedeed ed efd!d"Zed.d$ed%ed efd&d'Ze d e!fd(d)Z"ed/fd*d+ Z#Z$S)0ParallelStrategyz:Strategy for training with multiple processes in parallel.N acceleratorzpl.accelerators.Acceleratorparallel_devicescluster_environment checkpoint_ioprecision_plugincs(tj|||d||_||_d|_dS)N)rrr)super__init__rr _layer_sync)selfrrrrr __class__b/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/pytorch_lightning/strategies/parallel.pyr"s zParallelStrategy.__init__returncCsdS)zReturn the root device.Nrrrrr root_device/szParallelStrategy.root_devicecC|jdur |jSdSNr)r global_rankr rrrr$5zParallelStrategy.global_rankcCr"r#)r local_rankr rrrr&9r%zParallelStrategy.local_rankcCr"r#)r node_rankr rrrr'=r%zParallelStrategy.node_rankcCr")N)r world_sizer rrrr)Ar%zParallelStrategy.world_sizecCs |jdkSr#)r$r rrris_global_zeroEs zParallelStrategy.is_global_zerocCs|jSN_parallel_devicesr rrrrJsz!ParallelStrategy.parallel_devicescCs ||_dSr+r,)rrrrrrNs cCs |jdur t|jnd|jdS)Nr) num_replicasrank)rlenr$r rrrdistributed_sampler_kwargsRsz+ParallelStrategy.distributed_sampler_kwargsFtensorgroup sync_gradscCst|||dS)z&Perform a all_gather on all processes.)r3r4)r )rr2r3r4rrr all_gatherYszParallelStrategy.all_gatherTdecisionallcCsFtjt||jd}|j|tjd}|rt||jk}|St|}|S)aReduces a boolean decision over distributed processes. By default is analogous to ``all`` from the standard library, returning ``True`` only if all input decisions evaluate to ``True``. If ``all`` is set to ``False``, it behaves like ``any`` instead. Args: decision: A single input decision. all: Whether to logically emulate ``all`` or ``any``. Defaults to True. Returns: bool: The reduced boolean decision. )device) reduce_op) torchr2intr!reducer SUMboolr))rr6r7rrrreduce_boolean_decision^sz(ParallelStrategy.reduce_boolean_decisionccsTt|jtjjjr%|j dVWddS1swYdSdVdS)zBlocks ddp sync gradients behaviour on backwards pass. This is useful for skipping sync when accumulating gradients, reducing communication overhead Returns: context manager with sync behaviour off N) isinstancemodelpl utilitiestypesDistributedDataParallelno_syncr rrrblock_backward_syncts  " z$ParallelStrategy.block_backward_synccs&|jdusJ|jtdSr+)rteardownrr rrrrHs zParallelStrategy.teardown)NNNNN)NF)T)rN)%__name__ __module__ __qualname____doc__rlistr:r8r r rrpropertyrr r!r;r$r&r'r)r>r*rsetterdictstrrr1rr5r?rrrGrH __classcell__rrrrrs^  " r)abcrrcollections.abcr contextlibrtypingrrr:rtyping_extensionsr pytorch_lightningrBlightning_fabric.pluginsr r &lightning_fabric.utilities.distributedr r pytorch_lightning.pluginsr#pytorch_lightning.plugins.precisionr%pytorch_lightning.strategies.strategyrrrrrrs