o zi@sfddlZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z GdddZdS) N) DataParallel)DistributedDataParallel)ResumableDistributedSampler)ResumableSequentialSamplerc@seZdZdZddefddZddZdd Zd ej j fd d Z d dZ dej fddZdejjfddZddZ ddejdefddZeddZdS) AcceleratorafThis class is used to prepare models and dataloaders for usage with DDP or DP. Use the functions prepare_model, prepare_dataloader to prepare the respective objects. In the case of models, they are moved to the appropriate GPU and SyncBatchNorm is applied to them. In the case of dataloaders, a sampler is created and the dataloader is initialized with that sampler. If the world size is 1, prepare_model and prepare_dataloader are no-ops. If the environment variable ``LOCAL_RANK`` is not set, then the script was launched without ``torchrun``, and ``DataParallel`` will be used instead of ``DistributedDataParallel`` (not recommended), if the world size (number of GPUs) is greater than 1. Parameters ---------- amp : bool, optional Whether or not to enable automatic mixed precision, by default False FampcCstdd}tj|_|jdko|du|_|jdko|du|_|jdkr'dnd|_|jr;t |}t j dd|j|d|durAdn||_ ||_ Gd d d }|rVtjj n||_tjritj|j |_dSd|_dS) N LOCAL_RANKrcpucudancclzenv://) init_method world_sizerankc@s4eZdZddZddZddZddZd d Zd S) z)Accelerator.__init__..DummyScalercSdSNselfrrM/home/ubuntu/.local/lib/python3.10/site-packages/audiotools/ml/accelerator.py__init__6z2Accelerator.__init__..DummyScaler.__init__cSs |dSr)stepr optimizerrrrr9s z.Accelerator.__init__..DummyScaler.stepcS|Srrrlossrrrscale<rz/Accelerator.__init__..DummyScaler.scalecSrrrrrrrunscale_?rz2Accelerator.__init__..DummyScaler.unscale_cSrrrrrrrupdateBrz0Accelerator.__init__..DummyScaler.updateN)__name__ __module__ __qualname__rrrr r!rrrr DummyScaler5s  r%)osgetenvtorchr device_countruse_ddpuse_dpdeviceintdistinit_process_group local_rankr GradScalerscaler is_available device_ctx)rrr0r%rrrr!s*  zAccelerator.__init__cCs|jdur |j|Sr)r4 __enter__rrrrr5Js  zAccelerator.__enter__cCs"|jdur|j|||dSdSr)r4__exit__)rexc_type exc_value tracebackrrrr6Os zAccelerator.__exit__modelcKsV||j}|jrtjj|}t|fd|jgi|}|S|j r)t |fi|}|S)abPrepares model for DDP or DP. The model is moved to the device of the correct rank. Parameters ---------- model : torch.nn.Module Model that is converted for DDP or DP. Returns ------- torch.nn.Module Wrapped model, or original model if DDP and DP are turned off. device_ids) tor,r*r(nn SyncBatchNormconvert_sync_batchnormrr0r+r)rr:kwargsrrr prepare_modelSs zAccelerator.prepare_modelcOstjjj|jg|Ri|S)z^Context manager for autocasting. Arguments go to ``torch.cuda.amp.autocast``. )r(r rautocast)rargsr@rrrrBlszAccelerator.autocastrcCs|j|dS)zBackwards pass, after scaling the loss if ``amp`` is enabled. Parameters ---------- loss : torch.Tensor Loss value. N)r2rbackwardrrrrrDrs zAccelerator.backwardrcCs|j|dS)zSteps the optimizer, using a ``scaler`` if ``amp`` is enabled. Parameters ---------- optimizer : torch.optim.Optimizer Optimizer to step forward. N)r2rrrrrr}s zAccelerator.stepcCs|jdS)zUpdates the scale factor.N)r2r!rrrrr!szAccelerator.updateNdataset start_idxcKs||jr*t|||j|jd}d|vrt|d|jd|d<t|d|jd|d<nt||}tjjj |fd|i|}|S)aWraps a dataset with a DataLoader, using the correct sampler if DDP is enabled. Parameters ---------- dataset : typing.Iterable Dataset to build Dataloader around. start_idx : int, optional Start index of sampler, useful if resuming from some epoch, by default None Returns ------- _type_ _description_ ) num_replicasr num_workersr batch_sizesampler) r*DistributedSamplerrr0maxSequentialSamplerr(utilsdata DataLoader)rrErFr@rJ dataloaderrrrprepare_dataloaders zAccelerator.prepare_dataloadercCst|dr|jS|S)zUnwraps the model if it was wrapped in DDP or DP, otherwise just returns the model. Use this to unwrap the model returned by :py:func:`audiotools.ml.accelerator.Accelerator.prepare_model`. module)hasattrrS)r:rrrunwraps zAccelerator.unwrap)Fr)r"r#r$__doc__boolrr5r6r(r=ModulerArBTensorrDoptim Optimizerrr!typingIterabler-rR staticmethodrUrrrrr s$)   $r)r&r\r(torch.distributed distributedr.torch.nn.parallelrr data.datasetsrrKrrMrrrrrs