o yi-@sZddlmZddlmZddlmZmZddlZddlZddlZddl Z GdddZ dS))division)unicode_literals)IterableOptionalNc@s0eZdZdZ ddeejjdede fddZ de eejjdeejjfd d Z dde eejjdd fd d Z dde eejjdd fddZ dde eejjdd fddZ dde eejjdd fddZej dde eejjfddZdd ddZdefddZdedd fddZd S)!ExponentialMovingAveragea Maintains (exponential) moving average of a set of parameters. Args: parameters: Iterable of `torch.nn.Parameter` (typically from `model.parameters()`). Note that EMA is computed on *all* provided parameters, regardless of whether or not they have `requires_grad = True`; this allows a single EMA object to be consistantly used even if which parameters are trainable changes step to step. If you want to some parameters in the EMA, do not pass them to the object in the first place. For example: ExponentialMovingAverage( parameters=[p for p in model.parameters() if p.requires_grad], decay=0.9 ) will ignore parameters that do not require grad. decay: The exponential decay. use_num_updates: Whether to use number of updates when computing averages. T parametersdecayuse_num_updatescCs^|dks|dkr td||_|rdnd|_t|}dd|D|_d|_dd|D|_dS)N?Decay must be between 0 and 1rcSsg|]}|qS)clonedetach.0pr r A/home/ubuntu/.local/lib/python3.10/site-packages/torch_ema/ema.py 4s z5ExponentialMovingAverage.__init__..cSsg|]}t|qSr )weakrefrefrr r rr>s) ValueErrorr num_updateslist shadow_paramscollected_params _params_refs)selfrrr r r r__init__)sz!ExponentialMovingAverage.__init__returncCs\|durdd|jD}tdd|Drtd|St|}t|t|jkr,td|S)NcSg|]}|qSr r rr r rrEz.cs|]}|duVqdSNr rr r r Fz;ExponentialMovingAverage._get_parameters..z(One of) the parameters with which this ExponentialMovingAverage was initialized no longer exists (was garbage collected); please either provide `parameters` explicitly or keep the model to which they belong from being garbage collected.zNumber of parameters passed as argument is different from number of shadow parameters maintained by this ExponentialMovingAverage)ranyrrlenrrrr r r_get_parameters@sz(ExponentialMovingAverage._get_parametersNcCs||}|j}|jdur!|jd7_t|d|jd|j}d|}t"t|j|D]\}}||}||| |q0WddS1sNwYdS)a Update currently maintained parameters. Call this every time the parameters are updated, such as the result of the `optimizer.step()` call. Args: parameters: Iterable of `torch.nn.Parameter`; usually the same set of parameters used to initialize this object. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. N r ) r)rrmintorchno_gradziprmul_sub_)rrrone_minus_decays_paramparamtmpr r rupdateZs     "zExponentialMovingAverage.updatecCs2||}t|j|D] \}}|j|jq dS)aq Copy current averaged parameters into given collection of parameters. Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored moving averages. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. N)r)r/rdatacopy_)rrr3r4r r rcopy_tozs z ExponentialMovingAverage.copy_tocCs||}dd|D|_dS)a: Save the current parameters for restoring later. Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be temporarily stored. If `None`, the parameters of with which this `ExponentialMovingAverage` was initialized will be used. cSsg|]}|qSr )r)rr4r r rrsz2ExponentialMovingAverage.store..N)r)rr(r r rstores  zExponentialMovingAverage.storecCsD|jdur td||}t|j|D] \}}|j|jqdS)ad Restore the parameters stored with the `store` method. Useful to validate the model with EMA parameters without affecting the original optimization process. Store the parameters before the `copy_to` method. After validation (or model saving), use this to restore the former parameters. Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored parameters. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. NzGThis ExponentialMovingAverage has no `store()`ed weights to `restore()`)r RuntimeErrorr)r/r7r8)rrc_paramr4r r rrestores  z ExponentialMovingAverage.restorec csD||}||||z dVW||dS||w)a  Context manager for validation/inference with averaged parameters. Equivalent to: ema.store() ema.copy_to() try: ... finally: ema.restore() Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored parameters. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. N)r)r:r9r=r(r r raverage_parameterss   z+ExponentialMovingAverage.average_parameterscs>fdd|jD|_|jdurfdd|jD|_dS)zMove internal buffers of the ExponentialMovingAverage to `device`. Args: device: like `device` argument to `torch.Tensor.to` c.g|]}|r|jdn|jdqSdevicedtype)rBis_floating_pointtorrAr rr  z/ExponentialMovingAverage.to..Ncr?r@rDrrAr rrrG)rr)rrBrCr rArrFs   zExponentialMovingAverage.tocCs|j|j|j|jdS)z.z!shadow_params must all be Tensorsrzcollected_params must be a listcsrJr#rKrr r rr$rNz$collected_params must all be Tensorsz8collected_params and shadow_params had different lengthscSr r r rr r rrr!z.csr"r#r rr r rr$r%rAzTTried to `load_state_dict()` with the wrong number of parameters in the saved state.)copydeepcopyrrrrLintrrallrr'rr& enumeraterFrBrC)rrIparamsirr r rload_state_dicts^             z(ExponentialMovingAverage.load_state_dict)Tr#)NN)rN)__name__ __module__ __qualname____doc__rr-nn Parameterfloatboolrrr)r6r9r:r= contextlibcontextmanagerr>rFdictrIrVr r r rrs^     "     r) __future__rrtypingrrrrOr_r-rr r r rs