o ϯi7@sddlZddlZddlmZmZmZmZgdZGdddZGdddeZ Gdd d eZ Gd d d eZ Gd d d eZ GdddeZ GdddeZGdddeZGdddeZGdddeZdS)N)ListOptionalSequenceUnion) ParamSchedulerConstantParamSchedulerCosineParamSchedulerExponentialParamSchedulerLinearParamSchedulerCompositeParamSchedulerMultiStepParamSchedulerStepParamScheduler StepWithFixedGammaParamSchedulerPolynomialDecayParamSchedulerc@s&eZdZdZdZdedefddZdS)rz Base class for parameter schedulers. A parameter scheduler defines a mapping from a progress value in [0, 1) to a number (e.g. learning rate). gư>wherereturncCstd)a2 Get the value of the param for a given point at training. We update params (such as learning rate) based on the percent progress of training completed. This allows a scheduler to be agnostic to the exact length of a particular run (e.g. 120 epochs vs 90 epochs), as long as the relative progress where params should be updated is the same. However, it assumes that the total length of training is known. Args: where: A float in [0,1) that represents how far training has progressed z'Param schedulers must override __call__)NotImplementedErrorselfrrQ/home/ubuntu/.local/lib/python3.10/site-packages/fvcore/common/param_scheduler.py__call__!szParamScheduler.__call__N)__name__ __module__ __qualname____doc__ WHERE_EPSILONfloatrrrrrrsrc@s4eZdZdZdeddfddZdedefdd ZdS) rz/ Returns a constant value for a param. valuerNcCs ||_dSN)_value)rrrrr__init__7 zConstantParamScheduler.__init__rcCs|dkr td||jSN?z/where in ParamScheduler must be in [0, 1]: got ) RuntimeErrorr rrrrr:s zConstantParamScheduler.__call__rrrrrr!rrrrrr2src@8eZdZdZdededdfddZdedefd d ZdS) ra Cosine decay or cosine warmup schedules based on start and end values. The schedule is updated based on the fraction of training progress. The schedule was proposed in 'SGDR: Stochastic Gradient Descent with Warm Restarts' (https://arxiv.org/abs/1608.03983). Note that this class only implements the cosine annealing part of SGDR, and not the restarts. Example: .. code-block:: python CosineParamScheduler(start_value=0.1, end_value=0.0001) start_value end_valuerNcC||_||_dSr _start_value _end_valuerr(r)rrrr!Q zCosineParamScheduler.__init__rcCs*|jd|j|jdttj|S)Ng?)r-r,mathcospirrrrrYszCosineParamScheduler.__call__r&rrrrrB rc@r') r a Exponetial schedule parameterized by a start value and decay. The schedule is updated based on the fraction of training progress, `where`, with the formula `param_t = start_value * (decay ** where)`. Example: .. code-block:: python ExponentialParamScheduler(start_value=2.0, decay=0.02) Corresponds to a decreasing schedule with values in [2.0, 0.04). r(decayrNcCr*rr,_decay)rr(r5rrrr!nr/z"ExponentialParamScheduler.__init__rcCs|j|j|Srr6rrrrrvsz"ExponentialParamScheduler.__call__r&rrrrr _r4r c@r') r a Linearly interpolates parameter between ``start_value`` and ``end_value``. Can be used for either warmup or decay based on start and end values. The schedule is updated after every train step by default. Example: .. code-block:: python LinearParamScheduler(start_value=0.0001, end_value=0.01) Corresponds to a linear increasing schedule with values in [0.0001, 0.01) r(r)rNcCr*rr+r.rrrr!r/zLinearParamScheduler.__init__rcCs|j||jd|SNr0)r-r,rrrrrszLinearParamScheduler.__call__r&rrrrr zr4r c @sReZdZdZ  d deedeedeeeddfddZd edefd d Z dS) r a, Takes a predefined schedule for a param value, and a list of epochs or steps which stand for the upper boundary (excluded) of each range. Example: .. code-block:: python MultiStepParamScheduler( values=[0.1, 0.01, 0.001, 0.0001], milestones=[30, 60, 80, 120] ) Then the param value will be 0.1 for epochs 0-29, 0.01 for epochs 30-59, 0.001 for epochs 60-79, 0.0001 for epochs 80-120. Note that the length of values must be equal to the length of milestones plus one. Nvalues num_updates milestonesrcCs6|dur |dur td|dur2g}t|tt|}tt|dD] }|||dq%n!t|trEt|t|t |duksStdt|t |du|durb|d|dd}}|t|krltd||_ ||_ ||_ d}|j D]}||j krtd|j |f||krtd ||f|}qzdS) aW Args: values: param value in each range num_updates: the end of the last range. If None, will use ``milestones[-1]`` milestones: the boundary of each range. If None, will evenly split ``num_updates`` For example, all the following combinations define the same scheduler: * num_updates=90, milestones=[30, 60], values=[1, 0.1, 0.01] * num_updates=90, values=[1, 0.1, 0.01] * milestones=[30, 60, 90], values=[1, 0.1, 0.01] * milestones=[3, 6, 9], values=[1, 0.1, 0.01] (ParamScheduler is scale-invariant) Nz.num_updates and milestones cannot both be Noner0z3MultiStep scheduler requires a list of %d miletonesz?Total num_updates must be greater than length of param schedulerzTMilestone must be smaller than total number of updates: num_updates=%d, milestone=%dzHMilestone must be smaller than start epoch: start_epoch=%d, milestone=%d) ValueErrorr1ceilrlenrangeappend isinstancerint_param_schedule _num_updates _milestones)rr9r:r; step_widthidx start_epoch milestonerrrr!sR   z MultiStepParamScheduler.__init__rcCs>|dkr td|t||j|j}|jt|j|Sr#)r%rCrrErDbisect bisect_rightrF)rr epoch_numrrrrs z MultiStepParamScheduler.__call__)NN) rrrrrrrrCr!rrrrrr s  @r c@r') ras Decays the param value after every epoch according to a polynomial function with a fixed power. The schedule is updated after every train step by default. Example: .. code-block:: python PolynomialDecayParamScheduler(base_value=0.1, power=0.9) Then the param value will be 0.1 for epoch 0, 0.099 for epoch 1, and so on. base_valuepowerrNcCr*r _base_value_power)rrNrOrrrr!r/z&PolynomialDecayParamScheduler.__init__rcCs|jd||jSr8rPrrrrr sz&PolynomialDecayParamScheduler.__call__r&rrrrrs rc@sDeZdZdZdeeefdeeddfddZdedefd d Z dS) r a Takes a fixed schedule for a param value. If the length of the fixed schedule is less than the number of epochs, then the epochs are divided evenly among the param schedule. The schedule is updated after every train epoch by default. Example: .. code-block:: python StepParamScheduler(values=[0.1, 0.01, 0.001, 0.0001], num_updates=120) Then the param value will be 0.1 for epochs 0-29, 0.01 for epochs 30-59, 0.001 for epoch 60-89, 0.0001 for epochs 90-119. r:r9rNcCs8|dkrtdt|trt|dkstd||_dS)Nrz'Number of updates must be larger than 0z:Step scheduler requires a list of at least one param value)r=rBrr?rD)rr:r9rrrr! s zStepParamScheduler.__init__rcCs"t||jt|j}|j|Sr)rCrr?rD)rrindrrrr-s zStepParamScheduler.__call__) rrrrrrCrrr!rrrrrr s   r c @s@eZdZdZdededededdf dd Zd edefd d ZdS) ra Decays the param value by gamma at equal number of steps so as to have the specified total number of decays. Example: .. code-block:: python StepWithFixedGammaParamScheduler( base_value=0.1, gamma=0.1, num_decays=3, num_updates=120) Then the param value will be 0.1 for epochs 0-29, 0.01 for epochs 30-59, 0.001 for epoch 60-89, 0.0001 for epochs 90-119. rN num_decaysgammar:rNcCs||fD]}t|ttfr|dkstdq||fD]}t|tr%|dks)tdq||_||_||_||_|g}t|D] }| |d|q=t ||d|_ dS)Nrz-base_value and gamma must be positive numbersz4num_decays and num_updates must be positive integersr<)r:r9) rBrCrr=rNrTrUr:r@rAr _step_param_scheduler)rrNrTrUr:kr9_rrrr!Bs$    z)StepWithFixedGammaParamScheduler.__init__rcCs ||Sr)rVrrrrr\r"z)StepWithFixedGammaParamScheduler.__call__)rrrrrrCr!rrrrrr2s rc@sHeZdZdZdeedeedeeddfddZ d edefd d Z dS) r a Composite parameter scheduler composed of intermediate schedulers. Takes a list of schedulers and a list of lengths corresponding to percentage of training each scheduler should run for. Schedulers are run in order. All values in lengths should sum to 1.0. Each scheduler also has a corresponding interval scale. If interval scale is 'fixed', the intermediate scheduler will be run without any rescaling of the time. If interval scale is 'rescaled', intermediate scheduler is run such that each scheduler will start and end at the same values as it would if it were the only scheduler. Default is 'rescaled' for all schedulers. Example: .. code-block:: python schedulers = [ ConstantParamScheduler(value=0.42), CosineParamScheduler(start_value=0.42, end_value=1e-4) ] CompositeParamScheduler( schedulers=schedulers, interval_scaling=['rescaled', 'rescaled'], lengths=[0.3, 0.7]) The parameter value will be 0.42 for the first [0%, 30%) of steps, and then will cosine decay from 0.42 to 0.0001 for [30%, 100%) of training. schedulerslengthsinterval_scalingrNcCst|t|kr tdt|dkrtdtt|ddkr$tdt|dkr6dt|dd|d<|D] }|dvrEtd |q8||_||_||_dS) Nz*Schedulers and lengths must be same lengthrz?There must be at least one scheduler in the composite schedulerr$gMbP?z*The sum of all values in lengths must be 1r<)rescaledfixedzUnsupported interval_scaling: )r?r=abssum_lengths _schedulers_interval_scaling)rrYrZr[srrrr!s"   z CompositeParamScheduler.__init__rcCsd}|j|}||j|kr2|t|jdkr2|d7}||j|7}||j|kr2|t|jdks|j|}|}|j|}|dkrR||j|}|||j|}||S)Nrr0r\)r`rr?rarb)rri running_total schedulerscheduler_whereinterval_scalescheduler_startrrrrs*   z CompositeParamScheduler.__call__) rrrrrrrrstrr!rrrrrr `s r )rKr1typingrrrr__all__rrrr r r rr rr rrrrs]#.