o iW@sdZddlZddlmZddlmZGdddZGdddZiZd d Z d d Z e Gd ddeZ e GdddeZ e GdddeZ dS)z Schedulers.N)dynamic_import)fill_missing_argsc@seZdZddZddZdS) _PrefixParsercCs||_||_dSNparserprefix)selfrrr N/home/ubuntu/.local/lib/python3.10/site-packages/espnet/scheduler/scheduler.py__init__ s z_PrefixParser.__init__cKs4|dsJ|jj|j|ddfi|dS)N--) startswithr add_argumentr)r namekwargsr r r rs&z_PrefixParser.add_argumentN)__name__ __module__ __qualname__r rr r r r r s rc@s~eZdZdZdZdedejfddZddZ e ded ej fd d Z e d efd d Ze defddZdedefddZdS)SchedulerInterfacezScheduler interface.keyargscCsT||_|d|jd}t|D]\}}||r't||t|d|qdS)Initialize class._N)raliasvarsitemsrsetattrlen)r rrrkvr r r r s zSchedulerInterface.__init__cCs t|j|jd|jd|S)zGet argument without prefix.r)getattrrrr)r rr r r get_arg s zSchedulerInterface.get_argrc Cs8||jd}|t|d|d|jdd|S)zAdd arguments for CLI.z schedulerr -r)add_argument_groupr_add_argumentsr)clsrrgroupr r r add_arguments$s"z SchedulerInterface.add_argumentscCsdSrr rr r r r'+z!SchedulerInterface._add_argumentsc sJfdd}fdd|D}tjdi|}t||}|S)zInitialize this class with python-level args. Args: key (str): key of hyper parameter Returns: LMinterface: A new instance of LMInterface. cs |Sr)r*r+r(rr r add;s z%SchedulerInterface.build..addcs(i|]\}}djd||qS)r)r).0r!r"r-r r >s(z,SchedulerInterface.build..Nr )rargparse Namespacer)r(rrr.rr r-r build/s    zSchedulerInterface.buildn_iterreturncCst)zScale at `n_iter`. Args: n_iter (int): number of current iterations. Returns: float: current scale of learning rate. )NotImplementedErrorr r4r r r scaleCs zSchedulerInterface.scaleN)rrr__doc__rstrr1r2r r$ classmethodArgumentParserr* staticmethodrr'r3intfloatr8r r r r rsrcCs|jd|jt|j<|S)zRegister scheduler.:)rrSCHEDULER_DICTr)r(r r r register_schedulerSsrBcCs&t|t}t|tsJ|d|S)zImport Scheduler class dynamically. Args: module (str): module_name:class_name or alias in `SCHEDULER_DICT` Returns: type: Scheduler class z& does not implement SchedulerInterface)rrA issubclassr)module model_classr r r dynamic_import_schedulerYs rFc@seZdZdZdZddZdS) NoSchedulerzScheduler which does nothing.nonecCsdS) Scale of lr.g?r r7r r r r8pr,zNoScheduler.scaleN)rrrr9rr8r r r r rGjs rGcs>eZdZdZdZedefddZfddZdd Z Z S) NoamSchedulerznWarmup + InverseSqrt decay scheduler. Args: noam_warmup (int): number of warmup iterations. noamrcCs|jdtddddS)Add scheduler args.--warmupNumber of warmup iterations.typedefaulthelpNrr>r+r r r r's zNoamScheduler._add_argumentscs(t||d|j|jd|_dS)rN)superr warmup normalize)r rr __class__r r r szNoamScheduler.__init__cCs&|d7}|jt|d||jdS)rIrUgrV)rYminrX)r stepr r r r8szNoamScheduler.scale) rrrr9rr=rr'r r8 __classcell__r r rZr rJus rJc@s.eZdZdZdZedefddZddZdS) CyclicCosineScheduleravCyclic cosine annealing. Args: cosine_warmup (int): number of warmup iterations. cosine_total (int): number of total annealing iterations. Notes: Proposed in https://openreview.net/pdf?id=BJYwwY9ll (and https://arxiv.org/pdf/1608.03983.pdf). Used in the GPT2 config of Megatron-LM https://github.com/NVIDIA/Megatron-LM cosinercCs(|jdtddd|jdtddddS) rLrMrNrOrPz--totaliz%Number of total annealing iterations.NrTr+r r r r's z$CyclicCosineScheduler._add_argumentscCs,ddl}d||j||j|jdS)rIrNg?rU)mathcospirXtotal)r r4rar r r r8s$zCyclicCosineScheduler.scaleN) rrrr9rr=rr'r8r r r r r_s   r_)r9r1espnet.utils.dynamic_importrespnet.utils.fill_missing_argsrrrrArBrFrGrJr_r r r r s   =