o si@s|ddlZddlmZddlZddlmZGdddeZGdddeZ Gd d d eZ d d Z Gd ddej j ZeZdS)N) Optimizer)SinkPITLossWrapperc@s^eZdZdZddZddZddZdd Zdd d Zd dZ ddZ dddZ dddZ d S) BaseSchedulerzBase class for the step-wise scheduler logic. Args: optimizer (Optimize): Optimizer instance to apply lr schedule on. Subclass this and overwrite ``_get_lr`` to write your own step-wise scheduler. cCs||_d|_dSNr) optimizerstep_num)selfrr N/home/ubuntu/.local/lib/python3.10/site-packages/asteroid/engine/schedulers.py__init__s zBaseScheduler.__init__cCs|jdSN)r zero_gradr r r r rszBaseScheduler.zero_gradcCstr )NotImplementedErrorrr r r _get_lrszBaseScheduler._get_lrcCs|jjD]}||d<qdS)Nlr)r param_groups)r r param_groupr r r _set_lrs  zBaseScheduler._set_lrNcCs$|jd7_|}||dS)z5Update step-wise learning rate before optimizer.step.N)rrr)r metricsepochrr r r stepszBaseScheduler.stepcCs|j|dSr )__dict__update)r state_dictr r r load_state_dict%zBaseScheduler.load_state_dictcCsdd|jDS)NcSsi|] \}}|dkr||qS)rr ).0keyvaluer r r )sz,BaseScheduler.state_dict..)ritemsrr r r r(szBaseScheduler.state_dictr順cCs@g}t||D]}|jd7_||qd|_t|S)z0Returns the scheduler values from start to stop.rr)rangerappendrtorchtensor)r startstoplr_list_r r r as_tensor+s  zBaseScheduler.as_tensorcCs4ddlm}|j||d}|||dS)z-Plot the scheduler values from start to stop.rN)r)r*)matplotlib.pyplotpyplotr-plotnumpyshow)r r)r*pltall_lrr r r r04s  zBaseScheduler.plot)NN)rr$) __name__ __module__ __qualname____doc__r rrrrrrr-r0r r r r rs   rcs*eZdZdZdfdd ZddZZS) NoamScheduleraThe Noam learning rate scheduler, originally used in conjunction with the Adam optimizer in [1]. Args: optimizer (Optimizer): Optimizer instance to apply lr schedule on. d_model(int): The number of units in the layer output. warmup_steps (int): The number of steps in the warmup stage of training. scale (float): A fixed coefficient for rescaling the final learning rate. Schedule: The Noam scheduler increases the learning rate linearly for the first ``warmup_steps`` steps, and decreases it thereafter proportionally to the inverse square root of the step number: :math:`lr = scale\_factor * ( model\_dim^{-0.5} * adj\_step )` :math:`adj\_step = min(step\_num^{0.5}, step\_num * warmup\_steps^{-1.5})` References [1] Vaswani et al. (2017) "Attention is all you need". 31st Conference on Neural Information Processing Systems ?cs"t|||_||_||_dSr )superr d_modelscale warmup_steps)r rr<r>r= __class__r r r Ss  zNoamScheduler.__init__cCs0|j|jdt|jd|j|jd}|S)N)r=r<minrr>r rr r r rYszNoamScheduler._get_lr)r:r5r6r7r8r r __classcell__r r r?r r9=sr9cs2eZdZdZ    d fdd Zdd ZZS) DPTNetSchedulerahDual Path Transformer Scheduler used in [1] Args: optimizer (Optimizer): Optimizer instance to apply lr schedule on. steps_per_epoch (int): Number of steps per epoch. d_model(int): The number of units in the layer output. warmup_steps (int): The number of steps in the warmup stage of training. noam_scale (float): Linear increase rate in first phase. exp_max (float): Max learning rate in second phase. exp_base (float): Exp learning rate base in second phase. Schedule: This scheduler increases the learning rate linearly for the first ``warmup_steps``, and then decay it by 0.98 for every two epochs. References [1]: Jingjing Chen et al. "Dual-Path Transformer Network: Direct Context- Aware Modeling for End-to-End Monaural Speech Separation" Interspeech 2020. r:-C6:?\(\?cs:t|||_||_||_||_||_||_d|_dSr) r;r noam_scaler<r>exp_maxexp_basesteps_per_epochr)r rrNr<r>rKrLrMr?r r r ws  zDPTNetScheduler.__init__cCsx|j|jdkr|jd7_|j|jkr$|j|j|jdd}|S|j|jdt|jd|j|jd}|S)NrrrrArB) rrNrr>rLrMrKr<rCrDr r r rs zDPTNetScheduler._get_lr)rHr:rIrJrEr r r?r rGbsrGcCstd|dgS)NgRQ? )rC)rr r r sinkpit_default_beta_schedulerrPc@s$eZdZdZefddZddZdS)SinkPITBetaScheduleradScheduler of the beta value of SinkPITLossWrapper This module is used as a Callback function of `pytorch_lightning.Trainer`. Args: cooling_schedule (callable) : A callable that takes a parameter `epoch` (int) and returns the value of `beta` (float). The default function is ``sinkpit_default_beta_schedule``: :math:`\beta = min(1.02^{epoch}, 10)` Example >>> from pytorch_lightning import Trainer >>> from asteroid.losses import SinkPITBetaScheduler >>> # Default scheduling function >>> sinkpit_beta_schedule = SinkPITBetaSchedule() >>> trainer = Trainer(callbacks=[sinkpit_beta_schedule]) >>> # User-defined schedule >>> sinkpit_beta_schedule = SinkPITBetaScheduler(lambda ep: 1. if ep < 10 else 100.) >>> trainer = Trainer(callbacks=[sinkpit_beta_schedule]) cCs ||_dSr )cooling_schedule)r rRr r r r s zSinkPITBetaScheduler.__init__cCs<t|jtsJ|j|jksJ|j}||}||j_dSr ) isinstance loss_funcr current_epochrRbeta)r trainer pl_modulerrVr r r on_train_epoch_starts   z)SinkPITBetaScheduler.on_train_epoch_startN)r5r6r7r8rPr rYr r r r rQs  rQ)r'torch.optim.optimizerrpytorch_lightningpllossesrobjectrr9rGrP callbacksCallbackrQ_BaseSchedulerr r r r s  5%9"