o پi.@s,dZddlZddlmZGdddeZdS)an PyTorch LARS / LARC Optimizer An implementation of LARS (SGD) + LARC in PyTorch Based on: * PyTorch SGD: https://github.com/pytorch/pytorch/blob/1.7/torch/optim/sgd.py#L100 * NVIDIA APEX LARC: https://github.com/NVIDIA/apex/blob/master/apex/parallel/LARC.py Additional cleanup and modifications to properly support PyTorch XLA. Copyright 2021 Ross Wightman N) OptimizercsReZdZdZ         dfdd Zfd d Zedd d ZZ S)Larsa LARS for PyTorch Paper: `Large batch training of Convolutional Networks` - https://arxiv.org/pdf/1708.03888.pdf Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups. lr (float, optional): learning rate (default: 1.0). momentum (float, optional): momentum factor (default: 0) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) dampening (float, optional): dampening for momentum (default: 0) nesterov (bool, optional): enables Nesterov momentum (default: False) trust_coeff (float): trust coefficient for computing adaptive lr / trust_ratio (default: 0.001) eps (float): eps for division denominator (default: 1e-8) trust_clip (bool): enable LARC trust ratio clipping (default: False) always_adapt (bool): always apply LARS LR adapt, otherwise only when group weight_decay != 0 (default: False) ?rFMbP?:0yE>c s|dkr td||dkrtd||dkr!td||r/|dks+|dkr/tdt|||||||| | d } t|| dS)NgzInvalid learning rate: zInvalid momentum value: zInvalid weight_decay value: rz8Nesterov momentum requires a momentum and zero dampening) lrmomentum dampening weight_decaynesterov trust_coeffeps trust_clip always_adapt) ValueErrordictsuper__init__) selfparamsrrr r r r r rrdefaults __class__C/home/ubuntu/.local/lib/python3.10/site-packages/timm/optim/lars.pyr#s(  z Lars.__init__cs(t||jD]}|ddq dS)Nr F)r __setstate__ param_groups setdefault)rstategrouprrrrFs  zLars.__setstate__Nc Csd}|durt |}Wdn1swY|jD]}|d}|d}|d}|d}|d}|d} |dD]} | jdurFq>| j} |d ksQ|d r| d } | d } || | | || }t| d kt| d k|d d }|d rtj||dd d}| j| |d| ||d kr|j | }d|vrt |  }|d<n|d}||j| d |d|r| j ||d} n|} | j| |d dq>q |S)zPerforms a single optimization step. Args: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nr rr r r r rrrg@rrr)max)alphamomentum_buffer) torch enable_gradrgradnormwhereclampadd_mul_rclonedetachadd)rclosurelossrr rr r r r pr%w_normg_norm trust_ratio param_statebufrrrstepKsR        &z Lars.step) rrrrFrrFF)N) __name__ __module__ __qualname____doc__rrr#no_gradr6 __classcell__rrrrrs #r)r:r#torch.optim.optimizerrrrrrrs