o ‚o™i®$ã@sdddlZddlmZddlmZGdd„deƒZGdd„deƒZGdd „d eƒZGd d „d eƒZdS) éN)Ú Optimizer)ÚOptimizer1Statecs0eZdZ         d ‡fdd „ Z‡ZS) ÚLARSrFé Néédç{®Gáz”?c s<|dkrtdƒ‚tƒjd||||fd|||| | | dd dS)aG Base LARS optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`): The learning rate. momentum (`float`, defaults to 0): The momentum value speeds up the optimizer by taking bigger steps. dampening (`float`, defaults to 0): The dampening value reduces the momentum of the optimizer. weight_decay (`float`, defaults to 1e-2): The weight decay value for the optimizer. nesterov (`bool`, defaults to `False`): Whether to use Nesterov momentum. optim_bits (`int`, defaults to 32): The number of bits of the optimizer state. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. max_unorm (`float`, defaults to 0.02): The maximum gradient norm. rú'LARS without momentum is not supported!ÚlarsçF©Ú max_unormÚ block_wiseN©ÚNotImplementedErrorÚsuperÚ__init__) ÚselfÚparamsÚlrÚmomentumÚ dampeningÚ weight_decayÚnesterovÚ optim_bitsÚargsÚ min_8bit_sizeÚpercentile_clippingr ©Ú __class__©úK/home/ubuntu/.local/lib/python3.10/site-packages/bitsandbytes/optim/lars.pyr s ) ôz LARS.__init__) rrrFrNrrr©Ú__name__Ú __module__Ú __qualname__rÚ __classcell__r r rr!r sôrcó.eZdZ        d ‡fdd„ Z‡ZS) ÚLARS8bitrFNrrrc ó<|dkrtdƒ‚tƒjd||||fd|d||| | dd dS) aÝ 8-bit LARS optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`): The learning rate. momentum (`float`, defaults to 0): The momentum value speeds up the optimizer by taking bigger steps. dampening (`float`, defaults to 0): The dampening value reduces the momentum of the optimizer. weight_decay (`float`, defaults to 1e-2): The weight decay value for the optimizer. nesterov (`bool`, defaults to `False`): Whether to use Nesterov momentum. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. max_unorm (`float`, defaults to 0.02): The maximum gradient norm. rr r r éFr Nr© rrrrrrrrrrr rr r!rHó & ôzLARS8bit.__init__©rrrFNrrrr"r r rr!r(Góõr(cr') Ú LARS32bitrFNrrrc r)) aÞ 32-bit LARS optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`): The learning rate. momentum (`float`, defaults to 0): The momentum value speeds up the optimizer by taking bigger steps. dampening (`float`, defaults to 0): The dampening value reduces the momentum of the optimizer. weight_decay (`float`, defaults to 1e-2): The weight decay value for the optimizer. nesterov (`bool`, defaults to `False`): Whether to use Nesterov momentum. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. max_unorm (`float`, defaults to 0.02): The maximum gradient norm. rr r r rFr Nrr+rr r!rr,zLARS32bit.__init__r-r"r r rr!r/€r.r/csHeZdZ      d ‡fdd„ Z‡fdd„Ze ¡d d d „ƒZ‡ZS)Ú PytorchLARSç{®Gáz„?rFrc s„|dkr td|›ƒ‚|dkrtd|›ƒ‚|dkr!td|›ƒ‚t||||||d}|r9|dks5|dkr9tdƒ‚tƒ ||¡dS)Nr zInvalid learning rate: zInvalid momentum value: zInvalid weight_decay value: )rrrrrr rz8Nesterov momentum requires a momentum and zero dampening)Ú ValueErrorÚdictrr) rrrrrrrr Údefaultsrr r!rºs" úzPytorchLARS.__init__cs(tƒ |¡|jD]}| dd¡q dS)NrF)rÚ __setstate__Ú param_groupsÚ setdefault)rÚstateÚgrouprr r!r5×s  ÿzPytorchLARS.__setstate__NcCsˆd}|durt ¡ |ƒ}Wdƒn1swY|jD]¡}g}g}g}|d}|d}|d} |d} |d} |d} |dD]|} | jdurLqD|j| }| j}|d kr_|j| |d }|d kr| d d¡}|duryt |¡ ¡}||d <n |  |¡j |d | d | rŽ|||}n|}d }| dkr¶| j tj ksžJ‚t  |  ¡¡}t  |¡}|| |kr¶| ||}| j || |d qDq |S)z±Performs a single optimization step. Args: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nrrrrr rrr)ÚalphaÚmomentum_bufferégð?r )ÚtorchÚ enable_gradr6Úgradr8ÚaddÚgetÚcloneÚdetachÚmul_Úadd_ÚdtypeÚfloat32Únorm)rÚclosureÚlossr9Úparams_with_gradÚd_p_listÚmomentum_buffer_listrrrrr rÚpr8Úd_pÚbufÚupdateÚ update_scaleÚpnormÚunormr r r!ÚstepÜsR ÿ         á!zPytorchLARS.step)r1rrrFr)N) r#r$r%rr5r=Úno_gradrUr&r r rr!r0¹sø r0) r=Ú torch.optimrÚbitsandbytes.optim.optimizerrrr(r/r0r r r r!Ús  <99