o 3wi]-@spddlmZGdddeZGdddeZGdddeZGdd d eZGd d d eZGd d d eZdS))Optimizer1Statecs0eZdZ         d fd d ZZS) Lion-C6?g?gGz?r NdTFc s(tjd|||d|||||| | d dS)aj Base Lion optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-4): The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)): The beta values are the decay rates of the first and second-order moment of the optimizer. weight_decay (`float`, defaults to 0): The weight decay value for the optimizer. optim_bits (`int`, defaults to 32): The number of bits of the optimizer state. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. is_paged (`bool`, defaults to `False`): Whether the optimizer is a paged optimizer or not. lionis_pagedNsuper__init__) selfparamslrbetas weight_decay optim_bitsargs min_8bit_sizepercentile_clipping block_wiser  __class__T/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/bitsandbytes/optim/lion.pyr s& z Lion.__init__) rrrrNrrTF__name__ __module__ __qualname__r __classcell__rrrrrsrc.eZdZ        d fd d ZZS) Lion8bitrrrNrrTFc (tjd|||d|d||||| d dS)a 8-bit Lion optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-4): The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)): The beta values are the decay rates of the first and second-order moment of the optimizer. weight_decay (`float`, defaults to 0): The weight decay value for the optimizer. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. is_paged (`bool`, defaults to `False`): Whether the optimizer is a paged optimizer or not. r r r Nr rrrrrrrrrr rrrr@# zLion8bit.__init__rrrNrrTFrrrrrr$?r$cr#) Lion32bitrrrNrrTFc r%)a 32-bit Lion optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-4): The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)): The beta values are the decay rates of the first and second-order moment of the optimizer. weight_decay (`float`, defaults to 0): The weight decay value for the optimizer. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. is_paged (`bool`, defaults to `False`): Whether the optimizer is a paged optimizer or not. r r rr Nr r'rrrrtr(zLion32bit.__init__r)rrrrrr+sr*r+cr#) PagedLionrrrrNrrTc s(tjd|||d|||||| dd dS)a Paged Lion optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-4): The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)): The beta values are the decay rates of the first and second-order moment of the optimizer. weight_decay (`float`, defaults to 0): The weight decay value for the optimizer. optim_bits (`int`, defaults to 32): The number of bits of the optimizer state. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. r r Tr Nr ) rrrrrrrrrrrrrrr(zPagedLion.__init__)rrrrNrrTrrrrrr,r*r,c,eZdZ       d fdd ZZS) PagedLion8bitrrrNrrTc (tjd|||d|d||||dd dS)a Paged 8-bit Lion optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-4): The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)): The beta values are the decay rates of the first and second-order moment of the optimizer. weight_decay (`float`, defaults to 0): The weight decay value for the optimizer. optim_bits (`int`, defaults to 32): The number of bits of the optimizer state. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. r r r&Tr Nr rrrrrrrrrrrrr" zPagedLion8bit.__init__rrrNrrTrrrrrr.r.cr-) PagedLion32bitrrrNrrTc r/)a Paged 32-bit Lion optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-4): The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)): The beta values are the decay rates of the first and second-order moment of the optimizer. weight_decay (`float`, defaults to 0): The weight decay value for the optimizer. optim_bits (`int`, defaults to 32): The number of bits of the optimizer state. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. r r rTr Nr r0rrrrr1zPagedLion32bit.__init__r2rrrrrr4r3r4N)bitsandbytes.optim.optimizerrrr$r+r,r.r4rrrrs 74443