o 3wi8@spddlmZGdddeZGdddeZGdddeZGdd d eZGd d d eZGd d d eZdS))Optimizer2Statec4eZdZ         d fd d ZZS)AdamWMbP?g?g+?:0yE>{Gz?F NdTc s(tjd|||||||| | | | d dS)a Base AdamW optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-3): The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)): The beta values are the decay rates of the first and second-order moment of the optimizer. eps (`float`, defaults to 1e-8): The epsilon value prevents division by zero in the optimizer. weight_decay (`float`, defaults to 1e-2): The weight decay value for the optimizer. amsgrad (`bool`, defaults to `False`): Whether to use the [AMSGrad](https://hf.co/papers/1904.09237) variant of Adam that uses the maximum of past squared gradients instead. optim_bits (`int`, defaults to 32): The number of bits of the optimizer state. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. is_paged (`bool`, defaults to `False`): Whether the optimizer is a paged optimizer or not. adamis_pagedNsuper__init__ selfparamslrbetaseps weight_decayamsgrad optim_bitsargs min_8bit_sizepercentile_clipping block_wiser __class__U/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/bitsandbytes/optim/adamw.pyr , zAdamW.__init__ rrrrFr Nr r TF__name__ __module__ __qualname__r __classcell__r!r!rr"rrcr) AdamW8bitrrrrFr Nr r Tc (tjd|||||d|| | | | d dS)a 8-bit AdamW optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-3): The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)): The beta values are the decay rates of the first and second-order moment of the optimizer. eps (`float`, defaults to 1e-8): The epsilon value prevents division by zero in the optimizer. weight_decay (`float`, defaults to 1e-2): The weight decay value for the optimizer. amsgrad (`bool`, defaults to `False`): Whether to use the [AMSGrad](https://hf.co/papers/1904.09237) variant of Adam that uses the maximum of past squared gradients instead. optim_bits (`int`, defaults to 32): The number of bits of the optimizer state. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. is_paged (`bool`, defaults to `False`): Whether the optimizer is a paged optimizer or not. r r Nrrrr!r"rFr#zAdamW8bit.__init__r$r%r!r!rr"r+Er*r+cr) AdamW32bitrrrrFr Nr r Tc r,)a 32-bit AdamW optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-3): The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)): The beta values are the decay rates of the first and second-order moment of the optimizer. eps (`float`, defaults to 1e-8): The epsilon value prevents division by zero in the optimizer. weight_decay (`float`, defaults to 1e-2): The weight decay value for the optimizer. amsgrad (`bool`, defaults to `False`): Whether to use the [AMSGrad](https://hf.co/papers/1904.09237) variant of Adam that uses the maximum of past squared gradients instead. optim_bits (`int`, defaults to 32): The number of bits of the optimizer state. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. is_paged (`bool`, defaults to `False`): Whether the optimizer is a paged optimizer or not. r r r Nrrrr!r"rr#zAdamW32bit.__init__r$r%r!r!rr"r.r*r.c2eZdZ         d fd d ZZS) PagedAdamWrrrrFr Nr r Tc s(tjd|||||||| | | dd dS)a Paged AdamW optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-3): The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)): The beta values are the decay rates of the first and second-order moment of the optimizer. eps (`float`, defaults to 1e-8): The epsilon value prevents division by zero in the optimizer. weight_decay (`float`, defaults to 1e-2): The weight decay value for the optimizer. amsgrad (`bool`, defaults to `False`): Whether to use the [AMSGrad](https://hf.co/papers/1904.09237) variant of Adam that uses the maximum of past squared gradients instead. optim_bits (`int`, defaults to 32): The number of bits of the optimizer state. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. is_paged (`bool`, defaults to `False`): Whether the optimizer is a paged optimizer or not. r Tr Nr rrrrrrrrrrrrrr!r"r+ zPagedAdamW.__init__ rrrrFr Nr r Tr%r!r!rr"r0r0cr/)PagedAdamW8bitrrrrFr Nr r Tc (tjd|||||d|| | | dd dS)a Paged 8-bit AdamW optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-3): The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)): The beta values are the decay rates of the first and second-order moment of the optimizer. eps (`float`, defaults to 1e-8): The epsilon value prevents division by zero in the optimizer. weight_decay (`float`, defaults to 1e-2): The weight decay value for the optimizer. amsgrad (`bool`, defaults to `False`): Whether to use the [AMSGrad](https://hf.co/papers/1904.09237) variant of Adam that uses the maximum of past squared gradients instead. optim_bits (`int`, defaults to 32): The number of bits of the optimizer state. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. is_paged (`bool`, defaults to `False`): Whether the optimizer is a paged optimizer or not. r r-Tr Nrr1rr!r"rr2zPagedAdamW8bit.__init__r3r%r!r!rr"r5r4r5cr/)PagedAdamW32bitrrrrFr Nr r Tc r6)a Paged 32-bit AdamW optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-3): The learning rate. betas (`tuple(float, float)`, defaults to (0.9, 0.999)): The beta values are the decay rates of the first and second-order moment of the optimizer. eps (`float`, defaults to 1e-8): The epsilon value prevents division by zero in the optimizer. weight_decay (`float`, defaults to 1e-2): The weight decay value for the optimizer. amsgrad (`bool`, defaults to `False`): Whether to use the [AMSGrad](https://hf.co/papers/1904.09237) variant of Adam that uses the maximum of past squared gradients instead. optim_bits (`int`, defaults to 32): The number of bits of the optimizer state. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. is_paged (`bool`, defaults to `False`): Whether the optimizer is a paged optimizer or not. r r Tr Nrr1rr!r"r8r2zPagedAdamW32bit.__init__r3r%r!r!rr"r77r4r7N)bitsandbytes.optim.optimizerrrr+r.r0r5r7r!r!r!r"s ===<<