o oi@s@ddlmZGdddeZGdddeZGdddeZdS) )Optimizer1Statec2eZdZ          d fd d ZZS) Adagrad{Gz?r绽|= NdTc sd|ks td|d|kstd|d|ks!td||dkr)td|dkr1tdtd||d||||| | | d S) a Base Adagrad optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-2): The learning rate. lr_decay (`int`, defaults to 0): The learning rate decay. weight_decay (`float`, defaults to 0.0): The weight decay value for the optimizer. initial_accumulator_value (`int`, defaults to 0): The initial momemtum values. eps (`float`, defaults to 1e-10): The epsilon value prevents division by zero in the optimizer. optim_bits (`int`, defaults to 32): The number of bits of the optimizer state. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. Invalid learning rate: Invalid weight_decay value: Invalid epsilon value: /Initial accumulator value != 0.0 not supported!Lr Decay != 0.0 not supported!adagradr r N ValueErrorsuper__init__ selfparamslrlr_decay weight_decayinitial_accumulator_valueeps optim_bitsargs min_8bit_sizepercentile_clipping block_wise __class__N/home/ubuntu/.local/lib/python3.10/site-packages/bitsandbytes/optim/adagrad.pyr .)zAdagrad.__init__ rrrrrrNrr T__name__ __module__ __qualname__r __classcell__r%r%r#r&rrcr) Adagrad8bitrrrNrr Tc sd|ks td|d|kstd|d|ks!td||dkr)td|dkr1td| s5Jtd||d||d || | | d S) a 8-bit Adagrad optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-2): The learning rate. lr_decay (`int`, defaults to 0): The learning rate decay. weight_decay (`float`, defaults to 0.0): The weight decay value for the optimizer. initial_accumulator_value (`int`, defaults to 0): The initial momemtum values. eps (`float`, defaults to 1e-10): The epsilon value prevents division by zero in the optimizer. optim_bits (`int`, defaults to 8): The number of bits of the optimizer state. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. r r r r rrrrr0Nrrr#r%r&rLs0)zAdagrad8bit.__init__) rrrrrr0Nrr Tr)r%r%r#r&r/Kr.r/cr) Adagrad32bitrrrrNrr Tc sd|ks td|d|kstd|d|ks!td||dkr)td|dkr1tdtd||d||d || | | d S) a 32-bit Adagrad optimizer. Arguments: params (`torch.tensor`): The input parameters to optimize. lr (`float`, defaults to 1e-2): The learning rate. lr_decay (`int`, defaults to 0): The learning rate decay. weight_decay (`float`, defaults to 0.0): The weight decay value for the optimizer. initial_accumulator_value (`int`, defaults to 0): The initial momemtum values. eps (`float`, defaults to 1e-10): The epsilon value prevents division by zero in the optimizer. optim_bits (`int`, defaults to 32): The number of bits of the optimizer state. args (`object`, defaults to `None`): An object with additional arguments. min_8bit_size (`int`, defaults to 4096): The minimum number of elements of the parameter tensors for 8-bit optimization. percentile_clipping (`int`, defaults to 100): Adapts clipping threshold automatically by tracking the last 100 gradient norms and clipping the gradient at a certain percentile to improve stability. block_wise (`bool`, defaults to `True`): Whether to independently quantize each block of tensors to reduce outlier effects and improve stability. r r r r rrrrrNrrr#r%r&rr'zAdagrad32bit.__init__r(r)r%r%r#r&r1r.r1N)bitsandbytes.optim.optimizerrrr/r1r%r%r%r&s CD