o Ni,@sdZddlmZddlZddlmZddlmZddlZddl m Z ddl m Z ddlm Z ddlmZd d lmZd d lmZGd d d eZ ddddZdS) zC This module contains the implementation of the LoRA-FA optimizer. ) annotationsN)Iterable)Callable)is_bf16_available)autocast) Optimizer) PeftModel) infer_devicecsBeZdZdZ     ddfdd ZeddddZZS)LoraFAOptimizera; Implements the LoRA-FA optimizer designed specifically for training Low-Rank Adaptation (LoRA) parameters efficiently. Note that LoraFAOptimizer is based on adamw-hf in transformers, with only LoRA part modified. Without LoRA it will fall back to adamw-hf. Args: params (Iterable[nn.parameter.Parameter]): Parameters to optimize. lr (float, optional): Learning rate (default: 1e-3). betas (Tuple[float, float], optional): Coefficients for computing running averages of gradient and squared gradient (default: (0.9, 0.999)). eps (float, optional): Term added to denominator to improve numerical stability (default: 1e-6). weight_decay (float, optional): Weight decay (L2 penalty) (default: 0.0). correct_bias (bool, optional): Whether to apply bias correction as in original Adam (default: True). Args in sub-function step: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Reference: - LoRA-FA: https://huggingface.co/papers/2308.03303 MbP?g?g+?ư>Tparams Iterable[nn.parameter.Parameter]lrfloatbetastuple[float, float]eps weight_decay correct_biasboolcs|dkr td|dd|dkrdks#ntd|ddd|dkr/dks:ntd|ddd|ksFtd |d|||||d }t||dS) NrzInvalid learning rate: z - should be >= 0.0r?zInvalid beta parameter: z - should be in [0.0, 1.0)zInvalid epsilon value: )rrrrr) ValueErrorsuper__init__)selfrrrrrrdefaults __class__J/home/ubuntu/.local/lib/python3.10/site-packages/peft/optimizers/lorafa.pyr9s zLoraFAOptimizer.__init__Nclosurerc Csd}|dur |}|jD]}|d}g}g}t|d|dD]\}}d|vr/|jdur/q |j} d|vrZ||||t|dkrR|d|dd} n t|dkrYq n|} |j| } t| dkrt|dkrd| d <t|d| d <t|d| d <nd| d <t|| d <t|| d <t|dkr|d} |d} | j}d}| | j }tj ||t | j d| j}t}trt|tjdd|d||}Wdn1swYn d|d||}|j| jjkr|| jj}| d | d }}|d\}}| d d7<||j|d|d||j||d|d||d}|d}|dr_d|| d }d|| d }|t||}| j||| d|ddkr}| j| |d |ddg}g}q | d | d }}|d\}}| d d7<||j| d|d||j| | d|d||d}|d}|drd|| d }d|| d }|t||}|j||| d|ddkr|j||d |ddq q |S)z Performs a single optimization step. Arguments: closure (`Callable`, *optional*): A closure that reevaluates the model and returns the loss. Nscaling_factorrnameslorarrrstep exp_avg_B exp_avg_sq_Bexp_avg exp_avg_sqg:0yE>) device_typedtyperr)alpha)valuerrrrr) param_groupszipgradappendlenfindstatetorch zeros_likeTlinalgpinveyeshapetodevicer rrbfloat16r/mul_add_addcmul_sqrtmathaddcdiv_)rr%lossgroupr& param_list name_listpnr4namer8AB grad_B_orindeltaAA_TAA_T_invr.grad_Br*r+beta1beta2denom_B step_sizebias_correction1bias_correction2r,r-denomr#r#r$r)Ss         (     qzLoraFAOptimizer.step)r r rrT) rrrrrrrrrrrr)N)r%r) __name__ __module__ __qualname____doc__rr9no_gradr) __classcell__r#r#r!r$r #sr rFmodelr rint lora_alpharrr use_rslorarreturnrc Csl|D] \}}d|vr|dq|r|t|n||}||dd|D|d|dg} t| S)a+ Helper function to instantiate a lorafa optimizer specifically configured for a given model using the LoRA method. This function will: - Disable gradient updates for the "lora_A" parameters (these are typically frozen during LoRA training). - Compute the scaling factor based on provided `lora_alpha` and rank `r` for proper gradient projection. - Create and configure parameter groups for the optimizer including specified learning rate, weight decay, and additional optimizer options. For hyper-params, LoRA-FA uses the same hyper-params as AdamW, except for the LoRA hyper-params (r, lora_alpha, use_rslora). One can always use the same hyper-params such as lr and weight_decay, as AdamW in LoRA tuning. Args: model (PeftModel): The model containing LoRA-adapted parameters. r (int): Rank of the LoRA decomposition. lora_alpha (int): Scaling factor for LoRA parameterization. lr (float): Learning rate for optimizer updates. weight_decay (float): Weight decay for AdamW. use_rslora (bool): whether to use rslora. In rslora, the lora scaling factor becomes to lora_alpha / math.sqrt(r) instead of lora_alpha / r. Returns: Optimizer: Configured lorafa optimizer instance ready for training. lora_AFcSsg|]\}}|qSr#r#).0rO_r#r#r$ sz+create_lorafa_optimizer..r )rrr'r&rr)named_parametersrequires_grad_rGrF parametersr ) rdrergrrrhrOparam lora_scalingr2r#r#r$create_lorafa_optimizers  rs)rF)rdr rerfrgrfrrrrrhrrir)ra __future__rrGcollections.abcrtypingrr9torch.nnnnaccelerate.utils.importsrr torch.optimr peft_modelr utils.otherr r rsr#r#r#r$s          6