o ̳i@sddlZddlmZmZmZddlZddlmmZ ddlmZddl m Z m Z ddl mZddlmZGdddejeZd ejd dfd d Zd ejd dfd dZdS)N)ListOptionalUnion)nn) linear_nf4to_nf4)_register_nf4_dispatch_ops) AdapterModulecseZdZdZ   ddededededed ed effd d Zd ddee e e j efdefddZ ddZe ddZddZdee fddZde jde jfddZZS) DoRALinearaQDoRA linear layer as introduced in `DoRA: Weight-Decomposed Low-Rank Adaptation of Large Language Models `_. DoRA (Weight-Decomposed Low-Rank Adaptation) fine-tunes a layer by decomposing the pre-trained weights into two components: magnitude and direction. The magnitude component is a learnable scalar vector that scales each output channel, while the direction component, modified via LoRA, adjusts the orientation of weights. By scaling the LoRA update component :math:`BAx` with the `magnitude` vector, DoRA allows the model to apply distinct scaling adjustments across different output dimensions. Args: in_dim (int): input dimension out_dim (int): output dimension rank (int): rank of the low-rank approximation alpha (float): scaling factor for the low-rank approximation dropout (float): dropout probability. Default: 0.0 use_bias (bool): whether to include bias in the original linear layer. Default: False quantize_base (bool): Whether to quantize base linear weight or not. Default: False **quantization_kwargs: Keyword arguments to pass to `to_nf4` when quantizing the base linear weight. Examples of valid arguments are `block_size` and `scaler_block_size`, which control the granularity of weight quantization and scaler quantization respectively. This is only used if `quantize_base` is True. Default None Raises: ValueError: If ``quantize_base`` is False, but quantization kwargs are provided. Fin_dimout_dimrankalphadropoutuse_bias quantize_basec  s,t||_||_|||_||_||_|js+tdd|Dr+t d|t j |||jd} |js:| j nt | j fi|} |jrI| jnd} d|_|dt | |d| durct | nd|dkrpt j|d nt |_t j ||dd|_t j ||dd|_t t||_|dS) NcSsg|]}|qSr).0vrrO/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/modules/peft/dora.py Dsz'DoRALinear.__init__..zO``quantize_base`` is False, but received the following quantization arguments: ) in_features out_featuresbiasFweightrr )p)super__init__r r scalingr_quantize_baseanyvalues ValueErrorrLinearrrrdisabledregister_parameter ParameterDropoutIdentityrlora_alora_btorchempty magnitudeinitialize_parameters) selfr r rrrrrquantization_kwargslinearrr __class__rrr2s4   zDoRALinear.__init__T)recursedevicer5cCsR|jj||d|jj||dtjtj|j|d|jjd}tj |j||S)N)r6r5)r6) requires_grad) r*to_emptyr+rr'r, empty_liker.r7utils swap_tensors)r0r6r5r.rrrr8`szDoRALinear.to_emptycCst|jt|jdS)N)_lora_a_init_paramsr*_lora_b_init_paramsr+)r0rrrr/ms z DoRALinear.initialize_parameterscCs`t|jj|jjj|jjjgrtd|j|jjj}|jj|jj}|j | ||dS)a= DoRA initializes the magnitude vector such that its outputs are initially identical to standard LoRA's outputs. This must be called after loading/initializing base model and LoRA params. Raises: RuntimeError: If base or LoRA parameters are still on meta device. zUCannot initialize DoRA magnitude if base or LoRA parameters are still on meta device.N) r!ris_metar*r+ RuntimeErrortodtyper.copy__get_weight_norm)r0 base_weight lora_weightrrrinitialize_dora_magnitudess z$DoRALinear.initialize_dora_magnitudecCs*||j|}tjj|dd|j}|S)N)dim)rr,linalgnormr@rA)r0rrE weight_normrrrrCszDoRALinear._get_weight_normreturncCs gd}|S)z Return a list of strings corresponding to the names of the ``nn.Parameter`` s in the model coming from the adapter. For DoRA this means lora_a.weight, lora_b.weight, and magnitude. )z lora_a.weightz lora_b.weightr.r)r0adapter_paramsrrrrMszDoRALinear.adapter_paramsxc Cs|jrt||jd}|jr||j}n t||j|j}|jr!|S||}| | |}t j |j jj d|j jj|jd}| | |j}|j}|j|j}|||}|}||dd} | d|| ||j} | |S)z Args: x (torch.Tensor): input tensor with shape ``(..., in_dim)`` Returns: Tensor: output tensor with shape ``(..., out_dim)`` )inputrrG)r6rA)r rrrrFr2r%rr+r*r,eyeshaper6rATr.r@rCdetachviewr) r0rNbase_outlora_outx_eyerEr.rrKmag_norm_scaledora_outrrrforwards2   zDoRALinear.forward)r FF)__name__ __module__ __qualname____doc__intfloatboolrrrstrr,r6r8r/no_gradrFrCrrMTensorr\ __classcell__rrr3rr s@#/    r rNrLcCstjj|jtdddS)z6 Initialize LoRA A weight to Kaiming uniform. )aN)rinitkaiming_uniform_rmathsqrtrNrrrr<sr<cCstj|jdS)z, Initialize LoRA B weight to zeros. N)rrjzeros_rrnrrrr=sr=)rltypingrrrr,torch.nn.functionalr functionalrQtorchao.dtypes.nf4tensorrrtorchtune.modules.low_precisionrtorchtune.modules.peftr Moduler r$r<r=rrrrs   /