o wiu"@sddlmZmZddlmZmZmZddlZddlm Z ddl m Z ddl m Z mZddlmZddlmZdd lmZmZdd lmZmZdd lmZGd d d eZGdddeZeGdddeeZdS)) dataclassfield)ListLiteralOptionalN)ShardedStateDict)(gather_from_tensor_model_parallel_region)"make_sharded_tensor_for_checkpoint%make_tp_sharded_tensor_for_checkpoint)nn) ModuleMatcher)ParallelLinearAdapter"get_adapter_attributes_from_linear)PEFTAdapterWrapper)loggingc sJeZdZdZddZddZ dd ed ed ee d e ffd d Z Z S)ParallelLinearDoRAAdapterzT Adapter class for DoRA to handle the additional weight_magnitude parameter cCstj|dd|_dS)zl Initialize weight_magnitude with shape (d,), where d is the output dim of the linear layer T) requires_gradN)r Parameterweight_magnitude)selfvaluer[/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/llm/peft/dora.pyinit_weight_magnitude#sz/ParallelLinearDoRAAdapter.init_weight_magnitudecCs|jS)zG Public function to get the weight magnitude parameter )rrrrrget_weight_magnitude)sz.ParallelLinearDoRAAdapter.get_weight_magnituderNprefixsharded_offsetsmetadatareturncsPt|||}|d}|jrt|j||d}n t|j|d|d}|||<|S)z Sharded state dict implementation for DoRA adapter. Weight magnitude is TP sharded for linear_qkv and linear_fc1 only. r)prepend_offsetsr)supersharded_state_dictinput_is_parallelr rr )rrrr r$ magnitude_keymagnitude_sharded_tensor __class__rrr$/s  z,ParallelLinearDoRAAdapter.sharded_state_dict)rrN) __name__ __module__ __qualname____doc__rrstrtuplerdictrr$ __classcell__rrr(rrsrcs<eZdZdZdejdeffdd ZddZdd Z Z S) DoRALinearz An adapter wrapper that is designed to be used with DoRA It extends the AdapterWrapper class to provide a specific implementation of the forward method. to_wrapadaptercs4t||||j|j|_|j|dS)N)r#__init__alphadimscalingr4r_get_weight_norm)rr3r4r(rrr5NszDoRALinear.__init__cCst|jjrt|jjjjj}|jjj}n|jjj}t|jjjjj}|jj|j||}t j j |dd |j S)N)r7)r4r%r linear_outweightT linear_inr3r8torchlinalgnormtodtypedetach)rlinear_out_weightlinear_in_weightr<rrrr9Ts  zDoRALinear._get_weight_normcCs||\}}}||}|j|ddd}|jjdus&|js)d}n|d||j||d}||||j ||fS)a* Forward method for DoRA mag_norm_scale * (linear_output + adapter_output) = ||W_0 + B_0 A_0|| / ||W_0 + B A|| * (W_0 x + B A x) = ||W_0 + B_0 A_0|| ((W_0 + B A) / ||W_0 + B A||) x = m ((W_0 + B A) / ||W_0 + B A||) x = equation 5 in DoRA paper When dropout is used, equation becomes W_0 x + (m /||W_0 + B A|| - 1) W_0 dropout(x) + m /||W_0 + B A|| B A dropout(x) = ... = m /||W_0 + B A|| (W_0 x + B A dropout(x)) + (m /||W_0 + B A|| - 1) W_0 (dropout(x) - x) r:Nr) base_linear_forwardr4 contiguousrr9viewdropouttrainingreshapeshape)rx linear_outputbiaslayernorm_outputadapter_outputmag_norm_scaledropout_correctionrrrforward_s zDoRALinear.forward) r*r+r,r-r Modulerr5r9rVr1rrr(rr2Hs  r2c@seZdZUdZedddZeeed<dZ e ed<dZ e ed <d Z e ed <d Zed ed<dZeed<dZeed<ddZddejfddZdS)DoRAaK Implements the DoRA (Weight-Decomposed LowRank Adaptation) module for parameter-efficient fine-tuning. DoRA decomposes pre-trained weight into magnitude and direction, and uses a low-rank projection in the directional component to adapt the weights of a pre-trained model to a new downstream task. This class facilitates the application of DoRA to specific modules within the model architecture. Args: See LoRA class for a detailed explanation of the arguments. Example: -------- >>> from nemo.collections import llm >>> lora = llm.peft.DoRA(target_modules=['linear_qkv', 'linear_proj'], dim=32, alpha=64) >>> model = llm.Mistral7BModel(model_transform=lora) >>> # (set up trainer and data) >>> trainer.fit(model, data) References: ----------- Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Min-Hung Chen (2024). DoRA: Weight-Decomposed Low-Rank Adaptation. arXiv preprint arXiv:2402.09353. https://arxiv.org/abs/2402.09353 ) cCsgdS)N) linear_qkv linear_proj linear_fc1 linear_fc2rrrrrsz DoRA.)default_factorytarget_modules r7@r6grKpre)rbpostdropout_positionxavierlora_A_init_methodzerolora_B_init_methodcCs|jdks JddS)Nrbz`DoRA only supports pre-adapter dropout at this time.Please set DoRA(..., dropout_position='pre'))rdrrrr __post_init__s zDoRA.__post_init__Nmc Cs||||}durC|\}}t|\}}} } } td|t|| |j|dd|j|jd||j|j t |dd|j | | d} t || S|S)a Applies DoRA to a specific module within the model architecture. Args: m (nn.Module): The module to apply DoRA to. name (str, optional): Name of the module (if applicable). Defaults to None. prefix (str, optional): Prefix for the module name (if applicable). Defaults to None. Returns: nn.Module: The modified module with DoRA applied, or the original module if not a target. NzAdding DoRA to: identityFconfig) base_linear_name activation norm_typecolumn_init_methodrow_init_method gather_outputr%rKrdmodel_parallel_configr6disable_sequence_parallel_commbase_linear_is_parallel) matchrrinforr7rfrhrKrdgetattrr6r2) rrjnameransrv full_namer% in_features out_featuresdisable_sp_commrur4rrr transforms2    zDoRA.transform)NN)r*r+r,r-rr_rr.__annotations__r7intr6rKfloatrdrrfrhrir rWrrrrrrXs      rX) dataclassesrrtypingrrrr?(megatron.core.dist_checkpointing.mappingrmegatron.core.tensor_parallelrmegatron.core.utilsr r r (nemo.collections.llm.peft.module_matcherr nemo.collections.llm.peft.utilsr r%nemo.lightning.pytorch.callbacks.peftrr nemo.utilsrrr2rXrrrrs     *9