o wiX@sDddlZddlmZmZddlmZmZddlZddlm Z ej r*e d\Z Z ndZ dZ ddlmmZddlmZddlmZed d \ZZdd lmZdd lmZmZdd lmZmZddlmZGdddeZeryGdddej Z!Gdddej Z"      d ddZ#eGdddeeZ$GdddeZ%dS)!N) dataclassfield)ListLiteral) safe_import bitsandbytesF)nn)safe_import_fromtransformer_enginepytorch) ModuleMatcher)"get_adapter_attributes_from_linearis_expert_linear)PEFTAdapterWrapper)loggingc@seZdZdZddZdS) LoRALinearagAn adapter wrapper that adds the output of the adapter to the output of the wrapped module. This class is designed to be used with LoRA (Low-Rank Adaptation) and similar techniques where the adapter's output is added to the main module's output. It extends the AdapterWrapper class to provide a specific implementation of the forward method. cOsD|j|g|Ri|\}}}||}||j}|||fS)N)base_linear_forwardadapter contiguousreshapeshape)selfxargskwargs linear_outputbiaslayernorm_outputadapter_outputr [/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/llm/peft/lora.pyforward2s  zLoRALinear.forwardN)__name__ __module__ __qualname____doc__r"r r r r!r*s rcsZeZdZdZ      dfdd Zeje      dd d Zfd d Z Z S)TELinearAdaptera TELinear + LoRA, maintains ckpts structrue (i.e. Linear's weight/bias remain at the same FQN) The _init_wrapper and _forward methods provide the LoRA functionality. We want to be able to use those inside LinearAdapter but also for monkey-patching modules, without repeating the same code -> therefore those are decorated with @staticmethod. Args: orig_linear (nn.Module): the linear module to augment. dim (int): lora's dim in_features -> dim -> out_features. alpha (int): lora's scaling alpha. dropout (float): dropout prob (default: 0.0). dropout_position (str): where to apply dropout rel. to lora (choices= ['pre', 'post'], default=post) lora_A_init_method (str): init method for lora_A (choices= ['xavier', 'uniform']) lora_dtype (torch.dtype): weight's dtype, by default will use orig_linear's but if they are quantized weights (e.g. 4bit) needs to be specified explicitly.  postxavierNc s|jtjksJ|jduo|jjddk}tt|j|j|j ||j j |j j d|j j |j j |r<|jj |jj tj|||||||ddS)Nr) in_features out_featuresrdevice params_dtypedimalphadropoutdropout_positionlora_A_init_method lora_dtype) __class__teLinearrrsuperr'__init__r-r.weightr/dtypedatacopy_ _init_adapter) r orig_linearr2r3r4r5r6r7has_biasr8r r!r<Os*   zTELinearAdapter.__init__c C||_|||_|jj}d|j_|jdurd|j_|j}|j} |p$|jj} t j ||d| |d|_ t j || d| |d|_ |dkrJt j j|j jjnt jj|j jjtdd|j jjdt j|d|_|d vsoJ|||_dS) a%Adds LoRA weights to obj. The obj is either a LinearAdapter or an nn.Module (when monkey-patching). Args: obj (LinearAdapter | nn.Module): input module to adapt. dim (int): lora's dim in_features -> dim -> out_features. alpha (int): lora's scaling alpha. dropout (float): dropout prob (default: 0.0). dropout_position (str): where to apply dropout rel. to lora (choices= ['pre', 'post'], default=post) lora_A_init_method (str): init method for lora_A (choices= ['xavier', 'uniform']) lora_dtype (torch.dtype): weight's dtype, by default will use orig_linear's but if they are quantized weights (e.g. 4bit) needs to be specified explicitly. FNrr>r/r,arpprer+r2scaler=r/ requires_gradrr-r.r>rr:lora_alora_btorchinituniform_r?kaiming_uniform_mathsqrtfill_Dropoutr4r5 objr2r3r4r5r6r7r/r-r.r>r r r!rAr$    zTELinearAdapter._init_adaptercsZtt||}|jdkr||}|||}||j}|jdkr)||}||S)NrMr+)r;r'r"r5r4rRrQrO)rrreslora_resrDr r!r"s     zTELinearAdapter.forwardr(r)r*r+r,N r#r$r%r&r<rSno_grad staticmethodrAr" __classcell__r r rDr!r'<s&#.r'csVeZdZdZ      dfdd Zeje      dd d Zd d Z Z S) LinearAdaptera Linear + LoRA, maintains ckpts structrue (i.e. Linear's weight/bias remain at the same FQN) The _init_wrapper and _forward methods provide the LoRA functionality. We want to be able to use those inside LinearAdapter but also for monkey-patching modules, without repeating the same code -> therefore those are decorated with @staticmethod. Args: orig_linear (nn.Module): the linear module to augment. dim (int): lora's dim in_features -> dim -> out_features. alpha (int): lora's scaling alpha. dropout (float): dropout prob (default: 0.0). dropout_position (str): where to apply dropout rel. to lora (choices= ['pre', 'post'], default=post) lora_A_init_method (str): init method for lora_A (choices= ['xavier', 'uniform']) lora_dtype (torch.dtype): weight's dtype, by default will use orig_linear's but if they are quantized weights (e.g. 4bit) needs to be specified explicitly. r(r)r*r+r,Nc st|tjsJtt|j|j|j|jdu|j j |j j d|j j |j j |jdur5|jj |jj tj|||||||ddS)N)r-r.rr/r>r1) isinstancerr:r;rer<r-r.rr=r/r>r?r@rA)rrBr2r3r4r5r6r7rDr r!r<s(   zLinearAdapter.__init__c CrE) aAdds LoRA weights to obj. The obj is either a LinearAdapter or an nn.Module (when monkey-patching). Args: obj (LinearAdapter | nn.Module): input module to adapt. dim (int): lora's dim in_features -> dim -> out_features. alpha (int): lora's scaling alpha. dropout (float): dropout prob (default: 0.0). dropout_position (str): where to apply dropout rel. to lora (choices= ['pre', 'post'], default=post) lora_A_init_method (str): init method for lora_A (choices= ['xavier', 'uniform']) lora_dtype (torch.dtype): weight's dtype, by default will use orig_linear's but if they are quantized weights (e.g. 4bit) needs to be specified explicitly. FNrFr,rGrHrrJrLrNr[r r r!rAr]zLinearAdapter._init_adaptercCst|dd}dur||jksJ||}n t||j|j}|jdkr)||}|| |}||j }|jdkr@||}||S)N super_fwdrMr+) getattrr"Flinearr=rr5r4rRrQrO)rrfwdr^r_r r r!r"s      zLinearAdapter.forwardr`rar r rDr!res&!.rer(r)r*r+r,c Cst|tjs|jtjksJt|drJ|jt|tjr5t||||||||j}t dt|fi}n!|jtjkrRt ||||||||j}t dt |fi}nt dt |dddurj|j jtjjkrj|j|_||_|S)aMonkey-patches a nn.Linear (orig_linear param) to be a LinearAdapter, for all purposes think of this function as replacing a nn.Linear with a LinearAdapter defined above. The orig_linear might not contain valid weights, for example, the given orig_linear was initialized within a context-manager that uses a "meta" device. Therefore, we cannot copy the weight/bias from the orig_linear to the LinearAdapter, since those have not been allocated, To circumvent this scenario, LinearAdapter's additional functionality (_init_adapter, _forward) is based on static functions, so that we can use them for patching or when allocating a new LinearAdapter object. Args: orig_linear (nn.Linear): the module we add adapter to. dim (int, optional): Lora dim. Defaults to 8. alpha (int, optional): Lora alpha scale. Defaults to 32. dropout (float, optional): dropout prob. Defaults to 0.0. dropout_position (str, optional): location to apply dropout wrt lora. Defaults to 'post' (choices: 'pre', 'post'). lora_A_init_method (str, optional): lora_a init method. Defaults to 'xavier'. lora_dtype (_type_, optional): Lora weights' dtype. By default will use orig_linear's dtype but orig_linear might use non-trainable dtype (e.g., 4bit), in which case the user must specify the dtype manually. Defaults to None. Returns: (nn.Module): the monkey-patched (nn.Linear + LoRA) nn.Module rgPatchedLinearAdapterPatchedTELinearAdapterz8Expected isinstance(orig_linear, (nn.Linear, te.Linear)) quant_stateN)rfrr:r8r9hasattrrgrerAtyper'NotImplementedErrorrhrnr functional QuantStater") rBr2r3r4r5r6r7clsnew_clsr r r!patch_linear_module's$$  rvc@seZdZUdZedddZeeed<dZ e ed<dZ e ed<d Z e ed <d Zed ed <dZeed<dZeed<dZeed<dZejed<dZeed<ddejfddZdS)LoRAa Implements the LoRA (Low-Rank Adaptation) module for parameter-efficient fine-tuning. LoRA uses a low-rank projection to adapt the weights of a pre-trained model to a new downstream task. This class facilitates the application of LoRA to specific modules within the model architecture. Args: target_modules (List[str], optional): A list of module names to apply LoRA to. Defaults to all linear layers ['linear_qkv', 'linear_proj', 'linear_fc1', 'linear_fc2']. - 'linear_qkv': Apply LoRA to the fused linear layer used for query, key, and value projections in self-attention. - 'linear_proj': Apply LoRA to the linear layer used for projecting the output of self-attention. - 'linear_fc1': Apply LoRA to the first fully-connected layer in MLP. - 'linear_fc2': Apply LoRA to the second fully-connected layer in MLP. Target modules can also contain wildcards. For example, you can specify target_modules=['*.layers.0.*.linear_qkv', '*.layers.1.*.linear_qkv'] to add LoRA to only linear_qkv on the first two layers. exclude_modules (List[str], optional): A list of module names not to apply LoRa to. It will match all nn.Linear & nn.Linear-adjacent modules whose name does not match any string in exclude_modules. If used, will require target_modules to be empty list or None. dim (int): Dimension of the low-rank projection space. Defaults to 32. alpha (int): Weighting factor for the low-rank projection. Defaults to 32. dropout (float): Dropout rate for the low-rank projection. Defaults to 0.0. dropout_position (Literal['pre', 'post'], optional): Position for applying dropout. Can be 'pre' (before the low-rank projection) or 'post' (after). Defaults to 'pre'. a2a_experimental (bool): Enables the experimental All-to-All (A2A) communication strategy. Defaults to False. dropout_recompute (bool): Enables dropout recompute using Thunder JIT compilation. When True, applies thunder.jit() to the dropout layer for memory-efficient training by recomputing dropout activations during backward pass instead of storing them. lora_dtype (torch.dtype): Parameter data type for LoRA weights. Default None (will use model's dtype). Example: -------- >>> from nemo.collections import llm >>> lora = llm.peft.LoRA(target_modules=['linear_qkv', 'linear_proj'], dim=32) >>> model = llm.Mistral7BModel(model_transform=lora) >>> # (set up trainer and data) >>> trainer.fit(model, data) References: ----------- Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv preprint arXiv:2106.09685. https://arxiv.org/abs/2106.09685 ) cCsgdS)N) linear_qkv linear_proj linear_fc1 linear_fc2r r r r r!sz LoRA.)default_factorytarget_modulesr)r2r3r*r4rMrLr5r,r6zerolora_B_init_methodFa2a_experimentalNr7dropout_recomputemcCspddlm}||||}dur|\}}t|tjs!|jtjkrZ|js;t |j j ds;t |dddur>|j jtjjkr>t}n trI|jtjkrIt}nt}|||j|j|j|j|jdSt|\} } } } } td||| | |jfid|d d d dd |jd |jddd| d|jd|jdt |ddd|jdt|d|j d| d|j!d| }t"||S|S)a Applies LoRA to a specific module within the model architecture. Args: m (nn.Module): The module to apply LoRA to. name (str, optional): Name of the module (if applicable). Defaults to None. prefix (str, optional): Prefix for the module name (if applicable). Defaults to None. Returns: nn.Module: The modified module with LoRA applied, or the original module if not a target. r)ParallelLinearAdapterN _local_tensorrn)r2r3r4r6r7zAdding lora to: base_linear_name activationidentity norm_typecolumn_init_methodrow_init_method gather_outputFinput_is_parallelr4r5model_parallel_configconfigr3 is_expertrdisable_sequence_parallel_commrbase_linear_is_parallel)#nemo.collections.llm.peft.utilsrmatchrfrr:r8r9_add_via_setattrror=r?rhrnrrrrsrvHAVE_TEr'rer2r3r4r6r7r rinforr5rrrr)rrnameprefixransr full_namelora_clsrr-r.disable_sp_commrrr r r! transforms         zLoRA.transformNN)r#r$r%r&rr~rstr__annotations__r2intr3r4floatr5rr6rrboolr7rSr>rrModulerr r r r!rwfs 0       rwc@s*eZdZdZeddejfddZdS) LoRAMergea Implements the LoRA weight merge for parameter-efficient fine-tuning. Example: -------- >>> from nemo.collections.llm.peft.lora import LoRAMerge >>> lora_merge = LoRAMerge() >>> merged_model = lora_merge(trainer.strategy.megatron_parallel) NrcCst|ts|Std|r|ndd|r|nd|jj}|jj|jj|jj j |j |jj j |j }||}||jj_ |S)a Merges the LoRA adapter with the base model weights. Args: m (nn.Module): The module to apply LoRA merge to. name (str, optional): Name of the module to merge. Defaults to None. prefix (str, optional): Prefix for the module name. Defaults to None. Returns: nn.Module: The modified module with the LoRA adapter merged into the base model weights. zmerging .)rfrrrto_wrapr=rr3r2 linear_outtor/ linear_inr?)rrrr base_weight lora_weight merged_weightr r r!rs ( zLoRAMerge.transformr) r#r$r%r&rSrbrrrr r r r!rs rr`)&rW dataclassesrrtypingrrrSnemo.utils.import_utilsrcuda is_availablerHAVE_BNBtorch.nn.functionalrrrrir r9r(nemo.collections.llm.peft.module_matcherr rr r%nemo.lightning.pytorch.callbacks.peftrr nemo.utilsrrr:r'rervrwrr r r r!s@      sz ?