o }oi6@sddlmZmZddlmZmZmZmZddlZddl m Z ddlm Z ddl m Z mZddlmZddlmZmZdd lmZmZdd lmZGd d d e jZGd ddeZGdddeZeGdddeeZdS)) dataclassfield)ListLiteralOptionalTupleN)ShardedStateDict)nn) LinearAdapter LoRALinear) ModuleMatcher)"get_adapter_attributes_from_linearis_expert_linear)PEFTAdapterWrapper)loggingc @sDeZdZdZ   d dedeeeeefdeedd fd d Z dS) ModuleDictzR nn.ModuleDict with a sharded_state_dict implementation for checkpointing Nprefixsharded_offsetsmetadatareturnrcCs8i}|D]\}}||||d||q|S)aRetrieve the sharded state dictionary of the wrapped module and adapter. This method is used for distributed checkpointing, combining the sharded states of both the main module and the adapter. Args: prefix (str): A prefix added to parameter and buffer names. Defaults to ''. sharded_offsets (Tuple[Tuple[int, int, int]]): Offsets for sharded parameters. Defaults to an empty tuple. metadata (Optional[dict]): Additional metadata for the sharded state. Defaults to None. Returns: ShardedStateDict: The combined sharded state dictionary. .)itemsupdatesharded_state_dict)selfrrrrkeylayerrr\/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/llm/peft/canonical_lora.pyr"s zModuleDict.sharded_state_dict)rrN) __name__ __module__ __qualname____doc__strrintrdictrrrrr rsrc@eZdZdZddZdS)LoRALinearSplitQKVaAn adapter wrapper for `linear_qkv` where q, k, v are three separate adapters. This module that adds the output of the adapters to the output of the wrapped module while taking care of shape. This class is designed to be used with LoRA (Low-Rank Adaptation) and similar techniques where the adapter's output is added to the main module's output. It extends the AdapterWrapper class to provide a specific implementation of the forward method. c Cs||\}}}|j|}|j|}|j|}||jd|jdd|jjj }||jd|jdd|jjj } ||jd|jdd|jjj } t j || | gdd} | | jd| jdd} || |j|fS)Nrdim) base_linear_forwardadapter adapter_q adapter_k adapter_vreshapeshapeto_wrapconfig kv_channelstorchcat) rx linear_outputbiaslayernorm_outputqueryrvaluequery_4dkey_4dvalue_4dqkv_4dadapter_outputrrr forwardFs   """zLoRALinearSplitQKV.forwardNr!r"r#r$rFrrrr r)= r)c@r()LoRALinearSplitFC1UpGateaAn adapter wrapper for `linear_fc1` where up_proj and gate_proj are two separate adapters. This module that adds the output of the adapters to the output of the wrapped module while taking care of shape. This class is designed to be used with LoRA (Low-Rank Adaptation) and similar techniques where the adapter's output is added to the main module's output. It extends the AdapterWrapper class to provide a specific implementation of the forward method. cCsN||\}}}|j|}|j|}tj||gdd}|||j|fS)Nr,r-)r/r0 adapter_gate adapter_upr9r:r4r5)rr;r<r=r>adapter_output_gateadapter_output_uprErrr rF`s   z LoRALinearSplitFC1UpGate.forwardNrGrrrr rIWrHrIc@seZdZUdZedddZeeed<dZ e ed<dZ e ed<d Z e ed <d Zed ed <dZeed<dZeed<ddZddejfddZdS) CanonicalLoRAa* Implements the LoRA (Low-Rank Adaptation) module for parameter-efficient fine-tuning. Canonical LoRA applies LoRA on Q, K, V projection matrices separately, as well as Up and Gate projection matrices separately. This follows more closely with Huggingface's implementation of LoRA. Args: target_modules (List[str], optional): A list of module names to apply LoRA to. Defaults to all linear layers ['linear_q', 'linear_k', 'linear_v', 'linear_proj', 'linear_fc1_up', 'linear_fc1_gate', 'linear_fc2']. - 'linear_q', 'linear_k', 'linear_v': Apply LoRA to the linear layer used for query, key, and value projections in self-attention. This is fused into one matrix in NeMo LoRA, but left as three separate matrices in Canonical LoRA. - 'linear_proj': Apply LoRA to the linear layer used for projecting the output of self-attention. - 'linear_fc1_up', 'linear_fc1_proj': Apply LoRA to the Up proj and Gate proj layers. These two together constitute the first fully-connected layer in MLP in NeMo LoRA. - 'linear_fc2': Apply LoRA to the second fully-connected layer in MLP. Target modules can also contain wildcards. For example, you can specify target_modules=['*.layers.0.*.linear_q', '*.layers.1.*.linear_q'] to add LoRA to only linear_q on the first two layers. exclude_modules (List[str], optional): A list of module names not to apply LoRa to. It will match all nn.Linear & nn.Linear-adjacent modules whose name does not match any string in exclude_modules. If used, will require target_modules to be empty list or None. dim (int): Dimension of the low-rank projection space. Defaults to 32. alpha (int): Weighting factor for the low-rank projection. Defaults to 32. dropout (float): Dropout rate for the low-rank projection. Defaults to 0.0. dropout_position (Literal['pre', 'post'], optional): Position for applying dropout. Can be 'pre' (before the low-rank projection) or 'post' (after). Defaults to 'pre'. Example: -------- >>> from nemo.collections import llm >>> lora = llm.peft.CanonicalLoRA(target_modules=['linear_q', 'linear_k', 'linear_v', 'linear_fc1_up'], dim=32) >>> model = llm.Mistral7BModel(model_transform=lora) >>> # (set up trainer and data) >>> trainer.fit(model, data) References: ----------- Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv preprint arXiv:2106.09685. https://arxiv.org/abs/2106.09685 ) cCsgdS)N)linear_qlinear_klinear_v linear_proj linear_fc1_uplinear_fc1_gate linear_fc2rrrrr szCanonicalLoRA.)default_factorytarget_modules r.alphagdropoutpre)r\postdropout_positionxavierlora_A_init_methodzerolora_B_init_methodcCs|jD]q}|drJd|drJdd|vr(|j|dddqd|vr9|j|dddqd|vrJ|j|dddqd|vr[|j|dddqd |vrl|j|d dd q|j||qd S) a Construct a mapping from the target module as supported in LoRA() to the specific parts of the layer for which adapter is applied. For example, if user specifies target_module = ['linear_q', 'linear_k', 'linear_proj', 'linear_fc1_up'], then canonical_lora_mapping = { "linear_qkv": {'linear_q', 'linear_k'}, "linear_proj": {'linear_proj'}, # the value of this key does not matter "linear_fc1": {'linear_fc1_up'}, } If user specifies target_module = ['*.layers.0.*.linear_q', '*.layers.1.*.linear_q'], then canonical_lora_mapping = { "'*.layers.0.*.linear_qkv'": {'linear_q'}, "'*.layers.1.*.linear_qkv'": {'linear_q'}, } linear_qkvzCanonical LoRA does not support target 'linear_qkv'. Either use 'linear_qkv' with LoRA() or use ['linear_q', 'linear_k', 'linear_v'] with Canonical LoRA linear_fc1zCanonical LoRA does not support target 'linear_fc1'. Either use 'linear_fc1' with LoRA() or use ['linear_fc1_up', 'linear_fc1_gate'] with Canonical LoRArOrPrQrSrTN)rXendswithcanonical_mappingreplaceadd)rtargetrrr __post_init__s&   zCanonicalLoRA.__post_init__NmcCsddlm}||||}dur|\}}t|tjr(t||j|j|j |j dSt |\}} } } } t |j|dd|j |j d||j |jt|dd|jt|| | d} |d vrj|| | fi| }td |t||S|j|}td |d |d |d krd\}}}|jj|jj}d|vr|| | fi| }d|vr|| |fi| }d|vr|| |fi| }t|||d}t||S|dkrd\}}d|vr|| | dfi| }d|vr|| | dfi| }t||d}t||S|S)a Applies LoRA to a specific module within the model architecture. Args: m (nn.Module): The module to apply LoRA to. name (str, optional): Name of the module (if applicable). Defaults to None. prefix (str, optional): Prefix for the module name (if applicable). Defaults to None. Returns: nn.Module: The modified module with LoRA applied, or the original module if not a target. r)ParallelLinearAdapterN)r.rZr[r`identityFr7)r.base_linear_name activation norm_typecolumn_init_methodrow_init_method gather_outputinput_is_parallelr[r^model_parallel_configrZ is_expertdisable_sequence_parallel_commbase_linear_is_parallel)rRrUzAdding lora to: z ()rc)NNNrOrPrQ)r1r2r3rdNNrSr,rT)rKrJ)nemo.collections.llm.peft.utilsrlmatch isinstancer Linearr r.rZr[r`r r'rbr^getattrrrinfor rfr7r8num_query_groupsrr)rI)rrknamerrlansr| full_namert in_features out_featuresdisable_sp_commrxadapter_kwargsr0canonical_submodulesr1r2r3kv_out_featuresadaptersrKrJrrr transformsh         zCanonicalLoRA.transformrz)r!r"r#r$rrXrr%__annotations__r.r&rZr[floatr^rr`rbrjr Modulerrrrr rNis -    *rN) dataclassesrrtypingrrrrr9(megatron.core.dist_checkpointing.mappingrr nemo.collections.llm.peft.lorar r (nemo.collections.llm.peft.module_matcherr r{r r%nemo.lightning.pytorch.callbacks.peftrr nemo.utilsrrr)rIrNrrrr s