o wi@sddlmZddlmZmZddlmZmZmZddl m Z ddl m Z m Z ddlm Z ddlmZddlmZed d \ZZed d \ZZed d \ZZeeeefZeGd ddZdS)) defaultdict) dataclassfield)DictListSetN)ColumnParallelLinearRowParallelLinear)nnwildcard_match)safe_import_fromz+megatron.core.extensions.transformer_engineTEColumnParallelLinearTELayerNormColumnParallelLinearTERowParallelLinearc@sreZdZUdZedddZeeed<ee dZ eeed<edddZ e ee fed<d d ejfd d Zd S) ModuleMatchera Implements the LoRA (Low-Rank Adaptation) module for parameter-efficient fine-tuning. LoRA uses a low-rank projection to adapt the weights of a pre-trained model to a new downstream task. This class facilitates the application of LoRA to specific modules within the model architecture. Args: target_modules (List[str], optional): A list of module names to apply LoRA to. Defaults to all linear layers ['linear_qkv', 'linear_proj', 'linear_fc1', 'linear_fc2']. - 'linear_qkv': Apply LoRA to the fused linear layer used for query, key, and value projections in self-attention. - 'linear_proj': Apply LoRA to the linear layer used for projecting the output of self-attention. - 'linear_fc1': Apply LoRA to the first fully-connected layer in MLP. - 'linear_fc2': Apply LoRA to the second fully-connected layer in MLP. Target modules can also contain wildcards. For example, you can specify target_modules=['*.layers.0.*.linear_qkv', '*.layers.1.*.linear_qkv'] to add LoRA to only linear_qkv on the first two layers. cCsgdS)N) linear_qkv linear_proj linear_fc1 linear_fc2rrre/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/llm/peft/module_matcher.py=zModuleMatcher.)default_factorytarget_modulesexclude_modulescCsttSN)rsetrrrrr@rcanonical_mappingNmcs2|r |d|n|t|jpgdkr5 t|jdksJ|jD]}||ks,t|r2|fSq!dSt|jp:gdkr^t|jdksGJ|jD]}||ksUt|r[|fSqJdStttjg}t rk| t t rr| t try| tt|}||jvrtfdd|jDst||r|fSdS)a Determines whether a given module matches specified target patterns. This function checks if the provided module `m` should be included based on predefined mapping rules (`canonical_mapping`, `target_modules`, and `exclude_modules`). It returns the matching pattern if a match is found; otherwise, it returns `None`. Args: m (nn.Module): The module being checked. name (str, optional): The module's name. prefix (str, optional): A prefix to be used in constructing `full_name`. Returns: str or None: The matching pattern if a match is found; otherwise, `None`. Matching Logic: 1) If `canonical_mapping` is defined, it checks: - Whether `name` exactly matches a pattern. - Whether `full_name` matches any regex pattern in `canonical_mapping`. 2) If `target_modules` is defined, it follows the same logic as `canonical_mapping`. 3) If neither `canonical_mapping` nor `target_modules` are defined, it ensures: - `name` is not in `exclude_modules`. - `full_name` does not match any `target_modules` patterns. - `m` is an instance of `nn.Linear`. Notes: - `exclude_modules` should only be non-empty if neither `canonical_mapping` nor `target_modules` are set. - The function asserts that `exclude_modules` is empty when using `canonical_mapping` or `target_modules`. .rc3s|]}t|VqdSrr ).0pattern full_namerr ~sz&ModuleMatcher.match..N)lenrrr rrr r LinearHAVE_TE_COL_LINEARappendrHAVE_TE_LN_COL_LINEARrHAVE_TE_ROW_LINEARrtupleany isinstance)selfr nameprefixr# linear_typesrr$rmatchBs>         zModuleMatcher.match)NN)__name__ __module__ __qualname____doc__rrrstr__annotations__listrrrrr Moduler4rrrrr's r) collectionsr dataclassesrrtypingrrrtorch.nnr megatron.core.tensor_parallelrr torchnemo.collections.llm.peft.utilsr nemo.utils.import_utilsr rr)rr+rr,allHAVE_TErrrrrs*