o }oit@spddlZddlZddlmZmZddlmZddlmZddl m Z m Z m Z m Z mZmZddlmZddlZddlmZddlmZddlmZddlmZdd lmZdd lmZm Z m!Z!dd l"m#Z#dd l$m%Z%m&Z&dd l'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.m/Z/ddl0m1Z1ddl2m3Z3e rddl4m5Z5Gddde#ee*Z6Gdddej7Z8Gdddee3Z9dS)N)ABCabstractmethod)partial)Path) TYPE_CHECKINGAnyCallableDictOptionalTuple)_PATH)_WrappingCheckpointIO) TrainerFn)override)ADAPTER_META_FILENAMEHF_ADAPTER_CONFIG_FILENAMEHF_ADAPTER_PATH)IOMixin) ckpt_to_dirckpt_to_weights_subdir)MegatronParallel)ModelTransform)MegatronOptimizerModule)get_automodel_from_traineris_trainer_attached)logging)AsyncCompatibleCheckpointIO)ShardedStateDictcseZdZdZedddZdejdejfddZdejddfd d Z d d Z d e j de j deddffdd Zd e j ddfddZfddZddZdedefddZZS)PEFTaAbstract base class for Parameter-Efficient Fine-Tuning (PEFT) methods. This class defines the interface for PEFT methods, which are used to fine-tune large language models efficiently by modifying only a small subset of the model's parameters. Example: class MyPEFT(PEFT): def transform(self, module, name=None, prefix=None): # Implement the transform logic pass peft = MyPEFT() peft_model = LargeLanguageModel(model_transform=peft) NcCstd)aYTransform a single module according to the PEFT method. This method is called for each module in the model during the PEFT application process. It should be implemented by subclasses to define how individual modules are transformed for the specific PEFT technique. Args: module (nn.Module): The individual module to be transformed. name (Optional[str]): The name of the module within the model structure. Defaults to None. prefix (Optional[str]): A prefix to be added to the module name, typically used for nested modules. Defaults to None. Returns: nn.Module: The transformed module. This can be the original module with modifications, a new module replacing the original, or the original module if no transformation is needed for this specific module. Note: This method is automatically called for each module in the model when the PEFT instance is applied to the model using the __call__ method. z9The transform method should be implemented by subclasses.)NotImplementedError)selfmodulenameprefixr$Y/home/ubuntu/.local/lib/python3.10/site-packages/nemo/lightning/pytorch/callbacks/peft.py transform>szPEFT.transformmodelreturncCs||t|trt|dkr|D]}||jqnt|tjjj j r-|j |jn||jt |rD|j jjtjkrD|||S)a^Apply the PEFT method to the entire model. This method freezes the model parameters and walks through the model structure, applying the transform method to each module. Args: model (nn.Module): The model to be fine-tuned. Returns: nn.Module: The transformed model with PEFT applied. ) freeze_model isinstancerlenwalkr&torchnnparallel distributedDistributedDataParallelr!rtrainerstatefnrFITTINGr r' model_chunkr$r$r%__call__Ws   z PEFT.__call__cCs~t|trt|dkr|D]}|q t|tjjjjr#|j n|t |r;|j j j tjkr=|jdddSdSdS)aApply a default freeze method to the model. This method freezes all the model parameters. This method can be overridden by subclasses to implement custom freeze strategies (e.g. freeze only parts of the model) Args: model (nn.Module): The model to be fine-tuned. Returns: nn.Module: The transformed model with PEFT applied. r)T)modeN)r+rr,freezer.r/r0r1r2r!rr3r4r5rr6trainr7r$r$r%r*rs   zPEFT.freeze_modelcCs tt|dS)z This is a helper function to return a partial function that wraps the checkpoint I/O with the PEFT adapter. Can be overridden in each PEFT method class. )peft)rWrappedAdapterIOr r$r$r%get_wrappped_ios zPEFT.get_wrappped_ior3 pl_modulestagec s$ddlm}ddlm}tj||ddt|jv|_j _ | }d|_ d|_ |durQdd d }d d j_d d j_j j|_ d |_ dd j _ngd}fddtfdd |D}|dd|i|j _tj ddr|j jjnj j|_dj _|j rd j _dj _dS)zPTL callback setup function.r)create_checkpoint_io)r)rB HFAutoModelNF huggingfaceT) model_libraryloracSdSNTr$r$r$r$r%zPEFT.setup..cSrHrIr$r$r$r$r%rJrKcSrHrIr$xr$r$r%rJrK)save_ckpt_format async_savetorch_dist_multiprocassume_constant_structure parallel_saveparallel_save_within_dp parallel_loadload_directly_on_devicecsi|] }|tj|qSr$)getattrstrategy).0argr3r$r% s zPEFT.setup..cs tj|SN)hasattrrWrLrZr$r%rJ wrapping_ckpt_iorOr$)'nemo.lightning.pytorch.strategies.utilsrCnemo.lightning.pytorch.utilsrsupersetuptype__name___add_via_setattrrWr3r@automodel_setup_optimizerstransform_already_applied_checkpoint_connectorrestore_training_state restore_modelsetup_optimizersfilter_checkpoint_iorV wrapped_io_init_model_parallel_setup_optimizers) r r3rArBrCrrockpt_io_kwargsckpt_io_kwarg_names __class__rZr%rcs8           z PEFT.setupcCsjtdd|jD|_|jD] \}}t|dr2|D]\}}|dur1|j|d|qqdS)z Set params to be saved for PEFT. This function is called in apply_transform. Can be overridden in each PEFT method class. css|] \}}|jr|VqdSr\) requires_grad)rXr"paramr$r$r% s   z*PEFT.set_params_to_save..track_running_statsN.)setlightning_modulenamed_parametersparams_to_save named_modulesr] named_buffersadd)r r3 module_namer! buffer_namebufferr$r$r%set_params_to_saves  zPEFT.set_params_to_savecstdds t||jjdur+tjjjdtkr+ |jjj Stdddks8j durLj durJt d |d_ dSi}jjdurnt djjfdd |jD|d <t|jd r~t d |j|jjtjkrt d|j|jjdur|jr|jjdd g|d<|rjjjj|d}|jj|dd|jjtjkr|jj|dd|dd}durt|j |D] \}}|j!"|q|j#dddD]}t$|t%r|&||j'dSqt(|dur t )ddSdS)a  This function does the following: 1. Apply PEFT model transform. 2. Set up model parallel and optimizer, which were skipped in setup 3. Load weights and optimizer state dict 4. Set up `finalize_model_grads` from mcore. rhFNTzSetting up optimizerszLoading adapters from cs i|] \}}|r||qSr$)adapter_key_filterrXkvr?r$r%r[s z(PEFT.apply_transform.. state_dictinit_model_parallelzInitializing model parallel) is_loading optimizersharded_state_dictstrict)selective_restore lr_schedulerszmMegatronOptimizerModule not found in trainer callbacks. finalize_model_grads is not properly set up for PEFT.)*rVrbapply_transformrroadapter_ckpt_pathrpartsrrestore_automodelparentrgrinfor'ritemsr]rWrr4r5rr6rlshould_restore_optimizer_statesoptimizer_sharded_state_dictload_checkpointload_model_state_dictload_optimizer_state_dictgetziplr_scheduler_configs schedulerload_state_dict callbacksr+r on_fit_startr|rwarning)r r3adapter_sharded_state_dict adapter_staterconfig lrs_statecbrtr?r%rs^                zPEFT.apply_transformc s ddd|j|}|j}|dD]}||vs$J||fqddlm|jjdfdd|d Did d |j D] \}}| ||dvqF|j j tjkr|jd usdJd |||j||d d }d urt|j|D]\}} |j| qd Sd Sd S)z5restores automodel's adapter and optimizer state dictr'cSs,|d}|d|ks Jd|ddS)z0helper function to remove first "model" from fqnrzrr)N)splitjoin)fqnr#rr$r$r%pop_fqn_prefixs z.PEFT.restore_automodel..pop_fqn_prefixrr)to_cpucsi|] \}}||qSr$r$rrrr$r%r[-sz*PEFT.restore_automodel..FrNz/Expected automodel_setup_optimizers to be validr)r')rorr|rkeysr`rrWrrr}requires_grad_r4r5rr6rgrrrrrr) r r3pathrrkeyrwrrrr$rr%rs,      zPEFT.restore_automodelrcCs0t|tr |djS||jvpd|vp|dS)z Given a key in the state dict, return whether the key is an adapter (or base model). This function can be subclassed in each PEFT method class. r)z .adapter.z .adapters)r+tuplervr~endswith)r rr$r$r%r@s  zPEFT.adapter_key_filterNN)re __module__ __qualname____doc__rr&r/Moduler9r*r@plTrainerLightningModulestrrcrrrboolr __classcell__r$r$rtr%r,s "4 I&rc steZdZdZdejdejffdd ZddZdd d Z dde de e e e e fde e ddfddZZS)AdapterWrapperaAbstract base class for wrapping modules with adapters in Parameter-Efficient Fine-Tuning (PEFT). This class wraps a module and its associated adapter, providing methods for managing the state dictionaries of both the main module and the adapter. It does not implement the forward method, which must be implemented by concrete subclasses. Attributes: to_wrap (nn.Module): The main module to be wrapped. adapter (nn.Module): The adapter module to be applied. Note: This class is abstract and cannot be instantiated directly. Subclasses must implement the forward method. Example: class LoRALinear(AdapterWrapper): def __init__(self, to_wrap, adapter): super().__init__(to_wrap, adapter) def forward(self, x): return self.to_wrap(x) + self.adapter(x) main_module = nn.Linear(100, 100) adapter = nn.Linear(100, 100) parallel_adapter = LoRALinear(main_module, adapter) to_wrapadaptercstt|||_||_dSr\)rbr__init__rr)r rrrtr$r%rfs zAdapterWrapper.__init__cOs|j|g|Ri|}t|tsJ|jd| d}|}t|dkr:|\}}t|tr9t|dkr9|\}}n t|dkrE|\}}}|||fS)a_ Run the forward method of the linear module `to_wrap`. Return a tuple of three elements: linear_output, bias, layernorm_output x -> [layernorm/identity] -> layernorm_output -> [linear] -> linear_output, bias layernorm_output is different from input x only when linear layer is LayerNormColumnParallelLinear. z+ should return a tuple but instead returns N)rr+rr,)r rMargskwargs linear_outputbiaslayernorm_outputr$r$r%base_linear_forwardks"     z"AdapterWrapper.base_linear_forwardNFcCs:|duri}|jj|||d|jj||d|d|S)aRetrieve the state dictionary of the wrapped module and adapter. This method overrides the default state_dict behavior to include both the main module's state and the adapter's state under a special 'adapters' key. Args: destination (Optional[dict]): A dictionary to store the state. If None, a new dictionary is created. Defaults to None. prefix (str): A prefix added to parameter and buffer names. Defaults to ''. keep_vars (bool): If True, returns variables instead of tensor values. Defaults to False. Returns: dict: The state dictionary containing both the main module and adapter states. N) destinationr# keep_varsadapter.)rrr)r rr#rr$r$r%rs zAdapterWrapper.state_dictr$r#sharded_offsetsmetadatar(rcCs:i}||j|||||j|d|||S)aRetrieve the sharded state dictionary of the wrapped module and adapter. This method is used for distributed checkpointing, combining the sharded states of both the main module and the adapter. Args: prefix (str): A prefix added to parameter and buffer names. Defaults to ''. sharded_offsets (Tuple[Tuple[int, int, int]]): Offsets for sharded parameters. Defaults to an empty tuple. metadata (Optional[dict]): Additional metadata for the sharded state. Defaults to None. Returns: ShardedStateDict: The combined sharded state dictionary. r)updaterrr)r r#rrrr$r$r%rsz!AdapterWrapper.sharded_state_dict)NrF)rr$N)rerrrr/rrrrrr intr dictrrr$r$rtr%rJs" rc seZdZUdZdZeeed<dZee ed<dZ ee ed< ddeddeeddffd d Z e dd e eefd ed eeddfddZddZe   dd edeededeBde eeffddZZS)r>a A wrapper class for checkpoint I/O operations, specifically designed for PEFT (Parameter-Efficient Fine-Tuning). This class handles the complexities of saving and loading checkpoints for both initial PEFT training and resuming PEFT training. It ensures that only the necessary adapter weights are saved and loaded, while also preserving the base model weights. **Usage:** 1. **Initial PEFT Training:** - The class handles the saving of only adapter weights. - Metadata about the base model checkpoint is stored for future reference. 2. **PEFT Resume:** - The class loads both base model and adapter weights. - The previously stored metadata is used to locate the correct base model checkpoint. **Attributes:** - `peft`: The PEFT instance associated with the wrapped checkpoint I/O. - `model_ckpt_path`: The path to the base model checkpoint. - `adapter_ckpt_path`: The path to the adapter checkpoint. Note that the paths are set by save/load functions and users do not need to set them. **Methods:** - `save_checkpoint`: Saves the adapter weights and metadata to the specified path. - `load_checkpoint`: Loads the base model and adapter weights based on the specified path and metadata. Nr=model_ckpt_pathr checkpoint_io CheckpointIOr(cs||_t|dSr\)r=rbr)r rr=rtr$r%rszWrappedAdapterIO.__init__ checkpointrstorage_optionscsXjdusJd}dD] }||vr|}nq |dusJd||vs&Jd||}ttfdd|||<t||}jj|||d}ddlm } | rt |d d } dd l m } t j| r||} | jt} | jd d d | t}n| jd d d d tji} | t}t|d}t| |Wd|S1swY|S)N)rrzCExpected checkpoint to contain `sharded_state_dict` or `state_dict`z&Expected state_key to be in checkpointcsj|dS)Nr)r=r)itemr?r$r%rJsz2WrappedAdapterIO.save_checkpoint..)rr)is_global_rank_zeroT) is_saving)HFCheckpointIO)parentsexist_okrw)rpoprrmrlistrsave_checkpointnemo.utils.get_rankrrnemo.lightning.io.hfrr+_create_lora_hf_configrrmkdirrrrropenjsondump)r rrr state_keyrr ckpt_keysrequestrbase_dirrrhf_adapter_baseadapter_meta_pathfr$r?r%rs>          z WrappedAdapterIO.save_checkpointcCsxdd}ddlm}ddlm}|||jj}||jj||jj|jjt|j|d}| }d|d<d |d <||d <|S) z0Creates a HF lora config from a NeMo Lora configcSstttdd|}t|dkr|Sttdd|}dtdd|}t|}|D] }|t||q*t|S)a Extracts module names from a list of checkpoint keys that match the target modules. This function processes a list of target module patterns, where each pattern may or may not contain a wildcard (`'*'`). The function matches these patterns against the checkpoint keys, with the following behavior: - Patterns containing '*' will be expanded to match any sequence of characters except a dot (`.`). - Patterns without '*' are matched literally. Args: ckpt_keys (list of str): A list of strings representing checkpoint keys to be searched. target_modules (list of str): A list of target module patterns. Some patterns may contain wildcards (`'*'`), which match any characters except a dot. Returns: list of str: A list of module names from `target_modules` that match any of the `ckpt_keys`. The result is returned as a list of unique module names. Example: ckpt_keys = [ "model.model.layers.27.self_attn.k_proj", "model.model.layers.27.self_attn.v_proj", "model.model.layers.27.self_attn.mlp" ] target_modules = ["*proj"] extract_matched_module_names(ckpt_keys, target_modules) # Output: ['k_proj', 'v_proj'] Notes: - This function uses regular expressions to match the target patterns in the checkpoint keys. - Wildcards are expanded as `[^.]+` to ensure that the match doesn't cross dot (`.`) boundaries. cSsd|vSN*r$rLr$r$r%rJ6z_WrappedAdapterIO._create_lora_hf_config..extract_matched_module_names..rcSsd|vSrr$rLr$r$r%rJ9r|cSs |ddS)Nrz[^.]+)replacerLr$r$r%rJ;r^) rrmr,rmapr{rrefindall)rtarget_modulesre_target_modulesnon_re_target_modulescombined_patternansrr$r$r%extract_matched_module_namess&  zMWrappedAdapterIO._create_lora_hf_config..extract_matched_module_namesr) LoraConfig)DoRA)rr lora_alpha lora_dropoutuse_doraLORA peft_typeN megatron_corer) r=rnemo.collections.llm.peftrrdimalphadropoutr+to_dict)r rrrrmaterialized_module_names lora_configr$r$r%r s  2  z'WrappedAdapterIO._create_lora_hf_config map_locationrStrictHandlingc Cs|jdusJd}t|}t|ddr)tt||_|jj|id}|j}|_n>|t } rVt |d }t |} Wdn1sFwYt| d|_||_n|tt rd|t|_n||_|j||||} |dur|d| d|S| S)a ===================== Initial PEFT Training ===================== Initial PEFT training requires loading the base model weights. In this case, this function is called by trainer.strategy.setup() -> megatron_strategy.restore_model() -> megatron_strategy.load_checkpoint(). `path = PosixPath()`, and sharded_state_dict contains only base model weights =========== PEFT Resume =========== PEFT resume requires loading two set of model weights, 1) base model weights and 2) adapter weights Base model weights could be imported from e.g. HF, and is frozen during PEFT training. Adapter weights contains the training metadata that will need to be loaded. As such, this function will be entered twice during PEFT training resume. For the FIRST TIME this function is called by trainer._checkpoint_connector._restore_modules_and_callbacks. `path = AdapterPath(, base_model_path=)`, and sharded_state_dict contains only base model weights For the SECOND TIME this function is called by PEFT.apply_transform (above, in the same file). `path = PosixPath()`, and sharded_state_dict contains only adapter weights. Nbase_model_pathrrrr)rrrVrrrrrrrexistsrrloadrrr) r rrrr adapter_ckptbaserrr model_ckptr$r$r%rUs*     z WrappedAdapterIO.load_checkpointrr\)NNN)rerrrr=r r__annotations__rrrrrr rrr rrrrrrr$r$rtr%r>s: *$H  r>):rrabcrr functoolsrpathlibrtypingrrrr r r lightning.pytorchpytorchrr.torch.nnr/ lightning.fabric.utilities.typesr $lightning.pytorch.plugins.io.wrapperr lightning.pytorch.trainer.statesrtyping_extensionsrnemo.lightning.ckpt_utilsrrrnemo.lightning.io.mixinrnemo.lightning.io.plrr nemo.lightning.megatron_parallelr0nemo.lightning.pytorch.callbacks.model_transformr%nemo.lightning.pytorch.optim.megatronrrarr nemo.utilsr!nemo.utils.callbacks.dist_ckpt_ior(megatron.core.dist_checkpointing.mappingrrrrr>r$r$r$r%s:                 u