o }oi@sddlZddlmZddlmZddlmZmZmZm Z ddl m Z ddl mZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZm Z ddl!m"Z"dZ#GdddeZ$Gddde Z%dS)N)partial)Path)AnyCallableDictOptional)_PATH) TrainerFn) model_summary)override)base)ADAPTER_META_FILENAME) ckpt_to_dir)MegatronParallel)PEFTWrappedAdapterIO)loggingspeechlm_peft_resumecs^eZdZdeffdd ZddZ    dd d Zd d ZdddZde j fddZ Z S)SpeechToTextLLMPEFTpeftcst||_dS)N)super__init__r)selfr __class__c/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/speechlm/utils/model_transform.pyr%s  zSpeechToTextLLMPEFT.__init__cCs tt|dS)z} This is a helper function to return a partial function that wraps the checkpoint I/O with the PEFT adapter. )r)rSpeechLMWrappedAdapterIO)rrrrget_wrappped_io)s z#SpeechToTextLLMPEFT.get_wrappped_iomodel/nemo.collections.speechlm.model.SpeechToTextLLMreturncCs||j}td|jt|tr&t|dkr&|D]}||qn||t |dr=|j j j t jkr=||tdtj|dd|S)aRApply the PEFT method to the LLM. This method freezes the LLM parameters and walks through the model structure, applying the transform method to LLM module. Args: model (nn.Module): The model to be fine-tuned. Returns: nn.Module: The transformed model with PEFT applied. z&Applying PEFT to language model with: trainer ) max_depth) freeze_llmmodulerinfor isinstancerlen_transform_modulehasattrr#statefnr FITTING freeze_modelr summarize)rrr( model_chunkrrr__call__/s   zSpeechToTextLLMPEFT.__call__cCs2t|ds |j}t|drtj|j|jdddS)Nlanguage_modelT) _skip_map)r-r(r/walkr5 transform)rr(rrrr,Ps  z%SpeechToTextLLMPEFT._transform_moduleNcCs|jj|||dS)N)nameprefix)rr8)rr(r9r:rrrr8UszSpeechToTextLLMPEFT.transformr#cCs8|j}tdd|D|_t|jdkrtddS)z Set params that should be saved for PEFT, including some params that don't require gradients, such as the running mean and var of batchnorm. cSsg|]\}}|qSrr).0r9_rrr ^sz:SpeechToTextLLMPEFT.set_params_to_save..rz'No trainable parameters found for PEFT!N)lightning_modulesettrainable_parametersparams_to_saver+ RuntimeError)rr#rrrrset_params_to_saveXs z&SpeechToTextLLMPEFT.set_params_to_save)rr r!r )NN) __name__ __module__ __qualname__rrrr4r,r8plTrainerrC __classcell__rrrrr$s ! rc @sxeZdZe   d dedeededeBdee e ffddZ  ddedeed ededeBddf d d Z dS)rNpath map_locationstrictStrictHandlingr!c Cs|jdusJt|t}d}d}t|ddr/tt||_|jj|id}|j}|_ d}n,| rXt |d }t |} Wdn1sHwYt| d|_ ||_n||_ |j|||||d} |dur||d | d t| vrzd|t<|S| S) ad Overwrite the load_checkpoint method to handle PEFT resume for SpeechLM. ===================== Initial PEFT Training ===================== Initial PEFT training requires loading the base model weights. In this case, this function is called by trainer.strategy.setup() -> megatron_strategy.restore_model() -> megatron_strategy.load_checkpoint(). `path = PosixPath()`, and sharded_state_dict contains only base model weights =========== PEFT Resume =========== PEFT resume requires loading two set of model weights, 1) base model weights and 2) adapter weights Base model weights could be imported from e.g. HF, and is frozen during PEFT training. Adapter weights contains the training metadata that will need to be loaded. As such, this function will be entered twice during PEFT training resume. For the FIRST TIME this function is called by trainer._checkpoint_connector._restore_modules_and_callbacks. `path = AdapterPath(, base_model_path=)`, and sharded_state_dict contains only base model weights For the SECOND TIME this function is called by PEFT.apply_transform (above, in the same file). `path = PosixPath()`, and sharded_state_dict contains only adapter weights. NFbase_model_path)sharded_state_dictTrmodel_ckpt_path)rL state_dict) checkpoint_iorr getattrrstradapter_ckpt_pathload_checkpointrNrQexistsopenjsonload_load_checkpointupdateSPEECHLM_PEFT_RESUME) rrJrOrKrLadapter_meta_path adapter_ckpt load_basefmetadata model_ckptrrrrWes.!     z(SpeechLMWrappedAdapterIO.load_checkpointFracCs(|r dttdiS|j||||}|S)NrRT)dictr^rSrW)rrJrOrKrarLrdrrrr\sz)SpeechLMWrappedAdapterIO._load_checkpoint)NNN)NFN) rDrErFr rrrboolrrUrrWr\rrrrrcs8   F r)&rZ functoolsrpathlibrtypingrrrrlightning.pytorchpytorchrG lightning.fabric.utilities.typesr lightning.pytorch.trainer.statesr lightning.pytorch.utilitiesr typing_extensionsr nemo.collections.llm.fnr r/nemo.lightning.ckpt_utilsr nemo.lightning.io.plr nemo.lightning.megatron_parallelr%nemo.lightning.pytorch.callbacks.peftrr nemo.utilsrr^rrrrrrs$            ?