o 8wiT.@sddlmZddlZddlmZddlZddlmZddlmZddl m Z ddl m Z m Z mZmZddlmZmZmZdd lmZGd d d e ZdS) ) annotationsN)Optional)tqdm) PeftConfig) BaseTunerBaseTunerLayercheck_target_module_exists onload_layer)AuxiliaryTrainingWrapper_get_input_embeddings_name_get_submodules)TrainableTokensLayercseZdZUdZded<dCdDfdd ZdEfd d Zd d Z dFdGfdd ZddZ dHdd Z dId"d#Z dJd%d&Z e d'd(Zd)d*ZdKd+d,ZdLdMd.d/ZdNd0d1ZdNd2d3ZdOd5d6ZdPd8d9Z :dQdRd?d@Z    :dSdTdAdBZZS)UTrainableTokensModeltrainable_tokens_strprefixFlow_cpu_mem_usageboolcstj||||ddS)N)r)super__init__)selfmodelconfig adapter_namer __class___/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/peft/tuners/trainable_tokens/model.pyr"szTrainableTokensModel.__init__namecs.zt|WStyt|j|YSw)z1Forward missing attributes to the wrapped module.)r __getattr__AttributeErrorgetattrr)rrrrrr %s  z TrainableTokensModel.__getattr__cCs|jdur t|jd|_|S)N embed_tokens)target_modulesr r)r peft_config model_configrrr_prepare_adapter_config,s z,TrainableTokensModel._prepare_adapter_configTr nn.Modulerautocast_adapter_dtypereturnNonec stj||||d||}|ddre|jjdurgt|jtridd|jjD}|jj ddD]7\}fdd|D}|rdt |\} } } |j | } |j| d<| | || | | |d q3dSdSdSdS) N)rrr)rtie_word_embeddingsFcSs$g|]}d|dddqS).N)joinsplit).0nrrr Ks$z7TrainableTokensModel.inject_adapter..)remove_duplicatecsg|] }|r|qSr)endswith)r1 target_keyrrrr3Ns tied_adapterr)rinject_adapterget_model_configgetr_tied_weights_keys isinstanceget_input_embeddingsr named_modulesr r%to_dict_create_and_replace_dict) rrrr)rr& module_keysmodule matched_keysparenttarget target_namer%rr7rr93s>   z#TrainableTokensModel.inject_adaptercOsgSNr)rargskwargsrrr_get_tied_target_modules^sz-TrainableTokensModel._get_tied_target_modulesr%dictrFrGrE current_keyc CsN|}t|tr|j|fi|dS|j|||fi|}|||||dS)z The same as `_create_and_replace` but takes a dictionary instead of a peft config so that we can add keys that are not present in the config, such as `tied_adapter`. N)r=r update_layer_create_new_module_replace_module) rr%rrFrGrErMrJ new_modulerrrrAes  z-TrainableTokensModel._create_and_replace_dictrcCs |}|||||||dS)zc A private method to create and replace the target module with the adapter module. N)r@rA)rr%rrFrGrErMrJrrr_create_and_replacezs z(TrainableTokensModel._create_and_replacekeycCs t||SrH)r)rr%rSrrr_check_target_module_existss z0TrainableTokensModel._check_target_module_existscKs8t||fi|}|j||d|d|ddd|S)N init_weights token_indicesr8)rUrVr8)rrNr;)r%rrFrJrQrrrrOs z'TrainableTokensModel._create_new_modulecst|||t|dr|j}t|ds |j|_t|dr |j|_t|dddur>t|dr3|j|j_n|j|_||jjt d| D]\}}|j |vrdt fdd| Dsd||jjqGdS)N base_layerbiasstatemetac3s|]}|jkVqdSrH)device)r1prZrr sz7TrainableTokensModel._replace_module..)setattrhasattrrWweightrXr"rYtor[torchr?rany parameters)rrE child_namerQchildrrCrr]rrPs&        z$TrainableTokensModel._replace_modulecCs&|D] \}}|j|vrd|_qdS)NF)named_parametersr requires_grad)rrr2r\rrr _mark_only_adapters_as_trainables  z5TrainableTokensModel._mark_only_adapters_as_trainableenabledcCs,|jD]}t|ttfr||qdSrH)rmodulesr=rr enable_adapters)rrkrCrrr_set_adapter_layerss  z(TrainableTokensModel._set_adapter_layerscC|jdddS)zyEnable all adapters. Call this if you have previously disabled all adapters and want to re-enable them. TrkNrnrrrrenable_adapter_layersz*TrainableTokensModel.enable_adapter_layerscCro)zDisable all adapters. When disabling all adapters, the model output corresponds to the output of the base model. FrpNrqrrrrrdisable_adapter_layersrtz+TrainableTokensModel.disable_adapter_layersstr | list[str]cCsF|jD]}t|tr|jrtd|||q||_ dS)aSet the active adapter(s). Additionally, this function will set the specified adapters to trainable (i.e., requires_grad=True). If this is not desired, use the following code. ```py >>> for name, param in model_peft.named_parameters(): ... if ...: # some check on name (ex. if 'lora' in name) ... param.requires_grad = False ``` Args: adapter_name (`str` or `list[str]`): Name of the adapter(s) to be activated. zJAdapter cannot be set when the model is merged. Unmerging the model first.N) rrlr=rmergedwarningswarnunmerge set_adapteractive_adapter)rrrCrrrr{s    z TrainableTokensModel.set_adaptertorch.nn.ModulecCs |jddS)zh Gets back the base model by removing all the trainable tokens modules without merging. F)merge_unload_and_optionally_mergerrrrrunloads zTrainableTokensModel.unloadN progressbar safe_merge adapter_namesOptional[list[str]]cCs|j|||dS)a This method merges the trained tokens into the targeted embedding layer(s) of the base model. This is needed if someone wants to use the base model as a standalone model. Args: progressbar (`bool`): whether to show a progressbar indicating the unload and merge process safe_merge (`bool`): whether to activate the safe merging check to check if there is any potential Nan in the adapter weights adapter_names (`List[str]`, *optional*): The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults to `None`. )rrrr)rrrrrrrmerge_and_unloadsz%TrainableTokensModel.merge_and_unloadc sfddjD}d|rdndd}t|| |dD]Y}z tj|\}} } Wn ty4Yqwt| 6t| drP| j|||d } || | | nt| d rh|r^| j ||d || | | Wdn1srwYqjS) Ncsg|] \}}j|vr|qSr)r)r1rS_rrrrr3szETrainableTokensModel._unload_and_optionally_merge..z Unloading z and merging r)disabledesc"unload_and_optionally_merge_module)r~rrrW)rr) rr?rr r!r r`rrPr~get_base_layer) rr~rrrkey_listrrSrErFrGunloaded_modulerrrrrs,     z1TrainableTokensModel._unload_and_optionally_merge)F)rr)rr)TF) rr(rrr)rrrr*r+)r%rLrrrFr(rGrrEr(rMrr*r+)r%rrrrFr(rGrrEr(rMrr*r+)r%rrSrr*r)rr(r*r+)T)rkrr*r+)r*r+)rrvr*r+)r*r})FFN)rrrrrrr*r})TFFN)rrrrrr)__name__ __module__ __qualname__r__annotations__rr r'r9rKrArRrT staticmethodrOrPrjrnrsrur{rrr __classcell__rrrrrs6 +           r) __future__rrxtypingrrctorch.nnnnr peft.configrpeft.tuners.tuners_utilsrrrr peft.utilsr r r layerrrrrrrs