o 8wiH@sddlmZddlZddlmZddlmZddlmZddl Z ddl m Z ddl m Z ddl mZddlmZmZmZdd lmZmZmZd d lmZd d lmZmZGd ddeZdS)) annotationsN)asdict)Enum)Optional)tqdm)Conv1D) BaseTunerBaseTunerLayercheck_target_module_exists)4TRANSFORMERS_MODELS_TO_VBLORA_TARGET_MODULES_MAPPINGModulesToSaveWrapper_get_submodules) VBLoRAConfig)Linear VBLoRALayercs eZdZUdZdZded<dHdIfd d ZdJddZdKddZdLddZ e ddZ ddZ e ddZ dMddZe dd ZdNfd"d# ZdHdOd%d&ZdPdQd)d*ZdRd+d,ZdRd-d.ZdSd0d1Ze d2d3Z '   4dTdUd9d:ZdVd;d<Z 4dWdXd>d?Zd@dAZdYdZdDdEZdRdFdGZZS)[ VBLoRAModela Creates VBLoRA model from a pretrained transformers model. The method is described in detail in https://huggingface.co/papers/2405.15179. Args: model ([`~transformers.PreTrainedModel`]): The model to be adapted. config ([`VBLoRAConfig`]): The configuration of the VBLoRA model. adapter_name (`str`): The name of the adapter, defaults to `"default"`. low_cpu_mem_usage (`bool`, `optional`, defaults to `False`): Create empty adapter weights on meta device. Useful to speed up the loading process. Returns: `torch.nn.Module`: The VBLoRA model. Example: ```py >>> from transformers import AutoModelForCausalLM >>> from peft import VBLoRAConfig, get_peft_model >>> base_model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m") >>> config = VBLoRAConfig( ... task_type="SEQ_CLS", ... r=4, ... target_modules=["fc1", "fc2", "k_proj", "out_proj", "q_proj", "v_proj"], ... num_vectors=60, ... vector_length=256, ... save_only_topk_weights=True, ... ) >>> model = get_peft_model(base_model, config) ``` **Attributes**: - **model** ([`~transformers.PreTrainedModel`]) -- The model to be adapted. - **peft_config** ([`VBLoRAConfig`]): The configuration of the VBLoRAConfig model. vblora_strprefixFlow_cpu_mem_usageboolreturnNonecstj||||ddS)N)r)super__init__)selfmodelconfig adapter_namer __class__U/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/peft/tuners/vblora/model.pyrJszVBLoRAModel.__init__rrrcCs6t|j|j}tjj||j |j||j|<dSN) torchzeros num_vectors vector_lengthnninituniform_init_vector_bank_boundvblora_vector_bank)rrrr-r"r"r#_init_vblora_vector_bankMsz$VBLoRAModel._init_vblora_vector_bankr nn.ModulecCsti|_dSr$)r) ParameterDictr-)rrrrr"r"r#_pre_injection_hookRszVBLoRAModel._pre_injection_hookcCs2t|jdkr|jdkrt|jjddSdS)z A helper method to check the config when a new adapter is being added. Raise a ValueError if there is something wrong with the config or if it conflicts with existing adapters. rnonezf supports only 1 adapter with bias. When using multiple adapters, set bias to 'none' for all adapters.N)len peft_configbias ValueErrorr!__name__)rrr"r"r#_check_new_adapter_configUs   z%VBLoRAModel._check_new_adapter_configcCs t||Sr$)r ) vblora_configkeyr"r"r#_check_target_module_existses z'VBLoRAModel._check_target_module_existsc Cs|durtdt|do|jdu}|j|d}|||t|tr9|j||j|j |j |j |j |j |jddS|jd||j||d|} ||jvrQ| d|||| |dS)NzCurrent Key shouldn't be `None`r5)fan_in_fan_outr5)rr-rtopkr'r(vblora_dropoutinit_logits_std)r9r-rtargetFr")r6hasattrr5r<r. isinstancer update_layerr-r=r>r'r(r?r@_create_new_moduleactive_adapterrequires_grad__replace_module) rr9rrA target_nameparent current_keyr5kwargs new_moduler"r"r#_create_and_replaceis:       zVBLoRAModel._create_and_replacecst|||t|dr|j}t|ds |j|_t|dr |j|_t|dddur>t|dr3|j|j_n|j|_||jjt d| D]\}}d|vrct fdd| Dsc||jjqGdS)N base_layerr5statemetarc3s|]}|jkVqdSr$)device).0prQr"r# sz.VBLoRAModel._replace_module..) setattrrBrOweightr5getattrrPtorRr% named_modulesany parameters)rJ child_namerMchildnamemoduler"rUr#rHs&       zVBLoRAModel._replace_modulecCs|D] \}}|j|vrd|_q|jD]H}|j|j}|dkr!q|dkr6|D] \}}d|vr4d|_q)q|dkrU|D]}t|trSt |drS|jdurSd|j_q>qt d|ddS) NFr2allr5T vblora_onlyzRequested bias: z, is not implemented.) named_parametersr requires_gradactive_adaptersr4r5modulesrCrrBNotImplementedError)rrnrTrFr5mr"r"r# _mark_only_adapters_as_trainables,    z,VBLoRAModel._mark_only_adapters_as_trainablec Kst|tr |}n|}t|tjjr$|dr#tdd|d<|_n"t|t r>d|d<|ds=tdd|d<|_nt d|dtd ||||j |j |j |j|j|jd |}|S) Nr<zjfan_in_fan_out is set to True but the target module is `torch.nn.Linear`. Setting fan_in_fan_out to False.FTis_target_conv_1d_layerzafan_in_fan_out is set to False but the target module is `Conv1D`. Setting fan_in_fan_out to True.zTarget module z is not supported. Currently, only the following modules are supported: `torch.nn.Linear`, `transformers.pytorch_utils.Conv1D`.) rOr-rr=r'r(r>r?r@r")rCr get_base_layerr%r)rwarningswarnr<rr6r=r'r(r>r?r@)r9r-rrArLtarget_base_layerrMr"r"r#rEsF      zVBLoRAModel._create_new_moduler`cs8zt|WSty|dkrt|j|YSw)z1Forward missing attributes to the wrapped module.r)r __getattr__AttributeErrorrYr)rr`r r"r#rqs zVBLoRAModel.__getattr__ inferencecCsFi}|jD]\}}ddt|D}|rd|d<q|||<|S)NcSs&i|]\}}|t|tr|jn|qSr")rCrvalue)rSkvr"r"r# s&z7VBLoRAModel.get_peft_config_as_dict..Tinference_mode)r4itemsr)rrs config_dictr:rtrr"r"r#get_peft_config_as_dictsz#VBLoRAModel.get_peft_config_as_dictTenabledcCs,|jD]}t|ttfr||qdSr$)rrgrCr r enable_adapters)rr|rar"r"r#_set_adapter_layerss  zVBLoRAModel._set_adapter_layerscCs|jdddS)zyEnable all adapters. Call this if you have previously disabled all adapters and want to re-enable them. Tr|N)r~rr"r"r#enable_adapter_layerssz!VBLoRAModel.enable_adapter_layerscCsF|jD]}|j|j}|dkrd|d}t|q|jdddS)zDisable all adapters. When disabling all adapters, the model output corresponds to the output of the base model. r2z>Careful, disabling adapter layers with bias configured to be 'zL' does not produce the same output as the base model would without adaption.FrN)rfr4r5rnror~)rrFvalmsgr"r"r#disable_adapter_layers s    z"VBLoRAModel.disable_adapter_layersstr | list[str]cCsF|jD]}t|tr|jrtd|||q||_ dS)aSet the active adapter(s). Additionally, this function will set the specified adapters to trainable (i.e., requires_grad=True). If this is not desired, use the following code. ```py >>> for name, param in model_peft.named_parameters(): ... if ...: # some check on name (ex. if 'lora' in name) ... param.requires_grad = False ``` Args: adapter_name (`str` or `list[str]`): Name of the adapter(s) to be activated. zJAdapter cannot be set when the model is merged. Unmerging the model first.N) rrgrCrmergedrnrounmerge set_adapterrF)rrrar"r"r#rs    zVBLoRAModel.set_adaptercCs4|jdur|dtvrtdtt|d|_|S)N model_typez0Please specify `target_modules` in `peft_config`)target_modulesr r6set)r4 model_configr"r"r#_prepare_adapter_config3s   z#VBLoRAModel._prepare_adapter_configN progressbar safe_merge adapter_namesOptional[list[str]]c sfddjD}d|rdndd}t|| |dD]?}z tj|\}} } Wn ty4Yqwt| drN|rC| j||d || | | qt | t r]t || | j | j qjS) Ncg|] \}}j|vr|qSr"rrSr:_rr"r# Dz.z Unloading z and merging r)disabledescrO)rr)rr[rr rrrBmergerHrmrCr rWmodules_to_saverF) rrrrrkey_listrr:rJrArIr"rr#_unload_and_optionally_merge=s    z(VBLoRAModel._unload_and_optionally_mergecs|tjvrtd|dj|=fddjD}d}|D] }tj|\}}}t|trE| ||durE|j dd}q%|pIg_ j ||ddS)z Deletes an existing adapter. Args: adapter_name (str): Name of the adapter to be deleted. zAdapter z does not existcrr"rrrr"r#rbrz.VBLoRAModel.delete_adapter..N)new_active_adapters) listr4keysr6rr[r rCrdelete_adapterrF_delete_auxiliary_adapter)rrr new_adapterr:rrAr"rr#rWs   zVBLoRAModel.delete_adaptertorch.nn.ModulecCs|j|||dS)aJ This method merges the VBLoRA layers into the base model. This is needed if someone wants to use the base model as a standalone model. Args: progressbar (`bool`): whether to show a progressbar indicating the unload and merge process safe_merge (`bool`): whether to activate the safe merging check to check if there is any potential Nan in the adapter weights adapter_names (`list[str]`, *optional*): The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults to `None`. Example: ```py >>> from transformers import AutoModelForCausalLM >>> from peft import PeftModel >>> base_model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-40b") >>> peft_model_id = "smangrul/falcon-40B-int4-peft-lora-sfttrainer-sample" >>> model = PeftModel.from_pretrained(base_model, peft_model_id) >>> merged_model = model.merge_and_unload() ``` )rrrr)rrrrr"r"r#merge_and_unloadnszVBLoRAModel.merge_and_unloadcCs |jddS)z Gets back the base model by removing all the VBLoRA modules without merging. This gives back the original base model. F)rrrr"r"r#unloads zVBLoRAModel.unloaddefaulttuple[int, int]c Csd}d}d}|D]#\}}d|vr||7}q d|vr$||7}q |jr-||7}q |j|jr|j|j}d}|dkrCd}n|dkrJd}n |d krQd}nd }||j|j|j|jd} ||j|j|j|j|} t|| | } | |fS||} | |fS) z` Returns the number of savable VB-LoRA parameters and other savable parameters. r vblora_logitsr-rg?ig?l)rdnumelrer4save_only_topk_weightsr'r>int) radapter logits_paramsvector_bank_params other_paramsr`paramr'factortopk_weight_paramstopk_indices_params vblora_paramsr"r"r#get_nb_savable_parameterss:   z%VBLoRAModel.get_nb_savable_parameterscCs,|\}}td|dd||ddS)z_ Prints the number of savable VB-LoRA parameters and total savable parameters. z1VB-LoRA params to-be-saved (float32-equivalent): z,dz || total params to-be-saved: N)rprint)rrrr"r"r#print_savable_parameterss  z$VBLoRAModel.print_savable_parameters)F)rrrr)rrrrrr)rr/rrrrrr)rrrr)rr/rr)r`r)rsr)T)r|rrr)rr)rrrr)TFFN)rrrrrr)rrrr)FFN)rrrrrrrr)r)rr)r7 __module__ __qualname____doc__r__annotations__rr.r1r8 staticmethodr;rNrHrkrErqr{r~rrrrrrrrrr __classcell__r"r"r r#r!sD  &    ,   (       ! $r) __future__rrn dataclassesrenumrtypingrr%torch.nnr)rtransformers.pytorch_utilsrpeft.tuners.tuners_utilsrr r peft.utilsr r r rrlayerrrrr"r"r"r#s