o 8wiS@sddlmZddlZddlZddlmZmZddlmZddl m Z ddl Z ddl m Z ddl mZddlmZmZdd lmZmZmZdd lmZmZmZmZmZd d lmZmZmZm Z Gd ddeZ!dS)) annotationsN)asdictreplace)Enum)Optional)nn)Conv1D)is_bnb_4bit_availableis_bnb_available) BaseTunerBaseTunerLayercheck_target_module_exists)6TRANSFORMERS_MODELS_TO_IA3_FEEDFORWARD_MODULES_MAPPING1TRANSFORMERS_MODELS_TO_IA3_TARGET_MODULES_MAPPINGModulesToSaveWrapper_freeze_adapter_get_submodules)Conv2dConv3dIA3LayerLinearcseZdZUdZdZded<dDdEfdd Zed d Zed d Z dFddZ ddZ edGddZ ddZ dHfdd ZdDdIddZdJd!d"ZdKd#d$ZdKd%d&ZdLd)d*Zed+d,Z -dMdNd2d3ZdOdPd5d6ZdQd7d8ZdRd9d:ZdSd>d?ZdTdBdCZZS)UIA3ModelaP Creates a Infused Adapter by Inhibiting and Amplifying Inner Activations ((IA)^3) model from a pretrained transformers model. The method is described in detail in https://huggingface.co/papers/2205.05638 Args: model ([`~transformers.PreTrainedModel`]): The model to be adapted. config ([`IA3Config`]): The configuration of the (IA)^3 model. adapter_name (`str`): The name of the adapter, defaults to `"default"`. low_cpu_mem_usage (`bool`, `optional`, defaults to `False`): Create empty adapter weights on meta device. Useful to speed up the loading process. Returns: `torch.nn.Module`: The (IA)^3 model. Example: ```py >>> from transformers import AutoModelForSeq2SeqLM, ia3Config >>> from peft import IA3Model, IA3Config >>> config = IA3Config( ... peft_type="IA3", ... task_type="SEQ_2_SEQ_LM", ... target_modules=["k", "v", "w0"], ... feedforward_modules=["w0"], ... ) >>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base") >>> ia3_model = IA3Model(config, model) ``` **Attributes**: - **model** ([`~transformers.PreTrainedModel`]) -- The model to be adapted. - **peft_config** ([`ia3Config`]): The configuration of the (IA)^3 model. ia3_strprefixFlow_cpu_mem_usageboolcstj||||ddS)N)r)super__init__)selfmodelconfig adapter_namer __class__R/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/peft/tuners/ia3/model.pyrNszIA3Model.__init__cKstr ddl}ddlm}trddlm}|dd}|dd}|dd} t|tr2| } n|} |r\t| |j jr\| } | | j j| j j| jd |||fd| i| } | S|rt| |j jr| } | | j| jj| jjd |||fd| i| } | St|tj jrt||fd| i|} | St|tj jrt||fd| i|} | St| tj jr|d rtd d|d <|_t||fd| i|} | St| tr|d std d|d <|_t||f| dd|} | Std|d)Nrr) Linear8bitLt) Linear4bitloaded_in_8bitFloaded_in_4bitis_feedforward)has_fp16_weights thresholdindex) compute_dtypecompress_statistics quant_typefan_in_fan_outzjfan_in_fan_out is set to True but the target module is `torch.nn.Linear`. Setting fan_in_fan_out to False.zafan_in_fan_out is set to False but the target module is `Conv1D`. Setting fan_in_fan_out to True.T)r,is_target_conv_1d_layerzTarget module zd is not supported. Currently, only `torch.nn.Linear`, `torch.nn.Conv2d`, and `Conv1D` are supported.)r bitsandbytesbnbr(r r)pop isinstancer get_base_layerrcopyupdatestater-r.r/r0weightr1r2torchrrrwarningswarnr3r ValueError) ia3_configr#targetkwargsr6r(r)r*r+r,target_base_layereightbit_kwargs new_modulefourbit_kwargsr&r&r'_create_new_moduleQs|       %  zIA3Model._create_new_modulecCs t||SN)r )rBkeyr&r&r'_check_target_module_existss z$IA3Model._check_target_module_existsr! nn.ModulereturnNonecCs&|D] \}}|j|vrd|_qdS)NF)named_parametersr requires_grad)r r!npr&r&r' _mark_only_adapters_as_trainables  z)IA3Model._mark_only_adapters_as_trainablec Cs|||}|j|j|t|jddt|jddd}t|tr(|||jdS|j|||fi|} ||j vr=| d| ||| |dS)Nis_loaded_in_8bitFis_loaded_in_4bit)r3init_ia3_weightsr,r*r+) _check_target_module_feedforwardr3rWgetattrr!r8r update_layerrIactive_adaptersrequires_grad__replace_module) r rBr#rC target_nameparent current_keyr,rDrGr&r&r'_create_and_replaces      zIA3Model._create_and_replacecs>t|jtrtt|j}|Stfdd|jD}|S)z A helper private method that checks if the target module `key` matches with a feedforward module specified in `ia3_config` c3s|]}|VqdSrJ)endswith).0 target_keyrKr&r' z.)r8feedforward_modulesrrre fullmatchany)rBrKr,r&rer'rXs z)IA3Model._check_target_module_feedforwardcst|||t|dr|j}t|ds |j|_t|dr |j|_t|dddur>t|dr3|j|j_n|j|_||jjt d| D]\}}|j |vrdt fdd| Dsd||jjqGdS)N base_layerbiasr<metac3s|]}|jkVqdSrJ)device)rcrSrnr&r'rfrgz+IA3Model._replace_module..)setattrhasattrrlr=rmrYr<toror> named_modulesrrk parameters)r r_ child_namerGchildnamemoduler&rpr'r]s&        zIA3Model._replace_modulerxcs8zt|WSty|dkrt|j|YSw)z1Forward missing attributes to the wrapped module.r!)r __getattr__AttributeErrorrYr!)r rxr$r&r'rzs zIA3Model.__getattr__ inferencecCsFi}|jD]\}}ddt|D}|rd|d<q|||<|S)NcSs&i|]\}}|t|tr|jn|qSr&)r8rvalue)rckvr&r&r' s&z4IA3Model.get_peft_config_as_dict..Tinference_mode) peft_configitemsr)r r| config_dictrKr}r"r&r&r'get_peft_config_as_dictsz IA3Model.get_peft_config_as_dictTcCs,|jD]}t|ttfr||qdSrJ)r!modulesr8rrenable_adapters)r enabledryr&r&r'_set_adapter_layerss  zIA3Model._set_adapter_layerscC|jdddS)zyEnable all adapters. Call this if you have previously disabled all adapters and want to re-enable them. TrNrr r&r&r'enable_adapter_layerszIA3Model.enable_adapter_layerscCr)zDisable all adapters. When disabling all adapters, the model output corresponds to the output of the base model. FrNrrr&r&r'disable_adapter_layersrzIA3Model.disable_adapter_layersr#str | list[str]cCsF|jD]}t|tr|jrtd|||q||_ dS)aSet the active adapter(s). Additionally, this function will set the specified adapters to trainable (i.e., requires_grad=True). If this is not desired, use the following code. ```py >>> for name, param in model_peft.named_parameters(): ... if ...: # some check on name (ex. if 'lora' in name) ... param.requires_grad = False ``` Args: adapter_name (`str` or `list[str]`): Name of the adapter(s) to be activated. zJAdapter cannot be set when the model is merged. Unmerging the model first.N) r!rr8rmergedr?r@unmerge set_adapteractive_adapter)r r#ryr&r&r'r s    zIA3Model.set_adaptercCsd|jdur|dtvrtdtt|d|_|jdur0|dtvr'tdtt|d|_|S)N model_typez0Please specify `target_modules` in `peft_config`z5Please specify `feedforward_modules` in `peft_config`)target_modulesrrAsetrhr)r model_configr&r&r'_prepare_adapter_config!s      z IA3Model._prepare_adapter_configNmerge safe_merge adapter_namesOptional[list[str]]c stjddr tdtjddrtd|fddjD}|D]S}z tj|\}}}Wn ty?Yq)wt|drY|rN|j||d  ||| |q)t |t r||j |j} t| drv|rr| j||d | } t||| q)jS) a This method merges the (IA)^3 layers into the base model. This is needed if someone wants to use the base model as a standalone model. Args: safe_merge (`bool`, `optional`, defaults to `False`): If True, the merge operation will be performed in a copy of the original weights and check for NaNs before merging the weights. This is useful if you want to check if the merge operation will produce NaNs. Defaults to `False`. adapter_names (`List[str]`, *optional*): The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults to `None`. rUFz>Cannot merge ia3 layers when the model is loaded in 8-bit moderVz>Cannot merge ia3 layers when the model is loaded in 4-bit modecg|] \}}j|vr|qSr&rrcrK_rr&r' Hz9IA3Model._unload_and_optionally_merge..rlrr)rYr!rA_unloading_checksrtrr{rrrr]r9r8rmodules_to_saverrq) r rrrkey_listrKr_rCr^rGr&rr'_unload_and_optionally_merge1s2       z%IA3Model._unload_and_optionally_mergetorch.nn.ModulecCs|j||dS)u This method merges the IA³ layers into the base model. This is needed if someone wants to use the base model as a standalone model. Args: safe_merge (`bool`): whether to activate the safe merging check to check if there is any potential Nan in the adapter weights adapter_names (`List[str]`, *optional*): The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults to `None`. Example: ```py >>> from transformers import AutoModelForCausalLM >>> from peft import PeftModel >>> base_model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-40b") >>> peft_model_id = "smangrul/falcon-40B-int4-peft-lora-sfttrainer-sample" >>> model = PeftModel.from_pretrained(base_model, peft_model_id) >>> merged_model = model.merge_and_unload() ``` rr)r rrr&r&r'merge_and_unload_szIA3Model.merge_and_unloadcCs |jddS)u Gets back the base model by removing all the IA³ modules without merging. This gives back the original base model. F)rrrr&r&r'unloadzs zIA3Model.unloadcs|jvr td|dj|=fddjD}d}|D] }tj|\}}}t|trA|||durA|jdd}q!|pEg_ j ||ddS)z Deletes an existing adapter. Args: adapter_name (str): Name of the adapter to be deleted. Adapter  does not existcrr&rrrr&r'rrz+IA3Model.delete_adapter..N)new_active_adapters) rrAr!rtrr8rdelete_adapterr[r_delete_auxiliary_adapter)r r#r new_adapterrKrrCr&rr'rs    zIA3Model.delete_adapteradapters list[str]tuple[str, str]csD]}|jvrtd|dqddD}tfdd|Dr*tdfdd D}fd d D}t|d ksHt|d krLtd t|vr]d fddD}n tjfddD}t|vr}d fddD}||fStjfddD}||fS)z Helper function to check if the arguments to add_weighted_adapter are valid and compatible with the underlying model. rrcSsg|] }t|tr|qSr&)r8r)rcryr&r&r'rsz8IA3Model._check_add_weighted_adapter..c3s*|]tfddDdkVqdS)c3s|]}|jvVqdSrJ)rrcadapterwrapperr&r'rfrgzAIA3Model._check_add_weighted_adapter...rN)sum)rcrrr'rfs z7IA3Model._check_add_weighted_adapter..zLCannot add weighted adapters targeting the same module with modules_to_save.ch|] }tj|jqSr&)typerrrrr&r' z7IA3Model._check_add_weighted_adapter..crr&)rrrhrrr&r'rrrzQAll adapter configs should have the same type for target and feedforward modules.|c3$|] }dj|jdVqdS()Nrrrrr&r'rfs"c3|] }j|jVqdSrJrrrr&r'rfc3rrrrhrrr&r'rfs c3rrJrrrr&r'rfr) rrArrklenrjoinrunion)r rrmodules_to_save_wrapperstarget_module_typesfeedforward_module_typesnew_target_modulesnew_feedforward_modulesr&)rr r'_check_add_weighted_adapters2  z$IA3Model._check_add_weighted_adapterweights list[float]cs|tjvr dSj|d\}}tj|d||dj|<j|tj|fddjD}|D]B}t j|\}} }t | t r|| j vrX| j |} nq=| j | _ t||D]\} } | | j vrs| j | } nqd| j | j | 7_ qdq=dS)ac This method adds a new adapter by merging the given adapters with the given weights. Args: adapters (`list`): List of adapter names to be merged. weights (`list`): List of weights for each adapter. adapter_name (`str`): Name of the new adapter. Nrr)rrhcrr&rrrr&r'rrz1IA3Model.add_weighted_adapter..)listrkeysrrinject_adapterr!rrtrr8ria3_ldatazero_zip)r rrr#rrrrKrrC target_ia3_lrr=current_adapter_ia3_lr&rr'add_weighted_adapters6          zIA3Model.add_weighted_adapter)F)rr)r!rMrNrO)rNr)rxr)r|r)T)rNrO)r#rrNrO)TFN)rrrrrr)FN)rrrrrNr)rNr)r#rrNrO)rrrNr)rrrrr#rrNrO)__name__ __module__ __qualname____doc__r__annotations__r staticmethodrIrLrTrarXr]rzrrrrrrrrrrrr __classcell__r&r&r$r'r's8  $ C         .   (r)" __future__rrir? dataclassesrrenumrtypingrr>rtransformers.pytorch_utilsrpeft.import_utilsr r peft.tuners.tuners_utilsr r r peft.utilsrrrrrlayerrrrrrr&r&r&r's