o Ni<@sddlZddlZddlmZddlmZmZmZddlm Z m Z ddl m Z ddl mZmZmZmZmZmZddlmZdd lmZdd lmZmZmZGd d d e ZdS) N)Conv1D)is_bnb_4bit_availableis_bnb_availableis_gptqmodel_available) LoraConfig LoraModel)BaseTunerLayer)5TRANSFORMERS_MODELS_TO_ADALORA_TARGET_MODULES_MAPPING_freeze_adapter_get_submodulesget_auto_gptq_quant_linearget_gptqmodel_quant_linearget_quantization_config)gather_params_ctx)SVDQuantLinear) AdaLoraLayer RankAllocator SVDLinearcsxeZdZdZeZfddZdeddffdd Zd d Z e dd d Z d dZ ddZ ddZddZddZZS) AdaLoraModela Creates AdaLoRA (Adaptive LoRA) model from a pretrained transformers model. Paper: https://openreview.net/forum?id=lq62uWRJjiY Args: model ([`transformers.PreTrainedModel`]): The model to be adapted. config ([`AdaLoraConfig`]): The configuration of the AdaLora model. adapter_name (`str`): The name of the adapter, defaults to `"default"`. low_cpu_mem_usage (`bool`, `optional`, defaults to `False`): Create empty adapter weights on meta device. Useful to speed up the loading process. Returns: `torch.nn.Module`: The AdaLora model. Example:: >>> from transformers import AutoModelForSeq2SeqLM >>> from peft import LoraConfig, AdaLoraModel, AdaLoraConfig >>> config = AdaLoraConfig( peft_type="ADALORA", task_type="SEQ_2_SEQ_LM", init_r=12, lora_alpha=32, target_modules=["q", "v"], lora_dropout=0.01, ) >>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base") >>> model = AdaLoraModel(model, config, "default") **Attributes**: - **model** ([`transformers.PreTrainedModel`]) -- The model to be adapted. - **peft_config** ([`AdaLoraConfig`]): The configuration of the AdaLora model. c stj|||fi|d}|jD] }|js|d7}q|dkr%td|j|jr3t|j|dS||_t |j|j||j|_ dS)NrrzAdaLoraModel supports only 1 trainable adapter. When using multiple adapters, set inference_mode to True for all adapters except the one you want to train.) super__init__ peft_configvaluesinference_mode ValueErrorr modeltrainable_adapter_namer rankallocator)selfrconfig adapter_namekwargstraininable_mode_counter __class__M/home/ubuntu/.local/lib/python3.10/site-packages/peft/tuners/adalora/model.pyrEs zAdaLoraModel.__init__r returnNcsLt|d}|jD] }|js|d7}q |dkr$t|jjddS)z A helper method to check the config when a new adapter is being added. Raise a ValueError if there is something wrong with the config or if it conflicts with existing adapters. rrz supports only 1 trainable adapter. When using multiple adapters, set inference_mode to True for all adapters except the one you want to train.N)r_check_new_adapter_configrrrrr%__name__)rr r#config_r$r&r'r)Ys  z&AdaLoraModel._check_new_adapter_configc Cs|j|j|j|j|jt|jddt|jddd}|ds!|dr(ts(tdt |jdd }|dur7||d <t |t sit |jd rF|jj nd} |j|||fd | i|} ||jvr_| d|||| |dS|||j|j|j|jdS) Nis_loaded_in_8bitFis_loaded_in_4bit)r lora_alpha lora_dropoutfan_in_fan_outinit_lora_weightsloaded_in_8bitloaded_in_4bitr3r4zTo use AdaLora with 8-bit quantization, please install the `bitsandbytes` package. You can install it with `pip install bitsandbytes`.gptq)methodgptq_quantization_config hf_device_map device_map)init_rr/r0r1r2getattrrr ImportErrorr isinstancerhasattrr8_create_new_moduleactive_adaptersrequires_grad__replace_module update_layer) r lora_configr!target target_nameparent current_keyr"quantization_configr9 new_moduler&r&r'_create_and_replacens8       z AdaLoraModel._create_and_replacecKstr ddl}ddlm}trddlm}|dd}tr&t||d} nt |} | dd} | d d} t |t r@| } n|} | rdt | |jjrd|| jj| jj| jd |||fi|} | S| rtrt | |jjr|}|| j| jj| jjd |||fi|} | S| durt || rt||fi|} | St | tjjr|d rtd d|d <|_ nt | t!r|d stdd|d <|_ nt"d|dt#||fi|} | S)Nrr)SVDLinear8bitLt) SVDLinear4bitr7)r9r3Fr4)has_fp16_weights thresholdindex) compute_dtypecompress_statistics quant_typer1zjfan_in_fan_out is set to True but the target module is `torch.nn.Linear`. Setting fan_in_fan_out to False.zafan_in_fan_out is set to False but the target module is `Conv1D`. Setting fan_in_fan_out to True.TzTarget module zP is not supported. Currently, only `torch.nn.Linear` and `Conv1D` are supported.)$r bitsandbytesbnbrLrrMgetrr r popr=rget_base_layernn Linear8bitLtupdatestaterNrOrP Linear4bitcopyrQweightrRrSrtorchLinearwarningswarnr1rrr)rDr!rEr9r"rUrLrMr7 QuantLinearr3r4target_base_layerrJfourbit_kwargsr&r&r'r?sn       #  zAdaLoraModel._create_new_modulec Osf|jj|i|}t|dddurt|jtjr|j|jj }|dkr't dd}d}|j D]l\}}d|vs     8  z+AdaLoraModel.resize_modules_by_rank_patternc Cs|D]g\}}t|}||vrd|dddn d|ddd}dD]@}d|d|d|} |dkrQ||| jdkrJ|| |n|| || <q*||| jdkrd|| dd|fn|| || <q*q|S) Nrrrr)rrhrizbase_model.model.rir)rrrrrr) rr state_dictr!rrrprefixlayerrr&r&r'!resize_state_dict_by_rank_patterns8$, z.AdaLoraModel.resize_state_dict_by_rank_patterncCs|j|j}||j|jkr!|j|j|\}}|r||_dSdS||j|jkr?|jj|j|dd\}}||_|jdS||j|jkrR|j |j|jdSdS)aM This method updates Adalora budget and mask. This should be called in every training step after `loss.backward()` and before `zero_grad()`. `tinit`, `tfinal` and `deltaT` are handled with in the method. Args: global_step (`int`): The current training step, it is used to calculate adalora budget. Example: ```python >>> loss = model(**input).loss >>> loss.backward() >>> optimizer.step() >>> model.base_model.update_and_allocate(i_step) >>> optimizer.zero_grad() ``` T) force_maskN) rr total_steptfinalrupdate_and_allocaterr reset_iptmask_using_rank_pattern)r global_steprDrrr&r&r'r.s  z AdaLoraModel.update_and_allocatecOst|jjd)z;This method is not supported for AdaLoRA, use LoRA instead.z. does not support add_weighted_adapter method.) TypeErrorr%r*)rrzr"r&r&r'add_weighted_adapterXsz!AdaLoraModel.add_weighted_adapter)N)r* __module__ __qualname____doc__r target_module_mappingrrr)rK staticmethodr?rnrrrr __classcell__r&r&r$r'r%s - E*r)rbr`transformers.pytorch_utilsrpeft.import_utilsrrrpeft.tuners.lorarrpeft.tuners.tuners_utilsr peft.utilsr r r r r rpeft.utils.integrationsrr5rrrrrrr&r&r&r's