o 8wiD@sddlZddlmZddlmZddlmZddlZddlmZddl m Z ddl m Z m Z ddl mZmZmZmZdd lmZmZmZmZd d lmZd d lmZd d lmZd dlmZd dlm Z d dl!m"Z"d dl#m$Z$d dl%m&Z&m'Z'GdddeZ(dS)N)asdict)Enum)Optional)nn)tqdm)is_bnb_4bit_availableis_bnb_available) BaseTunerBaseTunerLayercheck_target_module_exists onload_layer)2TRANSFORMERS_MODELS_TO_LORA_TARGET_MODULES_MAPPINGModulesToSaveWrapper_get_submodulesget_quantization_config) dispatch_aqlm) dispatch_awq) OFTConfig) dispatch_eetq) dispatch_gptq) dispatch_hqq) dispatch_inc)OFTLayerdispatch_defaultc s\eZdZUdZdZeed<d7deddffdd Zd e ddfd d Z e d dZ ddZ ddZdejddfddZe ddZdeffdd Zd7defddZd8dd Zd9d!d"Zd#d$Zd%d&Zfd'd(Ze d)d*Z    d:d+ed,ed-eeefd.d/Zd0eddfd1d2Z d;d+ed,ed-eeedejjfd3d4Z dejjfd5d6Z!Z"S)<OFTModelaE Creates Orthogonal Finetuning model from a pretrained model. The method is described in https://huggingface.co/papers/2306.07280 Args: model (`torch.nn.Module`): The model to which the adapter tuner layers will be attached. config ([`OFTConfig`]): The configuration of the OFT model. adapter_name (`str`): The name of the adapter, defaults to `"default"`. low_cpu_mem_usage (`bool`, `optional`, defaults to `False`): Create empty adapter weights on meta device. Useful to speed up the loading process. Returns: `torch.nn.Module`: The OFT model. Example: ```py >>> from diffusers import StableDiffusionPipeline >>> from peft import OFTModel, OFTConfig >>> config_te = OFTConfig( ... r=8, ... target_modules=["k_proj", "q_proj", "v_proj", "out_proj", "fc1", "fc2"], ... module_dropout=0.0, ... init_weights=True, ... ) >>> config_unet = OFTConfig( ... r=8, ... target_modules=[ ... "proj_in", ... "proj_out", ... "to_k", ... "to_q", ... "to_v", ... "to_out.0", ... "ff.net.0.proj", ... "ff.net.2", ... ], ... module_dropout=0.0, ... init_weights=True, ... ) >>> model = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5") >>> model.text_encoder = OFTModel(model.text_encoder, config_te, "default") >>> model.unet = OFTModel(model.unet, config_unet, "default") ``` **Attributes**: - **model** ([`~torch.nn.Module`]) -- The model to be adapted. - **peft_config** ([`OFTConfig`]): The configuration of the OFT model. oft_prefixFlow_cpu_mem_usagereturnNcstj||||ddS)N)r)super__init__)selfmodelconfig adapter_namer __class__R/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/peft/tuners/oft/model.pyr!fszOFTModel.__init__r$cCs2t|jdkr|jdkrt|jjddSdS)z A helper method to check the config when a new adapter is being added. Raise a ValueError if there is something wrong with the config or if it conflicts with existing adapters. rnonezf supports only 1 adapter with bias. When using multiple adapters, set bias to 'none' for all adapters.N)len peft_configbias ValueErrorr'__name__)r"r$r(r(r)_check_new_adapter_configis   z"OFTModel._check_new_adapter_configcCs t||SN)r ) oft_configkeyr(r(r)_check_target_module_existsxs z$OFTModel._check_target_module_existscKs$|durtd|j|j|j|j|j|j|j|j|j |j t |j ddt |j ddd }gd} | D]} t |j | d} | durE| || d<q1t|tsxt|j d rU|j jnd} |j|||fd | i|} ||jvrn| d|||| |dS|j||j|j|j|j|j|j|j|j|j d dS) NzCurrent Key shouldn't be `None`is_loaded_in_8bitFis_loaded_in_4bit) roft_block_sizemodule_dropoutcofteps block_shareuse_cayley_neumannnum_cayley_neumann_termsfan_in_fan_out init_weightsloaded_in_8bitloaded_in_4bit)gptqaqlmawq)method_quantization_config hf_device_map device_map) r7r8r9r:r;r<r=r>r@)r.r7r8r9r:r;r<r=r>r?r@getattrr#r isinstancerhasattrrH_create_new_moduleactive_adaptersrequires_grad__replace_module update_layer)r"r2r%target target_nameparent current_keyoptional_kwargskwargs quant_methods quant_methodquantization_configrI new_moduler(r(r)_create_and_replace|sN       zOFTModel._create_and_replacecst|||t|dr|j}td|D]M\}}|j|vs$d|vrdt|dr-|j}n$t|dr6|j}nt|dr?|j }nt |dddurK|j }nt | }tfdd | Dsd||jqdS) N base_layermetaranknumqweightW_qweightin_proj_weightc3s|]}|jkVqdSr1)device).0pr^r(r) sz+OFTModel._replace_module..)setattrrLr]torchrd named_modulesrr`rarbrJrcnext parametersanyto)r"rT child_namer[childnamemodulerbr(rgr)rPs&        zOFTModel._replace_moduler#cCs|D] \}}|j|vrd|_q|jD]J}|j|j}|dkr!q|dkr6|D] \}}d|vr4d|_q)q|dkrW|D]\}}t|trUt |drU|jdurUd|j_q>qt d|ddS) NFr*allr-Toft_onlyzRequested bias: z, is not implemented.) named_parametersr requires_gradrNr,r-rkrKrrLNotImplementedError)r"r#nrfactive_adapterr-rrmr(r(r) _mark_only_adapters_as_trainables,   z)OFTModel._mark_only_adapters_as_trainablec Ksg}trddlm}||trddlm}|||ttt t t t t gd}|D]}|||fd|i|}|durAnq.|durNtd|d|S)Nr)dispatch_bnb_8bit)dispatch_bnb_4bitr2zTarget module zm is not supported. Currently, only the following modules are supported: `torch.nn.Linear`, `torch.nn.Conv2d`.)rbnbr}appendrr~extendrrrrrrrr.) r2r%rRrW dispatchersr}r~r[ dispatcherr(r(r)rMs8      zOFTModel._create_new_modulerrcs8zt|WSty|dkrt|j|YSw)z1Forward missing attributes to the wrapped module.r#)r __getattr__AttributeErrorrJr#)r"rrr&r(r)rs zOFTModel.__getattr__ inferencecCsFi}|jD]\}}ddt|D}|rd|d<q|||<|S)NcSs&i|]\}}|t|tr|jn|qSr()rKrvalue)rekvr(r(r) s&z4OFTModel.get_peft_config_as_dict..Tinference_mode)r,itemsr)r"r config_dictr3rr$r(r(r)get_peft_config_as_dictsz OFTModel.get_peft_config_as_dictTcCs,|jD]}t|ttfr||qdSr1)r#modulesrKr renable_adapters)r"enabledrsr(r(r)_set_adapter_layers!s  zOFTModel._set_adapter_layerscCs|jdddS)zyEnable all adapters. Call this if you have previously disabled all adapters and want to re-enable them. TrN)rr"r(r(r)enable_adapter_layers&szOFTModel.enable_adapter_layerscCsF|jD]}|j|j}|dkrd|d}t|q|jdddS)Nr*z>Careful, disabling adapter layers with bias configured to be 'zL' does not produce the same output as the base model would without adaption.Fr)rNr,r-warningswarnr)r"rzvalmsgr(r(r)disable_adapter_layers-s    zOFTModel.disable_adapter_layerscCsF|jD]}t|tr|jrtd|||q||_ dS)aSet the active adapter(s). Additionally, this function will set the specified adapters to trainable (i.e., requires_grad=True). If this is not desired, use the following code. ```py >>> for name, param in model_peft.named_parameters(): ... if ...: # some check on name (ex. if 'lora' in name) ... param.requires_grad = False ``` Args: adapter_name (`str` or `list[str]`): Name of the adapter(s) to be activated. zJAdapter cannot be set when the model is merged. Unmerging the model first.N) r#rrKrmergedrrunmerge set_adapterrz)r"r%rsr(r(r)r8s    zOFTModel.set_adaptercs<tt|jdddkrtd|jdrtddS)zVerify that the configuration supports merging. Currently gptq quantization and replicated layers do not support merging. quantization_methodNrCz8Cannot merge OFT layers when the model is gptq quantizedlayer_replicationz=Cannot merge OFT layers when base model layers are replicated)r _check_merge_allowedrJr#r.r,getrr&r(r)rOs  zOFTModel._check_merge_allowedcCs4|jdur|dtvrtdtt|d|_|S)N model_typez0Please specify `target_modules` in `peft_config`)target_modulesr r.set)r, model_configr(r(r)_prepare_adapter_configZs   z OFTModel._prepare_adapter_config progressbar safe_merge adapter_namesc s|rfddjD}d|rdndd}t|| |dD]Y}z tj|\}} } Wn ty:Yq$wt| 6t| drV| j|||d } || | | nt| d rn|rd| j ||d  || | | Wdn1sxwYq$jS) Ncg|] \}}j|vr|qSr(rrer3_rr(r) nz9OFTModel._unload_and_optionally_merge..z Unloading z and merging r#)disabledesc"unload_and_optionally_merge_module)mergerrr])rr) rr#rkrrrr rLrrPrget_base_layer) r"rrrrkey_listrr3rTrRrSunloaded_moduler(rr)_unload_and_optionally_mergeds0     z%OFTModel._unload_and_optionally_merger%cs|tjvrtd|dj|=fddjD}d}|D] }tj|\}}}t|trE| ||durE|j dd}q%|pIg_ j ||ddS)z Deletes an existing adapter. Args: adapter_name (str): Name of the adapter to be deleted. zAdapter z does not existcrr(rrrr(r)rrz+OFTModel.delete_adapter..N)new_active_adapters) listr,keysr.r#rkrrKrdelete_adapterrNrz_delete_auxiliary_adapter)r"r%r new_adapterr3rrRr(rr)rs   zOFTModel.delete_adaptercCs|j|||dS)a This method merges the OFT layers into the base model. This is needed if someone wants to use the base model as a standalone model. Args: progressbar (`bool`): whether to show a progressbar indicating the unload and merge process safe_merge (`bool`): whether to activate the safe merging check to check if there is any potential Nan in the adapter weights adapter_names (`List[str]`, *optional*): The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults to `None`. )rrrr)r"rrrr(r(r)merge_and_unloadszOFTModel.merge_and_unloadcCs |jddS)z Gets back the base model by removing all the oft modules without merging. This gives back the original base model. F)rrrr(r(r)unloads zOFTModel.unload)F)T)rN)TFFN)FFN)#r/ __module__ __qualname____doc__rstr__annotations__boolr!rr0 staticmethodr4r\rPrModuler|rMrrrrrrrrrrrrrjrr __classcell__r(r(r&r)r0sX  3 8 +        r))r dataclassesrenumrtypingrrjrrpeft.import_utilsrrpeft.tuners.tuners_utilsr r r r peft.utilsr rrrrDrrErr$reetqrrCrhqqrincrlayerrrrr(r(r(r)s&