o Ni<@s\ddlmZddlZddlZddlmZmZddlmZddl m Z m Z GdddeZ dS) ) annotationsN) BaseTunerBaseTunerLayer)3TRANSFORMERS_MODELS_TO_SHIRA_TARGET_MODULES_MAPPING)Linear ShiraLayerc@s:eZdZUdZdZded<eZeZ ddZ e ddZ d S) ShiraModela Creates a Sparse High Rank Adapter (SHiRA) Model from a pretrained model. Args: model ([`~transformers.PreTrainedModel`]): The model to be adapted. config ([`ShiraConfig`]): The configuration of the SHiRA model. adapter_name (`str`): The name of the adapter, defaults to `"default"`. Returns: `torch.nn.Module`: The SHiRA model. Example: ```py >>> from transformers import AutoModelForCausalLM >>> from peft import ShiraConfig, get_peft_model >>> base_model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m") >>> config = ShiraConfig(r=32) >>> model = get_peft_model(base_model, config) ``` **Attributes**: - **model** ([`~transformers.PreTrainedModel`]) -- The model to be adapted. - **peft_config** ([`ShiraConfig`]): The configuration of the SHiRA model. shira_strprefixcKs|durtdt|do|jdu}i} || d<|jdkr"|j| d<|D]\} } | | | <q&t|trT|jdurE|j|j |j fi| nd} |j || |j |j ddS|j |||fi| } ||jvri| d|||| |dS)NzCurrent Key shouldn't be `None`biasrandom random_seed) init_weightsF) ValueErrorhasattrr mask_typeritems isinstancermask_fn base_layerr update_layerr_create_new_moduleactive_adapterrequires_grad__replace_module)self shira_config adapter_nametarget target_nameparent current_keyoptional_kwargsr kwargskvmask new_moduler+K/home/ubuntu/.local/lib/python3.10/site-packages/peft/tuners/shira/model.py_create_and_replace=s0         zShiraModel._create_and_replacec Ks|j}|dd}t|tr|}n|}t|tjjr)|r(t dd}|_nt d|d|j durA|j ||j fi|nd}t||||j |fd|j i|}|S)Nr Fzjfan_in_fan_out is set to True but the target module is `torch.nn.Linear`. Setting fan_in_fan_out to False.zTarget module zZ is not supported. Currently, only the following modules are supported: `torch.nn.Linear`.r)fan_in_fan_outpoprrget_base_layertorchnnrwarningswarnrrrr) rr r!r&r._target_base_layerr)r*r+r+r,rfs>       zShiraModel._create_new_moduleN) __name__ __module__ __qualname____doc__r __annotations__rtuner_layer_clsrtarget_module_mappingr- staticmethodrr+r+r+r,r s  )r ) __future__rr3r1peft.tuners.tuners_utilsrr peft.utilsrlayerrrr r+r+r+r,s