o NiM@sddlmZddlZddlmZddlZddlmZddlmZddl m Z dZ d Z Gd d d ej Zd#ddZd#ddZd$ddZ d%d&d!d"ZdS)') annotationsN)Any)nn)PreTrainedModel) ArrowConfigtask_gks_csleZdZdZfddZeddZddd Zed d Z ed d Z dddZ ddZ Z S)ArrowLoraLinearLayerz_ This class represent the main logic of the arrow routing algorithm for linear layers. csft||_d|_|j|_|j|_|j|_|j |_|j |_ |j |_ d|_ g|_ ||_d|_dS)NFT)super__init__ in_features _protos_readytop_krouter_temperature temperaturerng_seedtask_adapter_namescopygks_adapter_namesuse_gksgks_donegks_added_adapter_namescast_input_dtype_enabled)selfr arrow_config __class__J/home/ubuntu/.local/lib/python3.10/site-packages/peft/tuners/lora/arrow.pyr #s  zArrowLoraLinearLayer.__init__csffdd|D}tjt|krdStjt|kr)fdd|D_|_d_dS)z Called when adapters are added/removed/renamed so Arrow can refresh its internal state before the next forward pass. cs>g|]}|vr|dkr|dr|tdds|qS) arrow_routerr N) startswithlenisdigit).0klora_Brr =s .z:ArrowLoraLinearLayer.on_adapter_change..Ncsg|] }|jvr|qSr)r)r$x)rrrr(HsF)keyssortedrr"rrr)rlora_Ar'all_ts_adapter_namesr)r'rron_adapter_change7s   z&ArrowLoraLinearLayer.on_adapter_change:0yE>c Cs|tj}|tj}|j|}d}|jdur(tj|jjd}|t |jtj | d|j |j|d} | | |} t|D]} |j||| } | | |} qB| S)uw Computes the top *right* singular vector of ΔW = B @ A without forming ΔW. Theory: For any matrix M, the right singular vectors are the eigenvectors of Mᵀ M. If ΔW = B @ A (with A ∈ ℝ^{r×in}, B ∈ ℝ^{out×r}), then ΔWᵀ ΔW = (B @ A)ᵀ (B @ A) = Aᵀ (Bᵀ B) A ∈ ℝ^{in×in}. Therefore, the dominant right singular vector of ΔW is the dominant eigenvector of M := Aᵀ (Bᵀ B) A. We find it by *power iteration* on the linear operator v ↦ Aᵀ (Bᵀ B) (A v), which avoids materializing ΔW (out×in) or M (in×in). The result lives in the input/token space (size = in_features), which is exactly what Arrow needs. (Right singular vectors ≡ eigenvectors of MᵀM; power iteration converges to the dominant eigenvector under mild conditions.) =============================== Practical notes: - We perform all iteration in float32 for numerical stability, then cast back to the LoRA dtype/device before storing/using the prototype. - Convergence is checked with a simple fixed-iter cap (`iters`) and/or `allclose` tolerance (`tol`). - The returned vector is unique up to sign (±), as with any singular vector. Downstream code should be sign-invariant. N)devicer)dtyper1 generator)totorchfloat32Tr Generatorr1type manual_seedintrandnsizer2normrange) rABitersepsA32B32Cgenv_wrrrtop_right_singular_vec_from_BAPs     z3ArrowLoraLinearLayer.top_right_singular_vec_from_BAc Cs~|jrdSg}|jD] }||j}||j}|||}|j|j|jd}||q tj |dd} |j d| ddd|_dS) a  Computes a prototype vector for each LoRA module in every layer by applying Singular Value Decomposition (SVD) to the `lora_A` matrix and extracting the top right singular vector. These prototypes are later used to calculate the cosine similarity between each input token and each expert. The resulting similarity scores serve as coefficients to compute a weighted average of the corresponding LoRA modules, effectively routing each token through its most relevant experts. ** This prototype computation is done is done once for all experts and is re-done on newly added adapters.** Args: lora_A : Matrices A in LoRA layer. lora_B (optional): Matrices B in LoRA layer. Defaults to None. N)r2r1rdim prototypesF) persistentT) rrweightrKr4r2r1appendr5stackregister_buffer) rr,r'protosnamer@rAproto32proto proto_stackrrrbuild_prototypes}s      z%ArrowLoraLinearLayer.build_prototypescs|jsdS|jr |js dStjfdd|jDddd}tjfdd|jDddd}|jdurO|jD]}|jj ||jj |q9n|jD]}|jj ||jj |qRd|_g|_dS) a\ This function performs General Knowledge Subtraction. It takes an average of provided general_adapters, and subtract it from each task_adapter. This subtraction tries to purify the task adapters, based on "forgetting-via-negation" principle. Forgetting-via-negation is a task-arithmetic operation, explained in: https://huggingface.co/papers/2212.04089 The task adapters will be more focused and isolated, enhancing the performance on new tasks. Args: lora_A : Matrices A in LoRA layer. lora_B : Matrices A in LoRA layer. Ncg|]}|jqSrrPr$nr,rrr(z5ArrowLoraLinearLayer.gen_know_sub..rrLcrZrr[r\r&rrr(r_FT) rrrr5rRrmeanrrPdatasub_)rr,r'avg_Aavg_BrUr)r,r'r gen_know_subs(      z!ArrowLoraLinearLayer.gen_know_subr2 torch.dtypecCs6|durdSt|dd}|r|j|kr|S|j|dS)a| Whether to cast the dtype of the input of the forward method. Usually, we want to enable this to align the input dtype with the dtype of the weight, but by setting layer.cast_input_dtype=False, this can be disabled if necessary. Enabling or disabling can be managed via the peft.helpers.disable_lora_input_dtype_casting context manager. NrT)r2)getattrr2r4)rr)r2rrrr_cast_input_dtypes    z&ArrowLoraLinearLayer._cast_input_dtypecsn|||jdjj}|j^}}}|d|} | d|jd} } tj fdd|jD| j | jd} t | |jj } tj | |jdd\}}| | | ftd}|d||tj||jdd}tjfd d|jDdd}tjfd d|jDdd}td | |}td ||}|| ddd}td ||}||}|d}|j|g||RS)u  Applies Arrow routing inside a LoRA layer. Steps: 1. Compute cosine similarity between each token representation and all adapter prototypes. 2. Select the top-k experts per token and normalize their scores with a softmax. 3. Project tokens into each selected expert’s low-rank space (A weights). 4. Map back to the output space (B weights). 5. Aggregate expert outputs via the weighted sum of their contributions. 6. Apply dropout, scaling, and return the reshaped delta. - Conceptually, this is a Mixture-of-Experts (MoE) over LoRA adapters, where coefficients are derived from prototype similarity. Returns: delta: LoRA output adjustment computed by Arrow routing. rcsg|]}|qSrrr\)scalingrrr(sz0ArrowLoraLinearLayer.forward..)r1r2rrLz-infcrZrr[r\r^rrr(r_crZrr[r\r&rrr(r_ztf, erf -> terzter, eor -> teoz te, teo -> to)rhrrPr2shapeviewr=rNr5tensorr1absr7topkrnew_fullfloatscatter_softmaxrrReinsum)rr)r,r'dropoutrjrArestF_intoktE scales_tenssimtop_vidx full_scorecoeffA_stackB_stackzy delta_flatdeltaout_dimr)r,r'rjrforwards.  zArrowLoraLinearLayer.forward)r/r0)r2rf)__name__ __module__ __qualname____doc__r r5no_gradr.rKrYrerhr __classcell__rrrrr s   - #  (r adapter_names list[str]c Cs@d}|D]}t}d}d}|D]1\}}t|drB||jvrB|j|j} |j|j} |dd} || | jd}| j| jf}q|durN|||d}q||dkrct d|d |d |d||d krxt d|d |d |d ||d krt d|dt |dt |d qt |d } | t |dfS)z After loading all adapters into `model`, check they share: - the same LoRA rank (r) - identical weight shapes - identical sets of target_modules Returns (sorted list of target module names, agreed rank r). Nr,.rir)rshapesmodulesr[z] rank mismatch: z != rz] shape mismatch: rz-] target_modules mismatch: this adapter -> z reference -> ) set named_moduleshasattrr,rPr'splitaddrk ValueErrorr+r;) modelr referencerU curr_modulescurr_r curr_shapes full_namemoduler@rAmod_nameagreed_modulesrrr%check_loaded_lora_compatibility_arrows>          rc CsZddlm}d}z ddl}|jj}Wn tyYnwd}zddlm}Wn ty/Ynw|j|jf}|dur?||f}|durH||f}g}| D]:\}} t | dr|D].} | t | divrt | ddpnt | dd} | duru| n| } t | |s| | |t| jfqYqN|rdg} |D]\} }}| d| d |d |qtd | dS) z Validate that every module holding LoRA weights for any of `adapter_names` is Linear-like: nn.Linear, bitsandbytes.nn.Linear4bit, nn.Conv1d, or transformers.models.gpt2.modeling_gpt2.Conv1D. If not, raise. rN)Conv1Dr, base_layeroriginal_modulezzLoRA adapters must only target Linear-like layers (nn.Linear, nn.Conv1d, HF Conv1D, or bitsandbytes.nn.Linear4bit). Found:z - adapter 'z ' on module 'z ' of type  )torch.nnr bitsandbytes Linear4bit ImportError&transformers.models.gpt2.modeling_gpt2rLinearConv1drrrg isinstancerQr9r TypeErrorjoin)rrrrbnbHFConv1D allowed_types offendersrrrUbaselayer_to_checklinestnamerrr)ensure_adapters_target_linear_layers_onlyHsH         rpathstrreturntuple[str, str | None]cCstj|rtjtj|dstd|d|dfS|dd}t|dkrKd|dd}t|dkrGd|dd}||fS|dfS|dfS)aq Resolve a user-provided adapter `path` into (model_id, subfolder). Supports: - Local path to a folder that contains `adapter_config.json` - Hub path with subfolder, e.g. "user/repo/ts_expert_0[/more/...]", which becomes: model_id="user/repo", subfolder="ts_expert_0[/more/...]" - Plain Hub repo id "user/repo" (no subfolder) zadapter_config.jsonzLocal adapter path 'z)' does not contain 'adapter_config.json'.N/) osrisdirisfilerrstriprr")rpartsmodel_id subfolderrrr_resolve_adapter_sourceys   r base_modelrtask_specific_adapter_pathsrrgeneral_adapter_pathslist[str] | Noneadapter_kwargsrcKs|dus t|dkrtdddlm}m}t|d\}}td} t|} |dur3d| vr3|| d<|j|f|| d| } t dt|D]+} t| } t|| \}}t|}|durfd|vrf||d<| j d|| d|qFdd t t|D|_ |j r|dust|dkrtd t t|D]+} t | }t|| \}}t|}|durd|vr||d<| j d||d|qd d t t|D|_ng|_t| |j |jd \}}t| |j |jd ||||d }| jd|d| d| S)NrzF`task_specific_adapter_paths` should contain at least one adapter path) LoraConfig PeftModel0r)r adapter_namercSg|]}t|qSr)TASK_ADAPTER_PREFIXr$irrrr(z&create_arrow_model..zDYou should provide general LoRA paths if you want to use GenKnowSub.cSrr)GKS_ADAPTER_PREFIXrrrrr(r)r)rtarget_modulesrr )r peft_configr)r"rpeftrrrrdictfrom_pretrainedr? load_adapterrrrrrr add_adapter set_adapter)rrrrrrr model_id0sub0initial_ts_expert_name first_kwargsrrts_expert_namemidsub more_kwargsgen_expert_name gks_kwargsrr router_cfgrrrcreate_arrow_modelsx       r)rr)rrrr)N) rrrrrrrrrr) __future__rrtypingrr5r transformersrconfigrrrModuler rrrrrrrrs"       + 1