o ۷i>@sdZddlZddlZddlZddlmZddlmZddlm Z m Z m Z ddl m Z eeZe r6ddlZdd Zd d Zd(d edBfd dZ d)ddZddZd*ddZddZddZd+defddZdeddfd d!Z d)d"d#Zd$d%Zd&d'Z dS),z3 PEFT utilities: Utilities related to peft library N)version)logging)is_peft_availableis_peft_versionis_torch_available)empty_device_cachec Csddlm}d}|D]}t||rt|d}nq |rTddlm}dd|D}|D]#}z |||\}}} Wn tyCYq.wt|drQt || | q.|Sddl m } | D]\} }tt|dkrpt|d} t|| rt|tjjrtjj|j|j|jd ud |jj} |j| _|jd ur|j| _d } n3t|| rt|tjjrtj|j|j|j|j|j |j!|j"|jj} |j| _|jd ur|j| _d } | rt || | ~t#q^|S) zd Recursively replace all instances of `LoraLayer` with corresponding new layers in `model`. rBaseTunerLayerF base_layer)_get_submodulescSsg|] \}}d|vr|qS)lora).0key_rrP/home/ubuntu/vllm_env/lib/python3.10/site-packages/diffusers/utils/peft_utils.py 2sz.recurse_remove_peft_layers..) LoraLayerN)biasT)$peft.tuners.tuners_utilsr modules isinstancehasattr peft.utilsr named_modulesAttributeErrorsetattrget_base_layerpeft.tuners.lorarnamed_childrenlenlistchildrenrecurse_remove_peft_layerstorchnnLinear in_features out_featuresrtoweightdeviceConv2d in_channels out_channels kernel_sizestridepaddingdilationgroupsr)modelr has_base_layer_patternmoduler key_listrparenttarget target_namernamemodule_replaced new_modulerrrr$#sv       .      r$cCs>ddlm}|dkr dS|D] }t||r||qdS)z Adjust the weightage given to the LoRA layers of the model. Args: model (`torch.nn.Module`): The model to scale. weight (`float`): The weight to be given to the LoRA layers. rr ?N)rr rr scale_layer)r5r+r r7rrrscale_lora_layersjs    rAr+cCshddlm}|dus|dkrdS|D]}t||r1|dkr%||q|jD]}||dq(qdS)a Removes the previously passed weight given to the LoRA layers of the model. Args: model (`torch.nn.Module`): The model to scale. weight (`float`, *optional*): The weight to be given to the LoRA layers. If no scale is passed the scale of the lora layer will be re-initialized to the correct value. If 0.0 is passed, we will re-initialize the scale with the correct value. rr Nr?)rr rr unscale_layeractive_adapters set_scale)r5r+r r7 adapter_namerrrunscale_lora_layers~s     rFTc s`i}i}t|dtt|dkr;t|ddttfdd| }dd| D}|durt|dkrtt|dkrt|ddttfdd| }|rudd| D}nd d| D}nt| td d | D}t d d |D} t dd |D} |||| | d} | S)Nrrc |dkSNrrx)rrr z!get_peft_kwargs..cSs i|] \}}|dd|qS)z.lora_B.rsplitrkvrrr  z#get_peft_kwargs..crGrHrrI) lora_alpharrrLrMcSs4i|]\}}d|ddddd|qS).z.lora_A.rz.alpha)joinrOreplacerPrrrrSs"cSs4i|]\}}d|ddddd|qS)rVz.down.rN)rXrOrPrrrrSs4cSsh|] }|ddqS)z.lorarrN)rr<rrr sz"get_peft_kwargs..css|]}d|vVqdS)lora_magnitude_vectorNrrrQrrr sz"get_peft_kwargs..css"|] }d|vo |dVqdS)lora_Bz.biasN)endswithr]rrrr^s )rKrU rank_pattern alpha_patterntarget_modulesuse_dora lora_bias) r"valuesr!set collectionsCounter most_commondictfilteritemspopkeysany) rank_dictnetwork_alpha_dictpeft_state_dictis_unetmodel_state_dictrErarbrcrdrelora_config_kwargsr)rUrKrget_peft_kwargss: rwcCs<ddlm}|D]}t||rdt|jSq dS)Nrr default_ default_0)rr rrr!rK)r5r r7rrrget_adapter_names   rzcCsHddlm}|D]}t||r!t|dr|j|dq | |_q dS)Nrr enable_adapters)enabled)rr rrrr{disable_adapters)r5r|r r7rrrset_adapter_layerss    r~cCsddlm}|D]}t||r t|dr||q tdq t|ddrAt|drC|j |dt |jdkrE|`d|_ dSdSdSdS)Nrr delete_adapterzdThe version of PEFT you are using is not compatible, please use a version that is greater than 0.6.1_hf_peft_config_loadedF peft_config) rr rrrr ValueErrorgetattrrrnr!r)r5rEr r7rrrdelete_adapter_layerss       rc Csvddlm}dd}|D]*\}}t||r8t|dr"||n||_t||D] \}}|||||q*qdS)Nrr cSsjt|ts|S|D] \}}||vr|Sq |d}|dd|dd|d}||d}|S)NrVrrz .attentions.r?)rrkrmrOget)weight_for_adapter module_name layer_nameweight_partsr block_weightrrrget_module_weights    z.get_module_weight set_adapter) rr rrrractive_adapterziprD) r5 adapter_namesweightsr rrr7rEr+rrr!set_weights_and_activate_adapterss    rjoint_attention_kwargs kwargs_namecsfdd}|S)a( Decorator to automatically handle LoRA layer scaling/unscaling in forward methods. This decorator extracts the `lora_scale` from the specified kwargs parameter, applies scaling before the forward pass, and ensures unscaling happens after, even if an exception occurs. Args: kwargs_name (`str`, defaults to `"joint_attention_kwargs"`): The name of the keyword argument that contains the LoRA scale. Common values include "joint_attention_kwargs", "attention_kwargs", "cross_attention_kwargs", etc. cstfdd}|S)Nc sddlm}d}|}|dur.|}||<|dd}|s.|dkr.tdd|r5t||z|g|Ri|}|W|rKt||SS|rTt||ww)Nr)USE_PEFT_BACKENDr?scalezPassing `scale` via `z1` when not using the PEFT backend is ineffective.) rWrrcopyrnloggerwarningrArF)selfargskwargsr lora_scaleattention_kwargsresult) forward_fnrrrwrapper$s*        z4apply_lora_scale..decorator..wrapper) functoolswraps)rrr)rr decorator#sz#apply_lora_scale..decoratorr)rrrrrapply_lora_scales !r min_versionreturncCs@tstdttjdt|k}|std|dS)z Checks if the version of PEFT is compatible. Args: version (`str`): The version of PEFT to check against. z@PEFT is not installed. Please install it with `pip install peft`peftz_The version of PEFT you are using is not compatible, please use a version that is greater than N)rrrparse importlibmetadata)ris_peft_version_compatiblerrrcheck_peft_versionGsrc Csddlm}|dur |}n t||||||d}t|d|vr,|dr,tddr,tdd|vr=|dr=td d r=td z|d i|WStyV} ztd | d} ~ ww)Nr) LoraConfig)rrrsrtrurErd= 0.9.0. Please upgrade.rez<=z0.13.2z2lora_bias requires PEFT >= 0.14.0. Please upgrade.z-`LoraConfig` class could not be instantiated.r)rrrw%_maybe_raise_error_for_ambiguous_keysrr TypeError) state_dictnetwork_alphasrrank_pattern_dictrtrurErrverrr_create_lora_config[s0     rcsh|d}|d}t|D]!fdd|D}fdd|D}|r1|r1tddr1tdqdS) Nrarccsg|]}|kr|qSrrrmodrrrrsz9_maybe_raise_error_for_ambiguous_keys..cs g|] }|vr|kr|qSrrrrrrrrTrz0.14.1zzThere are ambiguous keys present in this LoRA. To load it, please update your `peft` installation - `pip install -U peft`.)rr"rorr)configrarc exact_matchessubstring_matchesrrrr}s  rcsd}|dur@t|dd}|r"fdd|D}|r"dd|d}t|dd}|r@fd d|D}|r@|d d|d 7}|rIt|dSdS) NrWunexpected_keysc g|] }d|vr|vr|qSlora_rr]rErrrrTz2_maybe_warn_for_unhandled_keys..zSLoading adapter weights from state_dict led to unexpected keys found in the model: z, z. missing_keyscrrrr]rrrrrTzJLoading adapter weights from state_dict led to missing keys in the model: rV)rrXrr)incompatible_keysrEwarn_msgrlora_unexpected_keysrlora_missing_keysrrr_maybe_warn_for_unhandled_keyss,  r)N)TNN)T)r)!__doc__rhrr packagingrrWr import_utilsrrr torch_utilsr get_logger__name__rr%r$rAfloatrFrwrzr~rrstrrrrrrrrrrs4     G 2   1 "