o GiӲ@s2ddlZddlmZddlmZddlmZddlmZddl Z ddl Z ddl m m ZddlmZddlmZmZmZmZmZmZdd lmZdd lmZmZdd lmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'dd l(m)Z)d dl*m+Z+d dl,m-Z-m.Z.m/Z/m0Z0d dlm1Z1e'2e3Z4dZ5dZ6GdddZ7dS)N) defaultdict) nullcontext)Path)Callable)validate_hf_hub_args)ImageProjectionIPAdapterFaceIDImageProjection"IPAdapterFaceIDPlusImageProjectionIPAdapterFullImageProjectionIPAdapterPlusImageProjectionMultiIPAdapterImageProjection)load_model_dict_into_meta)_LOW_CPU_MEM_USAGE_DEFAULTload_state_dict) USE_PEFT_BACKEND_get_model_fileconvert_unet_state_dict_to_peft deprecateget_adapter_nameget_peft_kwargsis_accelerate_availableis_peft_versionis_torch_versionlogging)empty_device_cache#_func_optionally_disable_offloading)LORA_WEIGHT_NAMELORA_WEIGHT_NAME_SAFETEXT_ENCODER_NAME UNET_NAME)AttnProcsLayersz$pytorch_custom_diffusion_weights.binz,pytorch_custom_diffusion_weights.safetensorsc @seZdZdZeZeZede e e e j fBfddZ ddZddZed d Z dd e ejBdede dedef ddZddZefddZefddZefddZddZd S)UNet2DConditionLoadersMixinz: Load LoRA layers into a [`UNet2DCondtionModel`]. %pretrained_model_name_or_path_or_dictc Ksddlm}|dd}|dd}|dd}|dd}|d d}|d d} |d d} |d d} |d d} |dd} |dd}|dd}|dt}d}|ratddratd| durid} d}ddd}d}t|ts| r{| dus| dur| drzt || pt |||||| | |d }t j j |dd}Wnty}z |s|WYd}~nd}~ww|durt || pt|||||| | |d }t|}n|}tdd|D}tdd|D}d}d}d}|rd }td!d"||r|j|d#}n|r|j||j|| ||d$\}}}nt|d%|r8|dur8|j|d&\}}}|||j|j|jd'|rA|dS|rJ|dS|rb|j D]}t|t j!j"r`||qRdSdS)(a Load pretrained attention processor layers into [`UNet2DConditionModel`]. Attention processor layers have to be defined in [`attention_processor.py`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py) and be a `torch.nn.Module` class. Currently supported: LoRA, Custom Diffusion. For LoRA, one must install `peft`: `pip install -U peft`. Parameters: pretrained_model_name_or_path_or_dict (`str` or `os.PathLike` or `dict`): Can be either: - A string, the model id (for example `google/ddpm-celebahq-256`) of a pretrained model hosted on the Hub. - A path to a directory (for example `./my_model_directory`) containing the model weights saved with [`ModelMixin.save_pretrained`]. - A [torch state dict](https://pytorch.org/tutorials/beginner/saving_loading_models.html#what-is-a-state-dict). cache_dir (`str | os.PathLike`, *optional*): Path to a directory where a downloaded pretrained model configuration is cached if the standard cache is not used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, for example, `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. local_files_only (`bool`, *optional*, defaults to `False`): Whether to only load local model weights and configuration files or not. If set to `True`, the model won't be downloaded from the Hub. token (`str` or *bool*, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, the token generated from `diffusers-cli login` (stored in `~/.huggingface`) is used. revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, a commit id, or any identifier allowed by Git. subfolder (`str`, *optional*, defaults to `""`): The subfolder location of a model file within a larger model repository on the Hub or locally. network_alphas (`dict[str, float]`): The value of the network alpha used for stable learning and preventing underflow. This value has the same meaning as the `--network_alpha` option in the kohya-ss trainer script. Refer to [this link](https://github.com/darkstorm2150/sd-scripts/blob/main/docs/train_network_README-en.md#execute-learning). adapter_name (`str`, *optional*, defaults to None): Adapter name to be used for referencing the loaded adapter model. If not specified, it will use `default_{i}` where i is the total number of adapters being loaded. weight_name (`str`, *optional*, defaults to None): Name of the serialized state dict file. low_cpu_mem_usage (`bool`, *optional*): Speed up model loading by only loading the pretrained LoRA weights and not initializing the random weights. Example: ```py from diffusers import AutoPipelineForText2Image import torch pipeline = AutoPipelineForText2Image.from_pretrained( "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16 ).to("cuda") pipeline.unet.load_attn_procs( "jbilcke-hf/sdxl-cinematic-1", weight_name="pytorch_lora_weights.safetensors", adapter_name="cinematic" ) ``` r)*_maybe_remove_and_reapply_group_offloading cache_dirNforce_downloadFproxieslocal_files_onlytokenrevision subfolder weight_nameuse_safetensors adapter_name _pipelinenetwork_alphaslow_cpu_mem_usage<=z0.13.0zq`low_cpu_mem_usage=True` is not compatible with this `peft` version. Please update it with `pip install -U peft`.Tattn_procs_weightspytorch) file_type frameworkz .safetensors) weights_namer'r(r)r*r+r,r- user_agentcpu)devicecs|]}d|vVqdS)custom_diffusionN.0kr?r?J/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/loaders/unet.py z>UNet2DConditionLoadersMixin.load_attn_procs..css"|] }d|vp |dVqdS)loraz.alphaN)endswithr@r?r?rCrDs zUsing the `load_attn_procs()` method has been deprecated and will be removed in a future version. Please use `load_lora_adapter()`.load_attn_procs0.40.0) state_dict)rJunet_identifier_keyr2r0r1r3zQ does not seem to be in the correct format expected by Custom Diffusion training.r1dtyper<)#hooks.group_offloadingr&poprr ValueError isinstancedictrGrr safetensorstorch load_fileIOErrorrranykeysallr_process_custom_diffusion _process_lora unet_name_optionally_disable_offloadingset_attn_processortorNr<enable_model_cpu_offloadenable_sequential_cpu_offload componentsvaluesnnModule)selfr%kwargsr&r'r(r)r*r+r,r-r.r/r0r1r2r3 allow_pickler: model_filerJeis_custom_diffusionis_lorais_model_cpu_offloadis_sequential_cpu_offloadis_group_offloaddeprecation_messageattn_processors componentr?r?rCrHDs D                         z+UNet2DConditionLoadersMixin.load_attn_procsc Cs@ddlm}i}tt}|D]L\}}t|dkri||<qd|vr=d|dddd|ddd}}nd|dddd|ddd}}||||<q|D]<\}} t| dkrv|ddddd ||<qa| d jd } | d jd} d | vrd nd} |d | | | d ||<|| | qa|S)Nr)CustomDiffusionAttnProcessorrto_out.F)train_kv train_q_out hidden_sizecross_attention_dimzto_k_custom_diffusion.weightrzto_q_custom_diffusion.weightT) models.attention_processorrtrrSitemslenjoinsplitshaper) rgrJrtrrcustom_diffusion_grouped_dictkeyvalueattn_processor_keysub_key value_dictr|r{rzr?r?rCr[s4   42   z5UNet2DConditionLoadersMixin._process_custom_diffusioncststdddlm}m}m} t|} fdd| Dfdd|D} |durFfdd|Dfd d|D}d } d } d }t | dkrT| n|}t |dkrSt |d ivrmtd d t |}|duryt |}i}|D]\}}d|vr|j d||<qt |||dd}d|vr|drtddrtdn tddr|dd|vr|drtddrtdn tddr|d|d(i|}durt|||\} } }i}tddr||d<|||fdi|| ||fi|}d}|durKt |dd}|r+fd d|D}|r+d!d"|d#}t |d$d}|rKfd%d|D}|rK|d&d"|d'7}|rSt|| | |fS))Nz)PEFT backend is required for this method.r) LoraConfiginject_adapter_in_modelset_peft_model_state_dictcg|] }|r|qSr? startswithr@rKr?rC 4z=UNet2DConditionLoadersMixin._process_lora..cs,i|]\}}|vr|dd|qSrvreplacerArBv)rK unet_keysr?rC 5&z=UNet2DConditionLoadersMixin._process_lora..crr?rr@rr?rCr:rcs,i|]\}}|vr|dd|qSrrr) alpha_keysrKr?rCr;rF peft_configz Adapter name z? already in use in the Unet - please select a new adapter name.lora_BrT)is_unetuse_dora=z0.13.1r3r0runexpected_keysc g|] }d|vr|vr|qSlora_r?r@r0r?rCr zSLoading adapter weights from state_dict led to unexpected keys found in the model: z, z. missing_keyscrrr?r@rr?rCrrzJLoading adapter weights from state_dict led to missing keys in the model: rvr?)rrQpeftrrrlistrYr~rgetattrrrrrrPrr^rloggerwarning)rgrJrKr2r0r1r3rrrrYunet_state_dictrnrorpstate_dict_to_be_usedrankrvallora_config_kwargs lora_config peft_kwargsincompatible_keyswarn_msgrlora_unexpected_keysrlora_missing_keysr?)r0rrKrrCr\#s                  z)UNet2DConditionLoadersMixin._process_loracCs t|dS)NrLr)clsr1r?r?rCr^s z:UNet2DConditionLoadersMixin._optionally_disable_offloadingTNsave_directoryis_main_processr. save_functionsafe_serializationc  s`ddlmmmtj|rtd|ddSt fdd|j D}|rZ| }|durY|rYdd | D} t | d krPtd | d d | D}nd } tdd| tshtdd dlm} | |}|dur|r}dd}ntj}tj|dd|dur|r|rtnt}n|rtnt}t||} ||| td| dS)az Save attention processor layers to a directory so that it can be reloaded with the [`~loaders.UNet2DConditionLoadersMixin.load_attn_procs`] method. Arguments: save_directory (`str` or `os.PathLike`): Directory to save an attention processor to (will be created if it doesn't exist). is_main_process (`bool`, *optional*, defaults to `True`): Whether the process calling this is the main process or not. Useful during distributed training and you need to call this function on all processes. In this case, set `is_main_process=True` only on the main process to avoid race conditions. save_function (`Callable`): The function to use to save the state dictionary. Useful during distributed training when you need to replace `torch.save` with another method. Can be configured with the environment variable `DIFFUSERS_SAVE_MODE`. safe_serialization (`bool`, *optional*, defaults to `True`): Whether to save the model using `safetensors` or with `pickle`. Example: ```py import torch from diffusers import DiffusionPipeline pipeline = DiffusionPipeline.from_pretrained( "CompVis/stable-diffusion-v1-4", torch_dtype=torch.float16, ).to("cuda") pipeline.unet.load_attn_procs("path-to-save-model", weight_name="pytorch_custom_diffusion_weights.bin") pipeline.unet.save_attn_procs("path-to-save-model", weight_name="pytorch_custom_diffusion_weights.bin") ``` rrtCustomDiffusionAttnProcessor2_0$CustomDiffusionXFormersAttnProcessorzProvided path (z#) should be a directory, not a fileNc3s$|] \}}t|fVqdS)NrR)rA_xrr?rCrDs z>UNet2DConditionLoadersMixin.save_attn_procs..cSs"i|] \}}t|tjs||qSr?rRrUTensorrr?r?rCr"z?UNet2DConditionLoadersMixin.save_attn_procs..rzfSafetensors does not support saving dicts with non-tensor values. The following keys will be ignored: cSs"i|] \}}t|tjr||qSr?rrr?r?rCrrzUsing the `save_attn_procs()` method has been deprecated and will be removed in a future version. Please use `save_lora_adapter()`.save_attn_procsrIzOPEFT backend is required for saving LoRAs using the `save_attn_procs()` method.)get_peft_model_state_dictcSstjj||ddidS)Nformatpt)metadata)rTrU save_file)weightsfilenamer?r?rCrszBUNet2DConditionLoadersMixin.save_attn_procs..save_functionT)exist_okzModel weights saved in )r}rtrrospathisfilererrorrXrrr~ _get_custom_diffusion_state_dictrrrYrrrQ peft.utilsrrUsavemakedirs!CUSTOM_DIFFUSION_WEIGHT_NAME_SAFEr CUSTOM_DIFFUSION_WEIGHT_NAMErras_posixinfo) rgrrr.rrrhrlrJempty_state_dictrqr save_pathr?rrCrsJ)        z+UNet2DConditionLoadersMixin.save_attn_procscslddlmmmtfdd|jD}|}|jD]\}}t|dkr3i||<q#|S)Nrrcs&i|]\}}t|fr||qSr?r)rAyrrr?rCr szPUNet2DConditionLoadersMixin._get_custom_diffusion_state_dict..r) r}rtrrr#rrr~rJr)rg model_to_saverJnameattnr?rrCrszd?} | d@dA} | dBdC} | dDdE} | dFdG} | dHdI} dJ| vr| || dKdL<qdM| vr| || dNdO<qdP| vr| jdQddR}|d|| dPdS<|d|| dPdT<qdU| vr| || dUdV<qd| kr| |dW<qdX| kr| |dY<qdZ| kr| |d[<qd\| kr | |d]<q| || <qnd^|vrr|djd}|djd}||}d^}||jd} |dZjd| }|t| |||d_}Wdn 1sQwY| D]\} } | dd} | dd} | || <qZnI|d`jd}|dajd}|dbjd}|d`jdQ}tdcdd|D}|r|dejddn|dfjdd}|t|||||dg}Wdn 1swY| D]\} } | d dh} | didj} | dkdl} | dmdn} | dodp} | dqdr} | dsdt} | dudv} | dwdx} dP| vr7| dy}dz|dQ<dy|} | jdQddR}|d|| dPdS<|d|| dPdT<qd{| vrP| dy}dz|dQ<dy|} | || <qdU| vrm| dy}dz|dQ<dy|} | || dUdV<q| d|d}} | d~d} | dd} | dd} | dd} | dd} | dd} | dd} | dd} | dd} | dd} | dd} | || <q|s|j|dd|Sd|ji}t||||jdt|S)Nrinit_empty_weightsF,Cannot initialize model with low cpu memory usage because `accelerate` was not found in the environment. Defaulting to `low_cpu_mem_usage=False`. It is strongly recommended to install `accelerate` for faster and less memory-intense model loading. You can do so with: ``` pip install accelerate ``` .Tr1.9.0~Low memory initialization requires torch >= 1.9.0. Please either update your PyTorch version or set `low_cpu_mem_usage=False`. proj.weight)r|image_embed_dimnum_image_text_embedsproj image_embeds proj.3.weightz proj.0.weight)r|rzproj.0z ff.net.0.projzproj.2zff.net.2zproj.3norm"perceiver_resampler.proj_in.weightrz#perceiver_resampler.proj_out.weightz*perceiver_resampler.layers.0.0.to_q.weight@) embed_dims output_dims hidden_dimsheadsid_embeddings_dimzperceiver_resampler.rz0.tozattn.toz0.1.0.z0.ff.0.z 0.1.1.weightz0.ff.1.net.0.proj.weightz 0.1.3.weightz0.ff.1.net.2.weightz1.1.0.z1.ff.0.z 1.1.1.weightz1.ff.1.net.0.proj.weightz 1.1.3.weightz1.ff.1.net.2.weightz2.1.0.z2.ff.0.z 2.1.1.weightz2.ff.1.net.0.proj.weightz 2.1.3.weightz2.ff.1.net.2.weightz3.1.0.z3.ff.0.z 3.1.1.weightz3.ff.1.net.0.proj.weightz 3.1.3.weightz3.ff.1.net.2.weightz layers.0.0z layers.0.ln0z layers.0.1z layers.0.ln1z layers.1.0z layers.1.ln0z layers.1.1z layers.1.ln1z layers.2.0z layers.2.ln0z layers.2.1z layers.2.ln1z layers.3.0z layers.3.ln0z layers.3.1z layers.3.ln1norm1z0.norm10norm2z0.norm21to_kvr)dimto_kto_vruzto_out.0zproj.net.0.proj.weightz proj.0.biaszproj.net.0.proj.biasz proj.2.weightzproj.net.2.weightz proj.2.biaszproj.net.2.bias norm.weight)r|rmult num_tokenslatentszproj_in.weightzproj_out.weightcsr=)rNr?r@r?r?rCrDrEzZUNet2DConditionLoadersMixin._convert_ip_adapter_image_proj_to_diffusers..zlayers.0.attn.to_q.weightzlayers.0.0.to_q.weight)rrrr num_queriesz2.toz 0.0.norm1z0.ln0z 0.0.norm2z0.ln1z 1.0.norm1z1.ln0z 1.0.norm2z1.ln1z 2.0.norm1z2.ln0z 2.0.norm2z2.ln1z 3.0.norm1z3.ln0z 3.0.norm2z3.ln1rvrto_qz0.1.0z0.ff.0z0.1.1z0.ff.1.net.0.projz0.1.3z 0.ff.1.net.2z1.1.0z1.ff.0z1.1.1z1.ff.1.net.0.projz1.1.3z 1.ff.1.net.2z2.1.0z2.ff.0z2.1.1z2.ff.1.net.0.projz2.1.3z 2.ff.1.net.2z3.1.0z3.ff.0z3.1.1z3.ff.1.net.0.projz3.1.3z 3.ff.1.net.2)strict device_maprN)r acceleraterrrrNotImplementedErrorrrrr~rr r chunkr rXr rrrr<rrNr)rgrJr3rupdated_state_dictimage_projection init_contextrclip_embeddings_dimr|rrdiffusers_namerrrrrv_chunkid_embeddings_dim_inid_embeddings_dim_out multiplier norm_layerrattn_key_presentpartsrr?r?rC+_convert_ip_adapter_image_proj_to_diffusersst                                            -                                    zGUNet2DConditionLoadersMixin._convert_ip_adapter_image_proj_to_diffusersc Csddlm}m}m}|rtrddlm}nd}td|dur*t dd s*t d i}d }|r2|nt } |j D].} | d rCdn|jj} | d rS|jjd} n+| drkt| td} tt|jj| } n| dr~t| td} |jj| } | dusd| vr|j | j}||| <q9dt|j | jvr|}n ttdr|n|}g}|D]>}d|dvr|dg7}qd|dvr|dg7}qd|dvr|dg7}qd|dvr|dg7}q||ddjd g7}q| || | d|d|| <Wdn 1swYi}t|D])\}}|d |d!|d"|d#i|d$|d!|d"|d%iq|sD|| |n tt |!j"}tt |!j#}d&|i}t$|| |||d'|d7}q9t%|S)(Nr)IPAdapterAttnProcessorIPAdapterAttnProcessor2_0IPAdapterXFormersAttnProcessorrrFrTrrrrzattn1.processor mid_blockr up_blocksz up_blocks. down_blocksz down_blocks.motion_modulesXFormersscaled_dot_product_attentionr image_projrrirrrg?)r{r|scalerzto_k_ip.z.weight ip_adapterz.to_k_ip.weightzto_v_ip.z.to_v_ip.weightrr)&r}rrrrr rrrrr rrrrYrGconfigr|rblock_out_channelsintrrreversed __class__strhasattrFr enumerateupdaternextiterrdr<rNrr)rg state_dictsr3rrrr attn_procskey_idrrr|r{block_idattn_processor_classrrJrir<rNrr?r?rC%_convert_ip_adapter_attn_to_diffuserss                $( zAUNet2DConditionLoadersMixin._convert_ip_adapter_attn_to_diffuserscCst|ts|g}|jdur|jjdkrt|ds|j|_d|_|j||d}||g}|D]}|j |d|d}| |q/t ||_d|j_|j |j |jddS)N text_projtext_encoder_hid_proj)r3r" ip_image_projrM)rRrencoder_hid_projr%encoder_hid_dim_typer+r9r7r_rappendr r`rNr<)rgr1r3r2image_projection_layersrJimage_projection_layerr?r?rC_load_ip_adapter_weights\s&      z4UNet2DConditionLoadersMixin._load_ip_adapter_weightscCsi}t|jD]\}}t|D]\}}|d|dvr||vr&i||<||d|d|d|di||d|d|d|di||d|d|d|di||d|d|d|di||d|d|d|di||d|d|d|di||d|d |d|d i||d|d |d|d iqq |S) Nz.to_k_lora.down.weightr$zunet.z.to_q_lora.down.weightz.to_v_lora.down.weightz.to_out_lora.down.weightz.to_k_lora.up.weightz.to_q_lora.up.weightz.to_v_lora.up.weightz.to_out_lora.up.weight)r-rrrYr.)rgr1 lora_dictsr3rr6rJr?r?rC_load_ip_adapter_loras|s`0z2UNet2DConditionLoadersMixin._load_ip_adapter_loras)TNNT)__name__ __module__ __qualname____doc__r!text_encoder_namer"r]rr*rSrUrrHr[r\ classmethodr^rPathLikeboolrrrrrr7r@rBr?r?r?rCr$<s@="u  f  \ b r$)8r collectionsr contextlibrpathlibrtypingrrTrUtorch.nn.functionalre functionalr,huggingface_hub.utilsrmodels.embeddingsrr r r r r models.model_loading_utilsrmodels.modeling_utilsrrutilsrrrrrrrrrrutils.torch_utilsr lora_baser lora_pipelinerr r!r"r# get_loggerrCrrrr$r?r?r?rCs*       0