o pi{ @sddlZddlZddlZddlmZddlmZmZmZm Z m Z ddl Z ddl Z ddl mZddlmZddlmZddlmZmZddlmZmZmZmZmZmZmZmZmZm Z m!Z!eredd l"m#Z#erndd l$m%Z%er{dd l&m'Z'm(Z(m)Z)e*e+Z,d ddZ-ddZ.  d!de ee/e/fde dde e e0ee0edffddZ1d"de dfddZ2d"de dfddZ3ddZ4GdddZ5dS)#N)Path)CallableDictListOptionalUnion) model_info)HF_HUB_OFFLINE) ModelMixinload_state_dict) USE_PEFT_BACKEND_get_model_filedelete_adapter_layers deprecateis_accelerate_availableis_peft_availableis_transformers_availableloggingrecurse_remove_peft_layersset_adapter_layers!set_weights_and_activate_adapters)PreTrainedModel)BaseTunerLayer)AlignDevicesHook CpuOffloadremove_hook_from_module?FcCsd|i}|D]6}t|tr>|dkr||tt|jj}d|vr*||d<n d|vr6|dur6t d|jdi|qdS)ai Fuses LoRAs for the text encoder. Args: text_encoder (`torch.nn.Module`): The text encoder module to set the adapter layers for. If `None`, it will try to get the `text_encoder` attribute. lora_scale (`float`, defaults to 1.0): Controls how much to influence the outputs with the LoRA parameters. safe_fusing (`bool`, defaults to `False`): Whether to check fused weights for NaN values before fusing and if values are NaN not fusing them. adapter_names (`List[str]` or `str`): The names of the adapters to use. safe_merger adapter_namesNzThe `adapter_names` argument is not supported with your PEFT version. Please upgrade to the latest version of PEFT. `pip install -U peft`) modules isinstancer scale_layerlistinspect signaturemerge parameters ValueError) text_encoder lora_scale safe_fusingr merge_kwargsmodulesupported_merge_kwargsr r Y/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/loaders/lora_base.pyfuse_text_encoder_lora7s    r1cCs$|D] }t|tr|qdS)z Unfuses LoRAs for the text encoder. Args: text_encoder (`torch.nn.Module`): The text encoder module to set the adapter layers for. If `None`, it will try to get the `text_encoder` attribute. N)r!r"runmerge)r*r.r r r0unfuse_text_encoder_lora[s r3rr*rtext_encoder_weightscCsF|durtddd}t|tr|gn|}|||}t|||dS)a  Sets the adapter layers for the text encoder. Args: adapter_names (`List[str]` or `str`): The names of the adapters to use. text_encoder (`torch.nn.Module`, *optional*): The text encoder module to set the adapter layers for. If `None`, it will try to get the `text_encoder` attribute. text_encoder_weights (`List[float]`, *optional*): The weights to use for the text encoder. If `None`, the weights are set to `1.0` for all the adapters. NzrThe pipeline does not have a default `pipe.text_encoder` class. Please make sure to pass a `text_encoder` instead.cSsVt|ts |gt|}t|t|kr"tdt|dt|dd|D}|S)NLength of adapter names + is not equal to the length of the weights cSsg|] }|dur |ndqS)Nrr ).0wr r r0 zJset_adapters_for_text_encoder..process_weights..)r"r$lenr))rweightsr r r0process_weightss z6set_adapters_for_text_encoder..process_weights)r)r"strr)rr*r4r=r r r0set_adapters_for_text_encoderis r?cC |durtdt|dddS)a Disables the LoRA layers for the text encoder. Args: text_encoder (`torch.nn.Module`, *optional*): The text encoder module to disable the LoRA layers for. If `None`, it will try to get the `text_encoder` attribute. NText Encoder not found.Fenabledr)rr*r r r0disable_lora_for_text_encoder rFcCr@)a Enables the LoRA layers for the text encoder. Args: text_encoder (`torch.nn.Module`, *optional*): The text encoder module to enable the LoRA layers for. If `None`, it will try to get the `text_encoder` attribute. NrATrBrDrEr r r0enable_lora_for_text_encoderrGrHcCs*t|t|dddur|`d|_dSdS)N peft_config)rgetattrrI_hf_peft_config_loadedrEr r r0!_remove_text_encoder_monkey_patchs  rLc @seZdZdZgZdZddZeddZeddZ ed d Z ed d Z e d:ddZ ddZ gdddfdeedededeeefddZgfdeefddZ d;deeeefdeeeeeeeeffddZd d!Zd"d#Zdeeeeffd$d%Zd&eefd'd(Zd&eeeeffd)d*Zdeed+eejeefd&dfd,d-Z e!d.d/Z"e!d0eeej#fd1ed2ed3ed4e$d5ef d6d7Z%e&d&efd8d9Z'dS)< LoraBaseMixinz!Utility class for handling LoRAs.rcKtd)Nz)`load_lora_weights()` is not implemented.NotImplementedError)selfkwargsr r r0load_lora_weightsszLoraBaseMixin.load_lora_weightscKrN)Nz&`save_lora_weights()` not implemented.rOclsrRr r r0save_lora_weightszLoraBaseMixin.save_lora_weightscKrN)Nz'`lora_state_dict()` is not implemented.rOrTr r r0lora_state_dictrWzLoraBaseMixin.lora_state_dictcCsd}d}|durL|jdurL|jD]9\}}t|tjrKt|drK|s)t|jt}|s@t|jt p?t|jdo?t|jj dt }t dt ||dq||fS)ar Optionally removes offloading in case the pipeline has been already sequentially offloaded to CPU. Args: _pipeline (`DiffusionPipeline`): The pipeline to disable offloading for. Returns: tuple: A tuple indicating if `is_model_cpu_offload` or `is_sequential_cpu_offload` is True. FN_hf_hookhooksrzAccelerate hooks detected. Since you have called `load_lora_weights()`, the previous hooks will be first removed. Then the LoRA parameters will be loaded and the hooks will be applied again.)recurse) hf_device_map componentsitemsr"nnModulehasattrrYrrrZloggerinfor)rU _pipelineis_model_cpu_offloadis_sequential_cpu_offload_ componentr r r0_optionally_disable_offloadings$     z,LoraBaseMixin._optionally_disable_offloadingc  Csddlm} m}d}t|ts|r|dus|dur`|dr`z&|dur+|j|d|d}t||p0||||||| | | d }tj j |dd}Wnt tj fy_}z | sS|d}WYd}~nd}~ww|dur|durp|j|d|d}t||pu| |||||| | | d }t |}|S|}|S) NLORA_WEIGHT_NAMELORA_WEIGHT_NAME_SAFE .safetensors)file_extensionlocal_files_only) weights_name cache_dirforce_downloadproxiesrptokenrevision subfolder user_agentcpu)devicez.bin) lora_pipelinerlrmr"dictendswith_best_guess_weight_namer safetensorstorch load_fileIOErrorSafetensorErrorr )rU%pretrained_model_name_or_path_or_dict weight_nameuse_safetensorsrprrrsrtrurvrwrx allow_picklerlrm model_file state_dicter r r0_fetch_state_dictsj     zLoraBaseMixin._fetch_state_dictrnFcs,ddlmm|s trtdg}tj|rdStj|r-fddt |D}nt |j }fdd|D}t |dkrCdShdt tfd d |}tfd d |Drit tfd d |}ntfdd |Drt tfdd |}t |dkrtdd|d|d}|S)Nrjrkz>When using the offline mode, you must specify a `weight_name`.csg|] }|r|qSr r}r7fror r0r9Gs  z9LoraBaseMixin._best_guess_weight_name..csg|] }|jr|jqSr ) rfilenamer}rrr r0r9Lsr> optimizer scheduler checkpointcstfddDS)Nc3s|]}|vVqdSNr )r7 substringxr r0 UszJLoraBaseMixin._best_guess_weight_name....)allr)unallowed_substringsrr0Usz7LoraBaseMixin._best_guess_weight_name..c3|]}|VqdSrrrrlr r0rXz8LoraBaseMixin._best_guess_weight_name..c |Srrrrr r0rY c3rrrrrmr r0rZrcrrrrrr r0r[rz9Provided path contains more than one weights file in the z~ format. Either specify `weight_name` in `load_lora_weights` or make sure there's only one `.safetensors` or `.bin` file in .)r{rlrmr r)ospathisfileisdirlistdirrsiblingsr;r$filterany)rUrrorptargeted_files files_in_reporr )rlrmrorr0r~9s8      z%LoraBaseMixin._best_guess_weight_namecCsZtstd|jD]!}t||d}|dur*t|jtr |q t|jtr*t |q dS)z Unloads the LoRA parameters. Examples: ```python >>> # Assuming `pipeline` is already loaded with the LoRA parameters. >>> pipeline.unload_lora_weights() >>> ... ``` )PEFT backend is required for this method.N) r r)_lora_loadable_modulesrJ issubclass __class__r unload_lorarrLrQrhmodelr r r0unload_lora_weightsds      z!LoraBaseMixin.unload_lora_weightsrNr]r+r,rc Ksd|vr d}tdd|d|vrd}tdd|d|vr$d}tdd|t|dkr.td |D]8}||jvrBt|d |jd t||d }|d urht|jtrZ|j|||d t|jt rht ||||dq0|j d7_ d S)a Fuses the LoRA parameters into the original parameters of the corresponding blocks. This is an experimental API. Args: components: (`List[str]`): List of LoRA-injectable components to fuse the LoRAs into. lora_scale (`float`, defaults to 1.0): Controls how much to influence the outputs with the LoRA parameters. safe_fusing (`bool`, defaults to `False`): Whether to check fused weights for NaN values before fusing and if values are NaN not fusing them. adapter_names (`List[str]`, *optional*): Adapter names to be used for fusing. If nothing is passed, all active adapters will be fused. Example: ```py from diffusers import DiffusionPipeline import torch pipeline = DiffusionPipeline.from_pretrained( "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16 ).to("cuda") pipeline.load_lora_weights("nerijs/pixel-art-xl", weight_name="pixel-art-xl.safetensors", adapter_name="pixel") pipeline.fuse_lora(lora_scale=0.7) ``` fuse_unetzPassing `fuse_unet` to `fuse_lora()` is deprecated and will be ignored. Please use the `components` argument and provide a list of the components whose LoRAs are to be fused. `fuse_unet` will be removed in a future version.1.0.0fuse_transformerzPassing `fuse_transformer` to `fuse_lora()` is deprecated and will be ignored. Please use the `components` argument and provide a list of the components whose LoRAs are to be fused. `fuse_transformer` will be removed in a future version.fuse_text_encoderzPassing `fuse_text_encoder` to `fuse_lora()` is deprecated and will be ignored. Please use the `components` argument and provide a list of the components whose LoRAs are to be fused. `fuse_text_encoder` will be removed in a future version.r%`components` cannot be an empty list.- is not found in self._lora_loadable_modules=rN)r,r)r+r,rrj) rr;r)rrJrrr fuse_lorarr1num_fused_loras) rQr]r+r,rrR depr_messagefuse_componentrr r r0r{sH'     zLoraBaseMixin.fuse_loracKsd|vr d}tdd|d|vrd}tdd|d|vr$d}tdd|t|dkr.td |D]4}||jvrBt|d |jd t||d }|d urdt|jttfrd| D] }t |t rc| qXq0|j d 8_ d S)a Reverses the effect of [`pipe.fuse_lora()`](https://huggingface.co/docs/diffusers/main/en/api/loaders#diffusers.loaders.LoraBaseMixin.fuse_lora). This is an experimental API. Args: components (`List[str]`): List of LoRA-injectable components to unfuse LoRA from. unfuse_unet (`bool`, defaults to `True`): Whether to unfuse the UNet LoRA parameters. unfuse_text_encoder (`bool`, defaults to `True`): Whether to unfuse the text encoder LoRA parameters. If the text encoder wasn't monkey-patched with the LoRA parameters then it won't have any effect. unfuse_unetzPassing `unfuse_unet` to `unfuse_lora()` is deprecated and will be ignored. Please use the `components` argument. `unfuse_unet` will be removed in a future version.runfuse_transformerzPassing `unfuse_transformer` to `unfuse_lora()` is deprecated and will be ignored. Please use the `components` argument. `unfuse_transformer` will be removed in a future version.unfuse_text_encoderzPassing `unfuse_text_encoder` to `unfuse_lora()` is deprecated and will be ignored. Please use the `components` argument. `unfuse_text_encoder` will be removed in a future version.rrrrNrj)rr;r)rrJrrr rr!r"rr2r)rQr]rRrrrr.r r r0 unfuse_lorasD     zLoraBaseMixin.unfuse_loraadapter_weightsc st|tr|gn|}t|}t|ts|gt|}t|t|kr1tdt|dt||ddD}fdd|D}i}|j D]}t ||}t ||D]X\}} t| t r| |d} | dur|t||s|td|d|d | dur|||vrtd |d |d |d |d|d||d n| } ||g||| qXt|jtr||||qLt|jtrt||||qLdS)Nr5r6cSsh|] }|D]}|qqSr r )r7adaptersadapterr r r0 s z-LoraBaseMixin.set_adapters..cs$i|]fddDqS)csg|] \}}|vr|qSr r )r7partrrr r0r9r:z9LoraBaseMixin.set_adapters...)r^)r7 list_adaptersrr0 sz.LoraBaseMixin.set_adapters..zLora weight dict contains z< weights but will be ignored because pipeline does not have rzLora weight dict for adapter 'z ' contains z",but this will be ignored because z does not contain weights for z.Valid parts for z are: )r"r>copydeepcopyr$r;r)get_list_adaptersvaluesrrJzipr|poprarbwarning setdefaultappendrrr set_adaptersrr?) rQrr all_adaptersinvert_list_adapters_component_adapter_weightsrhr adapter_namer<component_adapter_weightsr rr0rs^          zLoraBaseMixin.set_adapterscCZtstd|jD]!}t||d}|dur*t|jtr |q t|jtr*t |q dSNr) r r)rrJrrr disable_lorarrFrr r r0rA     zLoraBaseMixin.disable_loracCrr) r r)rrJrrr enable_lorarrHrr r r0rMrzLoraBaseMixin.enable_loracCsxtstdt|tr|g}|jD](}t||d}|dur9t|jtr)| |qt|jt r9|D]}t ||q1qdS)a Args: Deletes the LoRA layers of `adapter_name` for the unet and text-encoder(s). adapter_names (`Union[List[str], str]`): The names of the adapter to delete. Can be a single string or a list of strings rN) r r)r"r>rrJrrr delete_adaptersrr)rQrrhrrr r r0rYs       zLoraBaseMixin.delete_adaptersreturncCsbtstdg}|jD]#}t||d}|dur.t|jtr.|D] }t|t r-|j }nq!q |S)a Gets the list of the current active adapters. Example: ```python from diffusers import DiffusionPipeline pipeline = DiffusionPipeline.from_pretrained( "stabilityai/stable-diffusion-xl-base-1.0", ).to("cuda") pipeline.load_lora_weights("CiroN2022/toy-face", weight_name="toy_face_sdxl.safetensors", adapter_name="toy") pipeline.get_active_adapters() ``` iPEFT backend is required for this method. Please install the latest version of PEFT `pip install -U peft`N) r r)rrJrrr r!r"ractive_adapters)rQrrhrr.r r r0get_active_adaptersos    z!LoraBaseMixin.get_active_adapterscCs`tstdi}|jD]"}t||d}|dur-t|jttfr-t|dr-t |j ||<q |S)zR Gets the current list of all available adapters in the pipeline. rNrI) r r)rrJrrr rrar$rIkeys)rQrrhrr r r0rs  zLoraBaseMixin.get_list_adaptersrzcCststd|jD]B}t||d}|durK|D]1}t|trJ|D]'}|j|||j ||t |drI|j durI|j |||j |<q"qq dS)a Moves the LoRAs listed in `adapter_names` to a target device. Useful for offloading the LoRA to the CPU in case you want to load multiple adapters and free some GPU memory. Args: adapter_names (`List[str]`): List of adapters to send device to. device (`Union[torch.device, str, int]`): Device to send the adapters to. Can be either a torch device, a str or an integer. rNlora_magnitude_vector) r r)rrJr!r"rlora_Atolora_Brar)rQrrzrhrr.rr r r0set_lora_devices$     zLoraBaseMixin.set_lora_devicecs4t|tjjr |n|}fdd|D}|S)Ncs i|] \}}d||qS)rr )r7 module_nameparamprefixr r0rs z.LoraBaseMixin.pack_weights..)r"rr_r`rr^)layersrlayers_weightslayers_state_dictr rr0 pack_weightsszLoraBaseMixin.pack_weightsrsave_directoryis_main_processr save_functionsafe_serializationc Csddlm}m}tj|rtd|ddS|dur'|r$dd}ntj }tj |dd|dur9|r7|}n|}t || }|||t d |dS) NrjrkzProvided path (z#) should be a directory, not a filecSstjj||ddidS)Nformatpt)metadata)rr save_file)r<filenamer r r0rsz6LoraBaseMixin.write_lora_layers..save_functionT)exist_okzModel weights saved in )r{rlrmrrrrberrorrsavemakedirsras_posixrc) rrrrrrrlrm save_pathr r r0write_lora_layerss    zLoraBaseMixin.write_lora_layerscCst|dr|jSdS)N _lora_scaler)rar)rQr r r0r+szLoraBaseMixin.lora_scale)rnFr)(__name__ __module__ __qualname____doc__rrrS classmethodrVrXrirr~rrr>floatboolrrrrrrrrrrrrrzintr staticmethodrTensorrrpropertyr+r r r r0rMsz   " L *  Q; =  !&   #rM)rFN)NNr)6rr%rpathlibrtypingrrrrrrrtorch.nnr_huggingface_hubrhuggingface_hub.constantsr models.modeling_utilsr r utilsr rrrrrrrrrr transformersrpeft.tuners.tuners_utilsraccelerate.hooksrrr get_loggerrrbr1r3r>r r?rFrHrLrMr r r r0sD    4    $ ,