o pi&@stddlZddlmZmZmZmZddlZddlmZddl m Z m Z m Z m Z ddlmZeeZGdddZdS)N)DictListTupleUnion) AttentionAttentionProcessor"PAGCFGIdentitySelfAttnProcessor2_0PAGIdentitySelfAttnProcessor2_0)loggingc@seZdZdZddZddZddZdd Zee ffd e e e e fd e eeffd d ZedefddZedefddZedefddZedefddZedee effddZdS)PAGMixinzRMixin class for [Pertubed Attention Guidance](https://arxiv.org/abs/2403.17377v1).c Cs|j}|dur td|r|dn|d}t|dr|j}n|j}dtjdtfdd }d d }|D]@}g} |D]"\} } || r[t || dur[||| s[t d | | | q9t| dkritd || D]} || _qkq1dS)zA Set the attention processor for the PAG layers. NzNo PAG attention processors have been set. Set the attention processors by calling `set_pag_applied_layers` and passing the relevant parameters.runetmodulereturncSst|to|j S)zY Check if the module is self-attention module based on its name. ) isinstanceris_cross_attention)rr_/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/pipelines/pag/pag_utils.py is_self_attn5sz6PAGMixin._set_pag_attn_processor..is_self_attncSs4|dd}|dd}|o|o||kS)N.)split isnumeric)layer_idnamerrris_fake_integral_match;sz@PAGMixin._set_pag_attn_processor..is_fake_integral_matchzApplying PAG to layer: z6Cannot find PAG layer to set attention processor for: )_pag_attn_processors ValueErrorhasattrr transformernnModulebool named_modulesresearchloggerdebugappendlen processor) selfpag_applied_layersdo_classifier_free_guidancepag_attn_processors pag_attn_procmodelrrrtarget_modulesrrrrr_set_pag_attn_processor$s8   z PAGMixin._set_pag_attn_processorcCs0|jr|j|jd|}|dkrd}|S|jS)z\ Get the scale factor for the perturbed attention guidance at timestep `t`. ir)do_pag_adaptive_scaling pag_scalepag_adaptive_scale)r,t signal_scalerrr_get_pag_scaleXs zPAGMixin._get_pag_scalec Cs`||}|r|d\}}}|||||||}|S|d\}}||||}|S)a Apply perturbed attention guidance to the noise prediction. Args: noise_pred (torch.Tensor): The noise prediction tensor. do_classifier_free_guidance (bool): Whether to apply classifier-free guidance. guidance_scale (float): The scale factor for the guidance term. t (int): The current time step. Returns: torch.Tensor: The updated noise prediction tensor after applying perturbed attention guidance. r)r9chunk) r, noise_predr.guidance_scaler7r5noise_pred_uncondnoise_pred_textnoise_pred_perturbrrr#_apply_perturbed_attention_guidancees   z,PAGMixin._apply_perturbed_attention_guidancecCs.tj|gddd}|rtj||gdd}|S)a Prepares the perturbed attention guidance for the PAG model. Args: cond (torch.Tensor): The conditional input tensor. uncond (torch.Tensor): The unconditional input tensor. do_classifier_free_guidance (bool): Flag indicating whether to perform classifier-free guidance. Returns: torch.Tensor: The prepared perturbed attention guidance tensor. r:r)dim)torchcat)r,conduncondr.rrr%_prepare_perturbed_attention_guidances z.PAGMixin._prepare_perturbed_attention_guidancer-r/cCst|dsd|_t|ts|g}|dur#t|trt|dkr#tdtt|D]}t||ts=tdt ||q)||_ ||_dS)a Set the the self-attention layers to apply PAG. Raise ValueError if the input is invalid. Args: pag_applied_layers (`str` or `List[str]`): One or more strings identifying the layer names, or a simple regex for matching multiple layers, where PAG is to be applied. A few ways of expected usage are as follows: - Single layers specified as - "blocks.{layer_index}" - Multiple layers as a list - ["blocks.{layers_index_1}", "blocks.{layer_index_2}", ...] - Multiple layers as a block name - "mid" - Multiple layers as regex - "blocks.({layer_index_1}|{layer_index_2})" pag_attn_processors: (`Tuple[AttentionProcessor, AttentionProcessor]`, defaults to `(PAGCFGIdentitySelfAttnProcessor2_0(), PAGIdentitySelfAttnProcessor2_0())`): A tuple of two attention processors. The first attention processor is for PAG with Classifier-free guidance enabled (conditional and unconditional). The second attention processor is for PAG with CFG disabled (unconditional only). rNr:z,Expected a tuple of two attention processorsz:Expected either a string or a list of string but got type ) rrrlisttupler*rrangestrtyper-)r,r-r/irrrset_pag_applied_layerss   zPAGMixin.set_pag_applied_layersrcC|jS)z:Get the scale factor for the perturbed attention guidance.) _pag_scaler,rrrr5zPAGMixin.pag_scalecCrO)zCGet the adaptive scale factor for the perturbed attention guidance.)_pag_adaptive_scalerQrrrr6rRzPAGMixin.pag_adaptive_scalecCs"|jdko|jdkot|jdkS)zNCheck if the adaptive scaling is enabled for the perturbed attention guidance.r)rSrPr*r-rQrrrr4s"z PAGMixin.do_pag_adaptive_scalingcCs|jdko t|jdkS)z5Check if the perturbed attention guidance is enabled.r)rPr*r-rQrrrdo_perturbed_attention_guidancesz(PAGMixin.do_perturbed_attention_guidancecCsx|jduriSdd|jD}i}t|dr|j}n t|dr#|j}ntd|jD] \}}|j|vr9|||<q,|S)z Returns: `dict` of PAG attention processors: A dictionary contains all PAG attention processors used in the model with the key as the name of the layer. NcSsh|]}|jqSr) __class__).0xrrr sz/PAGMixin.pag_attn_processors..rr zNo denoiser module found.)rrrr rattn_processorsitemsrU)r,valid_attn_processors processorsdenoiser_modulerprocrrrr/s    zPAGMixin.pag_attn_processorsN)__name__ __module__ __qualname____doc__r3r9rArGr r rrKrrrrNpropertyfloatr5r6r#r4rTrr/rrrrr !s04   ,r )r%typingrrrrrCtorch.nnr!models.attention_processorrrr r utilsr get_loggerr_r'r rrrrs