o Gi'@s\ddlZddlZddlmZddlmZmZmZmZddl m Z e e Z GdddZdS)N) AttentionAttentionProcessor"PAGCFGIdentitySelfAttnProcessor2_0PAGIdentitySelfAttnProcessor2_0)loggingc@seZdZdZddZddZ dddZd d Zee ffd e e e Bd e e e ffd dZedefddZedefddZedefddZedefddZedee e ffddZdS)PAGMixinzZMixin class for [Pertubed Attention Guidance](https://huggingface.co/papers/2403.17377v1).c Cs|j}|dur td|r|dn|d}t|dr|j}n|j}dtjdtfdd }d d }|D]@}g} |D]"\} } || r[t || dur[||| s[t d | | | q9t| dkritd || D]} || _qkq1dS)zA Set the attention processor for the PAG layers. NzNo PAG attention processors have been set. Set the attention processors by calling `set_pag_applied_layers` and passing the relevant parameters.runetmodulereturncSst|to|j S)zY Check if the module is self-attention module based on its name. ) isinstanceris_cross_attention)r rU/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/pipelines/pag/pag_utils.py is_self_attn4sz6PAGMixin._set_pag_attn_processor..is_self_attncSs4|dd}|dd}|o|o||kS)N.)split isnumeric)layer_idnamerrris_fake_integral_match:sz@PAGMixin._set_pag_attn_processor..is_fake_integral_matchzApplying PAG to layer: z6Cannot find PAG layer to set attention processor for: )_pag_attn_processors ValueErrorhasattrr transformernnModulebool named_modulesresearchloggerdebugappendlen processor) selfpag_applied_layersdo_classifier_free_guidancepag_attn_processors pag_attn_procmodelrrrtarget_modulesrr rrr_set_pag_attn_processor#s8   z PAGMixin._set_pag_attn_processorcCs0|jr|j|jd|}|dkrd}|S|jS)z\ Get the scale factor for the perturbed attention guidance at timestep `t`. ir)do_pag_adaptive_scaling pag_scalepag_adaptive_scale)r(t signal_scalerrr_get_pag_scaleWs zPAGMixin._get_pag_scaleFc Csj||}|r|d\}}} ||||||| }n|d\}} |||| }|r3||fS|S)a Apply perturbed attention guidance to the noise prediction. Args: noise_pred (torch.Tensor): The noise prediction tensor. do_classifier_free_guidance (bool): Whether to apply classifier-free guidance. guidance_scale (float): The scale factor for the guidance term. t (int): The current time step. return_pred_text (bool): Whether to return the text noise prediction. Returns: torch.Tensor | tuple[torch.Tensor, torch.Tensor]: The updated noise prediction tensor after applying perturbed attention guidance and the text noise prediction. r)r5chunk) r( noise_predr*guidance_scaler3return_pred_textr1noise_pred_uncondnoise_pred_textnoise_pred_perturbrrr#_apply_perturbed_attention_guidanceds   z,PAGMixin._apply_perturbed_attention_guidancecCs.tj|gddd}|rtj||gdd}|S)a Prepares the perturbed attention guidance for the PAG model. Args: cond (torch.Tensor): The conditional input tensor. uncond (torch.Tensor): The unconditional input tensor. do_classifier_free_guidance (bool): Flag indicating whether to perform classifier-free guidance. Returns: torch.Tensor: The prepared perturbed attention guidance tensor. r6r)dim)torchcat)r(conduncondr*rrr%_prepare_perturbed_attention_guidances z.PAGMixin._prepare_perturbed_attention_guidancer)r+cCst|dsd|_t|ts|g}|dur#t|trt|dkr#tdtt|D]}t||ts=tdt ||q)||_ ||_dS)a Set the self-attention layers to apply PAG. Raise ValueError if the input is invalid. Args: pag_applied_layers (`str` or `list[str]`): One or more strings identifying the layer names, or a simple regex for matching multiple layers, where PAG is to be applied. A few ways of expected usage are as follows: - Single layers specified as - "blocks.{layer_index}" - Multiple layers as a list - ["blocks.{layers_index_1}", "blocks.{layer_index_2}", ...] - Multiple layers as a block name - "mid" - Multiple layers as regex - "blocks.({layer_index_1}|{layer_index_2})" pag_attn_processors: (`tuple[AttentionProcessor, AttentionProcessor]`, defaults to `(PAGCFGIdentitySelfAttnProcessor2_0(), PAGIdentitySelfAttnProcessor2_0())`): A tuple of two attention processors. The first attention processor is for PAG with Classifier-free guidance enabled (conditional and unconditional). The second attention processor is for PAG with CFG disabled (unconditional only). rNr6z,Expected a tuple of two attention processorsz:Expected either a string or a list of string but got type ) rrr listtupler&rrangestrtyper))r(r)r+irrrset_pag_applied_layerss   zPAGMixin.set_pag_applied_layersr cC|jS)z:Get the scale factor for the perturbed attention guidance.) _pag_scaler(rrrr1zPAGMixin.pag_scalecCrL)zCGet the adaptive scale factor for the perturbed attention guidance.)_pag_adaptive_scalerNrrrr2rOzPAGMixin.pag_adaptive_scalecCs"|jdko|jdkot|jdkS)zNCheck if the adaptive scaling is enabled for the perturbed attention guidance.r)rPrMr&r)rNrrrr0s"z PAGMixin.do_pag_adaptive_scalingcCs|jdko t|jdkS)z5Check if the perturbed attention guidance is enabled.r)rMr&r)rNrrrdo_perturbed_attention_guidancesz(PAGMixin.do_perturbed_attention_guidancecCsx|jduriSdd|jD}i}t|dr|j}n t|dr#|j}ntd|jD] \}}|j|vr9|||<q,|S)z Returns: `dict` of PAG attention processors: A dictionary contains all PAG attention processors used in the model with the key as the name of the layer. NcSsh|]}|jqSr) __class__).0xrrr sz/PAGMixin.pag_attn_processors..r rzNo denoiser module found.)rrr rrattn_processorsitemsrR)r(valid_attn_processors processorsdenoiser_modulerprocrrrr+s    zPAGMixin.pag_attn_processorsN)F)__name__ __module__ __qualname____doc__r/r5r>rDrrrHrErFrrKpropertyfloatr1r2rr0rQdictr+rrrrr s24     ,r)r!r@torch.nnrmodels.attention_processorrrrrutilsr get_loggerr\r#rrrrrs