o GÆÏið–ã@snddlZddlZddlZddlmmZddlmZddlm Z ddl m Z m Z ddl mZ    dd ejd ed ed edededejfdd„Z    dždedeeeefBdedededejdBdedejfdd„Z  dŸdedeeeefBdedededejf dd„Z    d dejdBdefd!d"„Zd¡d#d$„Zd¢d%d&„Z d£d'd(„Zd)d*„Zd+d,„ZGd-d.„d.ej ƒZ!Gd/d0„d0ej ƒZ"Gd1d2„d2ej ƒZ#Gd3d4„d4ej ƒZ$ 5 6  d¤d7ed8ed9ed:eeefdBdejdBdejeejejfBf d;d<„Z% = d¥d>eeeefd7edejdBdejeejejfBfd?d@„Z& d¦dejdBdefdAdB„Z'd§dCdD„Z(d¨dEdF„Z)dŸdGdH„Z*dIdddd5ej+fdJedKejeBd7efdLdM„Z, 5 N d©dOejdPejeejBd8edQedRedeejejff dSdT„Z-dOejfdUdV„Z.GdWdX„dXej ƒZ/GdYdZ„dZej ƒZ0Gd[d\„d\ej ƒZ1Gd]d^„d^ej ƒZ2Gd_d`„d`ej ƒZ3Gdadb„dbej ƒZ4Gdcdd„ddej ƒZ5Gdedf„dfej ƒZ6Gdgdh„dhej ƒZ7Gdidj„djej ƒZ8Gdkdl„dlej ƒZ9Gdmdn„dnej ƒZ:Gdodp„dpej ƒZ;Gdqdr„drej ƒZGdwdx„dxej ƒZ?Gdydz„dzej ƒZ@Gd{d|„d|ej ƒZAGd}d~„d~ej ƒZBGdd€„d€ej ƒZCGdd‚„d‚ej ƒZDGdƒd„„d„ej ƒZEGd…d†„d†ej ƒZFd‡dˆ„ZGGd‰dŠ„dŠej ƒZHGd‹dŒ„dŒej ƒZIGddŽ„dŽej ƒZJGdd„dej ƒZKGd‘d’„d’ej ƒZLGd“d”„d”ej ƒZMGd•d–„d–ej ƒZNGd—d˜„d˜ej ƒZOGd™dš„dšej ƒZPGd›dœ„dœej ƒZQdS)ªéN)Únné)Ú deprecateé)ÚFP32SiLUÚget_activation)Ú AttentionFé'Ú timestepsÚ embedding_dimÚflip_sin_to_cosÚdownscale_freq_shiftÚscaleÚ max_periodÚreturnc Csöt|jƒdks Jdƒ‚|d}t |¡ tjd|tj|jd}|||}t |¡}|dd…df  ¡|ddd…f}||}tj t  |¡t  |¡gdd}|rktj |dd…|d…f|dd…d|…fgdd}|ddkrytj j |d ¡}|S) a& This matches the implementation in Denoising Diffusion Probabilistic Models: Create sinusoidal timestep embeddings. Args timesteps (torch.Tensor): a 1-D Tensor of N indices, one per batch element. These may be fractional. embedding_dim (int): the dimension of the output. flip_sin_to_cos (bool): Whether the embedding order should be `cos, sin` (if True) or `sin, cos` (if False) downscale_freq_shift (float): Controls the delta between frequencies between dimensions scale (float): Scaling factor applied to the embeddings. max_period (int): Controls the maximum frequency of the embeddings Returns torch.Tensor: an [N x dim] Tensor of positional embeddings. rzTimesteps should be a 1d-arrayrr)ÚstartÚendÚdtypeÚdeviceNéÿÿÿÿ©Údim)rrrr)ÚlenÚshapeÚmathÚlogÚtorchÚarangeÚfloat32rÚexpÚfloatÚcatÚsinÚcosrÚ functionalÚpad) r r r r rrÚhalf_dimÚexponentÚemb©r)úO/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/models/embeddings.pyÚget_timestep_embeddings ÿ  $2 r+çð?ÚnpÚ embed_dimÚ spatial_sizeÚ temporal_sizeÚspatial_interpolation_scaleÚtemporal_interpolation_scalerÚ output_typecCsh|dkr t|||||dS|ddkrtdƒ‚t|tƒr ||f}d|d}|d}tj|d|tjd|} tj|d|tjd|} tj| | d d } tj| dd } |   d d|d|dg¡} t || d d} tj||tjd|} t || d d}| ddd…dd…f} | j |d| j d|d} |dd…ddd…f}|j |d|ddd }tj|| gdd }|S)a˜ Creates 3D sinusoidal positional embeddings. Args: embed_dim (`int`): The embedding dimension of inputs. It must be divisible by 16. spatial_size (`int` or `tuple[int, int]`): The spatial dimension of positional embeddings. If an integer is provided, the same size is applied to both spatial dimensions (height and width). temporal_size (`int`): The temporal dimension of positional embeddings (number of frames). spatial_interpolation_scale (`float`, defaults to 1.0): Scale factor for spatial grid interpolation. temporal_interpolation_scale (`float`, defaults to 1.0): Scale factor for temporal grid interpolation. Returns: `torch.Tensor`: The 3D sinusoidal positional embeddings of shape `[temporal_size, spatial_size[0] * spatial_size[1], embed_dim]`. r-)r.r/r0r1r2érú"`embed_dim` must be divisible by 4ér©rrÚxy©ÚindexingrrÚpt©r3N©rÚ output_sizer)Ú_get_3d_sincos_pos_embed_npÚ ValueErrorÚ isinstanceÚintrrrÚmeshgridÚstackÚreshapeÚ!get_2d_sincos_pos_embed_from_gridÚ!get_1d_sincos_pos_embed_from_gridÚrepeat_interleaverÚconcat)r.r/r0r1r2rr3Úembed_dim_spatialÚembed_dim_temporalÚgrid_hÚgrid_wÚgridÚpos_embed_spatialÚgrid_tÚpos_embed_temporalÚ pos_embedr)r)r*Úget_3d_sincos_pos_embedPs@û   ÿÿrScCsLd}tdd|dd|ddkrtdƒ‚t|tƒr||f}d |d}|d}tj|d tjd |}tj|dtjd |} t | |¡} tj| dd } |   d d |d |dg¡} t || ƒ} tj|tjd |} t || ƒ} | tj dd…dd…f} tj | |dd } | dd…tj dd…f} tj | |d|d d d } tj| | gdd }|S)a– Creates 3D sinusoidal positional embeddings. Args: embed_dim (`int`): The embedding dimension of inputs. It must be divisible by 16. spatial_size (`int` or `tuple[int, int]`): The spatial dimension of positional embeddings. If an integer is provided, the same size is applied to both spatial dimensions (height and width). temporal_size (`int`): The temporal dimension of positional embeddings (number of frames). spatial_interpolation_scale (`float`, defaults to 1.0): Scale factor for spatial grid interpolation. temporal_interpolation_scale (`float`, defaults to 1.0): Scale factor for temporal grid interpolation. Returns: `np.ndarray`: The 3D sinusoidal positional embeddings of shape `[temporal_size, spatial_size[0] * spatial_size[1], embed_dim]`. z•`get_3d_sincos_pos_embed` uses `torch` and supports `device`. `from_numpy` is no longer required. Pass `output_type='pt' to use the new version now.úoutput_type=='np'ú0.33.0F©Ú standard_warnr4rr5r6r©r©ÚaxisrNr)rr@rArBr-rrrCrDrErFrGÚnewaxisÚrepeatÚ concatenate)r.r/r0r1r2Údeprecation_messagerJrKrLrMrNrOrPrQrRr)r)r*r?šs.ÿ      r?éc Cs|dkrd}tdd|ddt||||||dSt|tƒr!||f}tj|d|tjd |d||} tj|d |tjd |d ||} tj| | d d } tj| dd } |   dd |d |dg¡} t || |d} |r~|dkr~tj t  ||g¡| gdd } | S)aÒ Creates 2D sinusoidal positional embeddings. Args: embed_dim (`int`): The embedding dimension. grid_size (`int`): The size of the grid height and width. cls_token (`bool`, defaults to `False`): Whether or not to add a classification token. extra_tokens (`int`, defaults to `0`): The number of extra tokens to add. interpolation_scale (`float`, defaults to `1.0`): The scale of the interpolation. Returns: pos_embed (`torch.Tensor`): Shape is either `[grid_size * grid_size, embed_dim]` if not using cls_token, or `[1 + grid_size*grid_size, embed_dim]` if using cls_token r-ú•`get_2d_sincos_pos_embed` uses `torch` and supports `device`. `from_numpy` is no longer required. Pass `output_type='pt' to use the new version now.rTrUFrV)r.Ú grid_sizeÚ cls_tokenÚ extra_tokensÚinterpolation_scaleÚ base_sizerr7rr8r9rrr<) rÚget_2d_sincos_pos_embed_nprArBrrrrCrDrErFrIÚzeros) r.rarbrcrdrerr3r^rLrMrNrRr)r)r*Úget_2d_sincos_pos_embedÜsBÿú  ÿþÿ ÿþÿ rhcCs~|dkrd}tdd|ddt||dS|dd krtd ƒ‚t|d|d |d }t|d|d |d }tj||gd d }|S)aG This function generates 2D sinusoidal positional embeddings from a grid. Args: embed_dim (`int`): The embedding dimension. grid (`torch.Tensor`): Grid of positions with shape `(H * W,)`. Returns: `torch.Tensor`: The 2D sinusoidal positional embeddings with shape `(H * W, embed_dim)` r-zŸ`get_2d_sincos_pos_embed_from_grid` uses `torch` and supports `device`. `from_numpy` is no longer required. Pass `output_type='pt' to use the new version now.rTrUFrV)r.rNrrú embed_dim must be divisible by 2r<rr)rÚ$get_2d_sincos_pos_embed_from_grid_npr@rGrrI)r.rNr3r^Úemb_hÚemb_wr(r)r)r*rF s ÿþ rFc Cs|dkrd}tdd|ddt||dS|dd krtd ƒ‚|d ur.|jjd kr+tjntj}tj|d|j|d }||d}dd|}|  d¡}t  ||¡}t  |¡}t  |¡} tj || gdd} |r‚tj| d d …|dd …f| d d …d |d…fgdd} | S)aâ This function generates 1D positional embeddings from a grid. Args: embed_dim (`int`): The embedding dimension `D` pos (`torch.Tensor`): 1D tensor of positions with shape `(M,)` output_type (`str`, *optional*, defaults to `"np"`): Output type. Use `"pt"` for PyTorch tensors. flip_sin_to_cos (`bool`, *optional*, defaults to `False`): Whether to flip sine and cosine embeddings. dtype (`torch.dtype`, *optional*): Data type for frequency calculations. If `None`, defaults to `torch.float32` on MPS devices (which don't support `torch.float64`) and `torch.float64` on other devices. Returns: `torch.Tensor`: Sinusoidal positional embeddings of shape `(M, D)`. r-zŸ`get_1d_sincos_pos_embed_from_grid` uses `torch` and supports `device`. `from_numpy` is no longer required. Pass `output_type='pt' to use the new version now.rTz0.34.0FrV)r.ÚposrrriNÚmpsr7ç@r,r rrr)rÚ$get_1d_sincos_pos_embed_from_grid_npr@rÚtyperrÚfloat64rrEÚouterr"r#rIr!) r.rmr3r rr^ÚomegaÚoutÚemb_sinÚemb_cosr(r)r)r*rGAs(ÿ        :rGc CsÄt|tƒr ||f}tj|dtjd|d||}tj|dtjd|d||}t ||¡}tj|dd}| dd|d|dg¡}t||ƒ} |r`|dkr`tj t  ||g¡| gdd} | S)aÐ Creates 2D sinusoidal positional embeddings. Args: embed_dim (`int`): The embedding dimension. grid_size (`int`): The size of the grid height and width. cls_token (`bool`, defaults to `False`): Whether or not to add a classification token. extra_tokens (`int`, defaults to `0`): The number of extra tokens to add. interpolation_scale (`float`, defaults to `1.0`): The scale of the interpolation. Returns: pos_embed (`np.ndarray`): Shape is either `[grid_size * grid_size, embed_dim]` if not using cls_token, or `[1 + grid_size*grid_size, embed_dim]` if using cls_token rrXrrYr) rArBr-rrrCrDrErjr]rg) r.rarbrcrdrerLrMrNrRr)r)r*rfrs $$   rfcCsN|ddkr tdƒ‚t|d|dƒ}t|d|dƒ}tj||gdd}|S)aC This function generates 2D sinusoidal positional embeddings from a grid. Args: embed_dim (`int`): The embedding dimension. grid (`np.ndarray`): Grid of positions with shape `(H * W,)`. Returns: `np.ndarray`: The 2D sinusoidal positional embeddings with shape `(H * W, embed_dim)` rrrirrY)r@rpr-r])r.rNrkrlr(r)r)r*rj˜s rjcCs‚|ddkr tdƒ‚tj|dtjd}||d}dd|}| d¡}t d ||¡}t |¡}t |¡}tj||gd d }|S) a, This function generates 1D positional embeddings from a grid. Args: embed_dim (`int`): The embedding dimension `D` pos (`numpy.ndarray`): 1D tensor of positions with shape `(M,)` Returns: `numpy.ndarray`: Sinusoidal positional embeddings of shape `(M, D)`. rrrirXror,r rzm,d->mdrrY) r@r-rrrrEÚeinsumr"r#r])r.rmrtrurvrwr(r)r)r*rp®s      rpcsHeZdZdZ          d‡fd d „ Zd d„Zdd„Z‡ZS)Ú PatchEmbedaÍ 2D Image to Patch Embedding with support for SD3 cropping. Args: height (`int`, defaults to `224`): The height of the image. width (`int`, defaults to `224`): The width of the image. patch_size (`int`, defaults to `16`): The size of the patches. in_channels (`int`, defaults to `3`): The number of input channels. embed_dim (`int`, defaults to `768`): The output dimension of the embedding. layer_norm (`bool`, defaults to `False`): Whether or not to use layer normalization. flatten (`bool`, defaults to `True`): Whether or not to flatten the output. bias (`bool`, defaults to `True`): Whether or not to use bias. interpolation_scale (`float`, defaults to `1`): The scale of the interpolation. pos_embed_type (`str`, defaults to `"sincos"`): The type of positional embedding. pos_embed_max_size (`int`, defaults to `None`): The maximum size of the positional embedding. éàr_r6éFTrÚsincosNc stƒ ¡||||} ||_||_| |_tj||||f||d|_|r/tj|ddd|_ nd|_ ||_ |||||_ |_ |||_ | |_| rM| } nt| dƒ} | dur\d|_dS| dkr€t|| |j |jdd}| rod nd}|jd | ¡ d ¡|d dStd | ›ƒ‚)N©Ú kernel_sizeÚstrideÚbiasFgíµ ÷Æ°>)Úelementwise_affineÚepsçà?r|r;)rerdr3TrRr©Ú persistentzUnsupported pos_embed_type: )ÚsuperÚ__init__ÚflattenÚ layer_normÚpos_embed_max_sizerÚConv2dÚprojÚ LayerNormÚnormÚ patch_sizeÚheightÚwidthrerdrBrRrhÚregister_bufferr Ú unsqueezer@)Úselfrr‘rÚ in_channelsr.r‰rˆr€rdÚpos_embed_typerŠÚ num_patchesrarRr…©Ú __class__r)r*r‡Üs> ÿ   û zPatchEmbed.__init__cCsÜ|jdur tdƒ‚||j}||j}||jkr$td|›d|j›dƒ‚||jkr5td|›d|j›dƒ‚|j|d}|j|d}|j d|j|jd ¡}|dd…|||…|||…dd…f}| dd |jd ¡}|S) z2Crops positional embeddings for SD3 compatibility.Nz.`pos_embed_max_size` must be set for cropping.zHeight (z/) cannot be greater than `pos_embed_max_size`: Ú.zWidth (rrr)rŠr@rrRrEr)r”rr‘ÚtopÚleftÚspatial_pos_embedr)r)r*Úcropped_pos_embeds$    ÿ ÿ(zPatchEmbed.cropped_pos_embedcCsü|jdur|jdd…\}}n|jd|j|jd|j}}| |¡}|jr1| d¡ dd¡}|jr9| |¡}|jdurD|  |j ¡S|jrN|  ||¡}n(|j |ksX|j |krst|jjd||f|j|j|jdd}| ¡ d¡}n|j}||  |j ¡S)Néþÿÿÿrrrr;)r.rarerdrr3r)rŠrrrŒrˆÚ transposer‰rŽrRÚtorržrr‘rhrerdrr r“)r”Úlatentrr‘rRr)r)r*Úforward*s0 "     úzPatchEmbed.forward) rzrzr_r6r{FTTrr|N)Ú__name__Ú __module__Ú __qualname__Ú__doc__r‡ržr£Ú __classcell__r)r)r˜r*ryÊs ô7rycs*eZdZdZd ‡fdd„ Zdd „Z‡ZS) ÚLuminaPatchEmbedaz 2D Image to Patch Embedding with support for Lumina-T2X Args: patch_size (`int`, defaults to `2`): The size of the patches. in_channels (`int`, defaults to `4`): The number of input channels. embed_dim (`int`, defaults to `768`): The output dimension of the embedding. bias (`bool`, defaults to `True`): Whether or not to use bias. rr4r{Tcs.tƒ ¡||_tj|||||d|_dS)N©Ú in_featuresÚ out_featuresr€)r†r‡rrÚLinearrŒ)r”rr•r.r€r˜r)r*r‡Us   ýzLuminaPatchEmbed.__init__c CsÖ| |dj¡}|j}}| ¡\}}}}||||} } | ||| || |¡ dddddd¡}| d¡}| |¡}| dd¡}tj |j d|j dtj |jd} || ||fg||d| …d| …f dd¡  d¡fS) aÉ Patchifies and embeds the input tensor(s). Args: x (list[torch.Tensor] | torch.Tensor): The input tensor(s) to be patchified and embedded. Returns: tuple[torch.Tensor, torch.Tensor, list[tuple[int, int]], torch.Tensor]: A tuple containing the patchified and embedded tensor(s), the mask indicating the valid patches, the original image size(s), and the frequency tensor(s). rrr4rr6é©rrN) r¡rrÚsizeÚviewÚpermuterˆrŒrÚonesrÚint32r“) r”ÚxÚ freqs_cisÚ patch_heightÚ patch_widthÚ batch_sizeÚchannelrr‘Ú height_tokensÚ width_tokensÚmaskr)r)r*r£^s   ÿ   "  üzLuminaPatchEmbed.forward)rr4r{T©r¤r¥r¦r§r‡r£r¨r)r)r˜r*r©Js  r©c!sÂeZdZ          d'dededBdedededededededededededededdf ‡fdd„ Z d(dededed ejdBdej f d!d"„Z d#ej d$ej fd%d&„Z ‡Z S))ÚCogVideoXPatchEmbedrNr_é€éTéZé<é1r4éâçþ?r,rÚ patch_size_tr•r.Útext_embed_dimr€Ú sample_widthÚ sample_heightÚ sample_framesÚtemporal_compression_ratioÚmax_text_seq_lengthr1r2Úuse_positional_embeddingsÚ!use_learned_positional_embeddingsrcsÐtƒ ¡||_||_||_||_||_| |_| |_| |_ | |_ | |_ ||_ ||_ |dur;tj||||f||d|_n t |||||¡|_t ||¡|_|sS|rf|}| ||| ¡}|jd||ddSdS)Nr}Ú pos_embeddingr„)r†r‡rrÇr.rÊrÉrËrÌrÍr1r2rÎrÏrr‹rŒr­Ú text_projÚ_get_positional_embeddingsr’)r”rrÇr•r.rÈr€rÉrÊrËrÌrÍr1r2rÎrÏr…rÐr˜r)r*r‡s0  ÿýzCogVideoXPatchEmbed.__init__rc Csš||j}||j}|d|jd}|||}t|j||f||j|j|dd} |  dd¡} | jd|j||jdd} | j dd…|jd…f  | ¡| S)Nrr;)rr3rF©Ú requires_grad) rrÌrSr.r1r2rˆÚ new_zerosrÍÚdataÚcopy_) r”rÊrÉrËrÚpost_patch_heightÚpost_patch_widthÚpost_time_compression_framesr—rÐÚjoint_pos_embeddingr)r)r*rÒ²s&   ù ÿz.CogVideoXPatchEmbed._get_positional_embeddingsÚ text_embedsÚ image_embedsc Csš| |¡}|j\}}}}}|jdur>| d|||¡}| |¡}|j||g|jdd…¢RŽ}| d¡ dd¡}| dd¡}n:|j}|j} |  ddddd¡}| ||| | |||||||¡}|  ddddd ddd ¡ dd ¡ dd¡}| |¡}t j ||gdd   ¡} |j s‰|jrË|jrš|j|ks–|j|krštd ƒ‚|d|jd} |j|ks²|j|ks²|j| kr½|j||| | jd } n|j} | j| jd} | | } | S)a7 Args: text_embeds (`torch.Tensor`): Input text embeddings. Expected shape: (batch_size, seq_length, embedding_dim). image_embeds (`torch.Tensor`): Input image embeddings. Expected shape: (batch_size, num_frames, channels, height, width). Nrrr6rrr4r®ééraIt is currently not possible to generate videos at a different resolution that the defaults. This should only be the case with 'THUDM/CogVideoX-5b-I2V'.If you think this is incorrect, please open an issue at https://github.com/huggingface/diffusers/issues.©rrX)rÑrrÇrErŒr±rˆr rr²rr!Ú contiguousrÎrÏrÉrÊr@rÌrËrÒrrÐr¡r) r”rÜrÝr¹Ú num_framesÚchannelsrr‘ÚpÚp_tÚembedsÚpre_time_compression_framesrÐr)r)r*r£ËsJ   ÿ( ÿþ ÿ    ÿzCogVideoXPatchEmbed.forward)rNr_rÀrÁTrÂrÃrÄr4rÅrÆr,TT©N) r¤r¥r¦rBÚboolr r‡rrÚTensorrÒr£r¨r)r)r˜r*r¿€s|ðþýüûúùø ÷ ö õ ô óòñðï2ÿÿÿÿÿ þr¿c sZeZdZ     ddededed ed ef ‡fd d „ Zd ejdejdejfdd„Z‡ZS)ÚCogView3PlusPatchEmbedr_é rrÁé€r•Ú hidden_sizerÚtext_hidden_sizerŠcs‚tƒ ¡||_||_||_||_||_t ||d|¡|_ t ||¡|_ t |||dd}|  |||¡}|j d| ¡dddS)Nrr;)rer3rRFr„)r†r‡r•rîrrïrŠrr­rŒrÑrhrEr’r )r”r•rîrrïrŠrRr˜r)r*r‡s ÿzCogView3PlusPatchEmbed.__init__Ú hidden_statesÚencoder_hidden_statesrc Cs&|j\}}}}||jdks||jdkrtdƒ‚||j}||j}| ||||j||j¡}| dddddd¡ ¡}| |||||j|j¡}| |¡}| |¡}tj ||gdd}|jd}|j d|…d|…f  ||d ¡}tj ||j f|j|jd } tj | |gddd } ||  |j¡S) Nrz0Height and width must be divisible by patch sizerr4rr6r®rrr¯)N.)rrr@r±r²rárŒrÑrr!rRrErgrîrrr¡) r”rðrñr¹rºrr‘Ú text_lengthÚimage_pos_embedÚtext_pos_embedrRr)r)r*r£!s$     "ÿzCogView3PlusPatchEmbed.forward)r_rìrrÁrí© r¤r¥r¦rBr‡rrêr£r¨r)r)r˜r*rës$úþýüûú$rëTÚlinspaceÚthetaÚuse_realÚ grid_typeÚmax_sizec sø|durtdƒ‚|dkrY|\} } |\‰‰tj| d| dˆdˆˆ|tjd} tj| d| dˆdˆˆ|tjd} tjˆ|tjd} tjdˆˆdˆˆ|tjd} n,|dkr|\}}|\‰‰tj||tjd} tj||tjd} tjˆ|tjd} ntdƒ‚|d }|d d }|d d }t|| |dd }t|| |dd }t|| |dd }‡‡‡fd d„}|\}}|\}}|\}}|dkrì|dˆ…|dˆ…}}|dˆ…|dˆ…}}|dˆ…|dˆ…}}||||ƒ}||||ƒ}||fS)aÏ RoPE for video tokens with 3D structure. Args: embed_dim: (`int`): The embedding dimension size, corresponding to hidden_size_head. crops_coords (`tuple[int]`): The top-left and bottom-right coordinates of the crop. grid_size (`tuple[int]`): The grid size of the spatial positional embedding (height, width). temporal_size (`int`): The size of the temporal dimension. theta (`float`): Scaling factor for frequency computation. grid_type (`str`): Whether to use "linspace" or "slice" to compute grids. Returns: `torch.Tensor`: positional embedding with shape `(temporal_size * grid_size[0] * grid_size[1], embed_dim/2)`. TzJ `use_real = False` is not currently supported for get_3d_rotary_pos_embedrörrr7Úslicez%Invalid value passed for `grid_type`.r4ér6)r÷røcs˜|dd…dddd…f dˆˆd¡}|ddd…ddd…f ˆdˆd¡}|dddd…dd…f ˆˆdd¡}tj|||gdd}| ˆˆˆd¡}|S)Nrr)Úexpandrr!r±)Úfreqs_tÚfreqs_hÚfreqs_wÚfreqs©Ú grid_size_hÚ grid_size_wr0r)r*Úcombine_time_height_width‚s ÿÿÿ ÿ ÿz:get_3d_rotary_pos_embed..combine_time_height_widthN)r@rrörrÚget_1d_rotary_pos_embed)r.Ú crops_coordsrar0r÷rørùrúrrÚstoprLrMrPÚmax_hÚmax_wÚdim_tÚdim_hÚdim_wrþrÿrrÚt_cosÚt_sinÚh_cosÚh_sinÚw_cosÚw_sinr#r"r)rr*Úget_3d_rotary_pos_embed>sN ÿ ÿÿ    r©r,r,r,rdcCsú|\}}|\} } |\} } } tjd||d|||tjd}tj|d|d| d| | |tjd}tj|d|d| d| | |tjd}|d}|d}|d}t||| |ddd}t||| |ddd}t||| |ddd}||||||fS)Nrrr7r6TF)r÷røÚrepeat_interleave_real)rrörr)r.rrar0rdr÷rrrrrÚinterpolation_scale_tÚinterpolation_scale_hÚinterpolation_scale_wrPrLrMr r r rþrÿrr)r)r*Úget_3d_rotary_pos_embed_allegro£s2  ÿ ÿ ÿÿÿÿrc Csì|dkrd}tdd|ddt||||dS|\}}tj|d|d|dd |d|d|tjd } tj|d |d |d d |d |d |tjd } tj| | d d } tj| dd } |  dd g| jd d…¢¡} t || |d} | S)ak RoPE for image tokens with 2d structure. Args: embed_dim: (`int`): The embedding dimension size crops_coords (`tuple[int]`) The top-left and bottom-right coordinates of the crop. grid_size (`tuple[int]`): The grid size of the positional embedding. use_real (`bool`): If True, return real part and imaginary part separately. Otherwise, return complex numbers. device: (`torch.device`, **optional**): The device used to create tensors. Returns: `torch.Tensor`: positional embedding with shape `( grid_size * grid_size, embed_dim/2)`. r-r`rTrUFrV)r.rrarørrr7r8r9rrN©rø) rÚ_get_2d_rotary_pos_embed_nprrörrCrDrErÚ!get_2d_rotary_pos_embed_from_grid) r.rrarørr3r^rrrLrMrNrRr)r)r*Úget_2d_rotary_pos_embedÎs,ÿü,ÿ,ÿrc Cs”|\}}tj|d|d|ddtjd}tj|d|d|ddtjd}t ||¡}tj|dd}| ddg|jdd…¢¡}t|||d} | S) a RoPE for image tokens with 2d structure. Args: embed_dim: (`int`): The embedding dimension size crops_coords (`tuple[int]`) The top-left and bottom-right coordinates of the crop. grid_size (`tuple[int]`): The grid size of the positional embedding. use_real (`bool`): If True, return real part and imaginary part separately. Otherwise, return complex numbers. Returns: `torch.Tensor`: positional embedding with shape `( grid_size * grid_size, embed_dim/2)`. rF)ÚendpointrrrYrNr)r-rörrCrDrErr) r.rrarørrrLrMrNrRr)r)r*rs"" rcCsž|ddksJ‚t|d|d d¡|d}t|d|d d¡|d}|rDtj|d|dgdd}tj|d|dgdd}||fStj||gdd}|S)a½ Get 2D RoPE from grid. Args: embed_dim: (`int`): The embedding dimension size, corresponding to hidden_size_head. grid (`np.ndarray`): The grid of the positional embedding. use_real (`bool`): If True, return real part and imaginary part separately. Otherwise, return complex numbers. Returns: `torch.Tensor`: positional embedding with shape `( grid_size * grid_size, embed_dim/2)`. r4rrrrrr)rrErr!)r.rNrørkrlr#r"r(r)r)r*rsÿÿrcCs”|ddksJ‚t|d|||d}t|d|||d}| |d|dd¡ d|dd¡}| d||dd¡ |ddd¡}tj||gdd d¡}|S)as Get 2D RoPE from grid. Args: embed_dim: (`int`): The embedding dimension size, corresponding to hidden_size_head. grid (`np.ndarray`): The grid of the positional embedding. linear_factor (`float`): The linear factor of the positional embedding, which is used to scale the positional embedding in the linear layer. ntk_factor (`float`): The ntk factor of the positional embedding, which is used to scale the positional embedding in the ntk layer. Returns: `torch.Tensor`: positional embedding with shape `( grid_size * grid_size, embed_dim/2)`. r4rr)Ú linear_factorÚ ntk_factorrrr)rr±r\rr!rˆ)r.Úlen_hÚlen_wr r!rkrlr(r)r)r*Úget_2d_rotary_pos_embed_lumina>s ÿ ÿ  r$çˆÃ@rrmc Cs>|ddksJ‚t|tƒrt |¡}t|tjƒrt |¡}||}d|tjd|d||jd||}t ||¡}|jj dk} | rF|  ¡}|rp|rp|  ¡j dd|j ddd  ¡} | ¡j dd|j ddd  ¡} | | fS|r”tj|  ¡|  ¡gdd   ¡} tj| ¡| ¡gdd   ¡} | | fSt t |¡|¡} | S) a Precompute the frequency tensor for complex exponentials (cis) with given dimensions. This function calculates a frequency tensor with complex exponentials using the given dimension 'dim' and the end index 'end'. The 'theta' parameter scales the frequencies. The returned tensor contains complex values in complex64 data type. Args: dim (`int`): Dimension of the frequency tensor. pos (`np.ndarray` or `int`): Position indices for the frequency tensor. [S] or scalar theta (`float`, *optional*, defaults to 10000.0): Scaling factor for frequency computation. Defaults to 10000.0. use_real (`bool`, *optional*): If True, return real part and imaginary part separately. Otherwise, return complex numbers. linear_factor (`float`, *optional*, defaults to 1.0): Scaling factor for the context extrapolation. Defaults to 1.0. ntk_factor (`float`, *optional*, defaults to 1.0): Scaling factor for the NTK-Aware RoPE. Defaults to 1.0. repeat_interleave_real (`bool`, *optional*, defaults to `True`): If `True` and `use_real`, real part and imaginary part are each interleaved with themselves to reach `dim`. Otherwise, they are concateanted with themselves. freqs_dtype (`torch.float32` or `torch.float64`, *optional*, defaults to `torch.float32`): the dtype of the frequency tensor. Returns: `torch.Tensor`: Precomputed frequency tensor with complex exponentials. [S, D/2] rrr,r¯Únpurr=rr)rArBrrr-ÚndarrayÚ from_numpyrrsrqr r#rHrr"r!ÚpolarÚ ones_like) rrmr÷rør r!rÚ freqs_dtyperÚis_npuÚ freqs_cosÚ freqs_sinr¶r)r)r*r_s,$    $ÿ  ""rrrµr¶Úuse_real_unbind_dimÚ sequence_dimc CsÌ|r¼|\}}|dkr#|dddd…dd…f}|dddd…dd…f}n%|dkr@|ddd…ddd…f}|ddd…ddd…f}ntd|›dƒ‚| |j¡| |j¡}}|dkr}|jg|jdd…¢d‘d‘RŽ d¡\}}tj| |gdd d¡} n-|d kr¢|jg|jdd…¢d‘d‘RŽ d ¡\}}tj | |gdd} ntd |›d ƒ‚|  ¡||   ¡| |j ¡} | St  |  ¡jg|jdd…¢d‘d‘RŽ¡} |  d¡}t | |¡ d¡} |  |¡S) a3 Apply rotary embeddings to input tensors using the given frequency tensor. This function applies rotary embeddings to the given query or key 'x' tensors using the provided frequency tensor 'freqs_cis'. The input tensors are reshaped as complex numbers, and the frequency tensor is reshaped for broadcasting compatibility. The resulting tensors contain rotary embeddings and are returned as real tensors. Args: x (`torch.Tensor`): Query or key tensor to apply rotary embeddings. [B, H, S, D] xk (torch.Tensor): Key tensor to apply freqs_cis (`tuple[torch.Tensor]`): Precomputed frequency tensor for complex exponentials. ([S, D], [S, D],) Returns: tuple[torch.Tensor, torch.Tensor]: tuple of modified query tensor and key tensor with rotary embeddings. rNrz`sequence_dim=z` but should be 1 or 2.rrr6rŸz`use_real_unbind_dim=z` but should be -1 or -2.)r@r¡rrErÚunbindrrDrˆr!r rÚview_as_complexr“Ú view_as_realÚtype_as) rµr¶rør/r0r#r"Úx_realÚx_imagÚ x_rotatedruÚx_outr)r)r*Úapply_rotary_emb¢s.,, ,  r9c Cs€dd„}|\\}}\}}\}} |jddd\} } } || |d||ƒ} || |d||ƒ} || |d|| ƒ} tj| | | gdd}|S) NcSs²t ||¡dd…ddd…dd…f}t ||¡dd…ddd…dd…f}|dd|jdd…f|d|jddd…f}}tj| |fdd}| ¡|| ¡| |j¡S)N.rrr)ÚFÚ embeddingrrr!r r¡r)Útokensrmr#r"Úx1Úx2Útokens_rotatedr)r)r*Ú apply_1d_ropeÝs $$6 z/apply_rotary_emb_allegro..apply_1d_roper6rrrrr)Úchunkrr!) rµr¶Ú positionsr@rrrrrrÚtÚhÚwr)r)r*Úapply_rotary_emb_allegroÛsrFc sLeZdZ     d dedededededBf ‡fd d „ Zdd d „Z‡ZS)ÚTimestepEmbeddingÚsiluNTr•Útime_embed_dimÚact_fnÚout_dimÚ post_act_fnc sˆtƒ ¡t |||¡|_|durtj||dd|_nd|_t|ƒ|_|dur*|}n|}t |||¡|_|dur=d|_ dSt|ƒ|_ dS)NF©r€) r†r‡rr­Úlinear_1Ú cond_projrÚactÚlinear_2Úpost_act) r”r•rIrJrKrLÚ cond_proj_dimÚsample_proj_biasÚtime_embed_dim_outr˜r)r*r‡îs   zTimestepEmbedding.__init__cCsV|dur || |¡}| |¡}|jdur| |¡}| |¡}|jdur)| |¡}|Srè)rOrNrPrQrR)r”ÚsampleÚ conditionr)r)r*r£s      zTimestepEmbedding.forward)rHNNNTrè)r¤r¥r¦rBÚstrr‡r£r¨r)r)r˜r*rGís$øþýüûú rGc sFeZdZd dedededef‡fdd„ Zdejd ejfd d „Z ‡Z S) Ú TimestepsrÚ num_channelsr r rcs&tƒ ¡||_||_||_||_dSrè)r†r‡rZr r r)r”rZr r rr˜r)r*r‡s  zTimesteps.__init__r rcCst||j|j|j|jd}|S)N)r r r)r+rZr r r)r”r Út_embr)r)r*r£%sûzTimesteps.forward)r) r¤r¥r¦rBrér r‡rrêr£r¨r)r)r˜r*rYs rYcs6eZdZdZ d dedef‡fdd „ Zd d „Z‡ZS) ÚGaussianFourierProjectionz-Gaussian Fourier embeddings for noise levels.ér,TFÚembedding_sizercsftƒ ¡tjt |¡|dd|_||_||_|r1|`tjt |¡|dd|_ |j |_|` dSdS)NFrÓ) r†r‡rÚ ParameterrÚrandnÚweightrr ÚW)r”r^rÚset_W_to_weightrr r˜r)r*r‡3s ûz"GaussianFourierProjection.__init__cCs†|jrt |¡}|dd…df|jddd…fdtj}|jr2tjt |¡t |¡gdd}|Stjt |¡t |¡gdd}|S)Nrrr) rrrar-Úpir r!r#r")r”rµÚx_projrur)r)r*r£Bs ,ÿz!GaussianFourierProjection.forward)r]r,TTF) r¤r¥r¦r§rBr r‡r£r¨r)r)r˜r*r\0sÿÿÿr\cs4eZdZdZd dedef‡fdd„ Zdd„Z‡ZS) ÚSinusoidalPositionalEmbeddinga[Apply positional information to a sequence of embeddings. Takes in a sequence of embeddings with shape (batch_size, seq_length, embed_dim) and adds positional embeddings to them Args: embed_dim: (int): Dimension of the positional embedding. max_seq_length: Maximum sequence length to apply positional embeddings é r.Úmax_seq_lengthcs tƒ ¡t |¡ d¡}t t d|d¡t d¡ |¡}t d||¡}t  ||¡|ddd…ddd…f<t  ||¡|ddd…ddd…f<|  d|¡dS)Nrrrr%Úpe) r†r‡rrr“rrrrgr"r#r’)r”r.rhÚpositionÚdiv_termrir˜r)r*r‡[s $""z&SinusoidalPositionalEmbedding.__init__cCs*|j\}}}||jdd…d|…f}|Srè)rri)r”rµÚ_Ú seq_lengthr)r)r*r£ds z%SinusoidalPositionalEmbedding.forward)rg©r¤r¥r¦r§rBr‡r£r¨r)r)r˜r*rfOs  rfcs:eZdZdZdedededef‡fdd„ Zdd „Z‡ZS) ÚImagePositionalEmbeddingsa• Converts latent image classes into vector embeddings. Sums the vector embeddings with positional embeddings for the height and width of the latent space. For more details, see figure 10 of the dall-e paper: https://huggingface.co/papers/2102.12092 For VQ-diffusion: Output vector embeddings are used as input for the transformer. Note that the vector embeddings for the transformer are different than the vector embeddings from the VQVAE. Args: num_embed (`int`): Number of embeddings for the latent pixels embeddings. height (`int`): Height of the latent image i.e. the number of height embeddings. width (`int`): Width of the latent image i.e. the number of width embeddings. embed_dim (`int`): Dimension of the produced vector embeddings. Used for the latent pixel, height, and width embeddings. Ú num_embedrr‘r.csVtƒ ¡||_||_||_||_t |j|¡|_t |j|¡|_ t |j|¡|_ dSrè) r†r‡rr‘rpr.rÚ Embeddingr(Ú height_embÚ width_emb)r”rprr‘r.r˜r)r*r‡‚s z"ImagePositionalEmbeddings.__init__cCs¨| |¡}| tj|j|jd d|j¡¡}| d¡}| tj|j |jd d|j ¡¡}| d¡}||}| d|j|j d¡}||dd…d|j d…dd…f}|S)Nràrrr) r(rrrrrrr±r“rsr‘r)r”Úindexr(rrrsÚpos_embr)r)r*r£”s " " $z!ImagePositionalEmbeddings.forwardrnr)r)r˜r*rojsþýüûrocs<eZdZdZ‡fdd„Zd dd„Zd dejfdd „Z‡Z S) ÚLabelEmbeddinga7 Embeds class labels into vector representations. Also handles label dropout for classifier-free guidance. Args: num_classes (`int`): The number of classes. hidden_size (`int`): The size of the vector embeddings. dropout_prob (`float`): The probability of dropping a label. cs4tƒ ¡|dk}t |||¡|_||_||_dS©Nr)r†r‡rrqÚembedding_tableÚ num_classesÚ dropout_prob)r”ryrîrzÚuse_cfg_embeddingr˜r)r*r‡µs  zLabelEmbedding.__init__NcCsH|durtj|jd|jd|jk}nt |dk¡}t ||j|¡}|S)zB Drops labels to enable classifier-free guidance. Nrràr)rÚrandrrrzÚtensorÚwherery)r”ÚlabelsÚforce_drop_idsÚdrop_idsr)r)r*Ú token_drop¼s zLabelEmbedding.token_droprcCs6|jdk}|jr |s|dur| ||¡}| |¡}|Srw)rzÚtrainingr‚rx)r”rr€Ú use_dropoutÚ embeddingsr)r)r*r£Çs   zLabelEmbedding.forwardrè) r¤r¥r¦r§r‡r‚rÚ LongTensorr£r¨r)r)r˜r*rv«s   rvc sNeZdZ    ddedededef‡fdd „ Zd ejd ejfd d „Z‡ZS)ÚTextImageProjectionér{é rÈÚimage_embed_dimÚcross_attention_dimÚnum_image_text_embedscs6tƒ ¡||_t ||j|¡|_t ||¡|_dSrè)r†r‡rŒrr­rÝrÑ)r”rÈrŠr‹rŒr˜r)r*r‡Ðs zTextImageProjection.__init__rÜrÝcCs@|jd}| |¡}| ||jd¡}| |¡}tj||gddS)Nrrrr)rrÝrErŒrÑrr!)r”rÜrÝr¹Úimage_text_embedsr)r)r*r£Ýs   zTextImageProjection.forward)rˆr{r{r‰rõr)r)r˜r*r‡Ïsûþýüû r‡csBeZdZ   d dededef‡fdd„ Zdejfd d „Z‡ZS) ÚImageProjectionr{rgrŠr‹rŒcs4tƒ ¡||_t ||j|¡|_t |¡|_dSrè)r†r‡rŒrr­rÝrrŽ)r”rŠr‹rŒr˜r)r*r‡ës zImageProjection.__init__rÝcCs>|jd}| | |jjj¡¡}| ||jd¡}| |¡}|S)Nrr)rrÝr¡rarrErŒrŽ)r”rÝr¹r)r)r*r£÷s  zImageProjection.forward)r{r{rgrõr)r)r˜r*rŽêsüþýü rŽcs.eZdZd‡fdd„ Zdejfdd„Z‡ZS)ÚIPAdapterFullImageProjectionrˆcs8tƒ ¡ddlm}|||ddd|_t |¡|_dS©Nr©Ú FeedForwardÚgelu)ÚmultÚ activation_fn)r†r‡Ú attentionr’ÚffrrrŽ)r”rŠr‹r’r˜r)r*r‡s  z%IPAdapterFullImageProjection.__init__rÝcCs| | |¡¡Srè)rŽr—)r”rÝr)r)r*r£ sz$IPAdapterFullImageProjection.forward)rˆrˆ©r¤r¥r¦r‡rrêr£r¨r)r)r˜r*rsrcs.eZdZd‡fdd„ Zdejfdd„Z‡ZS) ÚIPAdapterFaceIDImageProjectionrˆrcsHtƒ ¡ddlm}||_||_|||||dd|_t |¡|_ dSr) r†r‡r–r’Ú num_tokensr‹r—rrrŽ)r”rŠr‹r”ršr’r˜r)r*r‡s  z'IPAdapterFaceIDImageProjection.__init__rÝcCs&| |¡}| d|j|j¡}| |¡S)Nr)r—rEršr‹rŽ)r”rÝrµr)r)r*r£s  z&IPAdapterFaceIDImageProjection.forward)rˆrˆrrr˜r)r)r˜r*r™ s r™cs(eZdZd‡fdd„ Zddd„Z‡ZS) ÚCombinedTimestepLabelEmbeddingsçš™™™™™¹?cs:tƒ ¡tdddd|_td|d|_t|||ƒ|_dS)Nr]Tr©rZr r ©r•rI)r†r‡rYÚ time_projrGÚtimestep_embedderrvÚclass_embedder)r”ryr Úclass_dropout_probr˜r)r*r‡s z(CombinedTimestepLabelEmbeddings.__init__NcCs2| |¡}| |j|d¡}| |¡}||}|S©NrX)rŸr r¡r¡)r”ÚtimestepÚ class_labelsÚ hidden_dtypeÚtimesteps_projÚ timesteps_embÚ conditioningr)r)r*r£%s  z'CombinedTimestepLabelEmbeddings.forward)rœrè©r¤r¥r¦r‡r£r¨r)r)r˜r*r›sr›có$eZdZ‡fdd„Zdd„Z‡ZS)Ú"CombinedTimestepTextProjEmbeddingscs<tƒ ¡tdddd|_td|d|_t||dd|_dS©Nr]TrrržrH©rJ)r†r‡rYrŸrGr ÚPixArtAlphaTextProjectionÚ text_embedder©r”r Úpooled_projection_dimr˜r)r*r‡1s z+CombinedTimestepTextProjEmbeddings.__init__cCs4| |¡}| |j|jd¡}| |¡}||}|Sr£)rŸr r¡rr°)r”r¤Úpooled_projectionr§r¨Úpooled_projectionsr©r)r)r*r£8s  z*CombinedTimestepTextProjEmbeddings.forwardrªr)r)r˜r*r¬0s r¬cr«)Ú*CombinedTimestepGuidanceTextProjEmbeddingscsJtƒ ¡tdddd|_td|d|_td|d|_t||dd|_dSr­) r†r‡rYrŸrGr Úguidance_embedderr¯r°r±r˜r)r*r‡Ds z3CombinedTimestepGuidanceTextProjEmbeddings.__init__c CsZ| |¡}| |j|jd¡}| |¡}| |j|jd¡}||}| |¡} || } | Sr£)rŸr r¡rr¶r°) r”r¤Úguidancer³r§r¨Ú guidance_projÚ guidance_embÚtime_guidance_embr´r©r)r)r*r£Ls   z2CombinedTimestepGuidanceTextProjEmbeddings.forwardrªr)r)r˜r*rµCs rµc s^eZdZddedededef‡fdd„ Zdejd ejd ejd ejd ejd ejf dd„Z‡Z S)Ú&CogView3CombinedTimestepSizeEmbeddingsr]r Ú condition_dimr²Ú timesteps_dimcsLtƒ ¡t|ddd|_t|ddd|_t||d|_t||dd|_dS)NTrrržrHr®) r†r‡rYrŸÚcondition_projrGr r¯Úcondition_embedder)r”r r¼r²r½r˜r)r*r‡\s z/CogView3CombinedTimestepSizeEmbeddings.__init__r¤Ú original_sizeÚ target_sizeÚ crop_coordsr¦rcCs¢| |¡}| | ¡¡ | d¡d¡}| | ¡¡ | d¡d¡}| | ¡¡ | d¡d¡} tj||| gdd} | |j|d¡} |  | j|d¡} | | } | S)NrrrrrX) rŸr¾rˆr±r°rr!r r¡r¿)r”r¤rÀrÁrÂr¦r§Úoriginal_size_projÚcrop_coords_projÚtarget_size_projr¾r¨Ú condition_embr©r)r)r*r£ds z.CogView3CombinedTimestepSizeEmbeddings.forward)r]) r¤r¥r¦rBr‡rrêrr£r¨r)r)r˜r*r»[s þýüûúùr»c s8eZdZd dedededef‡fdd„ Zdd „Z‡ZS) ÚHunyuanDiTAttentionPoolNÚ spacial_dimr.Ú num_headsÚ output_dimcsptƒ ¡t t |d|¡|d¡|_t ||¡|_t ||¡|_ t ||¡|_ t ||p0|¡|_ ||_ dS)Nrrƒ) r†r‡rr_rr`Úpositional_embeddingr­Úk_projÚq_projÚv_projÚc_projrÉ)r”rÈr.rÉrÊr˜r)r*r‡s   z HunyuanDiTAttentionPool.__init__c Cs| ddd¡}tj|jddd|gdd}||jdd…ddd…f |j¡}tjdid|dd…“d|“d |“d |j d “d |j “d |j j “d|j j “d|jj “dd“dt |j j|j j|jjg¡“dd“dd“dd“dd“d|jj “d|jj“dd“d|j“dd“Ž\}}| d¡S)NrrrT©rÚkeepdimrÚqueryÚkeyÚvalueÚembed_dim_to_checkrrÉÚ q_proj_weightÚ k_proj_weightÚ v_proj_weightÚin_proj_weightÚ in_proj_biasÚbias_kÚbias_vÚ add_zero_attnFÚ dropout_pÚout_proj_weightÚ out_proj_biasÚuse_separate_proj_weightrƒÚ need_weightsr))r²rr!ÚmeanrËr¡rr:Úmulti_head_attention_forwardrrÉrÍrarÌrÎr€rÏrƒÚsqueeze)r”rµrlr)r)r*r£ˆsV$  ÿþý üûúùø ÷ ö õ ô óòñðïîí zHunyuanDiTAttentionPool.forwardrè©r¤r¥r¦rBr‡r£r¨r)r)r˜r*rÇ|s  rÇcs0eZdZ    d ‡fdd„ Zd dd „Z‡ZS) Ú-HunyuanCombinedTimestepTextSizeStyleEmbeddingrˆr]éTcsŽtƒ ¡tdddd|_td|d|_tdddd|_t||d|d|_||_ |r8t   d|¡|_ d ||}n|}t ||d |d d |_dS) Nr]Trrržrü)rÉrÊrér4Ú silu_fp32)r«rîr¬rJ)r†r‡rYrŸrGr Ú size_projrÇÚpoolerÚ"use_style_cond_and_image_meta_sizerrqÚstyle_embedderr¯Úextra_embedder)r”r r²Úseq_lenr‹ríÚ extra_in_dimr˜r)r*r‡¥s$ ÿ üz6HunyuanCombinedTimestepTextSizeStyleEmbedding.__init__Nc Cs–| |¡}| |j|d¡}| |¡}|jr:| | d¡¡}|j|d}| dd¡}| |¡} tj ||| gdd} ntj |gdd} ||  | ¡} | S)NrXrrérr) rŸr r¡rìrírër±rîrr!rï) r”r¤rñÚimage_meta_sizeÚstyler¦r§r¨r´Ústyle_embeddingÚ extra_condr©r)r)r*r£Çs     z5HunyuanCombinedTimestepTextSizeStyleEmbedding.forward)rˆr]rèTrèrªr)r)r˜r*rç¤sú"rçcs&eZdZd‡fdd„ Zdd„Z‡ZS) Ú&LuminaCombinedTimestepCaptionEmbeddingrÁrèr]csLtƒ ¡t|ddd|_t||d|_t t |¡tj ||dd¡|_ dS)NTçrržrM) r†r‡rYrŸrGr rÚ Sequentialrr­Úcaption_embedder)r”rîr‹Úfrequency_embedding_sizer˜r)r*r‡âs ÿý þz/LuminaCombinedTimestepCaptionEmbedding.__init__c Csh| |¡}| |j|jd¡}| ¡ d¡}||jdd|jdd}| |¡}| |¡}||} | S©NrXrrr)rŸr r¡rr r“Úsumrù) r”r¤Ú caption_featÚ caption_maskÚ time_freqÚ time_embedÚcaption_mask_floatÚcaption_feats_poolÚ caption_embedr©r)r)r*r£ós   z.LuminaCombinedTimestepCaptionEmbedding.forward)rÁrèr]rªr)r)r˜r*röásröc sfeZdZ  ddedededededd f ‡fd d „ Z dd ejd ejdejdejd Bfdd„Z ‡Z S)Ú%MochiCombinedTimestepCaptionEmbeddingr]rür r²rÈrIÚnum_attention_headsrNcsJtƒ ¡t|ddd|_t||d|_t|||d|_t  ||¡|_ dS)NTr÷rrž)rr.rÊ) r†r‡rYrŸrGr ÚMochiAttentionPoolrìrr­Ú caption_proj)r”r r²rÈrIrr˜r)r*r‡s ÿz.MochiCombinedTimestepCaptionEmbedding.__init__r¤rñÚencoder_attention_maskr¦c CsB| |¡}| |j|d¡}| ||¡}| |¡}||} | |fSr£)rŸr r¡rìr) r”r¤rñrr¦rŸÚtime_embr´rr©r)r)r*r£s   z-MochiCombinedTimestepCaptionEmbedding.forward)r]rürè) r¤r¥r¦rBr‡rr†rêrr£r¨r)r)r˜r*rs4úþýüûúùûþýüûrcs4eZdZd dededef‡fdd„ Zdd„Z‡ZS) ÚTextTimeEmbeddingé@Ú encoder_dimrIrÉcs@tƒ ¡t |¡|_t||ƒ|_t ||¡|_t |¡|_ dSrè) r†r‡rrÚnorm1ÚAttentionPoolingÚpoolr­rŒÚnorm2)r”r rIrÉr˜r)r*r‡'s   zTextTimeEmbedding.__init__cCs,| |¡}| |¡}| |¡}| |¡}|Srè)r rrŒr)r”rðr)r)r*r£.s    zTextTimeEmbedding.forward)r rær)r)r˜r*r &sr csBeZdZd dededef‡fdd„ Zdejd ejfd d „Z‡ZS) ÚTextImageTimeEmbeddingr{rérÈrŠrIcs6tƒ ¡t ||¡|_t |¡|_t ||¡|_dSrè)r†r‡rr­rÑrÚ text_normÚ image_proj)r”rÈrŠrIr˜r)r*r‡7s  zTextImageTimeEmbedding.__init__rÜrÝcCs&| |¡}| |¡}| |¡}||Srè)rÑrr)r”rÜrÝÚtime_text_embedsÚtime_image_embedsr)r)r*r£=s   zTextImageTimeEmbedding.forward)r{r{rérõr)r)r˜r*r6srcs8eZdZd dedef‡fdd„ Zdejfdd „Z‡ZS) ÚImageTimeEmbeddingr{rérŠrIcs(tƒ ¡t ||¡|_t |¡|_dSrè)r†r‡rr­rrÚ image_norm©r”rŠrIr˜r)r*r‡Is zImageTimeEmbedding.__init__rÝcCs| |¡}| |¡}|Srè)rr)r”rÝrr)r)r*r£Ns  zImageTimeEmbedding.forward©r{rérõr)r)r˜r*rHsrcs>eZdZd dedef‡fdd„ Zdejdejfd d „Z‡ZS) ÚImageHintTimeEmbeddingr{rérŠrIcsâtƒ ¡t ||¡|_t |¡|_t tjdddddt  ¡tjdddddt  ¡tjddddddt  ¡tjdddddt  ¡tjddddddt  ¡tjdddddt  ¡tjdd ddddt  ¡tjd d ddd¡|_ dS) Nr6r_r)Úpaddingrgr)rré`r]r4) r†r‡rr­rrrrør‹ÚSiLUÚinput_hint_blockrr˜r)r*r‡Vs(   ñzImageHintTimeEmbedding.__init__rÝÚhintcCs&| |¡}| |¡}| |¡}||fSrè)rrr)r”rÝrrr)r)r*r£ls   zImageHintTimeEmbedding.forwardrrõr)r)r˜r*rUsrcs&eZdZd‡fdd„ Zdd„Z‡ZS)rNcs~tƒ ¡||_t t d|¡|d¡|_tj|||jd|_ tj|||jd|_ tj|||jd|_ ||_ ||j |_ dS)NrrƒrX)r†r‡rrr_rr`rËr­rÌrÍrÎrÉÚ dim_per_head)r”rÉr.rr˜r)r*r‡ws zAttentionPooling.__init__c sô| ¡\‰}}‡‡fdd„}|jdddˆj |j¡}tj||gdd}|ˆ |¡ƒ}|ˆ |¡ƒ}|ˆ  |¡ƒ}dt   t   ˆj ¡¡} t  d|| || ¡} tj|  ¡dd | j¡} t  d | |¡} |  ˆdd¡ dd ¡} | dd…d dd…fS) NcsF| ˆdˆjˆj¡}| dd¡}| ˆˆjdˆj¡}| dd¡}|S)Nrrr)r±rÉr r rE)rµ©Úbsr”r)r*r„s   z'AttentionPooling.forward..shaperTrÐrz bct,bcs->btsrz bts,bcs->bctrr)r°rãrËr¡rrr!rÍrÌrÎrÚsqrtr rxÚsoftmaxr rqrEr ) r”rµÚlengthr‘rÚ class_tokenÚqÚkÚvrraÚar)r!r*r£s zAttentionPooling.forwardrèrªr)r)r˜r*rts rc sxeZdZ ddedededBddf‡fdd„ Zedd œd ejd ejdejfd d „ƒZd ejd ej dejfdd„Z ‡Z S)rNrr.rÊrcsNtƒ ¡|p||_||_t |d|¡|_t ||¡|_t ||j¡|_dS)Nr) r†r‡rÊrrr­Úto_kvÚto_qÚto_out)r”rr.rÊr˜r)r*r‡§s  zMochiAttentionPool.__init__F©rÑrµr½cCs€| d¡| d¡ks J‚| d¡| d¡ksJ‚|dd…dd…dfj|jd}||jdddjdd}||jd|d}|S)a6 Pool tokens in x using mask. NOTE: We assume x does not require gradients. Args: x: (B, L, D) tensor of tokens. mask: (B, L) boolean tensor indicating which tokens are not padding. Returns: pooled: (B, D) tensor of pooled tokens. rrNrXTrÐ)Úmin)r°r¡rrüÚclamp)rµr½rÑÚpooledr)r)r*Ú pool_tokens¶s  zMochiAttentionPool.pool_tokensc Cs| d¡}|dd…dddd…f ¡}tj|ddd}|j||dd}tj||gdd}| |¡}| |dd…d f¡}||j }|  dd|j |f¡}|  dd ¡}|  d¡\} } |  d|j |f¡}|  d¡}tj|| | |d d }| d¡ dd¡}| |¡}|S) aP Args: x (`torch.Tensor`): Tensor of shape `(B, S, D)` of input tokens. mask (`torch.Tensor`): Boolean ensor of shape `(B, S)` indicating which tokens are not padding. Returns: `torch.Tensor`: `(B, D)` tensor of pooled tokens. rN)rrT)rÔr.rrrr6r÷)Ú attn_maskrÞ)r°rér:r%r2rr!r+r,rÚ unflattenr r1r“Úscaled_dot_product_attentionrårˆr-) r”rµr½ÚDr3Úx_poolÚkvr'Úhead_dimr(r)r)r)r*r£Ës"      zMochiAttentionPool.forwardrè) r¤r¥r¦rBr‡Ú staticmethodrrêr2Ú BoolTensorr£r¨r)r)r˜r*r¦süþýüû$$rcCsŠ|jdd…\}}dt |¡|}|dj|j|jd}|| d¡}tj| ¡|  ¡fdd}|  dd d d d¡  |||dd ¡}|S) zÉ Args: embed_dim: int box: a 3-D tensor [B x N x 4] representing the bounding boxes for GLIGEN pipeline Returns: [B x N x embed_dim] tensor of positional embeddings Nréd)NNNr7rrrrr6r4) rrrr¡rrr“rDr"r#r²rE)r.Úboxr¹Ú num_boxesr(r)r)r*Ú#get_fourier_embeds_from_boundingboxøs $r?cs2eZdZd‡fdd„ Z     d dd„Z‡ZS) ÚGLIGENTextBoundingboxProjectionú text-onlyrüc sftƒ ¡||_||_||_|dd|_t|tƒr|d}|dkrMt  t  |j|jd¡t  ¡t  dd¡t  ¡t  d|¡¡|_ t j t  |jg¡¡|_nX|dkr¥t  t  |j|jd¡t  ¡t  dd¡t  ¡t  d|¡¡|_t  t  |j|jd¡t  ¡t  dd¡t  ¡t  d|¡¡|_t j t  |jg¡¡|_t j t  |jg¡¡|_t j t  |jg¡¡|_dS)Nrr4rrAéz text-image)r†r‡Ú positive_lenrKÚfourier_embedder_dimÚ position_dimrAÚtuplerrør­rÚlinearsrr_rgÚnull_positive_featureÚ linears_textÚ linears_imageÚnull_text_featureÚnull_image_featureÚnull_position_feature)r”rCrKÚ feature_typeÚ fourier_freqsr˜r)r*r‡sD    û  û  ûz(GLIGENTextBoundingboxProjection.__init__NcCs$| d¡}t|j|ƒ}|j ddd¡} ||d|| }|durA|j ddd¡} ||d|| }| tj||gdd¡} | S| d¡}| d¡}|j  ddd¡} |j  ddd¡} ||d|| }||d|| }|  tj||gdd¡}|  tj||gdd¡}tj||gdd} | S)Nrrr) r“r?rDrMr±rHrGrr!rKrLrIrJ)r”ÚboxesÚmasksÚpositive_embeddingsÚ phrases_masksÚ image_masksÚphrases_embeddingsÚimage_embeddingsÚxyxy_embeddingÚ xyxy_nullÚ positive_nullÚobjsÚ text_nullÚ image_nullÚ objs_textÚ objs_imager)r)r*r£7s&  ñ z'GLIGENTextBoundingboxProjection.forward)rArü)NNNNNrªr)r)r˜r*r@ s-ør@cs0eZdZdZddef‡fdd„ Zdd„Z‡ZS) Ú)PixArtAlphaCombinedTimestepSizeEmbeddingszº For PixArt-Alpha. Reference: https://github.com/PixArt-alpha/PixArt-alpha/blob/0f55e922376d8b797edd44d25d0e7464b260dcab/diffusion/model/nets/PixArtMS.py#L164C9-L168C29 FÚuse_additional_conditionscsltƒ ¡||_tdddd|_td|d|_||_|r4tdddd|_td|d|_ td|d|_ dSdS)Nr]Trrrž) r†r‡ÚoutdimrYrŸrGr r`Úadditional_condition_projÚresolution_embedderÚaspect_ratio_embedder)r”r Ú size_emb_dimr`r˜r)r*r‡rs ýz2PixArtAlphaCombinedTimestepSizeEmbeddings.__init__c Cs| |¡}| |j|d¡}|jrD| | ¡¡ |¡}| |¡ |d¡}| | ¡¡ |¡} | | ¡ |d¡} |t j || gdd} | S|} | Srû) rŸr r¡r`rbrˆrcrErdrr!) r”r¤Ú resolutionÚ aspect_ratior¹r¦r§r¨Úresolution_embÚaspect_ratio_embr©r)r)r*r£s þz1PixArtAlphaCombinedTimestepSizeEmbeddings.forward©F)r¤r¥r¦r§rér‡r£r¨r)r)r˜r*r_js r_cs*eZdZdZd‡fdd„ Zdd„Z‡ZS) r¯zÊ Projects caption embeddings. Also handles dropout for classifier-free guidance. Adapted from https://github.com/PixArt-alpha/PixArt-alpha/blob/master/diffusion/model/nets/PixArt_blocks.py NÚ gelu_tanhcsŠtƒ ¡|dur |}tj||dd|_|dkr tjdd|_n|dkr*t ¡|_n|dkr3tƒ|_nt d|›ƒ‚tj||dd|_ dS) NTrªrkÚtanh)Ú approximaterHrêzUnknown activation function: ) r†r‡rr­rNÚGELUÚact_1rrr@rQ)r”r«rîr¬rJr˜r)r*r‡–s   z"PixArtAlphaTextProjection.__init__cCs"| |¡}| |¡}| |¡}|Srè)rNrorQ)r”Úcaptionrðr)r)r*r£¥s   z!PixArtAlphaTextProjection.forward)Nrkr¾r)r)r˜r*r¯sr¯c sDeZdZ    ddedededed d f ‡fd d „ Zd d„Z‡ZS)Ú!IPAdapterPlusImageProjectionBlockr{r r_r4Ú embed_dimsÚdim_headÚheadsÚ ffn_ratiorNc sftƒ ¡ddlm}t |¡|_t |¡|_t|||dd|_ t  t |¡|||d|dd¡|_ dS)Nrr‘F)Ú query_dimrsrtÚout_biasr“©r•r”r€) r†r‡r–r’rrÚln0Úln1rÚattnrør—)r”rrrsrtrur’r˜r)r*r‡­s    ü þz*IPAdapterPlusImageProjectionBlock.__init__cCsH| |¡}| |¡}tj||gdd}| ||¡|}| |¡|}|S)NrŸr)ryrzrr!r{r—)r”rµÚlatentsÚresidualrñr)r)r*r£Äs  z)IPAdapterPlusImageProjectionBlock.forward)r{r r_r4)r¤r¥r¦rBr r‡r£r¨r)r)r˜r*rq¬s"ûþýüûúrqcsneZdZdZ        dd ed ed ed ed ededededdf‡fdd„ Zdejdejfdd„Z ‡Z S)ÚIPAdapterPlusImageProjectionaäResampler of IP-Adapter Plus. Args: embed_dims (int): The feature dimension. Defaults to 768. output_dims (int): The number of output channels, that is the same number of the channels in the `unet.config.cross_attention_dim`. Defaults to 1024. hidden_dims (int): The number of hidden channels. Defaults to 1280. depth (int): The number of blocks. Defaults to 8. dim_head (int): The number of head channels. Defaults to 64. heads (int): Parallel attention heads. Defaults to 16. num_queries (int): The number of queries. Defaults to 8. ffn_ratio (float): The expansion ratio of feedforward network hidden layer channels. Defaults to 4. r{rˆér4r r_rürrÚ output_dimsÚ hidden_dimsÚdepthrsrtÚ num_queriesrurNc sxtƒ ¡t t d|ˆ¡ˆd¡|_t |ˆ¡|_t ˆ|¡|_ t  |¡|_ t  ‡‡‡‡fdd„t |ƒDƒ¡|_dS)Nrrƒcóg|] }tˆˆˆˆƒ‘qSr)©rq©Ú.0rl©rsrurtrr)r*Ú ñóz9IPAdapterPlusImageProjection.__init__..)r†r‡rr_rr`r|r­Úproj_inÚproj_outrÚnorm_outÚ ModuleListÚrangeÚlayers) r”rrr€rr‚rsrtrƒrur˜rˆr*r‡Ýs   ÿz%IPAdapterPlusImageProjection.__init__rµcCsP|j | d¡dd¡}| |¡}|jD] }|}||||ƒ}q| |¡}| |¡S)z‹Forward pass. Args: x (torch.Tensor): Input Tensor. Returns: torch.Tensor: Output Tensor. rr)r|r\r°r‹rrŒr)r”rµr|Úblockr}r)r)r*r£ôs    z$IPAdapterPlusImageProjection.forward)r{rˆrr4r r_rür4© r¤r¥r¦r§rBr r‡rrêr£r¨r)r)r˜r*r~Ís<÷þýüûúùø ÷ ör~cs€eZdZdZ           dd ed ed ed ededededededededdf‡fdd„ Zdejdejfdd„Z ‡Z S)Ú"IPAdapterFaceIDPlusImageProjectiona—FacePerceiverResampler of IP-Adapter Plus. Args: embed_dims (int): The feature dimension. Defaults to 768. output_dims (int): The number of output channels, that is the same number of the channels in the `unet.config.cross_attention_dim`. Defaults to 1024. hidden_dims (int): The number of hidden channels. Defaults to 1280. depth (int): The number of blocks. Defaults to 8. dim_head (int): The number of head channels. Defaults to 64. heads (int): Parallel attention heads. Defaults to 16. num_tokens (int): Number of tokens num_queries (int): The number of queries. Defaults to 8. ffn_ratio (float): The expansion ratio of feedforward network hidden layer channels. Defaults to 4. ffproj_ratio (float): The expansion ratio of feedforward network hidden layer channels (for ID embeddings). Defaults to 4. r{rrBr4r r_rürrrr€rÚid_embeddings_dimr‚rsrtršrƒruÚ ffproj_ratiorNc s¦tƒ ¡ddlm} ||_ˆ|_d|_d|_d|_| |ˆ|d| d|_ t   ˆ¡|_ t   |ˆ¡|_t   ˆ|¡|_t   |¡|_t  ‡‡‡‡fdd„t|ƒDƒ¡|_dS) Nrr‘Fr,r“)r•r”csg|] }tˆˆˆˆƒ‘qSr)r…r†©rsrrrurtr)r*r‰9 rŠz?IPAdapterFaceIDPlusImageProjection.__init__..)r†r‡r–r’ršr.Ú clip_embedsÚshortcutÚshortcut_scalerŒrrrŽr­r‹rŒrrŽrr) r”rrr€rr”r‚rsrtršrƒrur•r’r˜r–r*r‡ s     ÿz+IPAdapterFaceIDPlusImageProjection.__init__Ú id_embedscCs¦| |jj¡}| |¡}| d|j|j¡}| |¡}|}| |j¡}| d|j d|j d¡}|j D] }|}||||ƒ}q2|  |¡}|  |¡}|j rQ||j|}|S)zŸForward pass. Args: id_embeds (torch.Tensor): Input Tensor (ID embeds). Returns: torch.Tensor: Output Tensor. rrr6)r¡r—rrŒrEršr.rŽr‹rrrŒrr˜r™)r”ršr|r—rµr‘r}rur)r)r*r£< s      z*IPAdapterFaceIDPlusImageProjection.forward) r{r{rrBr4r r_r4rür4rr’r)r)r˜r*r“ sNôþýüûúùø ÷ ö õ ô ó#r“c sbeZdZdZ    ddededed ed d f ‡fd d „ Zdejdejdejd ejfdd„Z‡Z S)Ú!IPAdapterTimeImageProjectionBlocka¨Block for IPAdapterTimeImageProjection. Args: hidden_dim (`int`, defaults to 1280): The number of hidden channels. dim_head (`int`, defaults to 64): The number of head channels. heads (`int`, defaults to 20): Parallel attention heads. ffn_ratio (`int`, defaults to 4): The expansion ratio of feedforward network hidden layer channels. rr ér4Ú hidden_dimrsrtrurNcs¶tƒ ¡ddlm}t |¡|_t |¡|_t||||ddd|_ |||d|dd|_ t  ¡|_ t  |d|¡|_t |¡|_dt t |¡¡|j _|j  ¡d|j _d|j _dS)Nrr‘F)rvr‹rsrtr€rwr“rxr4)r†r‡r–r’rrryrzrr{r—rÚ adaln_silur­Ú adaln_projÚ adaln_normrr#rÚfuse_projectionsÚto_kÚto_v)r”rrsrtrur’r˜r)r*r‡f s(    ú    z*IPAdapterTimeImageProjectionBlock.__init__rµr|Ú timestep_embcCsÖ| | |¡¡}|jddd\}}}}|} | |¡}| |¡d|dd…df|dd…df}|jd} |j |¡} tj ||fdd} |j  | ¡jddd\} }| jd}||jj }|   | d|jj |¡  dd¡} |   | d|jj |¡  dd¡} |  | d|jj |¡  dd¡}| |jj| |jj  dd¡}tj| ¡dd |j¡}||}|  dd¡ | d|jj |¡}|jjd|ƒ}|jjd|ƒ}|| }|} | |¡d|dd…df|dd…df}| |¡| S) aAForward pass. Args: x (`torch.Tensor`): Image features. latents (`torch.Tensor`): Latent features. timestep_emb (`torch.Tensor`): Timestep embedding. Returns: `torch.Tensor`: Output latent features. r4rrNrrŸrr)rŸržrAryrzrr{r,rr!r+rtr±r rr$r rqrrEr-r r—)r”rµr|r¤r(Ú shift_msaÚ scale_msaÚ shift_mlpÚ scale_mlpr}r¹rÒÚkv_inputrÓrÔÚ inner_dimr9rar)r)r*r£‡ s0 .     .z)IPAdapterTimeImageProjectionBlock.forward)rr rœr4) r¤r¥r¦r§rBr‡rrêr£r¨r)r)r˜r*r›X s$ûþýüûú*!r›cseZdZdZ          dd ed ed edededededededededdf‡fdd„ Zdejdejde ejejffdd„Z ‡Z S)ÚIPAdapterTimeImageProjectionaŠResampler of SD3 IP-Adapter with timestep embedding. Args: embed_dim (`int`, defaults to 1152): The feature dimension. output_dim (`int`, defaults to 2432): The number of output channels. hidden_dim (`int`, defaults to 1280): The number of hidden channels. depth (`int`, defaults to 4): The number of blocks. dim_head (`int`, defaults to 64): The number of head channels. heads (`int`, defaults to 20): Parallel attention heads. num_queries (`int`, defaults to 64): The number of queries. ffn_ratio (`int`, defaults to 4): The expansion ratio of feedforward network hidden layer channels. timestep_in_dim (`int`, defaults to 320): The number of input channels for timestep embedding. timestep_flip_sin_to_cos (`bool`, defaults to True): Flip the timestep embedding order to `cos, sin` (if True) or `sin, cos` (if False). timestep_freq_shift (`int`, defaults to 0): Controls the timestep delta between frequencies between dimensions. é€é€ rr4r rœé@Trr.rÊrr‚rsrtrƒruÚtimestep_in_dimÚtimestep_flip_sin_to_cosÚtimestep_freq_shiftrNc s–tƒ ¡t t d|ˆ¡ˆd¡|_t |ˆ¡|_t ˆ|¡|_ t  |¡|_ t  ‡‡‡‡fdd„t |ƒDƒ¡|_t| | | ƒ|_t| ˆdd|_dS)Nrrƒcr„r))r›r†©rsrurtrr)r*r‰ì rŠz9IPAdapterTimeImageProjection.__init__..rHr®)r†r‡rr_rr`r|r­r‹rŒrrrŽrrrYrŸrGÚtime_embedding) r”r.rÊrr‚rsrtrƒrur¯r°r±r˜r²r*r‡Ø s  ÿz%IPAdapterTimeImageProjection.__init__rµr¤cCs†| |¡j|jd}| |¡}|j | d¡dd¡}| |¡}||dd…df}|jD]}||||ƒ}q,|  |¡}|  |¡}||fS)a#Forward pass. Args: x (`torch.Tensor`): Image features. timestep (`torch.Tensor`): Timestep in denoising process. Returns: `tuple`[`torch.Tensor`, `torch.Tensor`]: The pair (latents, timestep_emb). rXrrN) rŸr¡rr³r|r\r°r‹rrŒr)r”rµr¤r¤r|r‘r)r)r*r£ñ s     z$IPAdapterTimeImageProjection.forward) r¬r­rr4r rœr r4r®Tr) r¤r¥r¦r§rBrér‡rrêrFr£r¨r)r)r˜r*r«¼ sNôþýüûúùø ÷ ö õ ô ó.r«csXeZdZdeejeejBf‡fdd„ Zede fdd„ƒZ dee j fdd „Z ‡ZS) ÚMultiIPAdapterImageProjectionÚIPAdapterImageProjectionLayerscstƒ ¡t |¡|_dSrè)r†r‡rrŽÚimage_projection_layers)r”rµr˜r)r*r‡ s z&MultiIPAdapterImageProjection.__init__rcCs t|jƒS)zNumber of IP-Adapters loaded.)rr¶)r”r)r)r*Únum_ip_adapters s z-MultiIPAdapterImageProjection.num_ip_adaptersrÝcCsÚg}t|tƒsd}tdd|dd| d¡g}t|ƒt|jƒkr/tdt|ƒ›dt|jƒ›ƒ‚t||jƒD]5\}}|jd |jd}}|  ||f|jd d…¡}||ƒ}|  ||f|jdd…¡}|  |¡q5|S) NzÓYou have passed a tensor as `image_embeds`.This is deprecated and will be removed in a future release. Please make sure to update your script to pass `image_embeds` as a list of tensors to suppress this warning.zimage_embeds not a listú1.0.0FrVrzGimage_embeds must have the same length as image_projection_layers, got z and rr) rAÚlistrr“rr¶r@ÚziprrEÚappend)r”rÝÚprojected_image_embedsr^Ú image_embedÚimage_projection_layerr¹Ú num_imagesr)r)r*r£ s" ÿ ÿ z%MultiIPAdapterImageProjection.forward)r¤r¥r¦r¹rÚModulerFr‡ÚpropertyrBr·rrêr£r¨r)r)r˜r*r´ s "r´c@seZdZdd„ZdS)Ú FluxPosEmbedcOs*d}tdd|ƒddlm}||i|¤ŽS)NzœImporting and using `FluxPosEmbed` from `diffusers.models.embeddings` is deprecated. Please import it from `diffusers.models.transformers.transformer_flux`.rÂr¸r)rÂ)rÚtransformers.transformer_fluxrÂ)ÚclsÚargsÚkwargsr^rÂr)r)r*Ú__new__6 s  zFluxPosEmbed.__new__N)r¤r¥r¦rÇr)r)r)r*rÂ5 s rÂ)Frrr )r,r,Nr-)r,r,)Frr,r_Nr-)r-)r-FN)Frr,r_)r TröNN)rr N)TNr-)Trj)Trr)RrÚnumpyr-rÚtorch.nn.functionalrr$r:ÚutilsrÚ activationsrrÚattention_processorrrêrBrér r+rFrrXrSr'r?rhrFrGrfrjrprÀryr©r¿rërrrrrr$rrr9rFrGrYr\rfrorvr‡rŽrr™r›r¬rµr»rÇrçrörr rrrrrr?r@r_r¯rqr~r“r›r«r´rÂr)r)r)r*Ús|    úÿþýüûú ù:ùÿþýüûúù øNûÿþýüû úEøù ø D !2 ÿ&6=÷ûúùø ÷ öjù ûúù ø,ÿÿ ÿ 2  "$øÿþ ýFûÿþýüû ú90A$ !(="# 2R]%!;PdQ(