o Gi}V @srddlZddlZddlmZddlZddlmZddlmZm Z ddl m Z m Z ddl mZmZmZmZddlmZdd lmZmZdd lmZmZmZdd lmZdd lmZdd lm Z ddl!m"Z"ddl#m$Z$ddl%m&Z&m'Z'e(e)Z*GdddZ+GdddZ,Gdddejj-eZ.Gdddej-Z/eGdddej-Z0eGddde$eee e eZ1ddZ2dS)N)Any) ConfigMixinregister_to_config)FromOriginalModelMixinPeftAdapterMixin)apply_lora_scale deprecateis_torch_versionlogging)maybe_allow_in_graph)ContextParallelInputContextParallelOutput)AttentionMixinAttentionModuleMixin FeedForward)dispatch_attention_fn) CacheMixin)PixArtAlphaTextProjection)Transformer2DModelOutput) ModelMixin)AdaLayerNormSingleRMSNormc@seZdZddZdS)LTXVideoAttentionProcessor2_0cOsd}tdd|t|i|S)Nz~`LTXVideoAttentionProcessor2_0` is deprecated and this will be removed in a future version. Please use `LTXVideoAttnProcessor`rz1.0.0)r LTXVideoAttnProcessor)clsargskwargsdeprecation_messager a/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/models/transformers/transformer_ltx.py__new__)s z%LTXVideoAttentionProcessor2_0.__new__N)__name__ __module__ __qualname__r"r r r r!r(s rc @s`eZdZdZdZdZddZ   ddddejdejdBd ejdBd ejdBd ejf d d Z dS)rz Processor for implementing attention (SDPA is used by default if you're using PyTorch 2.0). This is used in the LTX model. It applies a normalization layer and rotary embedding on the query and key vector. NcCstddr tddS)NTepselementwise_affine)r\)superr)NotImplementedErrorhead_dim inner_dim inner_kv_dimrXr]use_biasr[out_dimr<rSnnrr@rALinearr=r>r? ModuleListrIappendDropout_default_processor_cls set_processor) r(rXr<rYrZr[r\r]r^r_ processornorm_epsnorm_elementwise_affine __class__r r!r)ws2  zLTXAttention.__init__r,r-r.r/r0c stt|jjjfdd|D}t|dkr,t d|d|jj j dfdd|D}|j|||||fi|S) Ncsg|] \}}|vr|qSr r ).0krLattn_parametersr r! sz(LTXAttention.forward..rzattention_kwargs z are not expected by z and will be ignored.csi|] \}}|vr||qSr r )rvrwwrxr r! sz(LTXAttention.forward..) setinspect signaturerqrP parameterskeysitemslenloggerwarningrur#)r(r,r-r.r/r unused_kwargsr rxr!forwards zLTXAttention.forward) rUrUrVr2TNTrWNrQ)r#r$r%rro_available_processorsintfloatboolstrr)rSrTr __classcell__r r rtr!r+ssZ  -r+cseZdZ      ddedededed ed ed ed d ffdd Zdededededeejeefdej d ejfddZ ddejded Bded Bded Bdeejeefd Bdejd Bd eejejffddZ Z S)LTXVideoRotaryPosEmbedr8@dimbase_num_frames base_height base_width patch_size patch_size_tthetar0Ncs8t||_||_||_||_||_||_||_dS)N) rcr)rrrrrrr)r(rrrrrrrrtr r!r)s  zLTXVideoRotaryPosEmbed.__init__rJ num_framesheightwidthrope_interpolation_scaledevicec Cs6tj|tj|d}tj|tj|d}tj|tj|d} tj| ||dd} tj| dd} | d|dddd} |dur| ddddf|d|j|j| ddddf<| ddddf|d|j |j | ddddf<| ddddf|d|j |j | ddddf<| dd  dd} | S) N)rHrij)indexingrrr8r r)rSarangefloat32meshgridstack unsqueezerepeatrrrrrrF transpose) r(rJrrrrrgrid_hgrid_wgrid_fgridr r r!_prepare_video_coordss 888z,LTXVideoRotaryPosEmbed._prepare_video_coordsr, video_coordscCs|d}|dur|j||||||jd}n%tj|dddf|j|dddf|j|dddf|jgdd}d} |j} |jtj t | |jt | |j|j d|jtj d } | t jd } | |ddd} | dd d} | jddd} | jddd} |j ddkrt| ddddd|j df}t| ddddd|j df}tj|| gdd} tj|| gdd} | | fS) Nr)rrr8r r1rg?)rrHg@)sizerrrSrrrrrlinspacemathlogrrpirrrFcosrepeat_interleavesin ones_like zeros_likecat)r(r,rrrrrrJrstartendfreqs cos_freqs sin_freqs cos_padding sin_paddingr r r!rsJ     &&zLTXVideoRotaryPosEmbed.forward)rrrr8r8r)NNNNN) r#r$r%rrr)tuplerSrTrrrrr r rtr!rsv  rcseZdZdZ      ddeded ed ed ed ed edededeffdd Z  dde j de j de j de e j e j fdBde j dBde j f ddZ Z S)LTXVideoTransformerBlocka Transformer block used in [LTX](https://huggingface.co/Lightricks/LTX-Video). Args: dim (`int`): The number of channels in the input and output. num_attention_heads (`int`): The number of heads to use for multi-head attention. attention_head_dim (`int`): The number of channels in each head. qk_norm (`str`, defaults to `"rms_norm"`): The normalization layer to use. activation_fn (`str`, defaults to `"gelu-approximate"`): Activation function to use in feed-forward. eps (`float`, defaults to `1e-6`): Epsilon value for normalization layers. rWgelu-approximateTư>Frnum_attention_headsattention_head_dimr]r_ activation_fnattention_biasattention_out_biasrarbc stt|| | d|_t|||||d||d|_t|| | d|_t||||||||d|_t||d|_ t t d||d|_dS)Nr`)rXr<rYrZr\r]r^r_)rXr]r<rYrZr\r^r_)rr?)rcr)rnorm1r+attn1norm2attn2rffrj ParameterrSrandnscale_shift_table) r(rrrr]r_rrrrarbrtr r!r)-s2   z!LTXVideoTransformerBlock.__init__Nr,r-tembr/encoder_attention_maskr0cCs|d}||}|jjd}|jd|j|||d|d} | jdd\} } } } }}|d| | }|j|d|d}||| }|j ||d|d}||}| |d|| }| |}|||}|S) NrNNr8r1r r)r,r-r/)r-r/r.) rrrr9rGrreshapeunbindrrrr)r(r,r-rr/rrJnorm_hidden_statesnum_ada_params ada_values shift_msa scale_msagate_msa shift_mlp scale_mlpgate_mlpattn_hidden_states ff_outputr r r!rXs2      z LTXVideoTransformerBlock.forward)rWrTTrFr)r#r$r%rRrrrrr)rSrTrrrr r rtr!rsX   0rc"sxeZdZdZdZdgZdgZeddddeddddedd ddd eddddeddddd eddd d Z e               d7de de de de de de de de de d e d!e d"ed#e d$e d%e d&d'f fd(d) Zed* ' ' ' ' ' ' d8d+ejd,ejd-ejd.ejd/e d'Bd0e d'Bd1e d'Bd2eeeefejBd'Bd3ejd'Bd*ee efd'Bd4e d&ejfd5d6ZZS)9LTXVideoTransformer3DModela A Transformer model for video-like data used in [LTX](https://huggingface.co/Lightricks/LTX-Video). Args: in_channels (`int`, defaults to `128`): The number of channels in the input. out_channels (`int`, defaults to `128`): The number of channels in the output. patch_size (`int`, defaults to `1`): The size of the spatial patches to use in the patch embedding layer. patch_size_t (`int`, defaults to `1`): The size of the tmeporal patches to use in the patch embedding layer. num_attention_heads (`int`, defaults to `32`): The number of heads to use for multi-head attention. attention_head_dim (`int`, defaults to `64`): The number of channels in each head. cross_attention_dim (`int`, defaults to `2048 `): The number of channels for cross attention heads. num_layers (`int`, defaults to `28`): The number of layers of Transformer blocks to use. activation_fn (`str`, defaults to `"gelu-approximate"`): Activation function to use in feed-forward. qk_norm (`str`, defaults to `"rms_norm_across_heads"`): The normalization layer to use. Tnormrr8rF) split_dim expected_dims split_outputr )r,r-r)rr8) gather_dimr)ropeproj_out rVrrrWr in_channels out_channelsrrrrr] num_layersrr_rsrrcaption_channelsrrr0Nc st|p|}t||_ttdd|_t dd|_ t | d|_ t ddd||dd |_t f d d t|D|_tjd dd |_t||_d|_dS)Nr rF)use_additional_conditions) in_features hidden_sizerrr)rrrrrrrcs(g|]}t d qS)) rrrr]r_rrrrarb)r)rvrL rrrrr]rfrsrrrr_r r!rzs z7LTXVideoTransformer3DModel.__init__..rr`)rcr)rjrkproj_inrrSrrr time_embedrcaption_projectionrrrlrangetransformer_blocks LayerNormnorm_outrgradient_checkpointing)r(rrrrrrr]rrr_rsrrrrrrtrr!r)s0    z#LTXVideoTransformer3DModel.__init__attention_kwargsr,r-timesteprrrrrr return_dictc  Cs||||||| } |dur"|jdkr"d||jd}|d}|d} ||}|j|| |jd\}}| | d|d}| | d|d}| |}| | d|d}|j D]}t rs|jrs|||||| |}q_||||| |d}q_|jd|dddddf}|dddddf|dddddf}}||}|d||}||}| s|fSt|d S) Nr r8gr)rJ hidden_dtyper1)r,r-rr/rr)sample)rndimrGrHrrrrrFr;rrrSis_grad_enabledr_gradient_checkpointing_funcrrrr)r(r,r-rrrrrrrrrr/rJrembedded_timestepblockscale_shift_valuesshiftscaleoutputr r r!rsP        .   z"LTXVideoTransformer3DModel.forward)rrr8r8rrVrrrrWFrrTT)NNNNNNT)r#r$r%rR _supports_gradient_checkpointing _skip_layerwise_casting_patterns_repeated_blocksrr_cp_planrrrrrr)rrSrT LongTensorrdictrrrr r rtr!rs            ?     rcCs\|\}}|ddd\}}tj| |gddd}|||||j}|S)Nr )r1r r1r)rCrrSrrFrrGrH)xrrrx_realx_imag x_rotatedoutr r r!rB2s  rB)3r~rtypingrrStorch.nnrjconfiguration_utilsrrloadersrrutilsrr r r utils.torch_utilsr _modeling_parallelrr attentionrrrattention_dispatchr cache_utilsr embeddingsrmodeling_outputsrmodeling_utilsr normalizationrr get_loggerr#rrrModuler+rrrrBr r r r!s<         C@ff   2