o Gi @sddlZddlmZddlmZddlZddlmZddlm Z m Z ddl m Z m Z ddlmZmZmZmZdd lmZmZdd lmZmZmZdd lmZdd lmZdd lmZm Z ddl!m"Z"ddl#m$Z$e%e&Z'dej(de)ej(ej(fdej(fddZ*dej(de)ej(ej(fdej(fddZ+eGdddeZ,Gdddej-Z.GdddZ/Gdddejj-eZ0Gdd d ej-Z1Gd!d"d"ej-Z2Gd#d$d$e"e ee e eZ3dS)%N) dataclass)Any) ConfigMixinregister_to_config)FromOriginalModelMixinPeftAdapterMixin) BaseOutputapply_lora_scaleis_torch_versionlogging)ContextParallelInputContextParallelOutput)AttentionMixinAttentionModuleMixin FeedForward)dispatch_attention_fn) CacheMixin))PixArtAlphaCombinedTimestepSizeEmbeddingsPixArtAlphaTextProjection) ModelMixin)RMSNormxfreqsreturncCs\|\}}|ddd\}}tj| |gddd}|||||j}|S)Nr )r rdim) unflattenunbindtorchstackflattenfloattodtype)rrcossinx_realx_imag x_rotatedoutr-b/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/models/transformers/transformer_ltx2.pyapply_interleaved_rotary_emb&s  r/cCsz|\}}|j}d}|jdkr(|jdkr(|j\}}}} ||||ddd}d}|jd} | ddkr;td| d | d} |jg|jddd| R} | d ddddf} | d ddddf}|d }|d }| |}|d ddddf}|d ddddf}|| |||| |jg|jdd | R}|r|dd||d}|j |d }|S) NFrr Trz6Expected x.shape[-1] to be even for split rotary, got ..r&) r&ndimshapereshapeswapaxes ValueErrorr$ unsqueezeaddcmul_r%)rrr'r(x_dtype needs_reshapebht_lastrsplit_xfirst_xsecond_xcos_usin_ur, first_out second_outr-r-r.apply_split_rotary_emb.s4  &    rKc@s"eZdZUdZded<ded<dS)AudioVisualModelOutputa Holds the output of an audiovisual model which produces both visual (e.g. video) and audio outputs. Args: sample (`torch.Tensor` of shape `(batch_size, num_channels, num_frames, height, width)`): The hidden states output conditioned on the `encoder_hidden_states` input, representing the visual output of the model. This is typically a video (spatiotemporal) output. audio_sample (`torch.Tensor` of shape `(batch_size, TODO)`): The audio output of the audiovisual model. z torch.Tensorsample audio_sampleN)__name__ __module__ __qualname____doc____annotations__r-r-r-r.rLWs  rLcseZdZdZddededeffdd Z dd ejd e e ejfd Bd ed Bd ej d Bde ejejejejejff ddZ ZS)LTX2AdaLayerNormSingleaB Norm layer adaptive layer norm single (adaLN-single). As proposed in PixArt-Alpha (see: https://huggingface.co/papers/2310.00426; Section 2.3) and adapted by the LTX-2.0 model. In particular, the number of modulation parameters to be calculated is now configurable. Parameters: embedding_dim (`int`): The size of each embedding vector. num_mod_params (`int`, *optional*, defaults to `6`): The number of modulation parameters which will be calculated in the first return argument. The default of 6 is standard, but sometimes we may want to have a different (usually smaller) number of modulation parameters. use_additional_conditions (`bool`, *optional*, defaults to `False`): Whether to use additional conditions for normalization or not. F embedding_dimnum_mod_paramsuse_additional_conditionscsJt||_t||d|d|_t|_tj||j|dd|_ dS)Nr) size_emb_dimrXTbias) super__init__rWrembnnSiLUsiluLinearlinear)selfrVrWrX __class__r-r.r]ys   zLTX2AdaLayerNormSingle.__init__Ntimestepadded_cond_kwargs batch_size hidden_dtypercCs>|pddd}|j|fi|||d}||||fS)N) resolution aspect_ratiorirj)r^rcra)rdrgrhrirjembedded_timestepr-r-r.forwardszLTX2AdaLayerNormSingle.forward)rUF)NNN)rOrPrQrRintboolr]r!Tensordictstrr&tuplero __classcell__r-r-rer.rThs"rTc@seZdZdZdZdZddZ    ddddejdejdBd ejdBd e ejejfdBd e ejejfdBd ejfd dZ dS)LTX2AudioVideoAttnProcessora= Processor for implementing attention (SDPA is used by default if you're using PyTorch 2.0) for the LTX-2.0 model. Compared to the LTX-1.0 model, we allow the RoPE embeddings for the queries and keys to be separate so that we can support audio-to-video (a2v) and video-to-audio (v2a) cross attention. NcCstddr tddS)Nr query_dimrkv_headsdim_headdropoutr[cross_attention_dimout_biasqk_normnorm_epsnorm_elementwise_affinercsLt| dkr td||_|||_|dur|jn|||_||_|dur*|n||_||_||_ ||_ ||_ | |_ t jj||| | d|_t jj||| | d|_t jj||j|d|_t jj|j|j|d|_t jj|j|j|d|_t jg|_|jt jj|j|j |d|jt j|| dur|} || dS)NrzIOnly 'rms_norm_across_heads' is supported as a valid value for `qk_norm`.epselementwise_affinerZ)r\r]NotImplementedErrorhead_dim inner_dim inner_kv_dimrruse_biasrout_dimrrr!r_rrrrbrrr ModuleListrappendDropout_default_processor_cls set_processor)rdrrrrrr[rrrrrr processorrer-r.r]s0  zLTX2Attention.__init__r{r|r}r~rrc stt|jjjfdd|D}t|dkr,t d|d|jj j dfdd|D}|j||||||fi|}|S) Ncsg|] \}}|vr|qSr-r-).0krAattn_parametersr-r. sz)LTX2Attention.forward..rzattention_kwargs z are not expected by z and will be ignored.csi|] \}}|vr||qSr-r-)rrwrr-r. sz)LTX2Attention.forward..) setinspect signaturerr parameterskeysitemslenloggerwarningrfrO)rdr{r|r}r~rkwargs unused_kwargsr-rr.ros   zLTX2Attention.forward) rrrrTNTrrTrNr)rOrPrQrRrwr_available_processorsrpr$rqrtr]r!rrrurorvr-r-rer.rzst     /rzc's@eZdZdZ       d.ded ed ed ed ed ededededededededeffdd Z        d/de j de j de j de j de j de j de j d e j d!e j d"e j d#e e j e j fdBd$e e j e j fdBd%e e j e j fdBd&e e j e j fdBd'e j dBd(e j dBd)e j dBd*e j dBd+e j f&d,d-Z Z S)0LTX2VideoTransformerBlocka Transformer block used in [LTX-2.0](https://huggingface.co/Lightricks/LTX-Video). Args: dim (`int`): The number of channels in the input and output. num_attention_heads (`int`): The number of heads to use for multi-head attention. attention_head_dim (`int`): The number of channels in each head. qk_norm (`str`, defaults to `"rms_norm"`): The normalization layer to use. activation_fn (`str`, defaults to `"gelu-approximate"`): Activation function to use in feed-forward. eps (`float`, defaults to `1e-6`): Epsilon value for normalization layers. rgelu-approximateTrFrrnum_attention_headsattention_head_dimr audio_dimaudio_num_attention_headsaudio_cross_attention_dimr activation_fnattention_biasattention_out_biasrrrc stt|| |d|_t||||| d| | |d |_t|| |d|_t||||| d| | |d |_t|| |d|_t|||||| | | |d |_ t|| |d|_ t|||||| | | |d |_ t|| |d|_ t|||||| | | |d |_ t|| |d|_t|||||| | | |d |_t|| |d|_t|| d|_t|| |d|_t|| d|_ttd||d|_ttd||d|_ttd||_ttd||_dS)Nr) rrrrr[rrrr) rrrrrr[rrr)rrU?)r\r]rnorm1rzattn1 audio_norm1 audio_attn1norm2attn2 audio_norm2 audio_attn2audio_to_video_normaudio_to_video_attnvideo_to_audio_normvideo_to_audio_attnnorm3rff audio_norm3audio_ffr_ Parameterr!randnscale_shift_tableaudio_scale_shift_table&video_a2v_cross_attn_scale_shift_table&audio_a2v_cross_attn_scale_shift_table)rdrrrrrraudio_attention_head_dimrrrrrrrrrer-r.r]:s      z"LTX2VideoTransformerBlock.__init__Nr{audio_hidden_statesr|audio_encoder_hidden_statestemb temb_audiotemb_ca_scale_shifttemb_ca_audio_scale_shift temb_ca_gatetemb_ca_audio_gatevideo_rotary_embaudio_rotary_embca_video_rotary_embca_audio_rotary_embencoder_attention_maskaudio_encoder_attention_maska2v_cross_attention_maskv2a_cross_attention_maskrc@Cs|d}||}|jjd}|jd|j|||d|d}|jdd\}}}}}}|d||}|j|d| d}|||}| |}|j jd}|j d|j|||d|d} | jdd\}!}"}#}$}%}&|d|"|!}|j |d| d}'||'|#}| |}|j ||d|d}||}||}|j||d|d}'||'}||}||}|jdd ddf}(|jd dddf})|(ddddd f|j|||jdd djdd}*|)ddddd f| j| || jdddjdd}+|*\},}-}.}/|+dd}0|jdd ddf}1|jd dddf}2|1ddddd f|j|||jdd djdd}3|2ddddd f| j| || jdddjdd}4|3\}5}6}7}8|4dd}9|d|,d|-d}:|d|5d|6d};|j|:|;| ||d }<||0|<}|d|.d|/d}:|d|7d|8d};|j|;|:|| |d }=||9|=}||d||}||}>||>|}||d|%|$}||}?||?|&}||fS) NrNNr1rr r)r{r|r~)r|r~r}r0.)r|r~rr})sizerrr6r%devicer7r rrrrrrrrrrrr&squeezerrrrrrr)@rdr{rr|rrrrrrrrrrrrrrrrinorm_hidden_statesnum_ada_params ada_values shift_msa scale_msagate_msa shift_mlp scale_mlpgate_mlpattn_hidden_statesnorm_audio_hidden_statesnum_audio_ada_paramsaudio_ada_valuesaudio_shift_msaaudio_scale_msaaudio_gate_msaaudio_shift_mlpaudio_scale_mlpaudio_gate_mlpattn_audio_hidden_statesvideo_per_layer_ca_scale_shiftvideo_per_layer_ca_gatevideo_ca_scale_shift_table video_ca_gatevideo_a2v_ca_scalevideo_a2v_ca_shiftvideo_v2a_ca_scalevideo_v2a_ca_shifta2v_gateaudio_per_layer_ca_scale_shiftaudio_per_layer_ca_gateaudio_ca_scale_shift_table audio_ca_gateaudio_a2v_ca_scaleaudio_a2v_ca_shiftaudio_v2a_ca_scaleaudio_v2a_ca_shiftv2a_gatemod_norm_hidden_statesmod_norm_audio_hidden_statesa2v_attn_hidden_statesv2a_attn_hidden_states ff_outputaudio_ff_outputr-r-r.ros                      z!LTX2VideoTransformerBlock.forward)rrTTrFr)NNNNNNNN)rOrPrQrRrprtrqr$r]r!rrrurorvr-r-rer.r's          rc!seZdZdZ           d3d ededededededededeedfdededededededdf fdd Z !d4d"ed#ed$ed%ed&e j d'ede j fd(d)Z *d5d"ed#ed&e j d+ede j f d,d-Zd.d/Z d6d0e j d&ee j BdBdee j e j ffd1d2ZZS)7LTX2AudioVideoRotaryPosEmbedax Video and audio rotary positional embeddings (RoPE) for the LTX-2.0 model. Args: causal_offset (`int`, *optional*, defaults to `1`): Offset in the temporal axis for causal VAE modeling. This is typically 1 (for causal modeling where the VAE treats the very first frame differently), but could also be 0 (for non-causal modeling). r1>r r2@videoTrr2r patch_size patch_size_tbase_num_frames base_height base_width sampling_rate hop_length scale_factors.theta causal_offsetmodalitydouble_precisionrrrNcst||_||_||_|dvrtd|d||_||_||_||_ ||_ ||_ ||_ t |t |t | d|_| |_| |_| |_| |_|jdvrWtd| d| |_dS)N)rrz rope_type=z9 not supported. Choose between 'interleaved' and 'split'.r)r4audioz Modality z@ is not supported. Supported modalities are `video` and `audio`.)r\r]rr5r6r9rr7rr8r9r:r;r$audio_latents_per_secondr<r=r>r?r@)rdrr5r6r7r8r9r:r;r<r=r>r?r@rrrer-r.r]]s*   z%LTX2AudioVideoRotaryPosEmbed.__init__8@ri num_framesheightwidthrfpscCshtjd||jtj|d}tjd||jtj|d}tjd||jtj|d} tj||| dd} tj| dd} |j|j|jf} tj| | j| j d} | | dddd} tj| | gd d}| dd}| d |ddd}tj|j|j d }dg|j}d |d<||j |}|d d dd f|j|jdjdd |d d dd f<|d d dd f||d d dd f<|S)a  Create per-dimension bounds [inclusive start, exclusive end) for each patch with respect to the original pixel space video grid (num_frames, height, width). This will ultimately have shape (batch_size, 3, num_patches, 2) where - axis 1 (size 3) enumerates (frame, height, width) dimensions (e.g. idx 0 corresponds to frames) - axis 3 (size 2) stores `[start, end)` indices within each dimension Args: batch_size (`int`): Batch size of the video latents. num_frames (`int`): Number of latent frames in the video latents. height (`int`): Latent height of the video latents. width (`int`): Latent width of the video latents. device (`torch.device`): Device on which to create the video grid. Returns: `torch.Tensor`: Per-dimension patch boundaries tensor of shape [batch_size, 3, num_patches, 2]. rstartendstepr&rij)indexingr)r&rrr1rrN.min)r!aranger6float32r5meshgridr"tensorr&rrr#r:repeatr<r5r>clamp)rdrirDrErFrrGgrid_fgrid_hgrid_wgridr5patch_size_delta patch_ends latent_coords scale_tensorbroadcast_shape pixel_coordsr-r-r.prepare_video_coordss$#  8$z1LTX2AudioVideoRotaryPosEmbed.prepare_video_coordsrshiftc Cstj||||jtj|d}|jd}||}||j|jdd}||j|j}||j|} | |j|jdd} | |j|j} tj || gdd} | d |dd} | d} | S)a Create per-dimension bounds [inclusive start, exclusive end) of start and end timestamps for each latent frame. This will ultimately have shape (batch_size, 3, num_patches, 2) where - axis 1 (size 1) represents the temporal dimension - axis 3 (size 2) stores `[start, end)` indices within each dimension Args: batch_size (`int`): Batch size of the audio latents. num_frames (`int`): Number of latent frames in the audio latents. device (`torch.device`): Device on which to create the audio grid. shift (`int`, *optional*, defaults to `0`): Offset on the latent indices. Different shift values correspond to different overlapping windows with respect to the same underlying latent grid. Returns: `torch.Tensor`: Per-dimension patch boundaries tensor of shape [batch_size, 1, num_patches, 2]. rHrrOrrr1) r!rQr6rRr<r>clipr;r:r"r:expand) rdrirDrrbrWaudio_scale_factorgrid_start_mel grid_start_s grid_end_mel grid_end_s audio_coordsr-r-r.prepare_audio_coordss  z1LTX2AudioVideoRotaryPosEmbed.prepare_audio_coordscOs8|jdkr |j|i|S|jdkr|j|i|SdS)Nr4rA)r?rark)rdargsrr-r-r.prepare_coords s  z+LTX2AudioVideoRotaryPosEmbed.prepare_coordscoordsc s|pj}jd}jdkr#jddd\}}||dd|jdkr1|j|j|jfn |jdkr:|jft j fd d t |Ddd |}|d}|j rXt jnt j}t |jt jd d |j|||d } | t jdj t jd} |ddd| } | ddd} |jdkr| jddd} | jddd} |j|dkrt | ddddd|j|f} t | ddddd|j|f}t j| | gdd} t j|| gdd} | | fS|jdkrd|jd}| jd}||}| }| }|dkr:t |ddddd|f} t |ddddd|f}t j| |gdd}t j||gdd}|jd}|jd}| |||j!d}| |||j!d}t "|dd} t "|dd} | | fS)Nr1r0r rrg@r4rAcs$g|]}dd|f|qSNr-)rirn max_positionsr-r.r%s$z8LTX2AudioVideoRotaryPosEmbed.forward..rg?)rIrJstepsr&rr4r3rrr)axis)#rr6r5chunkrr?r7r8r9r!r"ranger%r@float64rRpowr=linspacerpir: transposer#rr'repeat_interleaver( ones_like zeros_likecat concatenater7rr8)rdrnr num_pos_dims coords_start coords_endrZnum_rope_elems freqs_dtype pow_indicesr cos_freqs sin_freqs cos_padding sin_paddingexpected_freqs current_freqspad_sizecos_freqsin_freqr>r@r-rqr.ros^       ( &&       z$LTX2AudioVideoRotaryPosEmbed.forward)r1r1r-r.r.r/r0r1r3r1r4Trr2)rC)rro)rOrPrQrRrprur$rtrqr]r!rrrrarkrmrorvr-r-rer.r,Ss      8 K 6 r,cLs4eZdZdZdZdgZdgZeddddeddddedd ddd eddddeddddd eddd d Z e                                    d\d e d!e d"Bd#e d$e d%e d&e d'e d(e e e e fd)e d*e d+e d,e d-e d"Bd.e d/e d0e d1e d2e d3e d4e d5e d6e d7e d8e d9e d:ed;eded?ed@edAe dBe dCe dDe dEd"fJfdFdG ZedH " " " " " " I " " " " d]dJejdKejdLejdMejdNejdOejd"BdPejd"BdQejd"BdRe d"BdSe d"BdTe d"BdUedVe d"BdWejd"BdXejd"BdHee efd"BdYedEejf$dZd[ZZS)^LTX2VideoTransformer3DModela A Transformer model for video-like data used in [LTX](https://huggingface.co/Lightricks/LTX-Video). Args: in_channels (`int`, defaults to `128`): The number of channels in the input. out_channels (`int`, defaults to `128`): The number of channels in the output. patch_size (`int`, defaults to `1`): The size of the spatial patches to use in the patch embedding layer. patch_size_t (`int`, defaults to `1`): The size of the tmeporal patches to use in the patch embedding layer. num_attention_heads (`int`, defaults to `32`): The number of heads to use for multi-head attention. attention_head_dim (`int`, defaults to `64`): The number of channels in each head. cross_attention_dim (`int`, defaults to `2048 `): The number of channels for cross attention heads. num_layers (`int`, defaults to `28`): The number of layers of Transformer blocks to use. activation_fn (`str`, defaults to `"gelu-approximate"`): Activation function to use in feed-forward. qk_norm (`str`, defaults to `"rms_norm_across_heads"`): The normalization layer to use. Tnormrr1rF) split_dim expected_dims split_outputr )r{r|r)rr1) gather_dimr)ropeproj_outr2r1r-r.rr0r/r00rrrr3r in_channels out_channelsNr5r6rrrvae_scale_factorspos_embed_max_posr8r9audio_in_channelsaudio_out_channelsaudio_patch_sizeaudio_patch_size_trrrreaudio_pos_embed_max_posaudio_sampling_rateaudio_hop_length num_layersrrrrcaption_channelsrr rope_thetarope_double_precisionr>timestep_scale_multiplier$cross_attn_timestep_scale_multiplierrrc%&st|p|}| p | }  t| |_t| |_t| d|_t|d|_t ddd|_ t ddd|_ t ddd|_ t ddd|_ t ddd|_t ddd|_ttd  d|_ttdd|_t ||| | | |||!d |  d |_t||||||g||!d | d |_t| |}%t|||%| | ||!d |  d |_t|||%||||!d | d |_t fddt|D|_tj ddd|_t ||_ tjddd|_!t| |_"d|_#dS)N) in_features hidden_sizerUF)rWrXr0r1r rr4) rr5r6r7r8r9r<r=r>r?r@rrrA) rr5r6r7r:r;r<r=r>r?r@rr) rr5r6r7r8r9r=r>r?r@rr) rr5r6r7r:r;r=r>r?r@rrcs2g|]}t   dqS))rrrrrrrrrrrrrrr)r)rrArrrrrraudio_inner_dimrrrrrrrrr-r.r(s(z8LTX2VideoTransformer3DModel.__init__..rr)$r\r]r_rbproj_in audio_proj_inrcaption_projectionaudio_caption_projectionrT time_embedaudio_time_embedav_cross_attn_video_scale_shiftav_cross_attn_audio_scale_shiftav_cross_attn_video_a2v_gateav_cross_attn_audio_v2a_gaterr!rrrr,r audio_ropemaxcross_attn_ropecross_attn_audio_roperrvtransformer_blocks LayerNormnorm_outraudio_norm_outaudio_proj_outgradient_checkpointing)&rdrrr5r6rrrrrr8r9rrrrrrrrerrrrrrrrrrrrrr>rrrcross_attn_pos_embed_max_posrerr.r]s (  & z$LTX2VideoTransformer3DModel.__init__attention_kwargsrCr{rr|rrgaudio_timesteprrrDrErFrGaudio_num_frames video_coordsrj return_dictc*Cs|dur|n|}|dur |jdkr d||jd}|d}|dur8|jdkr8d||jd}|d}|d}|durN|jj|| | | |j| d}|dur[|j || |j}|j||jd}|j||jd}|j |ddddddf|jd}|j |ddddddf|jd}| |}| |}|jj|jj}|j|||jd\}}||d |d }||d |d }|j|||jd\}}||d |d }||d |d }|j|||jd\}}|j||||jd\}}||d |jd }||d |jd }|j|||jd\}}|j||||jd\} }||d |jd }| |d | jd } ||}||d |d }||}||d |d }|jD]\}!tr|jr| |!|||||||||| ||||||\}}qf|!did |d |d |d |d|d|d|d|d|d| d|d|d|d|d|d|\}}qf|j!d|dddddf}"|"dddddf|"dddddf}#}$|"|}|d|$|#}|#|}%|j$d|dddddf}&|&dddddf|&dddddf}'}(|%|}|d|(|'}|&|})|s<|%|)fSt'|%|)dS)a Forward pass for LTX-2.0 audiovisual video transformer. Args: hidden_states (`torch.Tensor`): Input patchified video latents of shape `(batch_size, num_video_tokens, in_channels)`. audio_hidden_states (`torch.Tensor`): Input patchified audio latents of shape `(batch_size, num_audio_tokens, audio_in_channels)`. encoder_hidden_states (`torch.Tensor`): Input video text embeddings of shape `(batch_size, text_seq_len, self.config.caption_channels)`. audio_encoder_hidden_states (`torch.Tensor`): Input audio text embeddings of shape `(batch_size, text_seq_len, self.config.caption_channels)`. timestep (`torch.Tensor`): Input timestep of shape `(batch_size, num_video_tokens)`. These should already be scaled by `self.config.timestep_scale_multiplier`. audio_timestep (`torch.Tensor`, *optional*): Input timestep of shape `(batch_size,)` or `(batch_size, num_audio_tokens)` for audio modulation params. This is only used by certain pipelines such as the I2V pipeline. encoder_attention_mask (`torch.Tensor`, *optional*): Optional multiplicative text attention mask of shape `(batch_size, text_seq_len)`. audio_encoder_attention_mask (`torch.Tensor`, *optional*): Optional multiplicative text attention mask of shape `(batch_size, text_seq_len)` for audio modeling. num_frames (`int`, *optional*): The number of latent video frames. Used if calculating the video coordinates for RoPE. height (`int`, *optional*): The latent video height. Used if calculating the video coordinates for RoPE. width (`int`, *optional*): The latent video width. Used if calculating the video coordinates for RoPE. fps: (`float`, *optional*, defaults to `24.0`): The desired frames per second of the generated video. Used if calculating the video coordinates for RoPE. audio_num_frames: (`int`, *optional*): The number of latent audio frames. Used if calculating the audio coordinates for RoPE. video_coords (`torch.Tensor`, *optional*): The video coordinates to be used when calculating the rotary positional embeddings (RoPE) of shape `(batch_size, 3, num_video_tokens, 2)`. If not supplied, this will be calculated inside `forward`. audio_coords (`torch.Tensor`, *optional*): The audio coordinates to be used when calculating the rotary positional embeddings (RoPE) of shape `(batch_size, 1, num_audio_tokens, 2)`. If not supplied, this will be calculated inside `forward`. attention_kwargs (`dict[str, Any]`, *optional*): Optional dict of keyword args to be passed to the attention processor. return_dict (`bool`, *optional*, defaults to `True`): Whether to return a dict-like structured output of type `AudioVisualModelOutput` or a tuple. Returns: `AudioVisualModelOutput` or `tuple`: If `return_dict` is `True`, returns a structured output of type `AudioVisualModelOutput`, otherwise a `tuple` is returned where the first element is the denoised video latent patch sequence and the second element is the denoised audio latent patch sequence. Nr r1gr)rGrNrmrr{rr|rrrrrrrrrrrrrr)rMrNr-)(r5r%r&r:rrrarrrkrrrrconfigrrrr#rrrrr6rrrrrr!is_grad_enabledr_gradient_checkpointing_funcrrrrrrrL)*rdr{rr|rrgrrrrDrErFrGrrrjrrrirrvideo_cross_attn_rotary_embaudio_cross_attn_rotary_emb%timestep_cross_attn_gate_scale_factorrrnraudio_embedded_timestepvideo_cross_attn_scale_shiftrAvideo_cross_attn_a2v_gateaudio_cross_attn_scale_shiftaudio_cross_attn_v2a_gateblockscale_shift_valuesrbscaleoutputaudio_scale_shift_values audio_shift audio_scale audio_outputr-r-r.roGsH   &                      .   .   z#LTX2VideoTransformer3DModel.forward)$rrr1r1r2rrr1r-r.r.rrr1r1r2rr.r0r-r/r0rrrFrrTTr3Tr1rrr) NNNNNNrCNNNNT)rOrPrQrR _supports_gradient_checkpointing _skip_layerwise_casting_patterns_repeated_blocksrr_cp_planrrprurtrqr$r]r r!rr LongTensorrsrrorvr-r-rer.r^sb             !"#$%&<     r)4r dataclassesrtypingrr!torch.nnr_configuration_utilsrrloadersrrutilsr r r r _modeling_parallelrr attentionrrrattention_dispatchr cache_utilsr embeddingsrrmodeling_utilsr normalizationr get_loggerrOrrrrur/rKrLModulerTrwrzrr,rr-r-r-r.s@        &&))KK.