o ۷i@sNddlZddlmZddlmZmZddlmZddlm Z ddl m Z ddl m Z dd lmZdd lmZd d lmZmZmZGd ddejZGdddejZGdddejZGdddejZGdddejZGdddejZGdddejZGdddejZGdddejZGdd d ejZ Gd!d"d"eeeeZ!dS)#N) ConfigMixinregister_to_config)FromOriginalModelMixin)apply_forward_hook)get_activation))PixArtAlphaCombinedTimestepSizeEmbeddings)AutoencoderKLOutput) ModelMixin)AutoencoderMixin DecoderOutputDiagonalGaussianDistributioncsPeZdZdZd dededdffdd Zdd ejdedBdejfd d Z Z S)PerChannelRMSNorma  Per-pixel (per-location) RMS normalization layer. For each element along the chosen dimension, this layer normalizes the tensor by the root-mean-square of its values across that dimension: y = x / sqrt(mean(x^2, dim=dim, keepdim=True) + eps) r :0yE> channel_dimepsreturnNcst||_||_dS)z Args: dim: Dimension along which to compute the RMS (typically channels). eps: Small constant added for numerical stability. N)super__init__rr)selfrr __class__g/home/ubuntu/vllm_env/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_ltx2.pyr(s  zPerChannelRMSNorm.__init__xcCs8|p|j}tj|d|jdd}t||j}||S)zI Apply RMS normalization along the configured dimension. rT)dimkeepdim)rtorchmeansqrtr)rrrmean_sqrmsrrrforward2s zPerChannelRMSNorm.forward)r rN) __name__ __module__ __qualname____doc__intfloatrrTensorr$ __classcell__rrrrrs ( rcseZdZ     ddededeeeeefBdeeeeefBdeeeeefBd ed effd d Zddejde dejfddZ Z S)LTX2VideoCausalConv3drr zeros in_channels out_channels kernel_sizestridedilationgroupsspatial_padding_modec st||_||_t|tr|n|||f|_t|tr|n|ddf}t|tr+|n|||f}|jdd}|jdd} d|| f} tj|||j|||| |d|_ dS)Nr rr)r3r4r5padding padding_mode) rrr0r1 isinstancetupler2nnConv3dconv) rr0r1r2r3r4r5r6 height_pad width_padr7rrrr@s&   zLTX2VideoCausalConv3d.__init__T hidden_statescausalrcCs|jd}|r.|ddddddddddfdd|dddf}tj||gdd}nH|ddddddddddfdd|ddddf}|ddddddddddfdd|ddddf}tj|||gdd}||}|S)Nrr rr)r2repeatr concatenater=)rr@rAtime_kernel_sizepad_left pad_rightrrrr$as :>> zLTX2VideoCausalConv3d.forward)rr r r r/T r&r'r(r*r:strrrr,boolr$r-rrrrr.?s,$!r.cseZdZdZ        dded edBd ed ed ed ededededdffdd Z   dde j de j dBde j dBdede j f ddZ Z S)LTX2VideoResnetBlock3da A 3D ResNet block used in the LTX 2.0 audiovisual model. Args: in_channels (`int`): Number of input channels. out_channels (`int`, *optional*): Number of output channels. If None, defaults to `in_channels`. dropout (`float`, defaults to `0.0`): Dropout rate. eps (`float`, defaults to `1e-6`): Epsilon value for normalization layers. elementwise_affine (`bool`, defaults to `False`): Whether to enable elementwise affinity in the normalization layers. non_linearity (`str`, defaults to `"swish"`): Activation function to use. conv_shortcut (bool, defaults to `False`): Whether or not to use a convolution shortcut. Nư>Fswishr/r0r1dropoutrelementwise_affine non_linearity inject_noisetimestep_conditioningr6rc st|p|}t||_t|_t||d| d|_t|_t ||_ t||d| d|_ d|_ d|_||krLt j||ddd|_ t j||ddd|_d|_d|_|rjt t|dd|_t t|dd|_d|_|rt td||d|_dSdS) Nr)r0r1r2r6T)rrRbiasr )r0r1r2r3?)rrr nonlinearityrnorm1r.conv1norm2r;DropoutrQconv2norm3 conv_shortcut LayerNormr<per_channel_scale1per_channel_scale2 Parameterrr/scale_shift_tablerandn) rr0r1rQrrRrSrTrUr6rrrrs@    zLTX2VideoResnetBlock3d.__init__Tinputstemb generatorrAc Cs|}||}|jdur)|dd|jd}|jdd\}}}} |d||}||}|j||d}|jdur]|jdd} tj | ||j |j dd} || |jdddddf}| |}|jduro|d| |}||}| |}|j||d}|jdur|jdd} tj | ||j |j dd} || |jdddddf}|jdur||dd d d}|jdur||}||}|S) Nr )rWrCN.NNNrBrA)ridevicedtype.rC)rZre unflattenunbindrYr[rbshaperrfrmrnr\rQr^rcr_movedimr`) rrgrhrirAr@shift_1scale_1shift_2scale_2 spatial_shape spatial_noiserrrr$sH              zLTX2VideoResnetBlock3d.forward)NrNrOFrPFFr/NNT)r&r'r(r)r*r+rLrKrrr, Generatorr$r-rrrrrMrsZ   7rMc sbeZdZ  ddededeeeeefBdeddf fd d Zdd ejd e dejfddZ Z S)LTXVideoDownsampler3dr r/r0r1r3r6rNcstt|tr |n|||f|_||d|d|d||_||jd|jd|jd}t||dd|d|_dSNrr rrr0r1r2r3r6)rrr9r:r3 group_sizer.r=)rr0r1r3r6rrrrs "" zLTXVideoDownsampler3d.__init__Tr@rAc Cstj|ddddd|jddf|gdd}|dd|jdfdd|jdfdd|jdf}|ddddd ddd dd}|dd|jf}|jdd}|j||d }|dd|jdfdd|jdfdd|jdf}|ddddd ddd dd}||}|S) Nrr rrBrWrCrrk) rcatr3ropermuteflattenr~r r=)rr@rAresidualrrrr$s 2   zLTXVideoDownsampler3d.forward)r r/rIrJrrrrr{s$r{c sjeZdZ    ddedeeeeefBdededed d f fd d Zddej ded ej fddZ Z S)LTXVideoUpsampler3dr Fr/r0r3rupscale_factorr6rNcshtt|tr |n|||f|_||_||_||d|d|d|}t||dd|d|_dSr|) rrr9r:r3rrr.r=)rr0r3rrr6r1rrrr!s   zLTXVideoUpsampler3d.__init__Tr@rAc Csh|j\}}}}}|jre||d|jd|jd|jd|||}|dddddddd ddd ddd}|jd|jd|jd|j} |d| ddd}|dddd|jdddf}|j||d }||d|jd|jd|jd|||}|dddddddd ddd ddd}|dddd|jdddf}|jr||}|S) NrCrr rrrrrrWrk) rqrreshaper3rrrrDr=) rr@rA batch_size num_channels num_framesheightwidthrrepeatsrrrr$9s$"0$$"0$zLTXVideoUpsampler3d.forward)r Fr r/rI) r&r'r(r*r:rLrKrrr,r$r-rrrrr s&$rcseZdZdZdZ        dd ed edBd ed edededededeffdd Z   dde j de j dBde j dBdede j f ddZ ZS)LTX2VideoDownBlock3Da Down block used in the LTXVideo model. Args: in_channels (`int`): Number of input channels. out_channels (`int`, *optional*): Number of output channels. If None, defaults to `in_channels`. num_layers (`int`, defaults to `1`): Number of resnet layers. dropout (`float`, defaults to `0.0`): Dropout rate. resnet_eps (`float`, defaults to `1e-6`): Epsilon value for normalization layers. resnet_act_fn (`str`, defaults to `"swish"`): Activation function to use. spatio_temporal_scale (`bool`, defaults to `True`): Whether or not to use a downsampling layer. If not used, output dimension would be same as input dimension. Whether or not to downsample across temporal dimension. is_causal (`bool`, defaults to `True`): Whether this layer behaves causally (future frames depend only on past frames) or not. TNr rNrOrPr=r/r0r1 num_layersrQ resnet_eps resnet_act_fnspatio_temporal_scaledownsample_typer6c st|p|}g} t|D]} | t|||||| dqt| |_d|_|rst|_|dkrA|jt ||dd| dn2|dkrR|jt ||d| dn!|d krc|jt ||d | dn|d krs|jt ||d| dd |_ dS) N)r0r1rQrrSr6r=rrrrr}spatial)r rr)r0r1r3r6temporal)rr r spatiotemporalF) rrrangeappendrMr; ModuleListresnets downsamplersr.r{gradient_checkpointing) rr0r1rrQrrrrr6r_rrrrmsp      zLTX2VideoDownBlock3D.__init__r@rhrirArcCslt|jD]\}}tr|jr||||||}q|||||d}q|jdur4|jD]}|||d}q+|S)z-Forward method of the `LTXDownBlock3D` class.rkN) enumeraterris_grad_enabledr_gradient_checkpointing_funcr)rr@rhrirAiresnet downsamplerrrrr$s   zLTX2VideoDownBlock3D.forward)Nr rNrOrPTr=r/ryr&r'r(r) _supports_gradient_checkpointingr*r+rKrLrrr,rzr$r-rrrrrSsX  MrcseZdZdZdZ       dd ed ed ed ed ededededdffdd Z   dde j de j dBde j dBdede j f ddZ ZS)LTX2VideoMidBlock3das A middle block used in the LTXVideo model. Args: in_channels (`int`): Number of input channels. num_layers (`int`, defaults to `1`): Number of resnet layers. dropout (`float`, defaults to `0.0`): Dropout rate. resnet_eps (`float`, defaults to `1e-6`): Epsilon value for normalization layers. resnet_act_fn (`str`, defaults to `"swish"`): Activation function to use. is_causal (`bool`, defaults to `True`): Whether this layer behaves causally (future frames depend only on past frames) or not. Tr rNrOrPFr/r0rrQrrrTrUr6rNc sjtd|_|rt|dd|_g} t|D]} | t||||||||dqt| |_ d|_ dS)NrWrr0r1rQrrSrTrUr6F) rr time_embedderr rrrMr;rrr) rr0rrQrrrTrUr6rrrrrrs(   zLTX2VideoMidBlock3d.__init__r@rhrirAcCs|jdur!|j|dd|d|jd}||ddddd}t|jD]\}}tr;|j r;| |||||}q&|||||d}q&|S)z,Forward method of the `LTXMidBlock3D` class.Nrtimestep resolution aspect_ratior hidden_dtyperCr rk) rrsizernviewrrrrrr)rr@rhrirArrrrrr$s zLTX2VideoMidBlock3d.forward)r rNrOrPFFr/ryrrrrrrsV  &rcseZdZdZdZ           dd ed edBd ed ededededededededeffdd Z   d de j de j dBde j dBdede j f ddZ ZS)!LTX2VideoUpBlock3da Up block used in the LTXVideo model. Args: in_channels (`int`): Number of input channels. out_channels (`int`, *optional*): Number of output channels. If None, defaults to `in_channels`. num_layers (`int`, defaults to `1`): Number of resnet layers. dropout (`float`, defaults to `0.0`): Dropout rate. resnet_eps (`float`, defaults to `1e-6`): Epsilon value for normalization layers. resnet_act_fn (`str`, defaults to `"swish"`): Activation function to use. spatio_temporal_scale (`bool`, defaults to `True`): Whether or not to use a downsampling layer. If not used, output dimension would be same as input dimension. Whether or not to downsample across temporal dimension. is_causal (`bool`, defaults to `True`): Whether this layer behaves causally (future frames depend only on past frames) or not. TNr rNrOrPFr/r0r1rrQrrrrTrUupsample_residualrr6c  st|p|}d|_| rt|dd|_d|_||kr*t||||||| | d|_d|_|r?tt || d| | | dg|_g} t |D]}| t||||||| | dqEt| |_ d|_ dS)NrWrrr)r3rrr6F)rrrr conv_inrM upsamplersr;rrrrrr)rr0r1rrQrrrrTrUrrr6rrrrrr>sZ    zLTX2VideoUpBlock3d.__init__r@rhrirArcCs|jdur|j||||d}|jdur/|j|dd|d|jd}||ddddd}|jdur@|jD]}|||d}q7t|jD]\}}t rZ|j rZ| |||||}qE|||||d}qE|S)NrkrrrCr ) rrrrrnrrrrrrrr)rr@rhrirA upsamplerrrrrrr$s&    zLTX2VideoUpBlock3d.forward) Nr rNrOrPTFFFr r/ryrrrrrr$sj     GrcseZdZdZ        d#dededeedfdeedfdeedfdeedfdeedfdededededeffdd Z d$de j dedBd e j fd!d"Z Z S)%LTX2VideoEncoder3daL The `LTXVideoEncoder3d` layer of a variational autoencoder that encodes input video samples to its latent representation. Args: in_channels (`int`, defaults to 3): Number of input channels. out_channels (`int`, defaults to 128): Number of latent channels. block_out_channels (`tuple[int, ...]`, defaults to `(256, 512, 1024, 2048)`): The number of output channels for each block. spatio_temporal_scaling (`tuple[bool, ...], defaults to `(True, True, True, True)`: Whether a block should contain spatio-temporal downscaling layers or not. layers_per_block (`tuple[int, ...]`, defaults to `(4, 6, 6, 2, 2)`): The number of layers per block. downsample_type (`tuple[str, ...]`, defaults to `("spatial", "temporal", "spatiotemporal", "spatiotemporal")`): The spatiotemporal downsampling pattern per block. Per-layer values can be - `"spatial"` (downsample spatial dims by 2x) - `"temporal"` (downsample temporal dim by 2x) - `"spatiotemporal"` (downsample both spatial and temporal dims by 2x) patch_size (`int`, defaults to `4`): The size of spatial patches. patch_size_t (`int`, defaults to `1`): The size of temporal patches. resnet_norm_eps (`float`, defaults to `1e-6`): Epsilon value for ResNet normalization layers. is_causal (`bool`, defaults to `True`): Whether this layer behaves causally (future frames depend only on past frames) or not. rirrrrTTTTrWrrrrrrrrrWr rOTr/r0r1block_out_channels.down_block_typesspatio_temporal_scalinglayers_per_blockr patch_size patch_size_tresnet_norm_eps is_causalr6c  s t||_| |_||d|_| |_|} t|j| dd| d|_t|}t g|_ t |D]/}| }||} ||dkrPt || ||| ||||| d}n td|||j |q0t| |d| | d |_t|_t |_t| |ddd| d|_d |_dS) Nrrr r}r)r0r1rrrrr6zUnknown down block type: rC)r0rrr6F)rrrrr0rr.rlenr;r down_blocksrr ValueErrorrr mid_blockrnorm_outSiLUconv_actconv_outr)rr0r1rrrrrrrrrr6output_channelnum_block_out_channelsr input_channel down_blockrrrrs\       zLTX2VideoEncoder3d.__init__Nr@rArc CsT|j}|j}|j\}}}}} ||} ||} | |} |p|j}|||| || || |}|dddddddddd}|j||d }t rc|j rc|j D] } | | |d d |}qL| |j |d d |}n|j D]} | ||d }qf|j ||d }||}||}|j||d }|d d d d f}|d|ddddd}tj||gdd }|S) z4The forward method of the `LTXVideoEncoder3d` class.rr rrrrrWrrkNrCrB)rrrqrrrrrrrrrrrrrrrDrr)rr@rApp_trrrrrpost_patch_num_framespost_patch_heightpost_patch_widthr last_channelrrrr$s4      zLTX2VideoEncoder3d.forward) rrrrrrrrWr rOTr/r%)r&r'r(r)r*r:rKrLr+rrr,r$r-rrrrrsP      (PrcseZdZdZ         d&dededeedfdeedfdeedfdededededeedfdedeedfdeedfdeddffdd Z  d'd!e j d"e j dBd#edBde j fd$d%Z Z S)(LTX2VideoDecoder3da The `LTXVideoDecoder3d` layer of a variational autoencoder that decodes its latent representation into an output sample. Args: in_channels (`int`, defaults to 128): Number of latent channels. out_channels (`int`, defaults to 3): Number of output channels. block_out_channels (`tuple[int, ...]`, defaults to `(128, 256, 512, 512)`): The number of output channels for each block. spatio_temporal_scaling (`tuple[bool, ...], defaults to `(True, True, True, False)`: Whether a block should contain spatio-temporal upscaling layers or not. layers_per_block (`tuple[int, ...]`, defaults to `(4, 3, 3, 3, 4)`): The number of layers per block. patch_size (`int`, defaults to `4`): The size of spatial patches. patch_size_t (`int`, defaults to `1`): The size of temporal patches. resnet_norm_eps (`float`, defaults to `1e-6`): Epsilon value for ResNet normalization layers. is_causal (`bool`, defaults to `False`): Whether this layer behaves causally (future frames depend only on past frames) or not. timestep_conditioning (`bool`, defaults to `False`): Whether to condition the model on timesteps. rrrrrTTTrrrrrWr rOFFFFrreflectr0r1r.rrrrrrrTrUrupsample_factorr6rNc st||_||_||d|_| |_tt|}tt|}tt|}tt| } tt| } tt| } |d}t||dd|d|_ t ||d|| d| |d|_ t |}t g|_t|D]2}|| |}||| |}t||||d|||| |d| | || ||d }|j|qdt|_t |_t||jdd|d|_d|_d|_d|_| rt tjdtjd |_t|dd|_t t d||d |_d |_!dS) Nrrrr r})r0rrrTrUr6) r0r1rrrrTrUrrr6g@@)rnrXF)"rrrrr1rr:reversedr.rrrrr;r up_blocksrrrrrrrrrretimestep_scale_multiplierrdrtensorfloat32r rfr)rr0r1rrrrrrrrTrUrrr6rrrrup_blockrrrrZs|                zLTX2VideoDecoder3d.__init__r@rhrAc Cs|p|j}|j||d}|jdur||j}tr7|jr7||j||d|}|jD] }||||d|}q*n|j|||d}|jD] }||||d}qB| |}|j dur|j | dd| d|j d}|| ddddddd}||jd}|jdd\}}|d||}||}|j||d}|j}|j}|j\} } } } } || d|||| | | }|ddd d d d d d d d  d d  d d}|S)NrkrrrCr )rrCrjrBrrrrWrr)rrrrrrrrrrrrrrnrrorerprrrrrqrr)rr@rhrArshiftscalerrrrrrrrrrr$sB         0zLTX2VideoDecoder3d.forward)rrrrrrWr rOFrFrrr)NN)r&r'r(r)r*r:rLr+rKrrr,r$r-rrrrr>st         `rc6seZdZdZdZe                   d_dedededeedfdeedfdeedfdeedfdeedfd ee dfd!ee dfd"ee dfd#eedfd$ee dfd%eedfd&e d'ed(ed)e d*e d+e d,e d-ed.ed/ed0ed1df4fd2d3 Z      d`d4edBd5edBd6edBd7e dBd8e dBd9e dBd1dfd:d;Z dadd?Ze dbdsz1AutoencoderKLLTX2Video.encode..rk) latent_dist)rrqsplitrrrrr )rrrArencoded_slicesh posteriorrrrencodes  zAutoencoderKLLTX2Video.encodezrhcCs|j\}}}}} |j|j} |j|j} |j|j} |jr*|| kr*|j||||dS|jr>| | ks5|| kr>|j ||||dS|j |||d} |sK| fSt | dS)NrArrksample) rqr rr r rr_temporal_tiled_decoder tiled_decoderr)rr!rhrArrrrrrtile_latent_min_heighttile_latent_min_widthtile_latent_min_num_framesdecrrr_decodes    zAutoencoderKLLTX2Video._decodecsjr4|jddkr4|dur!fdd|d|dfD}n fdd|dD}t|}n j||dj}|sB|fSt|dS) a Decode a batch of images. Args: z (`torch.Tensor`): Input batch of latent vectors. return_dict (`bool`, *optional*, defaults to `True`): Whether to return a [`~models.vae.DecoderOutput`] instead of a plain tuple. Returns: [`~models.vae.DecoderOutput`] or `tuple`: If return_dict is True, a [`~models.vae.DecoderOutput`] is returned, otherwise a plain `tuple` is returned. rr Ncs"g|] \}}j||djqSrr+r$)rz_slicet_slicerrrrsz1AutoencoderKLLTX2Video.decode..csg|] }j|djqSrr,)rr-rrrrsrkr#)rrqrrrr+r$r)rr!rhrArdecoded_slicesdecodedrrrdecodes   zAutoencoderKLLTX2Video.decodeab blend_extentc Cst|jd|jd|}t|D]@}|dddddd| |ddfd|||dddddd|ddf|||dddddd|ddf<q|S)Nrr minrqr)rr2r3r4yrrrblend_v  R&zAutoencoderKLLTX2Video.blend_vcCst|jd|jd|}t|D]@}|dddddddd| |fd|||dddddddd|f|||dddddddd|f<q|S)NrWr r5rr2r3r4rrrrblend_hr9zAutoencoderKLLTX2Video.blend_hc Cst|jd|jd|}t|D]@}|dddd| |ddddfd|||dddd|ddddf|||dddd|ddddf<q|S)Nr r5r:rrrblend_tr9zAutoencoderKLLTX2Video.blend_tc Cs|j\}}}}}||j}||j} |j|j} |j|j} |j|j} |j|j} | | }| | }g}td||jD]6}g}td||jD]%}|j|dddddd|||j|||jf|d}||qF||q;g}t |D]O\}}g}t |D]:\}}|dkr| ||d|||}|dkr| ||d||}||ddddddd| d| fq|t j |ddqxt j |ddddddddd|d| f}|S)zEncode a batch of images using a tiled encoder. Args: x (`torch.Tensor`): Input batch of videos. Returns: `torch.Tensor`: The latent representation of the encoded videos. rNrkr rWrBr)rqrr r r rrrrrr8r;rr)rrrArrrrr latent_height latent_widthr'r(tile_latent_stride_heighttile_latent_stride_width blend_height blend_widthrowsrrowjtime result_rows result_rowtilerrrrr's>      0  .0z#AutoencoderKLLTX2Video.tiled_encodec Cs|j\}}}}} ||j} | |j} |j|j} |j|j} |j|j}|j|j}|j|j}|j|j}g}td||D]4}g}td| |D]$}|j|dddddd||| ||| f||d}||qH||q>g}t |D]Q\}}g}t |D]<\}}|dkr| ||d|||}|dkr| ||d||}||ddddddd|jd|jfq|t j |ddqyt j |ddddddddd| d| f}|s|fSt|dS) a Decode a batch of images using a tiled decoder. Args: z (`torch.Tensor`): Input batch of latent vectors. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.vae.DecoderOutput`] instead of a plain tuple. Returns: [`~models.vae.DecoderOutput`] or `tuple`: If return_dict is True, a [`~models.vae.DecoderOutput`] is returned, otherwise a plain `tuple` is returned. rNrkr rWrBrr#)rqrr r r rrrrrr8r;rrr)rr!rhrArrrrrr sample_height sample_widthr'r(r@rArBrCrDrrErFrGrHrIrJr*rrrr&[s@        0  20 z#AutoencoderKLLTX2Video.tiled_decodec Cs|j\}}}}}|d|jd}|j|j} |j|j} | | } g} td||jD]R} |dddd| | |jdddddf}|jrY||jksQ||jkrY|j||d}n|j ||d}| dkrw|ddddddddddf}| |q*g}t | D]B\} }| dkr| | | d|| }| |ddddd| ddddfq| |ddddd| dddddfqt j|ddddddd|f}|S)Nr rrkrrB)rqrr rrrr r rrrrr=rr)rrrArrrrrlatent_num_framesr)tile_latent_stride_num_framesblend_num_framesrErrJrIrrrrrs,  0& .2$z-AutoencoderKLLTX2Video._temporal_tiled_encodec Cs|j\}}}}} |d|jd} |j|j} |j|j} |j|j} |j|j}|j|j}g}td||D]Y}|dddd||| dddddf}|jrl|jd| ksa|jd| krl|j |||ddj }n|j |||d}|dkr|ddddddddddf}| |q7g}t |D]F\}}|dkr|||d||}|ddddd|jddddf}| |q| |ddddd|jdddddfqtj|dd ddddd| f}|s|fSt|d S) Nr rrCrlTr"rkrrBr#)rqrr rr r rrrr&r$rrrr=rrr)rr!rhrArrrrrrnum_sample_framesr'r(r)rNrOrErrJr0rIr*rrrr%s6     ."& ( 4$ z-AutoencoderKLLTX2Video._temporal_tiled_decoder$sample_posteriorric CsN|}|j||dj} |r| j|d} n| } |j| ||d} |s%| jfS| S)Nrk)ri)r rr$moder1) rr$rhrQrrrrirrr!r*rrrr$s zAutoencoderKLLTX2Video.forward)rrrrrrrrrrrrrrFrWr rOrTTr/rNN)NNNNNNr%)NTry)NFNNTN)!r&r'r(r)rrr*r:rKrLr+rrrr,rrr rr rr+r1r8r;r=rr&rr%rzr$r-rrrrrs'            r "     %   5 > ) r)"rtorch.nnr;configuration_utilsrrloadersrutils.accelerate_utilsr activationsr embeddingsr modeling_outputsr modeling_utilsr vaer rrModulerr.rMr{rrrrrrrrrrrs0       !3~03|U.