o Gi,N@sddlZddlmZddlmZddlmZmZddlm Z ddl m Z dd l m Z dd lmZd d lmZeeZd Z ddejdejdBdefddZGdddeZdS)N)PipelineImageInput)AutoencoderKLLTX2Video) get_loggerreplace_example_docstring) randn_tensor)VideoProcessor)LTXPipelineOutput)DiffusionPipeline)LTX2LatentUpsamplerModela Examples: ```py >>> import torch >>> from diffusers import LTX2ImageToVideoPipeline, LTX2LatentUpsamplePipeline >>> from diffusers.pipelines.ltx2.export_utils import encode_video >>> from diffusers.pipelines.ltx2.latent_upsampler import LTX2LatentUpsamplerModel >>> from diffusers.utils import load_image >>> pipe = LTX2ImageToVideoPipeline.from_pretrained("Lightricks/LTX-2", torch_dtype=torch.bfloat16) >>> pipe.enable_model_cpu_offload() >>> image = load_image( ... "https://huggingface.co/datasets/a-r-r-o-w/tiny-meme-dataset-captioned/resolve/main/images/8.png" ... ) >>> prompt = "A young girl stands calmly in the foreground, looking directly at the camera, as a house fire rages in the background." >>> negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted" >>> frame_rate = 24.0 >>> video, audio = pipe( ... image=image, ... prompt=prompt, ... negative_prompt=negative_prompt, ... width=768, ... height=512, ... num_frames=121, ... frame_rate=frame_rate, ... num_inference_steps=40, ... guidance_scale=4.0, ... output_type="pil", ... return_dict=False, ... ) >>> latent_upsampler = LTX2LatentUpsamplerModel.from_pretrained( ... "Lightricks/LTX-2", subfolder="latent_upsampler", torch_dtype=torch.bfloat16 ... ) >>> upsample_pipe = LTX2LatentUpsamplePipeline(vae=pipe.vae, latent_upsampler=latent_upsampler) >>> upsample_pipe.vae.enable_tiling() >>> upsample_pipe.to(device="cuda", dtype=torch.bfloat16) >>> video = upsample_pipe( ... video=video, ... width=768, ... height=512, ... output_type="np", ... return_dict=False, ... )[0] >>> encode_video( ... video[0], ... fps=frame_rate, ... audio=audio[0].float().cpu(), ... audio_sample_rate=pipe.vocoder.config.output_sampling_rate, # should be 24000 ... output_path="video.mp4", ... ) ``` sampleencoder_output generator sample_modecCsRt|dr|dkr|j|St|dr|dkr|jSt|dr%|jStd)N latent_distrargmaxlatentsz3Could not access latents of provided encoder_output)hasattrrrmoderAttributeError)rrrrj/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/pipelines/ltx2/pipeline_ltx2_latent_upsample.pyretrieve_latents[s   rc!seZdZdZdededdffdd Z        d9d ejdBd e de de de de de dej dBdej dBdej dBdejdBdejfddZ d:dejdejdefddZdejdedejfdd Ze d:dejd!ejd"ejd#edejf d$d%Ze d;dejde de de d&e d'e dejfd(d)Zd*d+Zeee     , -  - -  . /dlatent_upsamplervaelatent_upsamplerreturnNcsft|j||dt|dddur|jjnd|_t|dddur'|jjnd|_t |jd|_ dS)N)rrr )vae_scale_factor) super__init__register_modulesgetattrrspatial_compression_ratiovae_spatial_compression_ratiotemporal_compression_ratiovae_temporal_compression_ratiorvideo_processor)selfrr __class__rrr#ks z#LTX2LatentUpsamplePipeline.__init__r yvideo batch_size num_framesheightwidthspatial_patch_sizetemporal_patch_sizedtypedevicerrc s| dur-| jdkr&|djd} |j} |j}| | | |||} | j| |dSj| jjdttr\t |krNt dt d|dfddt |D}n fd dD}t j |d d |}|S) Nrr r9r8z/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.cs,g|]}tj|d|qSrrrencode unsqueeze).0irr+r1rr s z>LTX2LatentUpsamplePipeline.prepare_latents..cs$g|]}tj|dqSr;r<)r?vid)rr+rrrBs$rdim)ndimr)r'_unpack_latentstorr8 isinstancelistlen ValueErrorrangetorchcat)r+r1r2r3r4r5r6r7r8r9rrlatent_num_frames latent_height latent_width init_latentsrrArprepare_latents|s.       z*LTX2LatentUpsamplePipeline.prepare_latents?reference_latentsfactorc Cs|}t|dD]8}t|dD].}tj|||fdd\}}tj|||fdd\} } |||f| | |||||f<qq t|||}|S)a5 Applies Adaptive Instance Normalization (AdaIN) to a latent tensor based on statistics from a reference latent tensor. Args: latent (`torch.Tensor`): Input latents to normalize reference_latents (`torch.Tensor`): The reference latents providing style statistics. factor (`float`): Blending factor between original and transformed latent. Range: -10.0 to 10.0, Default: 1.0 Returns: torch.Tensor: The transformed latent tensor rr NrD)clonerMsizerNstd_meanlerp) r+rrVrWresultr@cr_sdr_meani_sdi_meanrrradain_filter_latents&z.LTX2LatentUpsamplePipeline.adain_filter_latent compressioncCsD|d}t|}td||d}dd||}||}|S)aL Applies a non-linear tone-mapping function to latent values to reduce their dynamic range in a perceptually smooth way using a sigmoid-based compression. This is useful for regularizing high-variance latents or for conditioning outputs during generation, especially when controlling dynamic behavior with a `compression` factor. Args: latents : torch.Tensor Input latent tensor with arbitrary shape. Expected to be roughly in [-1, 1] or [0, 1] range. compression : float Compression strength in the range [0, 1]. - 0.0: No tone-mapping (identity transform) - 1.0: Full compression effect Returns: torch.Tensor The tone-mapped latent tensor of the same shape as input. g?g@rUg?)rNabssigmoid)r+rrc scale_factor abs_latents sigmoid_termscalesfilteredrrrtone_map_latentss  z+LTX2LatentUpsamplePipeline.tone_map_latents latents_mean latents_stdscaling_factorcCsP|ddddd|j|j}|ddddd|j|j}||||}|S)Nr )viewrHr9r8)rrlrmrnrrr_denormalize_latentssz/LTX2LatentUpsamplePipeline._denormalize_latents patch_size patch_size_tc CsV|d}|||||d|||}|dddddddd dd dddd}|S) Nrror r r)rYreshapepermuteflatten)rr3r4r5rrrsr2rrrrGs 0z*LTX2LatentUpsamplePipeline._unpack_latentscCs||jdks||jdkrtd|d|d|dur%|dur%td|dur1|dur1tdd|kr>dksCtdtddS) Nrz8`height` and `width` have to be divisible by 32 but are z and .z1Only one of `video` or `latents` can be provided.z/One of `video` or `latents` has to be provided.r z8`tone_map_compression_ratio` must be in the range [0, 1])r'rL)r+r1r4r5rtone_map_compression_ratiorrr check_inputssz'LTX2LatentUpsamplePipeline.check_inputsFpilTlatents_normalizeddecode_timestepdecode_noise_scale adain_factorr| output_type return_dictc CsD|j||||| d|durd}n|jd}|j}|dur[t|}||jdkrJ||j|jd}|d|}td|jdt|d|d|jj|||d }|j |t j d }|du}|j |||||||t j || |d }|r|r| ||jj|jj|jjj}| |jj}||}| d kr|||| }n|}| d kr||| }|d kr|}ne|jjjsd}nJt|j| ||jd}t| ts| g|} | dur| } n t| ts| g|} t j| ||jd }t j| ||jd ddddddf} d| || |}|jj||ddd}|jj||d}||s|fSt|dS)a Function invoked when calling the pipeline for generation. Args: video (`list[PipelineImageInput]`, *optional*) The video to be upsampled (such as a LTX 2.0 first stage output). If not supplied, `latents` should be supplied. height (`int`, *optional*, defaults to `512`): The height in pixels of the input video (not the generated video, which will have a larger resolution). width (`int`, *optional*, defaults to `768`): The width in pixels of the input video (not the generated video, which will have a larger resolution). num_frames (`int`, *optional*, defaults to `121`): The number of frames in the input video. spatial_patch_size (`int`, *optional*, defaults to `1`): The spatial patch size of the video latents. Used when `latents` is supplied if unpacking is necessary. temporal_patch_size (`int`, *optional*, defaults to `1`): The temporal patch size of the video latents. Used when `latents` is supplied if unpacking is necessary. latents (`torch.Tensor`, *optional*): Pre-generated video latents. This can be supplied in place of the `video` argument. Can either be a patch sequence of shape `(batch_size, seq_len, hidden_dim)` or a video latent of shape `(batch_size, latent_channels, latent_frames, latent_height, latent_width)`. latents_normalized (`bool`, *optional*, defaults to `False`) If `latents` are supplied, whether the `latents` are normalized using the VAE latent mean and std. If `True`, the `latents` will be denormalized before being supplied to the latent upsampler. decode_timestep (`float`, defaults to `0.0`): The timestep at which generated video is decoded. decode_noise_scale (`float`, defaults to `None`): The interpolation factor between random noise and denoised latents at the decode timestep. adain_factor (`float`, *optional*, defaults to `0.0`): Adaptive Instance Normalization (AdaIN) blending factor between the upsampled and original latents. Should be in [-10.0, 10.0]; supplying 0.0 (the default) means that AdaIN is not performed. tone_map_compression_ratio (`float`, *optional*, defaults to `0.0`): The compression strength for tone mapping, which will reduce the dynamic range of the latent values. This is useful for regularizing high-variance latents or for conditioning outputs during generation. Should be in [0, 1], where 0.0 (the default) means tone mapping is not applied and 1.0 corresponds to the full compression effect. generator (`torch.Generator` or `list[torch.Generator]`, *optional*): One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generate image. Choose between [PIL](https://pillow.readthedocs.io/en/stable/): `PIL.Image.Image` or `np.array`. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.ltx.LTXPipelineOutput`] instead of a plain tuple. Examples: Returns: [`~pipelines.ltx.LTXPipelineOutput`] or `tuple`: If `return_dict` is `True`, [`~pipelines.ltx.LTXPipelineOutput`] is returned, otherwise a `tuple` is returned where the first element is the upsampled video. )r1r4r5rr|Nr rz-Video length expected to be of the form `k * z + 1` but is z. Truncating to z frames.)r4r5r:) r1r2r3r4r5r6r7r8r9rrr~latent)rr9r8F)r)r)frames) r}shape_execution_devicerKr)loggerwarningr*preprocess_videorHrNfloat32rTrqrrlrmconfigrnrr8rbrktimestep_conditioningrrIrJtensordecodepostprocess_videomaybe_free_model_hooksr )r+r1r4r5r3r6r7rrrrrr|rrrr2r9latents_suppliedlatents_upsampledtimestepnoiserrr__call__sJ          z#LTX2LatentUpsamplePipeline.__call__) Nr r.r/r0r r NNNN)rU)r r )Nr/r0r.r r NFr~Nr~r~NrT)__name__ __module__ __qualname__model_cpu_offload_seqrr r#rNTensorintr8r9 GeneratorrTfloatrbrk staticmethodrqrGr}no_gradrEXAMPLE_DOC_STRINGrJrboolstrr __classcell__rrr,rrhs     ,            r)Nr)rNimage_processorrmodelsrutilsrrutils.torch_utilsrr*rltx.pipeline_outputr pipeline_utilsr rr rrrrrrrrrrrrs(       =