o iv[ @sddlZddlmZddlZddlmZddlmZddlm Z ddl m Z m Z m Z ddlmZddlmZdd lmZmZmZdd lmZdd lmZmZmZdd lmZmZmZm Z m!Z!dd l"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*m+Z+m,Z,ddl-m.Z.ddl/m0Z0m1Z1m2Z2ddl3m4Z4m5Z5m6Z6ddl7m8Z8GdddZ9e:d"ddZ;def ddZ?desz)ICLoraPipeline.__call__..)context conditionings)rg) denoisersigmasnoiserrKrJr_r`videoaudiorUz3[IC-LoRA] Skipping Stage 2 (--skip-stage-2 enabled)cstjj|jjdS)NrNrJrKrar5r-)r"rJrKr5r-rc)rNrBstage_2_output_shaper1rErfs)rgrh noise_scaleinitial_latent)rgrprq)r!rAr3 Generatorr- manual_seedrr6lenvideo_encodingaudio_encodingrr7Tensorrtor9r rKrJr'logginginfor<latentr=r;rr:item)rBrHrIrJrKrLrMrNrOrPrQrRrSrTrUrYrkctx_p video_context audio_contextstage_1_conditioningsstage_1_sigmas video_state audio_state decoded_video decoded_audioupscaled_video_latentdistilled_sigmasstage_2_conditioningsr1)rTrRrNrLrBrerorOrE__call__qs1           zICLoraPipeline.__call__rac Cs,t|||||j|jd} |j} | dkr-|| dks|| dkr-td|d|d| d|| } || } |D]O\} }t| ||jd}t|| | |j|j}||}t|j }|d urg|j ||d }||}n |d krn|}nd }t || |d }|d urt ||d }| |q7|rtdt|d| S)a! Create conditioning items for video generation. Args: conditioning_attention_strength: Scalar attention weight in [0, 1]. If conditioning_attention_mask is also provided, the downsampled mask is multiplied by this strength. Otherwise this scalar is passed directly as the attention mask. conditioning_attention_mask: Optional pixel-space attention mask with shape (B, 1, F_pixel, H_pixel, W_pixel) matching the reference video's pixel dimensions. Downsampled to latent space with causal temporal handling, then multiplied by conditioning_attention_strength. Returns: List of conditioning items. IC-LoRA conditionings are appended last. rnr2rzOutput dimensions (xz3) must be divisible by reference_downscale_factor ()r@ frame_capr-N)masktarget_latent_shaperG)r{downscale_factorstrength)attention_maskz[IC-LoRA] Added z video conditioning(s))r"r5r-r>rAr$r&rfrom_torch_shapeshape_downsample_mask_to_latentrrappendryrzrt)rBrNrOrJrKrLrarRrTrhrD ref_height ref_width video_pathr frame_genrl encoded_videoreference_video_shape latent_mask attn_maskcondr1r1rErbsP      z$ICLoraPipeline._create_conditioningsrrc Cs |jd}|j}|j}|j}|jd}tjjjt|d||fdd}t|d|d}|dddddd ddddf}|d kr|d kr|d |d } |d |d dksaJd |d |d t|ddddd dddddfd | d} | j dd} tj || gdd} n|} t| dS)a Downsample a pixel-space mask to latent space using VAE scale factors. Handles causal temporal downsampling: the first frame is kept separately (temporal scale factor = 1 for the first frame), while the remaining frames are downsampled by the VAE's temporal scale factor. Args: mask: Pixel-space mask of shape (B, 1, F_pixel, H_pixel, W_pixel). Values in [0, 1]. target_latent_shape: Expected latent shape after VAE encoding. Used to determine the target (F_latent, H_latent, W_latent). Returns: Flattened latent-space mask of shape (B, F_lat * H_lat * W_lat), matching the patchifier's token ordering (f, h, w). rr]zb 1 f h w -> (b f) 1 h warea)sizemodez(b f) 1 h w -> b 1 f h w)bNr2zPixel frames (z%) not compatible with latent frames (z/): (f_pix - 1) must be divisible by (f_lat - 1)zb 1 (f t) h w -> b 1 f t h w)t)dimzb 1 f h w -> b (f h w)) rr_rJrKr3nn functional interpolatermeancat) rrrf_lath_latw_latf_pix spatial_down first_framerrestrr1r1rEr_s,  &0  z)ICLoraPipeline._downsample_mask_to_latent)NNNF)FNrGFNN)rGN)__name__ __module__ __qualname____doc__strlistr r3r-rr boolrFintfloatrr8r rwrrrr rrb staticmethodrrr1r1r1rEr(-s C      (  Or(rVc CsBttjtdd}t|}t|d}|jdtdddd|jdt dd dd d |jd d dd| }d}d}|j durV|j \}}|}t ||j d|jd|jd}t|j|j|j|jret|jnd|j|jd}t} t|j| } ||j|j|j |j|j|j|j|j| ||j ||j!d \} } t"| |j| |j#| ddS)NT) distilled)paramsz--video-conditioningr])PATHSTRENGTH)actionnargsmetavarrequiredz--conditioning-attention-mask) MASK_PATHraOptional spatial attention mask: path to a grayscale mask video and attention strength. The mask video pixel values in [0,1] control per-region conditioning attention strength. The strength scalar is multiplied with the spatial mask. 0.0 = ignore IC-LoRA conditioning, 1.0 = full conditioning influence. When not provided, full conditioning strength (1.0) is used. Example: --conditioning-attention-mask path/to/mask.mp4 0.5)rrrdefaulthelpz--skip-stage-2 store_truezSkip Stage 2 upsampling and refinement. Output will be at half resolution (height//2, width//2). Useful for faster iteration or when GPU memory is limited.)rrrG) mask_pathrJrKrLr1)r)r*r+r,r.r0) rHrIrJrKrLrMrNrOrQrRrSrTrU)rlr`rm output_pathvideo_chunks_number)$ry getLoggersetLevelINFOrrr add_argumentrr parse_argsrT_load_mask_videorJrKrLr(r)r*r+rCr8r.compiler rr rHrIrMrNrOrSrUr%r) checkpoint_pathrparserargsrTrRr mask_strengthpipelinerQrrlrmr1r1rEmains       rrrJrKrLcCsLt}t|||d}t|||tj|}|jddd}|dd}|ddS)aLoad a mask video and return a pixel-space tensor of shape (1, 1, F, H, W). The mask video is loaded, resized to (height, width), converted to grayscale, and normalised to [0, 1]. Args: mask_path: Path to the mask video file. height: Target height in pixels. width: Target width in pixels. num_frames: Maximum number of frames to load. Returns: Tensor of shape ``(1, 1, F, H, W)`` with values in ``[0, 1]``. rr2T)rkeepdimrGg@rX)r#r$r&r3r4rclamp)rrJrKrLr-r mask_videorr1r1rErs   r lora_pathc Csz(t|dd}|p i}t|ddWdWS1s!wYWdStyF}ztd|d|WYd}~dSd}~ww)a]Read reference_downscale_factor from LoRA safetensors metadata. Some IC-LoRA models are trained with reference videos at lower resolution than the target output. This allows for more efficient training and can improve generalization. The downscale factor indicates the ratio between target and reference resolutions (e.g., factor=2 means reference is half the resolution). Args: lora_path: Path to the LoRA .safetensors file Returns: The reference downscale factor (1 if not specified in metadata, meaning reference and target have the same resolution) pt) frameworkr>r2Nz(Failed to read metadata from LoRA file 'z': )rmetadatarget Exceptionrywarning)rfrer1r1rEr?s  (r?__main__)rVN)Brycollections.abcrr3einopsr safetensorsrltx_core.components.noisersrltx_core.conditioningrrrltx_core.loaderr ltx_core.loader.registryr ltx_core.model.video_vaer r r ltx_core.quantizationrltx_core.typesrrrltx_pipelines.utils.argsrrrrrltx_pipelines.utils.blocksrrrrrrltx_pipelines.utils.constantsrrrltx_pipelines.utils.denoisersr ltx_pipelines.utils.helpersr!r"r#ltx_pipelines.utils.media_ior$r%r&ltx_pipelines.utils.typesr'r(inference_moderrrrwrr?rr1r1r1rEsN          j V