o }oi@sddlZddlmZmZmZddlZddlZddlm Z ddl m Z ddl m ZddlmZddlmZddlmZmZdd lmZdd lmZmZdd lmZdd lmZdd lmZddlm Z ddl!m"Z"Gdddej#Z$Gddde$Z%dS)N)ListOptionalUnion)Image) load_file) save_file)nn)tqdm)FrozenCLIPEmbedderFrozenT5Embedder)Flux)FluxControlNetFluxControlNetConfig)FlowMatchEulerDiscreteScheduler)flux_transformer_converter)FluxModelParams) AutoEncoder)loggingc%s&eZdZdZ      dGdedeedeedeedee d e f fd d Z dHd dZ ddddde jfdeeeefde dee jdee jde dee jdee jfddZede de de de jde jf ddZed d!Zed"d#Ze $ % & 'dId(e d)e d*ed+efd,d-Z dJd.d/Zed0d1Zed2d3Zed4d5Zed6d7Zdddd8dd9dddddd:dde j d d;dfdeeeefdee dee dedee d?eee j!ee j!fd@ee jdee jdee jdAeede de jde jdBe"dCe"dDef$dEdFZ#Z$S)KFluxInferencePipelinea7 A pipeline for performing image generation with flux. Args: params (FluxModelParams, optional): Configuration parameters for the model pipeline, including device settings and model configurations. flux (Flux, optional): A pre-initialized Flux model used for the transformation process. If None, a new Flux model is created using the configuration in `params`. vae (AutoEncoder, optional): A pre-initialized VAE (Variational Autoencoder) model. If None, a new VAE model is created using the configuration in `params.vae_config`. t5 (FrozenT5Embedder, optional): A pre-initialized FrozenT5Embedder model. If None, a new T5 model is created using the configuration in `params.t5_params`. clip (FrozenCLIPEmbedder, optional): A pre-initialized FrozenCLIPEmbedder model. If None, a new CLIP model is created using the configuration in `params.clip_params`. scheduler_steps (int, optional): The number of scheduler steps to use for inference. Default is 1000. Attributes: device (torch.device): The device (CPU or GPU) where the models will be placed. vae (AutoEncoder): The VAE model used for image reconstruction or generation. clip_encoder (FrozenCLIPEmbedder): The CLIP encoder for processing image-text inputs. t5_encoder (FrozenT5Embedder): The T5 encoder for processing text inputs. transformer (Flux): The Flux model used for image-text joint processing. vae_scale_factor (float): A scale factor for the VAE, based on the number of channels in the VAE. scheduler (FlowMatchEulerDiscreteScheduler): Scheduler used for controlling the flow of inference steps. params (FluxModelParams): Configuration parameters used for model setup. Methods: load_from_pretrained: Loads model weights from a checkpoint. encoder_prompt: Encodes text prompts and retrieves embeddings. _prepare_latent_image_ids: Prepares latent image ids for the generation process. _pack_latents: Packs latents into the desired format for input to the model. _unpack_latents: Unpacks latents from the model into image format. _calculate_shift: Calculates the shift parameter used for controlling sequence lengths in the model. prepare_latents: Prepares the latent tensors and latent image ids for generation. _generate_rand_latents: Generates random latents using a specified generator. numpy_to_pil: Converts a numpy array or a batch of images to PIL images. torch_to_numpy: Converts a tensor of images to a numpy array. denormalize: Denormalizes the image to the range [0, 1]. __call__: Runs the entire image generation process based on the input prompt, including encoding, latent preparation, inference, and output generation. Example: pipeline = FluxInferencePipeline(params) images = pipeline( prompt=["A beautiful sunset over a mountain range"], height=512, width=512, num_inference_steps=50, guidance_scale=7.5 ) Nparamsfluxvaet5clipscheduler_stepscst|j|_|j|j_|j|j_|dur"t|j|jn||_ |dur9t |jj |jj |jj |jjdn||_|durPt|jj |jj |jj|jjdn||_|durbt|j|jn||_dt|j jj|_t|d|_||_dS)a Initializes the FluxInferencePipeline with the provided models and configurations. Args: params (FluxModelParams, optional): Configuration parameters for the model pipeline, including device settings and model configurations. flux (Flux, optional): A pre-initialized Flux model used for the transformation process. If None, a new Flux model is created using the configuration in `params`. vae (AutoEncoder, optional): A pre-initialized VAE (Variational Autoencoder) model. If None, a new VAE model is created using the configuration in `params.vae_config`. t5 (FrozenT5Embedder, optional): A pre-initialized FrozenT5Embedder model. If None, a new T5 model is created using the configuration in `params.t5_params`. clip (FrozenCLIPEmbedder, optional): A pre-initialized FrozenCLIPEmbedder model. If None, a new CLIP model is created using the configuration in `params.clip_params`. scheduler_steps (int, optional): The number of scheduler steps to use for inference. Default is 1000. N)version max_lengthalways_return_pooleddevice)rrload_config_only)num_train_timesteps)super__init__r clip_params t5_paramsr vae_configtoevalrr rrr clip_encoderr r t5_encoderr flux_config transformerlenrch_multvae_scale_factorr scheduler)selfrrrrrr __class__c/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/diffusion/models/flux/pipeline.pyr$js6   $$   zFluxInferencePipeline.__init__TcCs|r"t||jj}|dur!tj|d}t||td|nt |}|jj |dd\}}dd|D}t |dkrOtd |td |dSdS) a Loads the model's weights from a checkpoint. If HF ckpt is provided, it will be converted to NeMo format and save it to local folder. Args: ckpt_path (str): Path to the checkpoint file. do_convert_from_hf (bool, optional): Whether to convert the checkpoint from Hugging Face format before loading. Default is True. save_converted_model_to (str, optional): Path to save the converted checkpoint if `do_convert_from_hf` is True. Default is None. Logs: The function logs information about missing or unexpected keys during checkpoint loading. N!nemo_flux_transformer.safetensors+saving converted transformer checkpoint to FstrictcSg|] }|ds|qS _extra_stateendswith.0kr5r5r6 z>FluxInferencePipeline.load_from_pretrained..rzThe following keys are missing during checkpoint loading, please check the ckpt provided or the image quality may be compromised. Found unexepected keys: ) rr-configospathjoinsave_safetensorsrinfoload_safetensorsload_state_dictr.)r2 ckpt_pathdo_convert_from_hfsave_converted_model_tockpt save_pathmissing unexpectedr5r5r6load_from_pretraineds$  z*FluxInferencePipeline.load_from_pretrainedcudapromptnum_images_per_prompt prompt_embedspooled_prompt_embedsmax_sequence_lengthrdtypec CsD|dur t|}n|dur|jd}ntd|dkr'|jj|kr'|j||dur2|j||d}|jd} |d|d}|||| dj|d}|dkr[|jj|kr[|j||durf||\} }|d|d}|||dj|d}|dur|n|jj }t ||jdd j||d } | |dd} | dd|| fS) ad Encodes a text prompt (or a batch of prompts) into embeddings using both T5 and CLIP models. Args: prompt (Union[str, List[str]]): The text prompt(s) to be encoded. Can be a string or a list of strings. num_images_per_prompt (int, optional): The number of images to generate per prompt. Default is 1. prompt_embeds (torch.FloatTensor, optional): Precomputed prompt embeddings, if available. Default is None. pooled_prompt_embeds (torch.FloatTensor, optional): Precomputed pooled prompt embeddings, if available. Default is None. max_sequence_length (int, optional): The maximum sequence length for the text model. Default is 512. device (torch.device, optional): The device (CPU or CUDA) on which the models are placed. Default is 'cuda'. dtype (torch.dtype, optional): The data type for tensor operations. Default is `torch.float`. Returns: Tuple[torch.FloatTensor, torch.FloatTensor, torch.FloatTensor]: - The prompt embeddings. - The pooled prompt embeddings. - The text IDs for the prompt. Raises: ValueError: If neither `prompt` nor `prompt_embeds` are provided. Nr0Either prompt or prompt_embeds must be provided.rX)r]rVr^rr^) r.shape ValueErrorr+rr(repeatviewr*r^torchzeros transpose) r2rYrZr[r\r]rr^ batch_sizeseq_len_text_idsr5r5r6encoder_prompts,&     z$FluxInferencePipeline.encoder_promptrkheightwidthc Cst|d|dd}|dt|ddddf|d<|dt|ddddf|d<|j\}}}|dddf|ddd}|||||}|j||dS)a Prepares latent image IDs for input into the model. These IDs represent the image grid. Args: batch_size (int): The number of samples in the batch. height (int): The height of the image. width (int): The width of the image. device (torch.device): The device to place the tensor. dtype (torch.dtype): The data type for the tensor. Returns: torch.FloatTensor: A tensor representing the latent image IDs. r!rb).rVN).r!rVrc)rhriarangerdrfreshaper() rkrprqrr^latent_image_idslatent_image_id_heightlatent_image_id_widthlatent_image_id_channelsr5r5r6_prepare_latent_image_ids s&&  z/FluxInferencePipeline._prepare_latent_image_idscCsR||||dd|dd}|dddddd}|||d|d|d}|S)a Packs latents into desired shape, e.g. (B, C, H, W) --> (B, (H//2)*(W//2), C * 4). Args: latents (torch.Tensor): The latents to be packed. batch_size (int): The number of samples in the batch. num_channels_latents (int): The number of channels in the latents. height (int): The height of the image. width (int): The width of the image. Returns: torch.Tensor: The packed latents. r!rrVrb)rgpermuters)latentsrknum_channels_latentsrprqr5r5r6 _pack_latents&sz#FluxInferencePipeline._pack_latentscCsh|j\}}}||}||}|||||ddd}|dddddd}|||d|d|d}|S)a Unpacks the latents from the model output into an image format suitable for further processing. The method reshapes and permutes the latents, adjusting their dimensions according to the specified `vae_scale_factor` to match the expected resolution of the image. Args: latents (torch.Tensor): The latents output from the model, typically in a compact, compressed format. height (int): The original height of the image before scaling, used to adjust the latent dimensions. width (int): The original width of the image before scaling, used to adjust the latent dimensions. vae_scale_factor (int): A scale factor used to adjust the resolution of the image when unpacking. This factor istypically the inverse of the VAE downsampling factor. Returns: torch.Tensor: The unpacked latents reshaped to match the expected dimensions for image reconstruction. The output tensor will have shape `(batch_size, channels, height * 2, width * 2)`. Notes: - This function is intended to convert latents back into a format that can be decoded into images by the VAE. ryr!rrbrVrz)rdrgr{rs)r|rprqr0rk num_patcheschannelsr5r5r6_unpack_latents;s z%FluxInferencePipeline._unpack_latents?(\? base_seq_len max_seq_len base_shift max_shiftcCs,||||}|||}|||}|SNr5) image_seq_lenrrrrmbmur5r5r6_calculate_shift^s  z&FluxInferencePipeline._calculate_shiftc Csdt||j}dt||j}||||f} |dur.||||||} |j||d| fSt|trFt||krFtdt|d|dtj | ||||d}| |||||}||||||} | dd | fS) a Prepares and optionally generates image latents for use in the image generation pipeline. This method can either use the provided latents (if already available) or generate new random latents using a random generator. The generated latents are then packed and prepared for the model to process. Args: batch_size (int): The number of samples in the batch. num_channels_latents (int): The number of channels in the latents (e.g., depth of the latent tensor). height (int): The height of the image to be generated (before scaling). width (int): The width of the image to be generated (before scaling). dtype (torch.dtype): The data type to use for the latents (e.g., `torch.float32`). device (torch.device): The device on which the latents will reside (e.g., 'cuda'). generator (Union[torch.Generator, List[torch.Generator]]): A random number generator or a list of generatorsfor generating random latents. If a list is provided, its length must match the batch size. latents (Optional[torch.FloatTensor]): An optional pre-existing latent tensor. If provided, it is used instead of generating new latents. Returns: tuple: A tuple containing: - latents (torch.Tensor): The prepared latents, with shape `(batch_size, num_channels_latents, height, width)`. - latent_image_ids (torch.Tensor): A tensor containing latent image IDs for each batch sample, used for indexing in the model. Raises: ValueError: If a list of generators is provided but its length does not match the batch size. r!Nrcz/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.) generatorrr^rkrrV) intr0rxr( isinstancelistr.rer_generate_rand_latentsr~rj) r2rkr}rprqr^rrr|rdrtr5r5r6prepare_latentsls$)   z%FluxInferencePipeline.prepare_latentscsfttr(dddfddt|D}tj|ddjd}|Stjd }|S) zY Create random latents using a random generator or a list of generators. )rVrVNcs"g|] }tj|dqS)rrr^)rhrandn)rAirr^rrdr5r6rCsz@FluxInferencePipeline._generate_rand_latents..rdimrr)rrrangerhcatr(r)rdrrr^rkr|r5rr6rs z,FluxInferencePipeline._generate_rand_latentscCs6|jdkr |d}|dd}dd|D}|S)zL Convert a numpy image or a batch of images to a PIL image. rb)N.uint8cSsg|]}t|qSr5)r fromarrayrAimager5r5r6rCsz6FluxInferencePipeline.numpy_to_pil..)ndimroundastype)images pil_imagesr5r5r6 numpy_to_pils z"FluxInferencePipeline.numpy_to_pilcCs |dddd}|S)zN Convert a torch image or a batch of images to a numpy image. rr!rbrV)floatcpur{numpy)r numpy_imagesr5r5r6torch_to_numpysz$FluxInferencePipeline.torch_to_numpycCs|ddddS)Nr!rrrV)clamp)rr5r5r6 denormalizesz!FluxInferencePipeline.denormalize@pilFnum_inference_steps timestepsguidance_scalerr| output_type save_to_diskoffload output_pathc! s|dksJd|durt|trd}|g}n"|dur%t|tr%t|}n| dur5t| tjr5| jd}ntd|j|| | | ||d\} } }|r[|j d|j dtj |jjd }||||||||| \} }td d||}| jd}t||jj|jj|jj|jj}|jj|||d |jj}|dkr||jkr|j |ttt|D]Y\}}| | jdj | j| j!d }|jj"rtj#|g|d  | jd}nd}tj$d| j!d |j| | | |d|||d}|j%||| d} Wdn 1s wYq|r|j dtj | dkr1| &ddWdS| dkr|'| &dd|||j(} |dkrT||jkrT|j) |tj$d| j!d|j)*| }Wdn 1snwY|r|j) dtj t+|}t,|}t-|}Wdn 1swY|rt.dt/j0|ddt|t1t|ksJfdd|D}t2||D]\} }|3t/j45|| dq|S)a+ Generates images based on a given text prompt and various model parameters. Optionally saves the images to disk. This method orchestrates the process of generating images by embedding the prompt, preparing the latent vectors, iterating through timesteps in the diffusion process, and then decoding the latent representation back into an image. It supports both the generation of latent representations or final images in a desired output format (e.g., PIL image). The images are optionally saved to disk with a unique filename based on the prompt. Args: prompt (Union[str, List[str]]): A text prompt or a list of text prompts to guide image generation. Each prompt generates one or more images based on the `num_images_per_prompt`. height (Optional[int]): The height of the output image. Default is 512. width (Optional[int]): The width of the output image. Default is 512. num_inference_steps (int): The number of steps for the diffusion process. Default is 28. timesteps (Optional[List[int]]): A list of specific timesteps for the diffusion process. If not provided, they are automatically calculated. guidance_scale (float): The scale of the guidance signal, typically used to control the strength of prompt conditioning. num_images_per_prompt (Optional[int]): The number of images to generate per prompt. Default is 1. generator (Optional[Union[torch.Generator, List[torch.Generator]]]): A random number generator or a list of generators for generating latents. If a list is provided, it should match the batch size. latents (Optional[torch.FloatTensor]): Pre-existing latents to use instead of generating new ones. prompt_embeds (Optional[torch.FloatTensor]): Optionally pre-computed prompt embeddings to skip the prompt encoding step. pooled_prompt_embeds (Optional[torch.FloatTensor]): Optionally pre-computed pooled prompt embeddings. output_type (Optional[str]): The format of the output. Can be "latent" or "pil" (PIL image). Default is "pil". max_sequence_length (int): The maximum sequence length for tokenizing the prompt. Default is 512. device (torch.device): The device on which the computation should take place (e.g., 'cuda'). Default is 'cuda'. dtype (torch.dtype): The data type of the latents and model weights. Default is `torch.float32`. save_to_disk (bool): Whether or not to save the generated images to disk. Default is True. offload (bool): Whether or not to offload model components to CPU to free up GPU memory during the process. Default is False. Returns: Union[List[Image.Image], torch.Tensor]: The generated images or latents, depending on the `output_type` argument. If `output_type` is "pil", a list of PIL images is returned. If "latent", the latents are returned. Raises: ValueError: If neither a `prompt` nor `prompt_embeds` is provided. Notes: - The model expects a device of 'cuda'. The method will raise an assertion error if a different device is provided. - The method handles both prompt-based and pre-embedded prompt input, providing flexibility for different usage scenarios. - If `save_to_disk` is enabled, images will be saved with a filename derived from the prompt text. rX4Transformer blocks in Mcore must run on cuda devicesNrVrr_rYr[r\rZr]rr^rry?sigmasrrrcr device_typer^r)imgtxtyrimg_idstxt_idsguidancelatentrSaving to diskTexist_okc0g|]}tD] }|ddd|qqSN(rmrrApidxrZr5r6rC0z2FluxInferencePipeline.__call__...png)6rstrrr.rh FloatTensorrdreror+r(r*rX empty_cacher- in_channelsrnplinspacerrr1base_image_seq_lenmax_image_seq_lenrr set_timestepsrrno_gradr enumerateexpandr^guidance_embedtensorautocaststeprjrr0rdecoderrrprintrGmakedirsrzipsaverHrI)!r2rYrprqrrrrZrr|r[r\rr]rr^rrrrkrnr}rtrrrrttimesteprpredr file_namer5rr6__call__sV                      $zFluxInferencePipeline.__call__)NNNNNrTN)rrrrr)%__name__ __module__ __qualname____doc__rrr rr r rr$rUrhrrrrrrr^ro staticmethodrxr~rrrrrrrfloat32 Generatorboolr __classcell__r5r5r3r6r$sG <% B$  "  A          rc.seZdZdZ        d:deedeededede d e d e d e ffd d Z d;ddZddZdejdejfddZddZddddddddddddddejddd d!dd!dfd"eeeefd#ee d$ee d%e d&eee d'ed(ee d)eeejeejfd*eejd+eejd,eejd-eed.e d/ejd0ejd1ed2ed3ed4ed5ee j ejfd6eeeefd7ef,d8d9Z!Z"S)<FluxControlNetInferencePipelinezu Flux Contronlnet inference pipeline initializes controlnet component in addition to a normal flux pipeline. Nrrcontorlnet_configrrrrrflux_controlnetc s6t|||||||durt||_dS||_dS)zI Same as Flux Inference Pipeline with controlnet object. N)r#r$r r) r2rrrrrrrrr3r5r6r$s z(FluxControlNetInferencePipeline.__init__Tc Cs.|r=t||jj}t||jj}|durr@r5r5r6rCrDzHFluxControlNetInferencePipeline.load_from_pretrained..rzThe following keys are missing during flux checkpoint loading, please check the ckpt provided or the image quality may be compromised. rEcSr;r<r>r@r5r5r6rCrDzThe following keys are missing during controlnet checkpoint loading, please check the ckpt provided or the image quality may be compromised. ) rr-rFrrGrHrIrJrrKrLrMr.) r2flux_ckpt_pathcontrolnet_ckpt_pathrOrP flux_ckptflux_controlnet_ckptrRrSrTr5r5r6rUs@    z4FluxControlNetInferencePipeline.load_from_pretrainedcCs0t|ts|g}dd|D}tj|dd}|S)z* PIL image to numpy array cSs"g|] }t|tjdqS)go@)rarrayrrrr5r5r6rCs"z@FluxControlNetInferencePipeline.pil_to_numpy..r)axis)rrrstackr2rr5r5r6 pil_to_numpys z,FluxControlNetInferencePipeline.pil_to_numpyrreturncCs,|jdkr |d}t|dddd}|S)z8 Convert numpy image into torch tensors rb).NrrVr!)rrh from_numpyrjrr5r5r6 numpy_to_pts z+FluxControlNetInferencePipeline.numpy_to_ptc st|tjrn'|dj|dj}} |ks| kr$fdd|D}||}||}|jd} | dkr:|} n|} |j| dd}|j ||d}|S)zS Preprocess image into torch tensor, also duplicate by batch size. rcsg|] }|jfddqS)rb)resample)resizerrprqr5r6rCszAFluxControlNetInferencePipeline.prepare_image..rVrrc) rrhTensorrprqrrrdrepeat_interleaver() r2rrprqrkrZrr^ orig_height orig_widthimage_batch_size repeat_byr5r r6 prepare_images    z-FluxControlNetInferencePipeline.prepare_imagerWrrrVrrXFgrrYrprqrrrrZrr|r[r\rr]rr^rrcontrol_guidance_startcontrol_guidance_end control_imagecontrolnet_conditioning_scalerc+s|dksJd|durt|trd}|g}n"|dur%t|tr%t|}n| dur5t| tjr5| jd}ntd|j|| | | ||d\} } }|r[|j d|j dtj |jjd }||||||||| \} }td d||}| jd}t||jj|jj|jj|jj}|jj|||d |jj}|j|||||tjd }|jd d\}}|jjdur|dkr|j |kr|j! |t"|j!#|j |d}Wdn1swY|jdd\}}|$|||||%dd}g} t&t|D]}!| 'd t(|!t||kp!|!dt||kq |dkr>||j kr>|j ||j |t"t)t*|D]u\}!}"|"+| jdj | j | j,d}#|jj-rqtj.|g|d+| jd}$nd}$| |!|}%tj/d| j,d4|j| || | |#d|||$|%d \}&}'|j| | | |#d|||$|&|'d }(|j0|(|"| d} Wdn 1swYqJ|r|j dtj | dkr| %ddWdS| dkr?|1| %dd|||j2} |dkr||j kr|j! |tj/d| j,d|j!3| })Wdn 1swY|r0|j! dtj t4|)})t5|)})t6|)})Wdn 1sJwY|rt7dt8j9|ddt|)t:t|kslJfdd|D}t;||)D]\}*})|)||*dqz|)S)a Generates images based on a given text prompt and optionally incorporates control images and ControlNet for guidance. This method generates images by embedding the prompt, preparing the latent vectors, iterating through timesteps in the diffusion process, and then decoding the latent representation back into an image. The method supports control images through ControlNet, where the `control_image` is used to condition the image generation. It also allows you to specify custom guidance scales and other parameters. Generated images can be saved to disk if requested. Args: prompt (Union[str, List[str]]): A text prompt or a list of text prompts to guide image generation. Each prompt generates one or more images based on the `num_images_per_prompt`. height (Optional[int]): The height of the output image. Default is 512. width (Optional[int]): The width of the output image. Default is 512. num_inference_steps (int): The number of steps for the diffusion process. Default is 28. timesteps (Optional[List[int]]): A list of specific timesteps for the diffusion process. If not provided, they are automatically calculated. guidance_scale (float): The scale of the guidance signal, typically used to control the strength of prompt conditioning. num_images_per_prompt (Optional[int]): The number of images to generate per prompt. Default is 1. generator (Optional[Union[torch.Generator, List[torch.Generator]]]): A random number generator or a list of generators for generating latents. If a list is provided, it should match the batch size. latents (Optional[torch.FloatTensor]): Pre-existing latents to use instead of generating new ones. prompt_embeds (Optional[torch.FloatTensor]): Optionally pre-computed prompt embeddings to skip the prompt encoding step. pooled_prompt_embeds (Optional[torch.FloatTensor]): Optionally pre-computed pooled prompt embeddings. output_type (Optional[str]): The format of the output. Can be "latent" or "pil" (PIL image). Default is "pil". max_sequence_length (int): The maximum sequence length for tokenizing the prompt. Default is 512. device (torch.device): The device on which the computation should take place (e.g., 'cuda'). Default is 'cuda'. dtype (torch.dtype): The data type of the latents and model weights. Default is `torch.float32`. save_to_disk (bool): Whether or not to save the generated images to disk. Default is True. offload (bool): Whether or not to offload model components to CPU to free up GPU memory during the process. Default is False. control_guidance_start (float): The start point for control guidance to apply during the diffusion process. control_guidance_end (float): The end point for control guidance to apply during the diffusion process. control_image (Union[Image.Image, torch.FloatTensor]): The image used for conditioning the generation process via ControlNet. controlnet_conditioning_scale (Union[float, List[float]]): Scaling factors to control the impact of the control image in the generation process. Can be a single value or a list for multiple images. Returns: Union[List[Image.Image], torch.Tensor]: The generated images or latents, depending on the `output_type` argument. If `output_type` is "pil", a list of PIL images is returned. If "latent", the latents are returned. Raises: ValueError: If neither a `prompt` nor `prompt_embeds` is provided. Notes: - The model expects a device of 'cuda'. The method will raise an assertion error if a different device is provided. - The method supports conditional image generation using ControlNet, where a `control_image` can guide the generation process. - If `save_to_disk` is enabled, images will be saved with a filename derived from the prompt text. rXrNrVrr_rrryrr)rrprqrkrZrr^rar!rcrrr) rcontrolnet_condrrrrrrconditioning_scale) rrrrrrrcontrolnet_double_block_samplescontrolnet_single_block_samplesrrrTrcrrrrrr5r6rCrz.r)?rrrr.rhrrdreror+r(r*rXrr-rrrrrrr1rrrrrrrrrinput_hint_blockrrrencoder~rjrappendrr rrr^rrrrrr0rrrrrrGrrrrrHrI)+r2rYrprqrrrrZrr|r[r\rr]rr^rrrrrrrrkrnr}rtrrrheight_control_imagewidth_control_imagecontrolnet_keeprrrrrrrrrrr5rr6r sc           (             (      4z(FluxControlNetInferencePipeline.__call__)NNNNNNrNr)#rrrrrrrr rr r rr r$rUrrndarrayrhr rrrrrrrrrrr^rrrrr5r5r3r6rs  (  #      r)&rGtypingrrrrrrhPILrsafetensors.torchrrLrrJrr /nemo.collections.diffusion.encoders.conditionerr r ,nemo.collections.diffusion.models.flux.modelr 7nemo.collections.diffusion.models.flux_controlnet.modelr rJnemo.collections.diffusion.sampler.flow_matching.flow_match_euler_discreter4nemo.collections.diffusion.utils.flux_ckpt_converterr4nemo.collections.diffusion.utils.flux_pipeline_utilsr*nemo.collections.diffusion.vae.autoencoderr nemo.utilsrModulerrr5r5r5r6s.           r