o Gi;@sddlZddlZddlmZddlmZmZddlm Z ddl m Z m Z m Z ddlmZdd lmZmZmZd d lmZd d lmZd d lmZe rWddlmmZdZndZe e Z!dZ"GdddeeZ#dS)N) CLIPTokenizer) AutoencoderKLUNet2DConditionModel) PNDMScheduler)is_torch_xla_availableloggingreplace_example_docstring) randn_tensor)DeprecatedPipelineMixinDiffusionPipelineImagePipelineOutput)BlipImageProcessor)Blip2QFormerModel)ContextCLIPTextModelTFah Examples: ```py >>> from diffusers.pipelines import BlipDiffusionPipeline >>> from diffusers.utils import load_image >>> import torch >>> blip_diffusion_pipe = BlipDiffusionPipeline.from_pretrained( ... "Salesforce/blipdiffusion", torch_dtype=torch.float16 ... ).to("cuda") >>> cond_subject = "dog" >>> tgt_subject = "dog" >>> text_prompt_input = "swimming underwater" >>> cond_image = load_image( ... "https://huggingface.co/datasets/ayushtues/blipdiffusion_images/resolve/main/dog.jpg" ... ) >>> guidance_scale = 7.5 >>> num_inference_steps = 25 >>> negative_prompt = "over-exposure, under-exposure, saturated, duplicate, out of frame, lowres, cropped, worst quality, low quality, jpeg artifacts, morbid, mutilated, out of frame, ugly, bad anatomy, bad proportions, deformed, blurry, duplicate" >>> output = blip_diffusion_pipe( ... text_prompt_input, ... cond_image, ... cond_subject, ... tgt_subject, ... guidance_scale=guidance_scale, ... num_inference_steps=num_inference_steps, ... neg_prompt=negative_prompt, ... height=512, ... width=512, ... ).images >>> output[0].save("image.png") ``` c!s$eZdZdZdZdZ   d3dededed e d e d e d e d e deedeeffdd ZddZd4ddZd5ddZd5ddZeee          !d6d"eed#ejjd$eed%eed&ejdBd'ed(e d)e d*e d+ejeejBdBd,edBd-ed.e d/edBd0efd1d2ZZ S)7BlipDiffusionPipelinea Pipeline for Zero-Shot Subject Driven Generation using Blip Diffusion. This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.) Args: tokenizer ([`CLIPTokenizer`]): Tokenizer for the text encoder text_encoder ([`ContextCLIPTextModel`]): Text encoder to encode the text prompt vae ([`AutoencoderKL`]): VAE model to map the latents to the image unet ([`UNet2DConditionModel`]): Conditional U-Net architecture to denoise the image embedding. scheduler ([`PNDMScheduler`]): A scheduler to be used in combination with `unet` to generate image latents. qformer ([`Blip2QFormerModel`]): QFormer model to get multi-modal embeddings from the text and image. image_processor ([`BlipImageProcessor`]): Image Processor to preprocess and postprocess the image. ctx_begin_pos (int, `optional`, defaults to 2): Position of the context token in the text encoder. z0.33.1z qformer->text_encoder->unet->vaer N tokenizer text_encodervaeunet schedulerqformerimage_processor ctx_begin_posmeanstdc s6t|j|||||||d|j|| | ddS)N)rrrrrrr)rrr)super__init__register_modulesregister_to_config) selfrrrrrrrrrr __class__n/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/pipelines/blip_diffusion/pipeline_blip_diffusion.pyrks  zBlipDiffusionPipeline.__init__cCs|j||ddS)NF) image_input text_input return_dict)r)r" input_image src_subjectr%r%r&get_query_embeddingssz*BlipDiffusionPipeline.get_query_embeddings?c CsNg}t||D]\}}d|d|}|d|gt||q|S)Nza  z, )zipstripappendjoinint)r"prompts tgt_subjectsprompt_strength prompt_repsrvprompt tgt_subjectr%r%r& _build_prompts  z#BlipDiffusionPipeline._build_promptc Cst||||f} t|trt||krtdt|d|d|dur+t| |||d}n|j||d}||jj}|S)Nz/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.) generatordevicedtype)r>r?) isinstancelistlen ValueErrorr torinit_noise_sigma) r" batch_size num_channelsheightwidthr?r>r=latentsshaper%r%r&prepare_latentss   z%BlipDiffusionPipeline.prepare_latentsc Csp|p|j}|jjjj}||jjj8}|j|dd|dd|}|j d}|jj g|}|j|j ||dd}|S)N max_lengthTpt)padding truncationrMreturn_tensorsr) input_idsctx_embeddingsr) _execution_devicer text_modelconfigmax_position_embeddingsrnum_query_tokensrrDrKrrR) r" query_embedsr:r>max_lentokenized_promptrFrtext_embeddingsr%r%r& encode_prompts,   z#BlipDiffusionPipeline.encode_prompt@2pilTr:reference_imagesource_subject_categorytarget_subject_categoryrJguidance_scalerHrInum_inference_stepsr= neg_promptr7r8 output_typer)c! Cs(|j}|jj||jj|jjddd}||}t|tr |g}t|tr(|g}t|tr0|g}t |}|j ||| | d}| ||}| |||}|dk}|rv|j jjj}|j| g|d|dd}|j |j|dd d }t||g}d t |jjjd }|j||jjj||||| ||jj|d }i}|jj| fi|t||jjD]A\}}|dk}|rt|gd n|}|j|||dddd}|r|d \}}||||}|j|||d}t rt!"q|j#j$||j#jj%ddd } |jj&| |d} |'|s| fSt(| dS)a Function invoked when calling the pipeline for generation. Args: prompt (`list[str]`): The prompt or prompts to guide the image generation. reference_image (`PIL.Image.Image`): The reference image to condition the generation on. source_subject_category (`list[str]`): The source subject category. target_subject_category (`list[str]`): The target subject category. latents (`torch.Tensor`, *optional*): Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor will be generated by random sampling. guidance_scale (`float`, *optional*, defaults to 7.5): Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://huggingface.co/papers/2207.12598). `guidance_scale` is defined as `w` of equation 2. of [Imagen Paper](https://huggingface.co/papers/2205.11487). Guidance scale is enabled by setting `guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`, usually at the expense of lower image quality. height (`int`, *optional*, defaults to 512): The height of the generated image. width (`int`, *optional*, defaults to 512): The width of the generated image. num_inference_steps (`int`, *optional*, defaults to 50): The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. generator (`torch.Generator` or `list[torch.Generator]`, *optional*): One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. neg_prompt (`str`, *optional*, defaults to ""): The prompt or prompts not to guide the image generation. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). prompt_strength (`float`, *optional*, defaults to 1.0): The strength of the prompt. Specifies the number of times the prompt is repeated along with prompt_reps to amplify the prompt. prompt_reps (`int`, *optional*, defaults to 20): The number of times the prompt is repeated along with prompt_strength to amplify the prompt. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generate image. Choose between: `"pil"` (`PIL.Image.Image`), `"np"` (`np.array`) or `"pt"` (`torch.Tensor`). return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.ImagePipelineOutput`] instead of a plain tuple. Examples: Returns: [`~pipelines.ImagePipelineOutput`] or `tuple` rN) image_mean image_stdrQ pixel_values)r5r6r7r8r-rM)rOrMrQN)rRrSrr r)rFrGrHrIr=rJr?r>)timestepencoder_hidden_statesdown_block_additional_residualsmid_block_additional_residualsample prev_sampleF)r))ri)images))rTr preprocessrVrrrDr@strrBr<r,r]rrUrWrrRtorchcatrblock_out_channelsrL in_channelsr?r set_timesteps enumerate progress_bar timestepschunkstep XLA_AVAILABLExm mark_steprdecodescaling_factor postprocessmaybe_free_model_hooksr)!r"r:rcrdrerJrfrHrIrgr=rhr7r8rir)r>rFrYr\do_classifier_free_guidancerM uncond_inputuncond_embeddingsscale_down_factorextra_set_kwargsitlatent_model_input noise_prednoise_pred_uncondnoise_pred_textimager%r%r&__call__sF          zBlipDiffusionPipeline.__call__)r NN)r-r.)N) Nr^r_r_r`Nrar-r.rbT)!__name__ __module__ __qualname____doc___last_supported_versionmodel_cpu_offload_seqrrrrrrrr4rAfloatrr,r<rLr]rvno_gradr EXAMPLE_DOC_STRINGruPILImageTensor Generatorboolr __classcell__r%r%r#r&rNs           r)$ PIL.Imagerrv transformersrmodelsrr schedulersrutilsrrr utils.torch_utilsr pipeline_utilsr r rblip_image_processingrmodeling_blip2rmodeling_ctx_cliprtorch_xla.core.xla_modelcore xla_modelrr get_loggerrloggerrrr%r%r%r&s$        (