o pik@sddlmZmZmZmZmZddlZddlmZm Z ddl m Z ddl m Z mZddlmZddlmZmZmZdd lmZd d lmZmZeeZd ZdddZGdddee ZdS))CallableDictListOptionalUnionN)T5EncoderModel T5Tokenizer)StableDiffusionLoraLoaderMixin)Kandinsky3UNetVQModel) DDPMScheduler) deprecateloggingreplace_example_docstring) randn_tensor)DiffusionPipelineImagePipelineOutputa Examples: ```py >>> from diffusers import AutoPipelineForText2Image >>> import torch >>> pipe = AutoPipelineForText2Image.from_pretrained( ... "kandinsky-community/kandinsky-3", variant="fp16", torch_dtype=torch.float16 ... ) >>> pipe.enable_model_cpu_offload() >>> prompt = "A photograph of the inside of a subway train. There are raccoons sitting on the seats. One of them is reading a newspaper. The window shows the city in the background." >>> generator = torch.Generator(device="cpu").manual_seed(0) >>> image = pipe(prompt, num_inference_steps=25, generator=generator).images[0] ``` cCsX||d}||ddkr|d7}||d}||ddkr$|d7}||||fS)Nrr)heightwidth scale_factor new_height new_widthrrp/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/pipelines/kandinsky3/pipeline_kandinsky3.pydownscale_height_and_width(s  rc%seZdZdZgdZdedededede f fdd Z d d Z e       d3dee jdee jdee jdee jfddZddZ      d4ddZeddZeddZeddZe eedd d!dd d"d"dddddd#d ddd$gfd%eeeefd&ed'ed(eeeeefd)eed*eed+eed,eee jee jfdee jdee jdee jdee jd-eed.ed/ee eee!gdfd0eef d1d2Z"Z#S)5Kandinsky3Pipelineztext_encoder->unet->movq)latents prompt_embedsnegative_prompt_embedsnegative_attention_maskattention_mask tokenizer text_encoderunet schedulermovqcs"t|j|||||ddS)N)r%r&r'r(r))super__init__register_modules)selfr%r&r'r(r) __class__rrr+<s   zKandinsky3Pipeline.__init__cCs`|r,t||dk||dk<|dd}|ddd|f}|ddd|f}||fS)Nrr)torch zeros_likesummax)r- embeddingsr$ cut_contextmax_seq_lengthrrrprocess_embedsJs z!Kandinsky3Pipeline.process_embedsTrNFr!r"r$r#c  Cs|dur|durt|t|urtdt|dt|d|dur&|j}|dur2t|tr2d} n|dur@t|tr@t|} n|jd} d} |dur||j|d| d d d } | j |}| j |} |j || d }|d}| || |\}} || d }|j dur|j j}nd}|j ||d}|j\}}}|d|d}||||d}| |d} |r4|dur4|durdg| }n$t|tr|g}n| t|krtd|dt|d|d| d |}|dur*|j|ddd d d d}|j |}|j |} |j || d }|d}|ddd|jdf}| ddd|jdf} || d }n t|}t| } |ra|jd}|j ||d}|j|jkr`|d|d}|| ||d}| |d} nd}d} ||| | fS)aX Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `List[str]`, *optional*): prompt to be encoded device: (`torch.device`, *optional*): torch device to place the resulting embeddings on num_images_per_prompt (`int`, *optional*, defaults to 1): number of images that should be generated per prompt do_classifier_free_guidance (`bool`, *optional*, defaults to `True`): whether to use classifier free guidance or not negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds`. instead. If not defined, one has to pass `negative_prompt_embeds`. instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. attention_mask (`torch.Tensor`, *optional*): Pre-generated attention mask. Must provide if passing `prompt_embeds` directly. negative_attention_mask (`torch.Tensor`, *optional*): Pre-generated negative attention mask. Must provide if passing `negative_prompt_embeds` directly. Nz?`negative_prompt` should be the same type to `prompt`, but got z != .rr max_lengthTpt)paddingr; truncationreturn_tensors)r$r)dtypedevicer0z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.)r=r;r>return_attention_maskr?)type TypeError_execution_device isinstancestrlistlenshaper% input_idstor$r&r8 unsqueezer@repeatview ValueErrorr1r2)r-promptdo_classifier_free_guidancenum_images_per_promptrAnegative_promptr!r" _cut_contextr$r# batch_sizer; text_inputstext_input_idsr@bs_embedseq_len_ uncond_tokens uncond_inputrrr encode_promptRs*                    z Kandinsky3Pipeline.encode_promptcCsR|dur t||||d}n|j|krtd|jd|||}||j}|S)N) generatorrAr@zUnexpected latents shape, got z , expected )rrKrQrMinit_noise_sigma)r-rKr@rAr`r r(rrrprepare_latentss   z"Kandinsky3Pipeline.prepare_latentsc s |durt|tr |dkrtd|dt|d|dur;tfdd|Ds;tdjdfd d |D|durN|durNtd |d |d |durZ|durZtd|durqt|tsqt|tsqtdt||dur|durtd|d|d |dur|dur|j|jkrtd|jd|jd|dur|durtd|dur|dur|jdd|jkrtd|jddd|jd|dur|durtd|dur|dur|jdd|jkrtd|jddd|jddSdSdS)Nrz5`callback_steps` has to be a positive integer but is z of type r9c3|]}|jvVqdSN_callback_tensor_inputs.0kr-rr   z2Kandinsky3Pipeline.check_inputs..2`callback_on_step_end_tensor_inputs` has to be in , but found cg|] }|jvr|qSrrergrjrr z3Kandinsky3Pipeline.check_inputs..zCannot forward both `prompt`: z and `prompt_embeds`: z2. Please make sure to only forward one of the two.zeProvide either `prompt` or `prompt_embeds`. Cannot leave both `prompt` and `prompt_embeds` undefined.z2`prompt` has to be of type `str` or `list` but is z'Cannot forward both `negative_prompt`: z and `negative_prompt_embeds`: zu`prompt_embeds` and `negative_prompt_embeds` must have the same shape when passed directly, but got: `prompt_embeds` z != `negative_prompt_embeds` zLPlease provide `negative_attention_mask` along with `negative_prompt_embeds`rz`negative_prompt_embeds` and `negative_attention_mask` must have the same batch_size and token length when passed directly, but got: `negative_prompt_embeds` z != `negative_attention_mask` z:Please provide `attention_mask` along with `prompt_embeds`z`prompt_embeds` and `attention_mask` must have the same batch_size and token length when passed directly, but got: `prompt_embeds` z != `attention_mask` ) rGintrQrDallrfrHrIrK) r-rRcallback_stepsrUr!r""callback_on_step_end_tensor_inputsr$r#rrjr check_inputssz    zKandinsky3Pipeline.check_inputscC|jSrd_guidance_scalerjrrrguidance_scale9z!Kandinsky3Pipeline.guidance_scalecCs |jdkS)NrrxrjrrrrS=s z.Kandinsky3Pipeline.do_classifier_free_guidancecCrwrd)_num_timestepsrjrrr num_timestepsAr{z Kandinsky3Pipeline.num_timestepsg@ipilr rRnum_inference_stepsrzrUrTrrr` output_type return_dictcallback_on_step_endruc& s|dd}|dd}|durtddd|dur tddd|durAtfdd|DsAtd jd fd d |Dd }j}|||| | || | |_|durat|t rad}n|durot|t rot |}n| j d}j |j|||| | || | d \} } } } jrt| | g} t| | g} jj||djj}t||d\}}||d||f| j|||j}tdrЈjdurЈjt ||jj}t |_j|d}t|D]\}}jrt|gdn|}j||| | ddd}jr| d\}} |d| ||}jj!||||dj"}|durai}!|D] }"t#|"|!|"<q1||||!}#|#d|}|#d| } |#d| } |#d| } |#d| } |t |dks||d|kr|djjdkr|$|dur||dkr|t%jd d}$||$||q| d!vrtd"| | d#ksj&j'|d d$d%}%| d&vr|%d'd'}%|%(dd}%|%)*ddd(d+,}%| d)kr-|%}%n|}%.|s|%fWdSt/|%d*WdS1swYdS)+u" Function invoked when calling the pipeline for generation. Args: prompt (`str` or `List[str]`, *optional*): The prompt or prompts to guide the image generation. If not defined, one has to pass `prompt_embeds`. instead. num_inference_steps (`int`, *optional*, defaults to 25): The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. timesteps (`List[int]`, *optional*): Custom timesteps to use for the denoising process. If not defined, equal spaced `num_inference_steps` timesteps are used. Must be in descending order. guidance_scale (`float`, *optional*, defaults to 3.0): Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://arxiv.org/abs/2207.12598). `guidance_scale` is defined as `w` of equation 2. of [Imagen Paper](https://arxiv.org/pdf/2205.11487.pdf). Guidance scale is enabled by setting `guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`, usually at the expense of lower image quality. negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. height (`int`, *optional*, defaults to self.unet.config.sample_size): The height in pixels of the generated image. width (`int`, *optional*, defaults to self.unet.config.sample_size): The width in pixels of the generated image. eta (`float`, *optional*, defaults to 0.0): Corresponds to parameter eta (η) in the DDIM paper: https://arxiv.org/abs/2010.02502. Only applies to [`schedulers.DDIMScheduler`], will be ignored for others. generator (`torch.Generator` or `List[torch.Generator]`, *optional*): One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. attention_mask (`torch.Tensor`, *optional*): Pre-generated attention mask. Must provide if passing `prompt_embeds` directly. negative_attention_mask (`torch.Tensor`, *optional*): Pre-generated negative attention mask. Must provide if passing `negative_prompt_embeds` directly. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generate image. Choose between [PIL](https://pillow.readthedocs.io/en/stable/): `PIL.Image.Image` or `np.array`. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.stable_diffusion.IFPipelineOutput`] instead of a plain tuple. callback (`Callable`, *optional*): A function that will be called every `callback_steps` steps during inference. The function will be called with the following arguments: `callback(step: int, timestep: int, latents: torch.Tensor)`. callback_steps (`int`, *optional*, defaults to 1): The frequency at which the `callback` function will be called. If not specified, the callback will be called at every step. clean_caption (`bool`, *optional*, defaults to `True`): Whether or not to clean the caption before creating embeddings. Requires `beautifulsoup4` and `ftfy` to be installed. If the dependencies are not installed, the embeddings will be created from the raw prompt. cross_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). Examples: Returns: [`~pipelines.ImagePipelineOutput`] or `tuple` callbackNrtz1.0.0zhPassing `callback` as an input argument to `__call__` is deprecated, consider use `callback_on_step_end`znPassing `callback_steps` as an input argument to `__call__` is deprecated, consider use `callback_on_step_end`c3rcrdrergrjrrrkrlz.Kandinsky3Pipeline.__call__..rmrncrorrergrjrrrprqz/Kandinsky3Pipeline.__call__..Trr)rTrArUr!r"rVr$r#)rArtext_encoder_offload_hook)totalrF)encoder_hidden_statesencoder_attention_maskrg?)r`r r!r"r$r#order)r<nprlatentzSOnly the output types `pt`, `pil`, `np` and `latent` are supported not output_type=r)force_not_quantizesample)rrg?r r)images)0poprrsrQrfrFrvryrGrHrIrJrKr_rSr1catboolr( set_timesteps timestepsrrbr@hasattrroffloadrr| progress_bar enumerater'chunkstep prev_samplelocalsupdategetattrr)decodeclampcpupermutefloatnumpy numpy_to_pilmaybe_free_model_hooksr)&r-rRrrzrUrTrrr`r!r"r$r#rrr rrukwargsrrtr6rArWrnum_warmup_stepsritlatent_model_input noise_prednoise_pred_uncondnoise_pred_textcallback_kwargsricallback_outputsstep_idximagerrjr__call__Es `             6        E&zKandinsky3Pipeline.__call__) TrNNNNFNN)NNNNNN)$__name__ __module__ __qualname__model_cpu_offload_seqrfrrr r r r+r8r1no_gradrTensorr_rbrvpropertyrzrSr}rEXAMPLE_DOC_STRINGrrHrrrr Generatorrrrr __classcell__rrr.rr2s   F        r)r) typingrrrrrr1 transformersrrloadersr modelsr r schedulersr utilsrrrutils.torch_utilsrpipeline_utilsrr get_loggerrloggerrrrrrrrs