o ۷iQ@sddlmZddlZddlZddlmZmZddlm Z ddl m Z m Z m Z mZddlmZdd lmZmZmZd d lmZd d lmZe rSddlmmZd ZndZe eZ dZ!GdddeeZ"dS))CallableN) CLIPTextModel CLIPTokenizer)DDPMWuerstchenScheduler) deprecateis_torch_xla_availableloggingreplace_example_docstring) randn_tensor)DeprecatedPipelineMixinDiffusionPipelineImagePipelineOutput) PaellaVQModel)WuerstchenDiffNeXtTFa Examples: ```py >>> import torch >>> from diffusers import WuerstchenPriorPipeline, WuerstchenDecoderPipeline >>> prior_pipe = WuerstchenPriorPipeline.from_pretrained( ... "warp-ai/wuerstchen-prior", torch_dtype=torch.float16 ... ).to("cuda") >>> gen_pipe = WuerstchenDecoderPipeline.from_pretrain("warp-ai/wuerstchen", torch_dtype=torch.float16).to( ... "cuda" ... ) >>> prompt = "an image of a shiba inu, donning a spacesuit and helmet" >>> prior_output = pipe(prompt) >>> images = gen_pipe(prior_output.image_embeddings, prompt=prompt) ``` csFeZdZdZdZgdZ d-dededede d e d e d d ffd d Z ddZ d.ddZeddZeddZeddZeeed dd dd dd d ddd dgf dejeejBd eeeBd!ed"ee d Bd#e d$eeeBd Bd%ed&ejeejBd Bdejd Bd'ed Bd(ed)eeegd fd Bd*eefd+d,ZZS)/WuerstchenDecoderPipelineaR Pipeline for generating images from the Wuerstchen model. This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.) Args: tokenizer (`CLIPTokenizer`): The CLIP tokenizer. text_encoder (`CLIPTextModel`): The CLIP text encoder. decoder ([`WuerstchenDiffNeXt`]): The WuerstchenDiffNeXt unet decoder. vqgan ([`PaellaVQModel`]): The VQGAN model. scheduler ([`DDPMWuerstchenScheduler`]): A scheduler to be used in combination with `prior` to generate image embedding. latent_dim_scale (float, `optional`, defaults to 10.67): Multiplier to determine the VQ latent space size from the image embeddings. If the image embeddings are height=24 and width=24, the VQ latent shape needs to be height=int(24*10.67)=256 and width=int(24*10.67)=256 in order to match the training conditions. ztext_encoder->decoder->vqgan)latentstext_encoder_hidden_statesnegative_prompt_embedsimage_embeddingsףp= W%@ tokenizer text_encoderdecoder schedulervqganlatent_dim_scalereturnNcs.t|j|||||d|j|ddS)N)rrrrr)r)super__init__register_modulesregister_to_config)selfrrrrrr __class__h/home/ubuntu/vllm_env/lib/python3.10/site-packages/diffusers/pipelines/wuerstchen/pipeline_wuerstchen.pyr![s z"WuerstchenDecoderPipeline.__init__cCsR|dur t||||d}n|j|krtd|jd|||}||j}|S)N) generatordevicedtypezUnexpected latents shape, got z , expected )r shape ValueErrortoinit_noise_sigma)r$r,r+r*r)rrr'r'r(prepare_latentsos   z)WuerstchenDecoderPipeline.prepare_latentsc Cs t|tr t|nd}|j|d|jjddd}|j}|j} |j|dddj} | jd|jdkrmt || sm|j | dd|jjddf} t d |jjd | |ddd|jjf}| ddd|jjf} |j ||| |d } | j} | j|d d } d}|r|durdg|}n;t|t|urtdt|dt|dt|tr|g}n|t|krtd|dt|d|d|d |}|j|d|jjddd}|j |j||j|d }|j}|jd}|d|d}||||d}| |fS)Nr max_lengthTpt)paddingr1 truncationreturn_tensorslongest)r3r5z\The following part of your input was truncated because CLIP can only handle sequences up to z tokens: )attention_maskrdimz?`negative_prompt` should be the same type to `prompt`, but got z != .z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.) isinstancelistlenrmodel_max_length input_idsr8r,torchequal batch_decodeloggerwarningrr.last_hidden_staterepeat_interleavetype TypeErrorstrr-repeatview)r$promptr*num_images_per_promptdo_classifier_free_guidancenegative_prompt batch_size text_inputstext_input_idsr8untruncated_ids removed_texttext_encoder_outputr!uncond_text_encoder_hidden_states uncond_tokens uncond_input*negative_prompt_embeds_text_encoder_outputseq_lenr'r'r( encode_promptzs~ $      z'WuerstchenDecoderPipeline.encode_promptcC|jSN_guidance_scaler$r'r'r(guidance_scalez(WuerstchenDecoderPipeline.guidance_scalecCs |jdkS)Nrr`rbr'r'r(rPs z5WuerstchenDecoderPipeline.do_classifier_free_guidancecCr^r_)_num_timestepsrbr'r'r( num_timestepsrdz'WuerstchenDecoderPipeline.num_timesteps grpilTrrrNnum_inference_steps timestepsrcrQrOr) output_type return_dictcallback_on_step_end"callback_on_step_end_tensor_inputsc% s|dd}|dd}|durtddd|dur tddd| durAtfdd| DsAtd jd fd d | Dj}jj}|_t |t sct |t rY|g}n t d t |djr|durt |t st |t rx|g}n t dt |dt |t rtj|dd}t |tjrtj||dj|d}t |tjst dt |dt |tst dt |d|||d|j|\}}|durt||gn|}jrt|t|gn|}t|djj}t|djj}|d|d||f}|dur"jj||djj}t|}n jj||djj}||||| j} t|dd_ t!"|ddD]\}}|#| d|}jjrjt| gdn| jrwt|gdn|||d}jr|$d\}}t%||j&}jj'||| |dj(} | duri} | D] }!t)|!| |!<q| ||| }"|"d| } |"d|}|"d |}|dur||dkr|t*jd!d"}#||#|| t+rt,-qM| d#vrtd$| | d%ks<j.jj/| } j.0| j12dd"}$| d&kr#|$3dddd"456}$n| d'kr;|$3dddd"456}$7|$}$n| }$8| sG|$St9|$S)(aq Function invoked when calling the pipeline for generation. Args: image_embedding (`torch.Tensor` or `list[torch.Tensor]`): Image Embeddings either extracted from an image or generated by a Prior Model. prompt (`str` or `list[str]`): The prompt or prompts to guide the image generation. num_inference_steps (`int`, *optional*, defaults to 12): The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. timesteps (`list[int]`, *optional*): Custom timesteps to use for the denoising process. If not defined, equal spaced `num_inference_steps` timesteps are used. Must be in descending order. guidance_scale (`float`, *optional*, defaults to 0.0): Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://huggingface.co/papers/2207.12598). `decoder_guidance_scale` is defined as `w` of equation 2. of [Imagen Paper](https://huggingface.co/papers/2205.11487). Guidance scale is enabled by setting `decoder_guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`, usually at the expense of lower image quality. negative_prompt (`str` or `list[str]`, *optional*): The prompt or prompts not to guide the image generation. Ignored when not using guidance (i.e., ignored if `decoder_guidance_scale` is less than `1`). num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. generator (`torch.Generator` or `list[torch.Generator]`, *optional*): One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. latents (`torch.Tensor`, *optional*): Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor will be generated by sampling using the supplied random `generator`. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generate image. Choose between: `"pil"` (`PIL.Image.Image`), `"np"` (`np.array`) or `"pt"` (`torch.Tensor`). return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.ImagePipelineOutput`] instead of a plain tuple. callback_on_step_end (`Callable`, *optional*): A function that calls at the end of each denoising steps during the inference. The function is called with the following arguments: `callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict)`. `callback_kwargs` will include a list of all tensors as specified by `callback_on_step_end_tensor_inputs`. callback_on_step_end_tensor_inputs (`list`, *optional*): The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the `._callback_tensor_inputs` attribute of your pipeline class. Examples: Returns: [`~pipelines.ImagePipelineOutput`] or `tuple` [`~pipelines.ImagePipelineOutput`] if `return_dict` is True, otherwise a `tuple`. When returning a tuple, the first element is a list with the generated image embeddings. callbackNcallback_stepsz1.0.0zhPassing `callback` as an input argument to `__call__` is deprecated, consider use `callback_on_step_end`znPassing `callback_steps` as an input argument to `__call__` is deprecated, consider use `callback_on_step_end`c3s|]}|jvVqdSr__callback_tensor_inputs.0krbr'r( 2s  z5WuerstchenDecoderPipeline.__call__..z2`callback_on_step_end_tensor_inputs` has to be in z , but found csg|] }|jvr|qSr'rqrsrbr'r( 6sz6WuerstchenDecoderPipeline.__call__..z2'prompt' must be of type 'list' or 'str', but got r<z;'negative_prompt' must be of type 'list' or 'str', but got rr9)r*)r+zI'image_embeddings' must be of type 'torch.Tensor' or 'np.array', but got z5'num_inference_steps' must be of type 'int', but got zo In Case you want to provide explicit timesteps, please use the 'timesteps' argument.r r)rjr*r7)reffnetclip) model_outputtimestepsampler)rrrorderr)r2nprhlatentzSOnly the output types `pt`, `np`, `pil` and `latent` are supported not output_type=rrrh):poprallr-rr_execution_devicerr+rar=r>rKrJrIrPrBcatrndarrayTensorr.intr]size zeros_likeconfigrr set_timestepsrjr?r0re enumerate progress_barexpandchunklerprcstep prev_samplelocalsgetattr XLA_AVAILABLExm mark_stepr scale_factordecoder~clamppermutecpufloatnumpy numpy_to_pilmaybe_free_model_hooksr)%r$rrNrirjrcrQrOr)rrkrlrmrnkwargsrorpr*r+ prompt_embedsrrrz latent_height latent_widthlatent_features_shapeitratiopredicted_latentspredicted_latents_textpredicted_latents_uncondcallback_kwargsrucallback_outputsstep_idximagesr'rbr(__call__s J                    z"WuerstchenDecoderPipeline.__call__)rr_) __name__ __module__ __qualname____doc__model_cpu_offload_seqrrrrrrrrr!r0r]propertyrcrPrfrBno_gradr EXAMPLE_DOC_STRINGrr>rKr Generatorboolrr __classcell__r'r'r%r(r;s R          r)#typingrrrrB transformersrr schedulersrutilsrrr r utils.torch_utilsr pipeline_utilsr rrmodeling_paella_vq_modelrmodeling_wuerstchen_diffnextrtorch_xla.core.xla_modelcore xla_modelrr get_loggerrrErrr'r'r'r(s"