o Gif@sddlmZddlZddlmZmZddlmZddlm Z ddl m Z m Z m Z mZddlmZd d lmZmZmZd d lmZe rOddlmmZd Znd Ze eZdZGdddeeZ dS))CallableN)CLIPTextModelWithProjection CLIPTokenizer)StableCascadeUNet)DDPMWuerstchenScheduler)is_torch_versionis_torch_xla_availableloggingreplace_example_docstring) randn_tensor)DeprecatedPipelineMixinDiffusionPipelineImagePipelineOutput) PaellaVQModelTFa Examples: ```py >>> import torch >>> from diffusers import StableCascadePriorPipeline, StableCascadeDecoderPipeline >>> prior_pipe = StableCascadePriorPipeline.from_pretrained( ... "stabilityai/stable-cascade-prior", torch_dtype=torch.bfloat16 ... ).to("cuda") >>> gen_pipe = StableCascadeDecoderPipeline.from_pretrain( ... "stabilityai/stable-cascade", torch_dtype=torch.float16 ... ).to("cuda") >>> prompt = "an image of a shiba inu, donning a spacesuit and helmet" >>> prior_output = pipe(prompt) >>> images = gen_pipe(prior_output.image_embeddings, prompt=prompt) ``` c$seZdZdZdZdZdZdZgdZ d5de de de d e d e d ed d ffdd ZddZ d6dejd Bdejd Bdejd Bdejd BfddZ d7ddZeddZeddZeddZd d!Zeeed d"d#d d d d d d$d d d%d&d d'gfd(ejeejBd)eeeBd*ed+ed,eeeBd Bdejd Bdejd Bdejd Bdejd Bd-ed.ej eej Bd Bd'ejd Bd/ed Bd0e!d1e"eegd fd Bd2eef d3d4Z#Z$S)8StableCascadeDecoderPipelinea_ Pipeline for generating images from the Stable Cascade model. This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.) Args: tokenizer (`CLIPTokenizer`): The CLIP tokenizer. text_encoder (`CLIPTextModelWithProjection`): The CLIP text encoder. decoder ([`StableCascadeUNet`]): The Stable Cascade decoder unet. vqgan ([`PaellaVQModel`]): The VQGAN model. scheduler ([`DDPMWuerstchenScheduler`]): A scheduler to be used in combination with `prior` to generate image embedding. latent_dim_scale (float, `optional`, defaults to 10.67): Multiplier to determine the VQ latent space size from the image embeddings. If the image embeddings are height=24 and width=24, the VQ latent shape needs to be height=int(24*10.67)=256 and width=int(24*10.67)=256 in order to match the training conditions. z0.35.2decoder text_encoderztext_encoder->decoder->vqgan)latentsprompt_embeds_poolednegative_prompt_embedsimage_embeddingsףp= W%@ tokenizer schedulervqganlatent_dim_scalereturnNcs.t|j|||||d|j|ddS)N)rrrrr)r)super__init__register_modulesregister_to_config)selfrrrrrr __class__n/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/pipelines/stable_cascade/pipeline_stable_cascade.pyr ^s z%StableCascadeDecoderPipeline.__init__c Cs|j\} } } } ||dt| |jjt| |jjf} |dur(t| |||d}n|j| kr8td|jd| ||}||j}|S)N) generatordevicedtypezUnexpected latents shape, got z , expected )shapeintconfigrr ValueErrortoinit_noise_sigma)r# batch_sizernum_images_per_promptr+r*r)rr_channelsheightwidth latents_shaper&r&r'prepare_latentsqs   z,StableCascadeDecoderPipeline.prepare_latents prompt_embedsrrnegative_prompt_embeds_pooledc  Cs|dur|j|d|jjddd} | j} | j} |j|dddj}|jd| jdkrft| |sf|j|dd|jjddf}t d |jjd || ddd|jjf} | ddd|jjf} |j | || |dd }|j d}|dur|j d}|j |j j|d }|j |j j|d }|j|d d}|j|d d}| dur|r|durdg|}n;t|t|urtdt|dt|dt|tr|g}n|t|krtd|dt|d|d|d |}|j|d|jjddd}|j |j ||j |dd }|j d} |j d} |rV| jd}| j |j j|d } | d|d} | |||d} | jd}| j |j j|d } | d|d} | |||d} ||| | fS)N max_lengthTpt)paddingr< truncationreturn_tensorslongest)r>r@z\The following part of your input was truncated because CLIP can only handle sequences up to z tokens: )attention_maskoutput_hidden_states)r+r*rdimz?`negative_prompt` should be the same type to `prompt`, but got z != .z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.)rmodel_max_length input_idsrDr,torchequal batch_decodeloggerwarningrr0 hidden_states text_embeds unsqueezer+repeat_interleavetype TypeError isinstancestrlenr/repeatview)r#r*r2r3do_classifier_free_guidancepromptnegative_promptr:rrr; text_inputstext_input_idsrDuntruncated_ids removed_texttext_encoder_output uncond_tokens uncond_input*negative_prompt_embeds_text_encoder_outputseq_lenr&r&r' encode_prompts               z*StableCascadeDecoderPipeline.encode_promptcs|dur!tfdd|Ds!tdjdfdd|D|dur4|dur4td|d|d |dur@|dur@td |durWt|tsWt|tsWtd t||durj|durjtd |d |d |dur|dur|j|jkrtd|jd|jddSdSdS)Nc3s|]}|jvVqdSN_callback_tensor_inputs.0kr#r&r' s  z.z2`callback_on_step_end_tensor_inputs` has to be in z , but found csg|] }|jvr|qSr&rjrlror&r' sz=StableCascadeDecoderPipeline.check_inputs..zCannot forward both `prompt`: z and `prompt_embeds`: z2. Please make sure to only forward one of the two.zeProvide either `prompt` or `prompt_embeds`. Cannot leave both `prompt` and `prompt_embeds` undefined.z2`prompt` has to be of type `str` or `list` but is z'Cannot forward both `negative_prompt`: z and `negative_prompt_embeds`: zu`prompt_embeds` and `negative_prompt_embeds` must have the same shape when passed directly, but got: `prompt_embeds` z != `negative_prompt_embeds` rI)allr/rkrWrXlistrUr,)r#r]r^r:r"callback_on_step_end_tensor_inputsr&ror' check_inputssB z)StableCascadeDecoderPipeline.check_inputscC|jSri_guidance_scaleror&r&r'guidance_scalez+StableCascadeDecoderPipeline.guidance_scalecCs |jdkS)NrCrwror&r&r'r\s z8StableCascadeDecoderPipeline.do_classifier_free_guidancecCrvri)_num_timestepsror&r&r' num_timesteps"rzz*StableCascadeDecoderPipeline.num_timestepscCstdg}ddg}t|d|tjdd}||}|j|}||j||j}}||dtjdd||}|S)NgMb?rrCg?r )rLtensorcospiclampr0r*acos)r#talphas_cumprods clamp_rangemin_varvarratior&r&r'get_timestep_ratio_conditioning&s   &z 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`, usually at the expense of lower image quality. negative_prompt (`str` or `list[str]`, *optional*): The prompt or prompts not to guide the image generation. Ignored when not using guidance (i.e., ignored if `decoder_guidance_scale` is less than `1`). prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. prompt_embeds_pooled (`torch.Tensor`, *optional*): Pre-generated pooled text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, pooled text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. negative_prompt_embeds_pooled (`torch.Tensor`, *optional*): Pre-generated negative pooled text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds_pooled will be generated from `negative_prompt` input argument. num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. generator (`torch.Generator` or `list[torch.Generator]`, *optional*): One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. latents (`torch.Tensor`, *optional*): Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor will be generated by sampling using the supplied random `generator`. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generate image. Choose between: `"pil"` (`PIL.Image.Image`), `"np"` (`np.array`) or `"pt"` (`torch.Tensor`). return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.ImagePipelineOutput`] instead of a plain tuple. callback_on_step_end (`Callable`, *optional*): A function that calls at the end of each denoising steps during the inference. The function is called with the following arguments: `callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict)`. `callback_kwargs` will include a list of all tensors as specified by `callback_on_step_end_tensor_inputs`. callback_on_step_end_tensor_inputs (`list`, *optional*): The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the `._callback_tensor_inputs` attribute of your pipeline class. Examples: Returns: [`~pipelines.ImagePipelineOutput`] or `tuple` [`~pipelines.ImagePipelineOutput`] if `return_dict` is True, otherwise a `tuple`. When returning a tuple, the first element is a list with the generated image embeddings. =2.2.0 when using `torch.bfloat16` dtype.)r^r:rrtrrFNrC) r]r*r2r3r\r^r:rrr;)r*rB clip_sampleFz set `clip_sample` to be Falsebetasg?r )sampletimestep_ratioclip_text_pooledeffnetr) model_outputtimesteprr)rr:r)r=nprlatentzSOnly the output types `pt`, `np`, `pil` and `latent` are supported not output_type=rrrr)=_execution_devicerr+rxrrLbfloat16r/rurWrscatrXrYr,rhr\ zeros_liker set_timesteps timestepsr9rhasattrr.rrOrPrcumprodr{ enumerate progress_barrlongcpuexpandsizer0floatdivchunklerprystep prev_samplelocalspop XLA_AVAILABLExm mark_stepr scale_factordecoderrpermutenumpy numpy_to_pilmaybe_free_model_hooksr)#r#rr]rryr^r:rrr;r3r)rrrrrtr*r+r2r4rralphasrirrpredicted_latentspredicted_latents_textpredicted_latents_uncondcallback_kwargsrncallback_outputsimagesr&r&r'__call__0sX         *          z%StableCascadeDecoderPipeline.__call__)r)NNNNNN)NNNN)%__name__ __module__ __qualname____doc___last_supported_version unet_nametext_encoder_namemodel_cpu_offload_seqrkrrrrrrr r9rLTensorrhrupropertyryr\r|rno_gradr EXAMPLE_DOC_STRINGrsrXr- Generatorboolrr __classcell__r&r&r$r'r:s   n )          r)!typingrrL transformersrrmodelsr schedulersrutilsrr r r utils.torch_utilsr pipeline_utilsrrr#wuerstchen.modeling_paella_vq_modelrtorch_xla.core.xla_modelcore xla_modelrr get_loggerrrOrrr&r&r&r's