o Gi @sddlZddlmZmZddlZddlmZmZmZm Z m Z ddl m Z ddl mZmZmZmZddlmZmZmZmZmZddlmZmZmZdd lmZdd lmZm Z m!Z!m"Z"m#Z#m$Z$dd l%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+dd l,m-Z-dd l.m/Z/ddl0m1Z1ddl2m3Z3m4Z4ddl5m6Z6e'rddl7m8m9Z:dZ;ndZ;e(dZ?d!ddZ@    d"deAdBdeBejCBdBdeDeAdBdeDeEdBfddZFGdd d e3e4eeeee1 ZGdS)#N)AnyCallable)CLIPImageProcessor CLIPTextModelCLIPTextModelWithProjection CLIPTokenizerCLIPVisionModelWithProjection)PipelineImageInput)FromSingleFileMixinIPAdapterMixin StableDiffusionXLLoraLoaderMixinTextualInversionLoaderMixin) AutoencoderKLImageProjection MotionAdapterUNet2DConditionModelUNetMotionModel)AttnProcessor2_0FusedAttnProcessor2_0XFormersAttnProcessor)adjust_lora_scale_text_encoder) DDIMSchedulerDPMSolverMultistepSchedulerEulerAncestralDiscreteSchedulerEulerDiscreteSchedulerLMSDiscreteScheduler PNDMScheduler)USE_PEFT_BACKENDis_torch_xla_availableloggingreplace_example_docstringscale_lora_layersunscale_lora_layers) randn_tensor)VideoProcessor) FreeInitMixin)DiffusionPipelineStableDiffusionMixin)AnimateDiffPipelineOutputTFa$ Examples: ```py >>> import torch >>> from diffusers.models import MotionAdapter >>> from diffusers import AnimateDiffSDXLPipeline, DDIMScheduler >>> from diffusers.utils import export_to_gif >>> adapter = MotionAdapter.from_pretrained( ... "a-r-r-o-w/animatediff-motion-adapter-sdxl-beta", torch_dtype=torch.float16 ... ) >>> model_id = "stabilityai/stable-diffusion-xl-base-1.0" >>> scheduler = DDIMScheduler.from_pretrained( ... model_id, ... subfolder="scheduler", ... clip_sample=False, ... timestep_spacing="linspace", ... beta_schedule="linear", ... steps_offset=1, ... ) >>> pipe = AnimateDiffSDXLPipeline.from_pretrained( ... model_id, ... motion_adapter=adapter, ... scheduler=scheduler, ... torch_dtype=torch.float16, ... variant="fp16", ... ).to("cuda") >>> # enable memory savings >>> pipe.enable_vae_slicing() >>> pipe.enable_vae_tiling() >>> output = pipe( ... prompt="a panda surfing in the ocean, realistic, high quality", ... negative_prompt="low quality, worst quality", ... num_inference_steps=20, ... guidance_scale=8, ... width=1024, ... height=1024, ... num_frames=16, ... ) >>> frames = output.frames[0] >>> export_to_gif(frames, "animation.gif") ``` cCsX|jttd|jdd}|jttd|jdd}|||}||d||}|S)a Rescales `noise_cfg` tensor based on `guidance_rescale` to improve image quality and fix overexposure. Based on Section 3.4 from [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://huggingface.co/papers/2305.08891). Args: noise_cfg (`torch.Tensor`): The predicted noise tensor for the guided diffusion process. noise_pred_text (`torch.Tensor`): The predicted noise tensor for the text-guided diffusion process. guidance_rescale (`float`, *optional*, defaults to 0.0): A rescale factor applied to the noise predictions. Returns: noise_cfg (`torch.Tensor`): The rescaled noise prediction tensor. r*T)dimkeepdim)stdlistrangendim) noise_cfgnoise_pred_textguidance_rescalestd_textstd_cfgnoise_pred_rescaledr9m/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/pipelines/animatediff/pipeline_animatediff_sdxl.pyrescale_noise_cfg|s  r;num_inference_stepsdevice timestepssigmascKs|dur |dur td|dur>dtt|jjv}|s(td|jd|jd ||d||j}t |}||fS|durpdtt|jjv}|sZtd|jd|jd ||d ||j}t |}||fS|j|fd |i||j}||fS) a Calls the scheduler's `set_timesteps` method and retrieves timesteps from the scheduler after the call. Handles custom timesteps. Any kwargs will be supplied to `scheduler.set_timesteps`. Args: scheduler (`SchedulerMixin`): The scheduler to get timesteps from. num_inference_steps (`int`): The number of diffusion steps used when generating samples with a pre-trained model. If used, `timesteps` must be `None`. device (`str` or `torch.device`, *optional*): The device to which the timesteps should be moved to. If `None`, the timesteps are not moved. timesteps (`list[int]`, *optional*): Custom timesteps used to override the timestep spacing strategy of the scheduler. If `timesteps` is passed, `num_inference_steps` and `sigmas` must be `None`. sigmas (`list[float]`, *optional*): Custom sigmas used to override the timestep spacing strategy of the scheduler. If `sigmas` is passed, `num_inference_steps` and `timesteps` must be `None`. Returns: `tuple[torch.Tensor, int]`: A tuple where the first element is the timestep schedule from the scheduler and the second element is the number of inference steps. NzYOnly one of `timesteps` or `sigmas` can be passed. Please choose one to set custom valuesr>zThe current scheduler class zx's `set_timesteps` does not support custom timestep schedules. Please check whether you are using the correct scheduler.)r>r=r?zv's `set_timesteps` does not support custom sigmas schedules. Please check whether you are using the correct scheduler.)r?r=r=r9) ValueErrorsetinspect signature set_timesteps parameterskeys __class__r>len) schedulerr<r=r>r?kwargsaccepts_timesteps accept_sigmasr9r9r:retrieve_timestepss2  rMcJseZdZdZdZgdZgdZ   dldeded e d e d e d e e Bd e deeBeBeBeBeBdededeffdd Z            dmdededBdejdBdedededBdedBdejdBdejdBdejdBdejdBd edBd!edBfd"d#Zdnd$d%Zd&d'Z d(d)Z!d*d+Z"       dod,d-Z# dnd.d/Z$ dnd0d1Z%d2d3Z&d4ej'fd5ejd6ed7ej(d8ejfd9d:Z)e*d;d<Z+e*d=d>Z,e*d?d@Z-e*dAdBZ.e*dCdDZ/e*dEdFZ0e*dGdHZ1e*dIdJZ2e3e4e5dddKdddLddddMddddNdddddddddOdddNddPdddPddddQgf#dee6eBdee6eBdBdRedSedBdTedBdUedVe6edWe6edXedBdYedee6eBdBdee6eBdBdedBdZed[ej7e6ej7BdBdQejdBdejdBdejdBdejdBdejdBd\e8dBd]e6ejdBd^edBd_ed`e9ee:fdBdaedbe;eefdBdce;eefdde;eefdBdee;eefdBdfe;eefdge;eefdBd!edBdheS)pAnimateDiffSDXLPipelinea Pipeline for text-to-video generation using Stable Diffusion XL. This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.) The pipeline also inherits the following loading methods: - [`~loaders.TextualInversionLoaderMixin.load_textual_inversion`] for loading textual inversion embeddings - [`~loaders.FromSingleFileMixin.from_single_file`] for loading `.ckpt` files - [`~loaders.StableDiffusionXLLoraLoaderMixin.load_lora_weights`] for loading LoRA weights - [`~loaders.StableDiffusionXLLoraLoaderMixin.save_lora_weights`] for saving LoRA weights - [`~loaders.IPAdapterMixin.load_ip_adapter`] for loading IP Adapters Args: vae ([`AutoencoderKL`]): Variational Auto-Encoder (VAE) Model to encode and decode images to and from latent representations. text_encoder ([`CLIPTextModel`]): Frozen text-encoder. Stable Diffusion XL uses the text portion of [CLIP](https://huggingface.co/docs/transformers/model_doc/clip#transformers.CLIPTextModel), specifically the [clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14) variant. text_encoder_2 ([` CLIPTextModelWithProjection`]): Second frozen text-encoder. Stable Diffusion XL uses the text and pool portion of [CLIP](https://huggingface.co/docs/transformers/model_doc/clip#transformers.CLIPTextModelWithProjection), specifically the [laion/CLIP-ViT-bigG-14-laion2B-39B-b160k](https://huggingface.co/laion/CLIP-ViT-bigG-14-laion2B-39B-b160k) variant. tokenizer (`CLIPTokenizer`): Tokenizer of class [CLIPTokenizer](https://huggingface.co/docs/transformers/v4.21.0/en/model_doc/clip#transformers.CLIPTokenizer). tokenizer_2 (`CLIPTokenizer`): Second Tokenizer of class [CLIPTokenizer](https://huggingface.co/docs/transformers/v4.21.0/en/model_doc/clip#transformers.CLIPTokenizer). unet ([`UNet2DConditionModel`]): Conditional U-Net architecture to denoise the encoded image latents. scheduler ([`SchedulerMixin`]): A scheduler to be used in combination with `unet` to denoise the encoded image latents. Can be one of [`DDIMScheduler`], [`LMSDiscreteScheduler`], or [`PNDMScheduler`]. force_zeros_for_empty_prompt (`bool`, *optional*, defaults to `"True"`): Whether the negative prompt embeddings shall be forced to always be set to 0. Also see the config of `stabilityai/stable-diffusion-xl-base-1-0`. z6text_encoder->text_encoder_2->image_encoder->unet->vae) tokenizer tokenizer_2 text_encodertext_encoder_2 image_encoderfeature_extractor)latents prompt_embedsnegative_prompt_embedsadd_text_embeds add_time_idsnegative_pooled_prompt_embedsnegative_add_time_idsNTvaerQrRrOrPunetmotion_adapterrIrSrTforce_zeros_for_empty_promptc stt|trt||}|j||||||||| | d |j| dt|ddr6dt |j j j dnd|_ t|j d|_t|drY|jdurYt|jj d rY|jj j|_dSd |_dS) N) r\rQrRrOrPr]r^rIrSrT)r_r\r&r*)vae_scale_factorr] sample_size)super__init__ isinstancerr from_unet2dregister_modulesregister_to_configgetattrrHr\configblock_out_channelsrar%video_processorhasattrr]rbdefault_sample_size) selfr\rQrRrOrPr]r^rIrSrTr_rGr9r:res2    (z AnimateDiffSDXLPipeline.__init__r*promptprompt_2r=num_videos_per_promptdo_classifier_free_guidancenegative_promptnegative_prompt_2rVrWpooled_prompt_embedsrZ lora_scale clip_skipc! Cs:|p|j}| dur9t|tr9| |_|jdur%tst|j| nt|j| |jdur9ts3t|j| nt|j| t|t rA|gn|}|durLt |}n|j d}|j dur\|j |j gn|j g}|jdurk|j|jgn|jg}|dur|pw|}t|t r|gn|}g}||g}t|||D]\}}}t|tr|||}||d|jddd}|j}||dddj}|j d |j d krt||s||dd|jd d f}td |jd ||||dd }| dur|djdkr|d} | dur|jd}n|j| d }||qtj|d d}|duo|jj}|r6| dur6|r6t|} t| } n|r| dur|pBd}|pG|}t|t rS||gn|}t|t r`||gn|}|durt |t |urt!dt |dt |d|t |krt"d|dt |d|d|d ||g}g}t|||D]E\}}}t|tr|||}|j d }||d|ddd}||j|dd } | dur| djdkr| d} | jd} || qtj|d d} |jdur|j|jj#|d}n |j|j$j#|d}|j \}}} |%d |d }|&|||d }|rR| j d }|jdur9| j|jj#|d} n | j|j$j#|d} | %d |d } | &|||d } | %d |&||d } |rm| %d |&||d } |jdurt|trtrt'|j| |jdurt|trtrt'|j| || | | fS)a\ Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `list[str]`, *optional*): prompt to be encoded prompt_2 (`str` or `list[str]`, *optional*): The prompt or prompts to be sent to the `tokenizer_2` and `text_encoder_2`. If not defined, `prompt` is used in both text-encoders device: (`torch.device`): torch device num_videos_per_prompt (`int`): number of images that should be generated per prompt do_classifier_free_guidance (`bool`): whether to use classifier free guidance or not negative_prompt (`str` or `list[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). negative_prompt_2 (`str` or `list[str]`, *optional*): The prompt or prompts not to guide the image generation to be sent to `tokenizer_2` and `text_encoder_2`. If not defined, `negative_prompt` is used in both text-encoders prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. pooled_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated pooled text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, pooled text embeddings will be generated from `prompt` input argument. negative_pooled_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative pooled text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, pooled negative_prompt_embeds will be generated from `negative_prompt` input argument. lora_scale (`float`, *optional*): A lora scale that will be applied to all LoRA layers of the text encoder if LoRA layers are loaded. clip_skip (`int`, *optional*): Number of layers to be skipped from CLIP while computing the prompt embeddings. A value of 1 means that the output of the pre-final layer will be used for computing the prompt embeddings. Nr max_lengthTpt)paddingr{ truncationreturn_tensorslongest)r}rr*z\The following part of your input was truncated because CLIP can only handle sequences up to z tokens: output_hidden_statesr&r-z?`negative_prompt` should be the same type to `prompt`, but got z != .z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.)dtyper=)(_execution_devicerfr _lora_scalerQrrr"rRstrrHshaperOrPziprmaybe_convert_promptmodel_max_length input_idstorchequal batch_decodeloggerwarningtor2 hidden_statesappendconcatrkr_ zeros_liketype TypeErrorr@rr]repeatviewr#)!rprrrsr=rtrurvrwrVrWrxrZryrz batch_size tokenizers text_encodersprompt_embeds_listpromptsrOrQ text_inputstext_input_idsuntruncated_ids removed_textzero_out_negative_prompt uncond_tokensnegative_prompt_embeds_listr{ uncond_inputbs_embedseq_len_r9r9r: encode_promptGs :                                  z%AnimateDiffSDXLPipeline.encode_promptc Cst|jj}t|tjs|j|ddj}|j ||d}|rH|j|ddj d}|j |dd}|jt |ddj d}|j |dd}||fS||j }|j |dd}t |} || fS) Nr|)rr=rTrrrr)nextrSrErrfrTensorrT pixel_valuesrrrepeat_interleaver image_embeds) rpimager=num_images_per_promptrrimage_enc_hidden_statesuncond_image_enc_hidden_statesruncond_image_embedsr9r9r: encode_image6s(    z$AnimateDiffSDXLPipeline.encode_imagecCslg}|rg}|duret|ts|g}t|t|jjjkr/tdt|dt|jjjdt||jjjD],\}} t| t } | ||d| \} } | | dddf|rc| | dddfq7n|D]} |rw| d\} } | | | | qgg}t |D]0\} } t j| g|dd} |rt j|| g|dd} t j| | gdd} | j|d} | | q|S) NzK`ip_adapter_image` must have same length as the number of IP Adapters. Got z images and z IP Adapters.r*r&rr)r=)rfr0rHr]encoder_hid_projimage_projection_layersr@rrrrchunk enumeratercatr)rpip_adapter_imageip_adapter_image_embedsr=rrurnegative_image_embedssingle_ip_adapter_imageimage_proj_layeroutput_hidden_statesingle_image_embedssingle_negative_image_embedsir9r9r:prepare_ip_adapter_image_embedsOsH        z7AnimateDiffSDXLPipeline.prepare_ip_adapter_image_embedsc Csd|jjj|}|j\}}}}}|ddddd|||||}|j|j}|dddf||df|jddddddd}|}|S)Nr*rr&r r) r\rkscaling_factorrpermutereshapedecodesamplefloat) rprUrchannels num_framesheightwidthrvideor9r9r:decode_latents}s"8z&AnimateDiffSDXLPipeline.decode_latentscCsXdtt|jjjv}i}|r||d<dtt|jjjv}|r*||d<|S)Neta generator)rArBrCrIsteprErF)rprr accepts_etaextra_step_kwargsaccepts_generatorr9r9r:prepare_extra_step_kwargssz1AnimateDiffSDXLPipeline.prepare_extra_step_kwargsc s|ddks |ddkrtd|d|d| dur8tfdd| Ds8tdjd fd d | D|durK|durKtd |d |d|dur^|dur^td|d |d|durj|durjtd|durt|tst|tstdt||durt|tst|tstdt||dur|durtd|d|d|dur|durtd|d|d|dur|dur|j|jkrtd|jd|jd|dur| durtd|dur| durtddSdS)Nr`rz7`height` and `width` have to be divisible by 8 but are z and rc3s|]}|jvVqdSN_callback_tensor_inputs.0krpr9r: s  z7AnimateDiffSDXLPipeline.check_inputs..z2`callback_on_step_end_tensor_inputs` has to be in z , but found csg|] }|jvr|qSr9rrrr9r: sz8AnimateDiffSDXLPipeline.check_inputs..zCannot forward both `prompt`: z and `prompt_embeds`: z2. Please make sure to only forward one of the two.z Cannot forward both `prompt_2`: zeProvide either `prompt` or `prompt_embeds`. Cannot leave both `prompt` and `prompt_embeds` undefined.z2`prompt` has to be of type `str` or `list` but is z4`prompt_2` has to be of type `str` or `list` but is z'Cannot forward both `negative_prompt`: z and `negative_prompt_embeds`: z)Cannot forward both `negative_prompt_2`: zu`prompt_embeds` and `negative_prompt_embeds` must have the same shape when passed directly, but got: `prompt_embeds` z != `negative_prompt_embeds` zIf `prompt_embeds` are provided, `pooled_prompt_embeds` also have to be passed. Make sure to generate `pooled_prompt_embeds` from the same text encoder that was used to generate `prompt_embeds`.zIf `negative_prompt_embeds` are provided, `negative_pooled_prompt_embeds` also have to be passed. Make sure to generate `negative_pooled_prompt_embeds` from the same text encoder that was used to generate `negative_prompt_embeds`.)r@allrrfrr0rr) rprrrsrrrvrwrVrWrxrZ"callback_on_step_end_tensor_inputsr9rr: check_inputssl z$AnimateDiffSDXLPipeline.check_inputsc Cs~|||||j||jf} t|tr%t||kr%tdt|d|d| dur2t| |||d} n| |} | |jj} | S)Nz/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.)rr=r) rarfr0rHr@r$rrIinit_noise_sigma) rprnum_channels_latentsrrrrr=rrUrr9r9r:prepare_latentss"   z'AnimateDiffSDXLPipeline.prepare_latentsc Csdt|||}|jjjt||}|jjjj}||kr(td|d|dt j |g|d}|S)Nz7Model expects an added time embedding vector of length z, but a vector of z was created. The model has an incorrect config. Please check `unet.config.time_embedding_type` and `text_encoder_2.config.projection_dim`.r) r0r]rkaddition_time_embed_dimrH add_embeddinglinear_1 in_featuresr@rtensor) rp original_sizecrops_coords_top_left target_sizertext_encoder_projection_dimrYpassed_add_embed_dimexpected_add_embed_dimr9r9r:_get_add_time_idss z)AnimateDiffSDXLPipeline._get_add_time_idscCsp|jj}|jjtjdt|jjjjdj t t t f}|r6|jj ||jjj||jjj|dSdS)Nrr)r\rrrfloat32rfdecoder mid_block attentions processorrrrpost_quant_convconv_in)rpruse_torch_2_0_or_xformersr9r9r: upcast_vaes z"AnimateDiffSDXLPipeline.upcast_vaeiw embedding_dimrreturncCst|jdks J|d}|d}ttd|d}ttj||d| }||dddf|dddf}tjt |t |gdd}|ddkrZtj j |d}|j|jd |fksfJ|S) a See https://github.com/google-research/vdm/blob/dc27b98a554f65cdc654b800da5aa1846545d41b/model_vdm.py#L298 Args: w (`torch.Tensor`): Generate embedding vectors with a specified guidance scale to subsequently enrich timestep embeddings. embedding_dim (`int`, *optional*, defaults to 512): Dimension of the embeddings to generate. dtype (`torch.dtype`, *optional*, defaults to `torch.float32`): Data type of the generated embeddings. Returns: `torch.Tensor`: Embedding vectors with shape `(len(w), embedding_dim)`. r*g@@r&g@rNr)rr*r)rHrrlogrexparangerrsincosnn functionalpad)rprrrhalf_dimembr9r9r:get_guidance_scale_embedding"s& z4AnimateDiffSDXLPipeline.get_guidance_scale_embeddingcC|jSr)_guidance_scalerr9r9r:guidance_scale@z&AnimateDiffSDXLPipeline.guidance_scalecCrr)_guidance_rescalerr9r9r:r5Drz(AnimateDiffSDXLPipeline.guidance_rescalecCrr) _clip_skiprr9r9r:rzHrz!AnimateDiffSDXLPipeline.clip_skipcCs|jdko |jjjduS)Nr*)rr]rktime_cond_proj_dimrr9r9r:ruOsz3AnimateDiffSDXLPipeline.do_classifier_free_guidancecCrr)_cross_attention_kwargsrr9r9r:cross_attention_kwargsSrz.AnimateDiffSDXLPipeline.cross_attention_kwargscCrr)_denoising_endrr9r9r: denoising_endWrz%AnimateDiffSDXLPipeline.denoising_endcCrr)_num_timestepsrr9r9r: num_timesteps[rz%AnimateDiffSDXLPipeline.num_timestepscCrr) _interruptrr9r9r: interrupt_rz!AnimateDiffSDXLPipeline.interrupt2g@r,pil)rrrUrrrr<r>r?rrrrrr output_type return_dictrr5rrrnegative_original_sizenegative_crops_coords_top_leftnegative_target_sizecallback_on_step_endrc$@s|p|j|j}|p|j|j}d} |p||f}|p||f}|||||| | |||||# | |_||_|!|_||_| |_d|_|durKt |t rKd}$n|durYt |t rYt |}$n|j d}$|j}%|jdurm|jddnd}&|j|||%| |j| | |||||&|jd \}}}}t|j||%||\}}|jjj}'||$| |'||||j|%|| }|||}(|})|jdurt|j d}*n|jjj}*|j||||j|*d}+|dur| dur|j||| |j|*d},n|+},|jrt j!||gdd }t j!||)gdd })t j!|,|+gdd }+|j"|dd }|#|%}|)#|%})|+#|%$|$| d}+|dus'|dur3|%|||%|$| |j}-|j&durpt |j&t'rp|j&dkrp|j&dkrptt(|jjj)|j&|jjj)t t t*fd d |}|d|}d}.|jjj+durt ,|j-d$|$| }/|j.|/|jjj+d j#|%|jd}.|j/r|j0nd}0t1|0D]}1|j/r|2||1||%|j|\}}t ||_3|j4|j3d}2t5|D]\}3}4|j6rҐq|jrt !|gdn|}5|j7|5|4}5|)|+d}6|dus|r|-|6d<|j|5|4||.|j|6ddd}7|jr|78d\}8}9|8|j-|9|8}7|jr-|j9dkr-t:|7|9|j9d}7|jj;|7|4|fi|(ddid}|"duri}:|#D] };t<|;|:|;<qH|"||3|4|:}<|<=d|}|<=d|}|<=d|}|<=d|)})|<=d|}|<=d|+}+|<=d|,},|2>t?rt@AqWdn 1swYq|jBjt jCko|jBjjD}=|=r|E|#tFtG|jBjHIj}|dkr|}>n |J|}?|jKjL|?|d}>|=r|jBj#t jCd |M|s|>fStN|>d!S)"u%1 Function invoked when calling the pipeline for generation. Args: prompt (`str` or `list[str]`, *optional*): The prompt or prompts to guide the video generation. If not defined, one has to pass `prompt_embeds`. instead. prompt_2 (`str` or `list[str]`, *optional*): The prompt or prompts to be sent to the `tokenizer_2` and `text_encoder_2`. If not defined, `prompt` is used in both text-encoders num_frames: The number of video frames that are generated. Defaults to 16 frames which at 8 frames per seconds amounts to 2 seconds of video. height (`int`, *optional*, defaults to self.unet.config.sample_size * self.vae_scale_factor): The height in pixels of the generated video. This is set to 1024 by default for the best results. Anything below 512 pixels won't work well for [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) and checkpoints that are not specifically fine-tuned on low resolutions. width (`int`, *optional*, defaults to self.unet.config.sample_size * self.vae_scale_factor): The width in pixels of the generated video. This is set to 1024 by default for the best results. Anything below 512 pixels won't work well for [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) and checkpoints that are not specifically fine-tuned on low resolutions. num_inference_steps (`int`, *optional*, defaults to 50): The number of denoising steps. More denoising steps usually lead to a higher quality video at the expense of slower inference. timesteps (`list[int]`, *optional*): Custom timesteps to use for the denoising process with schedulers which support a `timesteps` argument in their `set_timesteps` method. If not defined, the default behavior when `num_inference_steps` is passed will be used. Must be in descending order. sigmas (`list[float]`, *optional*): Custom sigmas to use for the denoising process with schedulers which support a `sigmas` argument in their `set_timesteps` method. If not defined, the default behavior when `num_inference_steps` is passed will be used. denoising_end (`float`, *optional*): When specified, determines the fraction (between 0.0 and 1.0) of the total denoising process to be completed before it is intentionally prematurely terminated. As a result, the returned sample will still retain a substantial amount of noise as determined by the discrete timesteps selected by the scheduler. The denoising_end parameter should ideally be utilized when this pipeline forms a part of a "Mixture of Denoisers" multi-pipeline setup, as elaborated in [**Refining the Image Output**](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/stable_diffusion_xl#refining-the-image-output) guidance_scale (`float`, *optional*, defaults to 5.0): Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://huggingface.co/papers/2207.12598). `guidance_scale` is defined as `w` of equation 2. of [Imagen Paper](https://huggingface.co/papers/2205.11487). Guidance scale is enabled by setting `guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`, usually at the expense of lower video quality. negative_prompt (`str` or `list[str]`, *optional*): The prompt or prompts not to guide the video generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). negative_prompt_2 (`str` or `list[str]`, *optional*): The prompt or prompts not to guide the video generation to be sent to `tokenizer_2` and `text_encoder_2`. If not defined, `negative_prompt` is used in both text-encoders num_videos_per_prompt (`int`, *optional*, defaults to 1): The number of videos to generate per prompt. eta (`float`, *optional*, defaults to 0.0): Corresponds to parameter eta (η) in the DDIM paper: https://huggingface.co/papers/2010.02502. Only applies to [`schedulers.DDIMScheduler`], will be ignored for others. generator (`torch.Generator` or `list[torch.Generator]`, *optional*): One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. latents (`torch.Tensor`, *optional*): Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for video generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor will be generated by sampling using the supplied random `generator`. prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. pooled_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated pooled text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, pooled text embeddings will be generated from `prompt` input argument. negative_pooled_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative pooled text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, pooled negative_prompt_embeds will be generated from `negative_prompt` input argument. ip_adapter_image: (`PipelineImageInput`, *optional*): Optional image input to work with IP Adapters. ip_adapter_image_embeds (`list[torch.Tensor]`, *optional*): Pre-generated image embeddings for IP-Adapter. If not provided, embeddings are computed from the `ip_adapter_image` input argument. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generated video. Choose between [PIL](https://pillow.readthedocs.io/en/stable/): `PIL.Image.Image` or `np.array`. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.stable_diffusion_xl.AnimateDiffPipelineOutput`] instead of a plain tuple. cross_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). guidance_rescale (`float`, *optional*, defaults to 0.0): Guidance rescale factor proposed by [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://huggingface.co/papers/2305.08891) `guidance_scale` is defined as `φ` in equation 16. of [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://huggingface.co/papers/2305.08891). Guidance rescale factor should fix overexposure when using zero terminal SNR. original_size (`tuple[int]`, *optional*, defaults to (1024, 1024)): If `original_size` is not the same as `target_size` the image will appear to be down- or upsampled. `original_size` defaults to `(height, width)` if not specified. Part of SDXL's micro-conditioning as explained in section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). crops_coords_top_left (`tuple[int]`, *optional*, defaults to (0, 0)): `crops_coords_top_left` can be used to generate an image that appears to be "cropped" from the position `crops_coords_top_left` downwards. Favorable, well-centered images are usually achieved by setting `crops_coords_top_left` to (0, 0). Part of SDXL's micro-conditioning as explained in section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). target_size (`tuple[int]`, *optional*, defaults to (1024, 1024)): For most cases, `target_size` should be set to the desired height and width of the generated image. If not specified it will default to `(height, width)`. Part of SDXL's micro-conditioning as explained in section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). negative_original_size (`tuple[int]`, *optional*, defaults to (1024, 1024)): To negatively condition the generation process based on a specific image resolution. Part of SDXL's micro-conditioning as explained in section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). For more information, refer to this issue thread: https://github.com/huggingface/diffusers/issues/4208. negative_crops_coords_top_left (`tuple[int]`, *optional*, defaults to (0, 0)): To negatively condition the generation process based on a specific crop coordinates. Part of SDXL's micro-conditioning as explained in section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). For more information, refer to this issue thread: https://github.com/huggingface/diffusers/issues/4208. negative_target_size (`tuple[int]`, *optional*, defaults to (1024, 1024)): To negatively condition the generation process based on a target image resolution. It should be as same as the `target_size` for most cases. Part of SDXL's micro-conditioning as explained in section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). For more information, refer to this issue thread: https://github.com/huggingface/diffusers/issues/4208. callback_on_step_end (`Callable`, *optional*): A function that calls at the end of each denoising steps during the inference. The function is called with the following arguments: `callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict)`. `callback_kwargs` will include a list of all tensors as specified by `callback_on_step_end_tensor_inputs`. callback_on_step_end_tensor_inputs (`list`, *optional*): The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the `._callback_tensor_inputs` attribute of your pipeline class. Examples: Returns: [`~pipelines.animatediff.pipeline_output.AnimateDiffPipelineOutput`] or `tuple`: If `return_dict` is `True`, [`~pipelines.animatediff.pipeline_output.AnimateDiffPipelineOutput`] is returned, otherwise a `tuple` is returned where the first element is a list with the generated frames. r*FNrscale) rrrsr=rtrurvrwrVrWrxrZryrzr)rrr)repeatsr-cs|kSrr9)tsdiscrete_timestep_cutoffr9r:sz2AnimateDiffSDXLPipeline.__call__..)rr)totalr&) text_embedstime_idsr)encoder_hidden_states timestep_condradded_cond_kwargsr%r,)r5r%rUrVrWrXrZrYr[latent)rr$r)frames)Ororarrrrrrrrfrr0rHrrrgetrrurzrMrIr]rk in_channelsrrrrRintprojection_dimrrrrrrrrrroundnum_train_timestepsfilterrrrrfree_init_enabled_free_init_num_itersr1_apply_free_initr progress_barrr scale_model_inputrr5r;rlocalspopupdate XLA_AVAILABLExm mark_stepr\float16 force_upcastrriterrrErrmpostprocess_videomaybe_free_model_hooksr+)@rprrrsrrrr<r>r?rrrvrwrtrrrUrVrWrxrZrrr$r%rr5rrrr&r'r(rzr)rrr=ryrrrXrrYr[rr4guidance_scale_tensornum_free_init_itersfree_init_iterrBrtlatent_model_inputr5 noise_prednoise_pred_uncondr4callback_kwargsrcallback_outputsneeds_upcastingr video_tensorr9r-r:__call__cs>                     $       =   z AnimateDiffSDXLPipeline.__call__)NNT) NNr*TNNNNNNNNr)NNNNNNN)?__name__ __module__ __qualname____doc__model_cpu_offload_seq_optional_componentsrrrrrrrrrrrrrrrrboolrerrr=r:rrrrrrrrrrrrrrpropertyrr5rzrurrrr no_gradr!EXAMPLE_DOC_STRINGr0 Generatorr dictrtuplerrZ __classcell__r9r9rqr:rNs * 2       p.  J                    !"#$rN)r,)NNNN)HrBtypingrrr transformersrrrrrimage_processorr loadersr r r rmodelsrrrrrmodels.attention_processorrrr models.lorar schedulersrrrrrrutilsrrr r!r"r#utils.torch_utilsr$rmr%free_init_utilsr'pipeline_utilsr(r)pipeline_outputr+torch_xla.core.xla_modelcore xla_modelrHrG get_loggerr[rrdr;r:rr=r0rrMrNr9r9r9r:sZ          2    ;