o pi @sddlZddlmZmZmZmZmZmZmZddl Z ddl m Z m Z m Z mZmZddlmZddlmZmZmZmZddlmZmZmZmZmZddlmZmZm Z dd l!m"Z"dd l#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)dd l*m+Z+m,Z,m-Z-m.Z.m/Z/dd l0m1Z1dd l2m3Z3ddl4m5Z5ddl6m7Z7m8Z8ddl9m:Z:e,;e<Z=dZ>dddZ?    d dee@deeeAe jBfdeee@deeeCfddZDGddde7e8eeeee5 ZEdS)!N)AnyCallableDictListOptionalTupleUnion)CLIPImageProcessor CLIPTextModelCLIPTextModelWithProjection CLIPTokenizerCLIPVisionModelWithProjection)PipelineImageInput)FromSingleFileMixinIPAdapterMixin StableDiffusionXLLoraLoaderMixinTextualInversionLoaderMixin) AutoencoderKLImageProjection MotionAdapterUNet2DConditionModelUNetMotionModel)AttnProcessor2_0FusedAttnProcessor2_0XFormersAttnProcessor)adjust_lora_scale_text_encoder) DDIMSchedulerDPMSolverMultistepSchedulerEulerAncestralDiscreteSchedulerEulerDiscreteSchedulerLMSDiscreteScheduler PNDMScheduler)USE_PEFT_BACKENDloggingreplace_example_docstringscale_lora_layersunscale_lora_layers) randn_tensor)VideoProcessor) FreeInitMixin)DiffusionPipelineStableDiffusionMixin)AnimateDiffPipelineOutputa$ Examples: ```py >>> import torch >>> from diffusers.models import MotionAdapter >>> from diffusers import AnimateDiffSDXLPipeline, DDIMScheduler >>> from diffusers.utils import export_to_gif >>> adapter = MotionAdapter.from_pretrained( ... "a-r-r-o-w/animatediff-motion-adapter-sdxl-beta", torch_dtype=torch.float16 ... ) >>> model_id = "stabilityai/stable-diffusion-xl-base-1.0" >>> scheduler = DDIMScheduler.from_pretrained( ... model_id, ... subfolder="scheduler", ... clip_sample=False, ... timestep_spacing="linspace", ... beta_schedule="linear", ... steps_offset=1, ... ) >>> pipe = AnimateDiffSDXLPipeline.from_pretrained( ... model_id, ... motion_adapter=adapter, ... scheduler=scheduler, ... torch_dtype=torch.float16, ... variant="fp16", ... ).to("cuda") >>> # enable memory savings >>> pipe.enable_vae_slicing() >>> pipe.enable_vae_tiling() >>> output = pipe( ... prompt="a panda surfing in the ocean, realistic, high quality", ... negative_prompt="low quality, worst quality", ... num_inference_steps=20, ... guidance_scale=8, ... width=1024, ... height=1024, ... num_frames=16, ... ) >>> frames = output.frames[0] >>> export_to_gif(frames, "animation.gif") ``` cCsX|jttd|jdd}|jttd|jdd}|||}||d||}|S)z Rescale `noise_cfg` according to `guidance_rescale`. Based on findings of [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://arxiv.org/pdf/2305.08891.pdf). See Section 3.4 r.T)dimkeepdim)stdlistrangendim) noise_cfgnoise_pred_textguidance_rescalestd_textstd_cfgnoise_pred_rescaledr=w/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/pipelines/animatediff/pipeline_animatediff_sdxl.pyrescale_noise_cfgss  r?num_inference_stepsdevice timestepssigmascKs|dur |dur td|dur>dtt|jjv}|s(td|jd|jd ||d||j}t |}||fS|durpdtt|jjv}|sZtd|jd|jd ||d ||j}t |}||fS|j|fd |i||j}||fS) a Calls the scheduler's `set_timesteps` method and retrieves timesteps from the scheduler after the call. Handles custom timesteps. Any kwargs will be supplied to `scheduler.set_timesteps`. Args: scheduler (`SchedulerMixin`): The scheduler to get timesteps from. num_inference_steps (`int`): The number of diffusion steps used when generating samples with a pre-trained model. If used, `timesteps` must be `None`. device (`str` or `torch.device`, *optional*): The device to which the timesteps should be moved to. If `None`, the timesteps are not moved. timesteps (`List[int]`, *optional*): Custom timesteps used to override the timestep spacing strategy of the scheduler. If `timesteps` is passed, `num_inference_steps` and `sigmas` must be `None`. sigmas (`List[float]`, *optional*): Custom sigmas used to override the timestep spacing strategy of the scheduler. If `sigmas` is passed, `num_inference_steps` and `timesteps` must be `None`. Returns: `Tuple[torch.Tensor, int]`: A tuple where the first element is the timestep schedule from the scheduler and the second element is the number of inference steps. NzYOnly one of `timesteps` or `sigmas` can be passed. Please choose one to set custom valuesrBzThe current scheduler class zx's `set_timesteps` does not support custom timestep schedules. Please check whether you are using the correct scheduler.)rBrArCzv's `set_timesteps` does not support custom sigmas schedules. Please check whether you are using the correct scheduler.)rCrArAr=) ValueErrorsetinspect signature set_timesteps parameterskeys __class__rBlen) schedulerr@rArBrCkwargsaccepts_timesteps accept_sigmasr=r=r>retrieve_timestepss2  rQcKseZdZdZdZgdZgdZ   dldeded e d e d e d e e e fd ede eeeeeefdededeffdd Z            dmdedeedeejdededeedeedeejdeejdeejdeejd eed!eefd"d#Z dnd$d%Z!d&d'Z"d(d)Z#d*d+Z$       dod,d-Z% dnd.d/Z& dnd0d1Z'd2d3Z(d4ej)fd5ejd6ed7ej*d8ejfd9d:Z+e,d;d<Z-e,d=d>Z.e,d?d@Z/e,dAdBZ0e,dCdDZ1e,dEdFZ2e,dGdHZ3e,dIdJZ4e5e6e7dddKdddLddddMddddNdddddddddOdddNddPdddPddddQgf#de ee8efdee ee8efdRedSeedTeedUedVe8edWe8edXeedYedee ee8efdee ee8efdeedZed[ee ej9e8ej9fdQeejdeejdeejdeejdeejd\ee:d]ee8ejd^eed_ed`ee;eeeee;gdfdie8efFdjdkZ?Z@S)pAnimateDiffSDXLPipelinea Pipeline for text-to-video generation using Stable Diffusion XL. This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.) The pipeline also inherits the following loading methods: - [`~loaders.TextualInversionLoaderMixin.load_textual_inversion`] for loading textual inversion embeddings - [`~loaders.FromSingleFileMixin.from_single_file`] for loading `.ckpt` files - [`~loaders.StableDiffusionXLLoraLoaderMixin.load_lora_weights`] for loading LoRA weights - [`~loaders.StableDiffusionXLLoraLoaderMixin.save_lora_weights`] for saving LoRA weights - [`~loaders.IPAdapterMixin.load_ip_adapter`] for loading IP Adapters Args: vae ([`AutoencoderKL`]): Variational Auto-Encoder (VAE) Model to encode and decode images to and from latent representations. text_encoder ([`CLIPTextModel`]): Frozen text-encoder. Stable Diffusion XL uses the text portion of [CLIP](https://huggingface.co/docs/transformers/model_doc/clip#transformers.CLIPTextModel), specifically the [clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14) variant. text_encoder_2 ([` CLIPTextModelWithProjection`]): Second frozen text-encoder. Stable Diffusion XL uses the text and pool portion of [CLIP](https://huggingface.co/docs/transformers/model_doc/clip#transformers.CLIPTextModelWithProjection), specifically the [laion/CLIP-ViT-bigG-14-laion2B-39B-b160k](https://huggingface.co/laion/CLIP-ViT-bigG-14-laion2B-39B-b160k) variant. tokenizer (`CLIPTokenizer`): Tokenizer of class [CLIPTokenizer](https://huggingface.co/docs/transformers/v4.21.0/en/model_doc/clip#transformers.CLIPTokenizer). tokenizer_2 (`CLIPTokenizer`): Second Tokenizer of class [CLIPTokenizer](https://huggingface.co/docs/transformers/v4.21.0/en/model_doc/clip#transformers.CLIPTokenizer). unet ([`UNet2DConditionModel`]): Conditional U-Net architecture to denoise the encoded image latents. scheduler ([`SchedulerMixin`]): A scheduler to be used in combination with `unet` to denoise the encoded image latents. Can be one of [`DDIMScheduler`], [`LMSDiscreteScheduler`], or [`PNDMScheduler`]. force_zeros_for_empty_prompt (`bool`, *optional*, defaults to `"True"`): Whether the negative prompt embeddings shall be forced to always be set to 0. Also see the config of `stabilityai/stable-diffusion-xl-base-1-0`. z6text_encoder->text_encoder_2->image_encoder->unet->vae) tokenizer tokenizer_2 text_encodertext_encoder_2 image_encoderfeature_extractor)latents prompt_embedsnegative_prompt_embedsadd_text_embeds add_time_idsnegative_pooled_prompt_embedsnegative_add_time_idsNTvaerUrVrSrTunetmotion_adapterrMrWrXforce_zeros_for_empty_promptc stt|trt||}|j||||||||| | d |j| ddt|j j j d|_ t |j d|_|jj j|_dS)N) r`rUrVrSrTrarbrMrWrX)rcr*r.)vae_scale_factor)super__init__ isinstancerr from_unet2dregister_modulesregister_to_configrLr`configblock_out_channelsrdr)video_processorra sample_sizedefault_sample_size) selfr`rUrVrSrTrarbrMrWrXrcrKr=r>rfs&    z AnimateDiffSDXLPipeline.__init__r.promptprompt_2rAnum_videos_per_promptdo_classifier_free_guidancenegative_promptnegative_prompt_2rZr[pooled_prompt_embedsr^ lora_scale clip_skipc! Cs|p|j}| dur9t|tr9| |_|jdur%tst|j| nt|j| |jdur9ts3t|j| nt|j| t|t rA|gn|}|durLt |}n|j d}|j dur\|j |j gn|j g}|jdurk|j|jgn|jg}|dur |pw|}t|t r|gn|}g}||g}t|||D]u\}}}t|tr|||}||d|jddd}|j}||dddj}|j d |j d krt||s||dd|jd d f}td |jd ||||dd }|d} | dur|jd}n|j| d }||qtj|d d}|duo|jj}|r*| dur*|r*t|} t| } n|r| dur|p6d}|p;|}t|t rG||gn|}t|t rT||gn|}|durst|t|urst dt|dt|d|t |krt!d|dt |d|d|d ||g}g}t|||D]8\}}}t|tr|||}|j d }||d|ddd}||j|dd } | d} | jd} || qtj|d d} |jdur|j|jj"|d}n |j|j#j"|d}|j \}}} |$d |d }|%|||d }|r9| j d }|jdur | j|jj"|d} n | j|j#j"|d} | $d |d } | %|||d } | $d |%||d } |rT| $d |%||d } |jdurit|tritrit&|j| |jdur~t|tr~tr~t&|j| || | | fS)a\ Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `List[str]`, *optional*): prompt to be encoded prompt_2 (`str` or `List[str]`, *optional*): The prompt or prompts to be sent to the `tokenizer_2` and `text_encoder_2`. If not defined, `prompt` is used in both text-encoders device: (`torch.device`): torch device num_videos_per_prompt (`int`): number of images that should be generated per prompt do_classifier_free_guidance (`bool`): whether to use classifier free guidance or not negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). negative_prompt_2 (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the image generation to be sent to `tokenizer_2` and `text_encoder_2`. If not defined, `negative_prompt` is used in both text-encoders prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. pooled_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated pooled text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, pooled text embeddings will be generated from `prompt` input argument. negative_pooled_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative pooled text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, pooled negative_prompt_embeds will be generated from `negative_prompt` input argument. lora_scale (`float`, *optional*): A lora scale that will be applied to all LoRA layers of the text encoder if LoRA layers are loaded. clip_skip (`int`, *optional*): Number of layers to be skipped from CLIP while computing the prompt embeddings. A value of 1 means that the output of the pre-final layer will be used for computing the prompt embeddings. Nr max_lengthTpt)paddingr{ truncationreturn_tensorslongest)r}rr.z\The following part of your input was truncated because CLIP can only handle sequences up to z tokens: output_hidden_statesr*r1z?`negative_prompt` should be the same type to `prompt`, but got z != .z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.)dtyperA)'_execution_devicergr _lora_scalerUr#rr&rVstrrLshaperSrTziprmaybe_convert_promptmodel_max_length input_idstorchequal batch_decodeloggerwarningto hidden_statesappendconcatrkrc zeros_liketype TypeErrorrDrrarepeatviewr')!rprrrsrArtrurvrwrZr[rxr^ryrz batch_size tokenizers text_encodersprompt_embeds_listpromptsrSrU text_inputstext_input_idsuntruncated_ids removed_textzero_out_negative_prompt uncond_tokensnegative_prompt_embeds_listr{ uncond_inputbs_embedseq_len_r=r=r> encode_prompt0s :                                 z%AnimateDiffSDXLPipeline.encode_promptc Cst|jj}t|tjs|j|ddj}|j ||d}|rH|j|ddj d}|j |dd}|jt |ddj d}|j |dd}||fS||j }|j |dd}t |} || fS) Nr|)rrArTrrrr)nextrWrIrrgrTensorrX pixel_valuesrrrepeat_interleaver image_embeds) rpimagerAnum_images_per_promptrrimage_enc_hidden_statesuncond_image_enc_hidden_statesruncond_image_embedsr=r=r> encode_images(    z$AnimateDiffSDXLPipeline.encode_imagecCslg}|rg}|duret|ts|g}t|t|jjjkr/tdt|dt|jjjdt||jjjD],\}} t| t } | ||d| \} } | | dddf|rc| | dddfq7n|D]} |rw| d\} } | | | | qgg}t |D]0\} } t j| g|dd} |rt j|| g|dd} t j| | gdd} | j|d} | | q|S) NzK`ip_adapter_image` must have same length as the number of IP Adapters. Got z images and z IP Adapters.r.r*rr)rA)rgr4rLraencoder_hid_projimage_projection_layersrDrrrrchunk enumeratercatr)rpip_adapter_imageip_adapter_image_embedsrArrurnegative_image_embedssingle_ip_adapter_imageimage_proj_layeroutput_hidden_statesingle_image_embedssingle_negative_image_embedsir=r=r>prepare_ip_adapter_image_embeds4sH        z7AnimateDiffSDXLPipeline.prepare_ip_adapter_image_embedsc Csd|jjj|}|j\}}}}}|ddddd|||||}|j|j}|dddf||df|jddddddd}|}|S)Nr.rr*rr) r`rkscaling_factorrpermutereshapedecodesamplefloat) rprYrchannels num_framesheightwidthrvideor=r=r>decode_latentsbs"8z&AnimateDiffSDXLPipeline.decode_latentscCsXdtt|jjjv}i}|r||d<dtt|jjjv}|r*||d<|S)Neta generator)rErFrGrMsteprIrJ)rprr accepts_etaextra_step_kwargsaccepts_generatorr=r=r>prepare_extra_step_kwargsosz1AnimateDiffSDXLPipeline.prepare_extra_step_kwargsc s|ddks |ddkrtd|d|d| dur8tfdd| Ds8tdjd fd d | D|durK|durKtd |d |d|dur^|dur^td|d |d|durj|durjtd|durt|tst|tstdt||durt|tst|tstdt||dur|durtd|d|d|dur|durtd|d|d|dur|dur|j|jkrtd|jd|jd|dur| durtd|dur| durtddSdS)Nrz7`height` and `width` have to be divisible by 8 but are z and rc3s|]}|jvVqdSN_callback_tensor_inputs.0krpr=r> s  z7AnimateDiffSDXLPipeline.check_inputs..z2`callback_on_step_end_tensor_inputs` has to be in z , but found csg|] }|jvr|qSr=rrrr=r> sz8AnimateDiffSDXLPipeline.check_inputs..zCannot forward both `prompt`: z and `prompt_embeds`: z2. Please make sure to only forward one of the two.z Cannot forward both `prompt_2`: zeProvide either `prompt` or `prompt_embeds`. Cannot leave both `prompt` and `prompt_embeds` undefined.z2`prompt` has to be of type `str` or `list` but is z4`prompt_2` has to be of type `str` or `list` but is z'Cannot forward both `negative_prompt`: z and `negative_prompt_embeds`: z)Cannot forward both `negative_prompt_2`: zu`prompt_embeds` and `negative_prompt_embeds` must have the same shape when passed directly, but got: `prompt_embeds` z != `negative_prompt_embeds` zIf `prompt_embeds` are provided, `pooled_prompt_embeds` also have to be passed. Make sure to generate `pooled_prompt_embeds` from the same text encoder that was used to generate `prompt_embeds`.zIf `negative_prompt_embeds` are provided, `negative_pooled_prompt_embeds` also have to be passed. Make sure to generate `negative_pooled_prompt_embeds` from the same text encoder that was used to generate `negative_prompt_embeds`.)rDallrrgrr4rr) rprrrsrrrvrwrZr[rxr^"callback_on_step_end_tensor_inputsr=rr> check_inputssl z$AnimateDiffSDXLPipeline.check_inputsc Cs~|||||j||jf} t|tr%t||kr%tdt|d|d| dur2t| |||d} n| |} | |jj} | S)Nz/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.)rrAr) rdrgr4rLrDr(rrMinit_noise_sigma) rprnum_channels_latentsrrrrrArrYrr=r=r>prepare_latentss"   z'AnimateDiffSDXLPipeline.prepare_latentsc Csdt|||}|jjjt||}|jjjj}||kr(td|d|dt j |g|d}|S)Nz7Model expects an added time embedding vector of length z, but a vector of z was created. The model has an incorrect config. Please check `unet.config.time_embedding_type` and `text_encoder_2.config.projection_dim`.r) r4rarkaddition_time_embed_dimrL add_embeddinglinear_1 in_featuresrDrtensor) rp original_sizecrops_coords_top_left target_sizertext_encoder_projection_dimr]passed_add_embed_dimexpected_add_embed_dimr=r=r>_get_add_time_idss z)AnimateDiffSDXLPipeline._get_add_time_idscCsp|jj}|jjtjdt|jjjjdj t t t f}|r6|jj ||jjj||jjj|dSdS)Nrr)r`rrrfloat32rgdecoder mid_block attentions processorrrrpost_quant_convconv_in)rpruse_torch_2_0_or_xformersr=r=r> upcast_vaes z"AnimateDiffSDXLPipeline.upcast_vaeiw embedding_dimrreturncCst|jdks J|d}|d}ttd|d}ttj||d| }||dddf|dddf}tjt |t |gdd}|ddkrZtj j |d}|j|jd |fksfJ|S) a See https://github.com/google-research/vdm/blob/dc27b98a554f65cdc654b800da5aa1846545d41b/model_vdm.py#L298 Args: w (`torch.Tensor`): Generate embedding vectors with a specified guidance scale to subsequently enrich timestep embeddings. embedding_dim (`int`, *optional*, defaults to 512): Dimension of the embeddings to generate. dtype (`torch.dtype`, *optional*, defaults to `torch.float32`): Data type of the generated embeddings. Returns: `torch.Tensor`: Embedding vectors with shape `(len(w), embedding_dim)`. r.g@@r*g@rNr)rr.r)rLrrlogrexparangerrsincosnn functionalpad)rprrrhalf_dimembr=r=r>get_guidance_scale_embeddings& z4AnimateDiffSDXLPipeline.get_guidance_scale_embeddingcC|jSr)_guidance_scalerr=r=r>guidance_scale%z&AnimateDiffSDXLPipeline.guidance_scalecCrr)_guidance_rescalerr=r=r>r9)rz(AnimateDiffSDXLPipeline.guidance_rescalecCrr) _clip_skiprr=r=r>rz-rz!AnimateDiffSDXLPipeline.clip_skipcCs|jdko |jjjduS)Nr.)rrarktime_cond_proj_dimrr=r=r>ru4sz3AnimateDiffSDXLPipeline.do_classifier_free_guidancecCrr)_cross_attention_kwargsrr=r=r>cross_attention_kwargs8rz.AnimateDiffSDXLPipeline.cross_attention_kwargscCrr)_denoising_endrr=r=r> denoising_end<rz%AnimateDiffSDXLPipeline.denoising_endcCrr)_num_timestepsrr=r=r> num_timesteps@rz%AnimateDiffSDXLPipeline.num_timestepscCrr) _interruptrr=r=r> interruptDrz!AnimateDiffSDXLPipeline.interrupt2g@r0pil)rrrYrrrr@rBrCrrrrrr output_type return_dictrr9rrrnegative_original_sizenegative_crops_coords_top_leftnegative_target_sizecallback_on_step_endrc$@s|p|j|j}|p|j|j}d} |p||f}|p||f}|||||| | |||||# | |_||_|!|_||_| |_d|_|durKt |t rKd}$n|durYt |t rYt |}$n|j d}$|j}%|jdurm|jddnd}&|j|||%| |j| | |||||&|jd \}}}}t|j||%||\}}|jjj}'||$| |'||||j|%|| }|||}(|})|jdurt|j d}*n|jjj}*|j||||j|*d}+|dur| dur|j||| |j|*d},n|+},|jrt j!||gdd }t j!||)gdd })t j!|,|+gdd }+|"|%}|)"|%})|+"|%#|$| d}+|dus |dur,|$|||%|$| |j}-|j%durit |j%t&ri|j%dkri|j%dkritt'|jjj(|j%|jjj(t t t)fd d |}|d|}d}.|jjj*durt +|j,d#|$| }/|j-|/|jjj*d j"|%|jd }.|j.r|j/nd}0t0|0D]}1|j.r|1||1||%|j|\}}t ||_2|j3|j2d}2t4|D]\}3}4|j5rːq|jrt !|gdn|}5|j6|5|4}5|)|+d}6|dus|r|-|6d<|j|5|4||.|j|6ddd}7|jr|77d\}8}9|8|j,|9|8}7|jr&|j8dkr&t9|7|9|j8d}7|jj:|7|4|fi|(ddid}|"dur}i}:|#D] };t;|;|:|;<qA|"||3|4|:}<|<jt j?ko|j>jj@}=|=r|A|"tBtC|j>jDEj}|dkr|}>n |F|}?|jGjH|?|d}>|=r|j>j"t j?d|I|s|>fStJ|>d S)!u0 Function invoked when calling the pipeline for generation. Args: prompt (`str` or `List[str]`, *optional*): The prompt or prompts to guide the video generation. If not defined, one has to pass `prompt_embeds`. instead. prompt_2 (`str` or `List[str]`, *optional*): The prompt or prompts to be sent to the `tokenizer_2` and `text_encoder_2`. If not defined, `prompt` is used in both text-encoders num_frames: The number of video frames that are generated. Defaults to 16 frames which at 8 frames per seconds amounts to 2 seconds of video. height (`int`, *optional*, defaults to self.unet.config.sample_size * self.vae_scale_factor): The height in pixels of the generated video. This is set to 1024 by default for the best results. Anything below 512 pixels won't work well for [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) and checkpoints that are not specifically fine-tuned on low resolutions. width (`int`, *optional*, defaults to self.unet.config.sample_size * self.vae_scale_factor): The width in pixels of the generated video. This is set to 1024 by default for the best results. Anything below 512 pixels won't work well for [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) and checkpoints that are not specifically fine-tuned on low resolutions. num_inference_steps (`int`, *optional*, defaults to 50): The number of denoising steps. More denoising steps usually lead to a higher quality video at the expense of slower inference. timesteps (`List[int]`, *optional*): Custom timesteps to use for the denoising process with schedulers which support a `timesteps` argument in their `set_timesteps` method. If not defined, the default behavior when `num_inference_steps` is passed will be used. Must be in descending order. sigmas (`List[float]`, *optional*): Custom sigmas to use for the denoising process with schedulers which support a `sigmas` argument in their `set_timesteps` method. If not defined, the default behavior when `num_inference_steps` is passed will be used. denoising_end (`float`, *optional*): When specified, determines the fraction (between 0.0 and 1.0) of the total denoising process to be completed before it is intentionally prematurely terminated. As a result, the returned sample will still retain a substantial amount of noise as determined by the discrete timesteps selected by the scheduler. The denoising_end parameter should ideally be utilized when this pipeline forms a part of a "Mixture of Denoisers" multi-pipeline setup, as elaborated in [**Refining the Image Output**](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/stable_diffusion_xl#refining-the-image-output) guidance_scale (`float`, *optional*, defaults to 5.0): Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://arxiv.org/abs/2207.12598). `guidance_scale` is defined as `w` of equation 2. of [Imagen Paper](https://arxiv.org/pdf/2205.11487.pdf). Guidance scale is enabled by setting `guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`, usually at the expense of lower video quality. negative_prompt (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the video generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). negative_prompt_2 (`str` or `List[str]`, *optional*): The prompt or prompts not to guide the video generation to be sent to `tokenizer_2` and `text_encoder_2`. If not defined, `negative_prompt` is used in both text-encoders num_videos_per_prompt (`int`, *optional*, defaults to 1): The number of videos to generate per prompt. eta (`float`, *optional*, defaults to 0.0): Corresponds to parameter eta (η) in the DDIM paper: https://arxiv.org/abs/2010.02502. Only applies to [`schedulers.DDIMScheduler`], will be ignored for others. generator (`torch.Generator` or `List[torch.Generator]`, *optional*): One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. latents (`torch.Tensor`, *optional*): Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for video generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor will ge generated by sampling using the supplied random `generator`. prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. pooled_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated pooled text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, pooled text embeddings will be generated from `prompt` input argument. negative_pooled_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative pooled text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, pooled negative_prompt_embeds will be generated from `negative_prompt` input argument. ip_adapter_image: (`PipelineImageInput`, *optional*): Optional image input to work with IP Adapters. ip_adapter_image_embeds (`List[torch.Tensor]`, *optional*): Pre-generated image embeddings for IP-Adapter. If not provided, embeddings are computed from the `ip_adapter_image` input argument. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generated video. Choose between [PIL](https://pillow.readthedocs.io/en/stable/): `PIL.Image.Image` or `np.array`. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.stable_diffusion_xl.AnimateDiffPipelineOutput`] instead of a plain tuple. cross_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). guidance_rescale (`float`, *optional*, defaults to 0.0): Guidance rescale factor proposed by [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://arxiv.org/pdf/2305.08891.pdf) `guidance_scale` is defined as `φ` in equation 16. of [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://arxiv.org/pdf/2305.08891.pdf). Guidance rescale factor should fix overexposure when using zero terminal SNR. original_size (`Tuple[int]`, *optional*, defaults to (1024, 1024)): If `original_size` is not the same as `target_size` the image will appear to be down- or upsampled. `original_size` defaults to `(height, width)` if not specified. Part of SDXL's micro-conditioning as explained in section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). crops_coords_top_left (`Tuple[int]`, *optional*, defaults to (0, 0)): `crops_coords_top_left` can be used to generate an image that appears to be "cropped" from the position `crops_coords_top_left` downwards. Favorable, well-centered images are usually achieved by setting `crops_coords_top_left` to (0, 0). Part of SDXL's micro-conditioning as explained in section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). target_size (`Tuple[int]`, *optional*, defaults to (1024, 1024)): For most cases, `target_size` should be set to the desired height and width of the generated image. If not specified it will default to `(height, width)`. Part of SDXL's micro-conditioning as explained in section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). negative_original_size (`Tuple[int]`, *optional*, defaults to (1024, 1024)): To negatively condition the generation process based on a specific image resolution. Part of SDXL's micro-conditioning as explained in section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). For more information, refer to this issue thread: https://github.com/huggingface/diffusers/issues/4208. negative_crops_coords_top_left (`Tuple[int]`, *optional*, defaults to (0, 0)): To negatively condition the generation process based on a specific crop coordinates. Part of SDXL's micro-conditioning as explained in section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). For more information, refer to this issue thread: https://github.com/huggingface/diffusers/issues/4208. negative_target_size (`Tuple[int]`, *optional*, defaults to (1024, 1024)): To negatively condition the generation process based on a target image resolution. It should be as same as the `target_size` for most cases. Part of SDXL's micro-conditioning as explained in section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). For more information, refer to this issue thread: https://github.com/huggingface/diffusers/issues/4208. callback_on_step_end (`Callable`, *optional*): A function that calls at the end of each denoising steps during the inference. The function is called with the following arguments: `callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict)`. `callback_kwargs` will include a list of all tensors as specified by `callback_on_step_end_tensor_inputs`. callback_on_step_end_tensor_inputs (`List`, *optional*): The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the `._callback_tensor_inputs` attribute of your pipeline class. Examples: Returns: [`~pipelines.animatediff.pipeline_output.AnimateDiffPipelineOutput`] or `tuple`: If `return_dict` is `True`, [`~pipelines.animatediff.pipeline_output.AnimateDiffPipelineOutput`] is returned, otherwise a `tuple` is returned where the first element is a list with the generated frames. r.FNrscale) rrrsrArtrurvrwrZr[rxr^ryrzr)rrrcs|kSrr=)tsdiscrete_timestep_cutoffr=r>sz2AnimateDiffSDXLPipeline.__call__..)rr)totalr*) text_embedstime_idsr)encoder_hidden_states timestep_condradded_cond_kwargsr&r0)r9r&rYrZr[r\r^r]r_latent)rr%r)frames)Krordrrrrrrr rgrr4rLrrrgetrrurzrQrMrark in_channelsrrrrVintprojection_dimrrrrrrrrroundnum_train_timestepsfilterrrrrfree_init_enabled_free_init_num_itersr5_apply_free_initr progress_barrr!scale_model_inputrr9r?rlocalspopupdater`float16 force_upcastrriterrrIrrmpostprocess_videomaybe_free_model_hooksr/)@rprrrsrrrr@rBrCrrrvrwrtrrrYrZr[rxr^rrr%r&rr9rrrr'r(r)rzr*rrrAryrrr\rr]r_rr4guidance_scale_tensornum_free_init_itersfree_init_iterrBrtlatent_model_inputr5 noise_prednoise_pred_uncondr8callback_kwargsrcallback_outputsneeds_upcastingr video_tensorr=r-r>__call__Hs=                     $        :   z AnimateDiffSDXLPipeline.__call__)NNT) NNr.TNNNNNNNNr)NNNNNNN)A__name__ __module__ __qualname____doc__model_cpu_offload_seq_optional_componentsrrr r r rrrrrr"r!r rrr r boolrfrrrrAr:rrrrrrrrrrrrrrpropertyrr9rzrurrrr!no_gradr%EXAMPLE_DOC_STRINGr GeneratorrrrrrrW __classcell__r=r=rqr>rRs *  0       l.  J                   !"#$rR)r0)NNNN)FrFtypingrrrrrrrr transformersr r r r r image_processorrloadersrrrrmodelsrrrrrmodels.attention_processorrrr models.lorar schedulersrrrr r!r"utilsr#r$r%r&r'utils.torch_utilsr(rmr)free_init_utilsr+pipeline_utilsr,r-pipeline_outputr/ get_loggerrXrrar?r:rrArrQrRr=r=r=r>sR$         2   ;