o pi.@spddlmZmZddlZddlmZmZddlmZm Z m Z ddl m Z ddl mZe eZGdd d ZdS) )OptionalUnionN)BasicTransformerBlockFreeNoiseTransformerBlock)CrossAttnDownBlockMotionDownBlockMotion UpBlockMotion)logging) randn_tensorc@seZdZdZdeeeeffddZdeeeeffddZ  d"de d e d e d e d e d e j de j dee jdee jfddZ    d#dee de dededdf ddZd$ddZed d!ZdS)%AnimateDiffFreeNoiseMixinz>Mixin class for [FreeNoise](https://arxiv.org/abs/2310.15169).blockcCs|jD]i}t|j}t|D]]}t|j|tr&|j||j|j|j qt|j|t s0J|j|}t|j |j |j |j|j|j|j|j|j|j|j|j|j|j dj|j|jd|j|<|j|j|ddqqdS)z:Helper function to enable FreeNoise in transformer blocks.)dimnum_attention_headsattention_head_dimdropoutcross_attention_dim activation_fnattention_biasonly_cross_attentiondouble_self_attentionpositional_embeddingsnum_positional_embeddingscontext_lengthcontext_strideweighting_schemedevicedtypeTstrictN)motion_moduleslentransformer_blocksrange isinstancerset_free_noise_properties_free_noise_context_length_free_noise_context_stride_free_noise_weighting_schemerrrrrrrrrrrrtorrload_state_dict state_dict)selfr motion_modulenum_transformer_blocksibasic_transfomer_blockr2b/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/pipelines/free_noise_utils.py_enable_free_noise_in_block#sD       z5AnimateDiffFreeNoiseMixin._enable_free_noise_in_blockcCs|jD]K}t|j}t|D]?}t|j|trM|j|}t|j|j|j |j |j |j |j |j|j|j|jd j|j|jd|j|<|j|j|ddqqdS)z;Helper function to disable FreeNoise in transformer blocks.) rrrrrrrrrrrrTrN)r!r"r#r$r%rrrrrrrrrrrrrr*rrr+r,)r-r r.r/r0free_noise_transfomer_blockr2r2r3_disable_free_noise_in_blockIs4     z6AnimateDiffFreeNoiseMixin._disable_free_noise_in_blockN batch_sizenum_channels_latents num_framesheightwidthrr generatorlatentsc  Cs.t|trt||krtdt|d|d|jdkr |jn|} ||| ||j||jf} | durCt| |||d} |jdkrB| Sn'| d|krL| S| d|jkretd|d |jd | d| |} |jd krt |j||j D]u} t d | |j} t || |j }|| }|d krnZttt | |}|tj||d }| }t |||}|||kr| dddd|f| dddd||f<qw||}|d|}| dddd|f| dddd||f<qwn|jdkr||jd|j}tj| g|dd} | ddddd|f} | S)Nz/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.repeat_context)r<rrrandomrz_You have passed `latents` as a parameter to FreeNoise. The expected number of frames is either z or z , but found shuffle_contextr)r<)r)r%listr" ValueErrorr'vae_scale_factorr _free_noise_noise_typesizer*r$r(maxmintorch LongTensorrandpermcat)r-r7r8r9r:r;rrr<r=context_num_framesshaper0 window_start window_end window_lengthindicesshuffled_indices current_start current_end prefix_length num_repeatsr2r2r3_prepare_latents_free_noisees`      . . z5AnimateDiffFreeNoiseMixin._prepare_latents_free_noisepyramidr@rrr noise_typereturnc Csdg}gd}||jjjkrtd|d|jjjd||vr+td|d|||vr9td|d ||p?|jjj|_||_||_||_ g|j j |j j |j j }|D]}||qZd S) a. Enable long video generation using FreeNoise. Args: context_length (`int`, defaults to `16`, *optional*): The number of video frames to process at once. It's recommended to set this to the maximum frames the Motion Adapter was trained with (usually 16/24/32). If `None`, the default value from the motion adapter config is used. context_stride (`int`, *optional*): Long videos are generated by processing many frames. FreeNoise processes these frames in sliding windows of size `context_length`. Context stride allows you to specify how many frames to skip between each window. For example, a context length of 16 and context stride of 4 would process 24 frames as: [0, 15], [4, 19], [8, 23] (0-based indexing) weighting_scheme (`str`, defaults to `pyramid`): Weighting scheme for averaging latents after accumulation in FreeNoise blocks. The following weighting schemes are supported currently: - "pyramid" Peforms weighted averaging with a pyramid like weight pattern: [1, 2, 3, 2, 1]. noise_type (`str`, defaults to "shuffle_context"): TODO r[)r@r>r?zYou have set context_length=zH which is greater than self.motion_adapter.config.motion_max_seq_length=z*. This can lead to bad generation results.z0The parameter `weighting_scheme` must be one of z, but got weighting_scheme=z*The parameter `noise_type` must be one of z, but got noise_type=N)motion_adapterconfigmotion_max_seq_lengthloggerwarningrCr'r(r)rEunet down_blocks mid_block up_blocksr4) r-rrrr\allowed_weighting_schemeallowed_noise_typeblocksr r2r2r3enable_free_noises( z+AnimateDiffFreeNoiseMixin.enable_free_noisecCs:d|_g|jj|jj|jj}|D]}||qdS)N)r'rcrdrerfr6)r-rir r2r2r3disable_free_noises  z,AnimateDiffFreeNoiseMixin.disable_free_noisecCst|do |jduS)Nr')hasattrr')r-r2r2r3free_noise_enabledsz,AnimateDiffFreeNoiseMixin.free_noise_enabled)NN)rYrZr[r@)r]N)__name__ __module__ __qualname____doc__rrrr r4r6intrIrrr GeneratorTensorrXstrrjrkpropertyrmr2r2r2r3r sV&%  L 4r )typingrrrImodels.attentionrrmodels.unets.unet_motion_modelrrr utilsr utils.torch_utilsr get_loggerrnrar r2r2r2r3s