o pi@sddlZddlmZmZmZmZddlZddlZddl m Z m Z ddl m Z ddlmZmZmZ  dd d ZGd d d ee ZdS)N)ListOptionalTupleUnion) ConfigMixinregister_to_config) deprecate)KarrasDiffusionSchedulersSchedulerMixinSchedulerOutput+?cosinecCs|dkr dd}n|dkrdd}ntd|g}t|D]}||}|d|}|td|||||qtj|tjdS) a Create a beta schedule that discretizes the given alpha_t_bar function, which defines the cumulative product of (1-beta) over time from t = [0,1]. Contains a function alpha_bar that takes an argument t and transforms it to the cumulative product of (1-beta) up to that part of the diffusion process. Args: num_diffusion_timesteps (`int`): the number of betas to produce. max_beta (`float`): the maximum beta to use; use values lower than 1 to prevent singularities. alpha_transform_type (`str`, *optional*, default to `cosine`): the type of noise schedule for alpha_bar. Choose from `cosine` or `exp` Returns: betas (`np.ndarray`): the betas used by the scheduler to step the model outputs rcSs t|ddtjddS)NgMb?gT㥛 ?r)mathcospitrl/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/schedulers/scheduling_deis_multistep.py alpha_bar_fn7s z)betas_for_alpha_bar..alpha_bar_fnexpcSst|dS)Ng()rrrrrrr<sz"Unsupported alpha_transform_type: r dtype) ValueErrorrangeappendmintorchtensorfloat32)num_diffusion_timestepsmax_betaalpha_transform_typerbetasit1t2rrrbetas_for_alpha_bars    "r)c"@s8eZdZdZddeDZdZe          dXdede de de de e j dede dede de de de d ed!e ed"e d#ef d$d%Zed&d'Zed(d)ZdYd*efd+d,ZdZd-ed.ee ejffd/d0Zd1ejd2ejfd3d4Zd5d6Zd7d8Zd9ejd2ejfd:d;Zd d<d=ejd1ejd2ejfd>d?Zd d<d=ejd1ejd2ejfd@dAZd d<dBeejd1ejd2ejfdCdDZ d d<dBeejd1ejd2ejfdEdFZ!dZdGdHZ"dIdJZ# d[d=ejdKeeejfd1ejdLed2ee$e%ff dMdNZ&d1ejd2ejfdOdPZ'dQejdRejdSej(d2ejfdTdUZ)dVdWZ*d S)\DEISMultistepScheduleru `DEISMultistepScheduler` is a fast high order solver for diffusion ordinary differential equations (ODEs). This model inherits from [`SchedulerMixin`] and [`ConfigMixin`]. Check the superclass documentation for the generic methods the library implements for all schedulers such as loading and saving. Args: num_train_timesteps (`int`, defaults to 1000): The number of diffusion steps to train the model. beta_start (`float`, defaults to 0.0001): The starting `beta` value of inference. beta_end (`float`, defaults to 0.02): The final `beta` value. beta_schedule (`str`, defaults to `"linear"`): The beta schedule, a mapping from a beta range to a sequence of betas for stepping the model. Choose from `linear`, `scaled_linear`, or `squaredcos_cap_v2`. trained_betas (`np.ndarray`, *optional*): Pass an array of betas directly to the constructor to bypass `beta_start` and `beta_end`. solver_order (`int`, defaults to 2): The DEIS order which can be `1` or `2` or `3`. It is recommended to use `solver_order=2` for guided sampling, and `solver_order=3` for unconditional sampling. prediction_type (`str`, defaults to `epsilon`): Prediction type of the scheduler function; can be `epsilon` (predicts the noise of the diffusion process), `sample` (directly predicts the noisy sample`) or `v_prediction` (see section 2.4 of [Imagen Video](https://imagen.research.google/video/paper.pdf) paper). thresholding (`bool`, defaults to `False`): Whether to use the "dynamic thresholding" method. This is unsuitable for latent-space diffusion models such as Stable Diffusion. dynamic_thresholding_ratio (`float`, defaults to 0.995): The ratio for the dynamic thresholding method. Valid only when `thresholding=True`. sample_max_value (`float`, defaults to 1.0): The threshold value for dynamic thresholding. Valid only when `thresholding=True`. algorithm_type (`str`, defaults to `deis`): The algorithm type for the solver. lower_order_final (`bool`, defaults to `True`): Whether to use lower-order solvers in the final steps. Only valid for < 15 inference steps. use_karras_sigmas (`bool`, *optional*, defaults to `False`): Whether to use Karras sigmas for step sizes in the noise schedule during the sampling process. If `True`, the sigmas are determined according to a sequence of noise levels {σi}. timestep_spacing (`str`, defaults to `"linspace"`): The way the timesteps should be scaled. Refer to Table 2 of the [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://huggingface.co/papers/2305.08891) for more information. steps_offset (`int`, defaults to 0): An offset added to the inference steps, as required by some model families. cCsg|]}|jqSr)name).0errr ysz!DEISMultistepScheduler.r -C6?{Gz?linearNrepsilonFףp= ??deislogrhoTlinspacernum_train_timesteps beta_startbeta_end beta_schedule trained_betas solver_orderprediction_type thresholdingdynamic_thresholding_ratiosample_max_valuealgorithm_type solver_typelower_order_finaluse_karras_sigmastimestep_spacing steps_offsetcCs|durtj|tjd|_n:|dkrtj|||tjd|_n*|dkr4tj|d|d|tjdd|_n|dkr>t||_n t|d|jd|j|_tj |jd d |_ t |j |_ t d |j |_ t|j t|j |_d |j |j d|_d|_| d vr| d vr|jddn t| d|j| dvr| dvr|jddn td| d|jd|_tjd |d |tjdddd}t||_dg||_d |_d|_d|_|jd|_dS)Nrr2 scaled_linear?rsquaredcos_cap_v2z is not implemented for r5rdimr )r6) dpmsolverz dpmsolver++r6)rC)r7)midpointheunbh1bh2r7)rDz solver type cpu)rr r!r%r8r)NotImplementedError __class__alphascumprodalphas_cumprodsqrtalpha_tsigma_tloglambda_tsigmasinit_noise_sigmarnum_inference_stepsnpcopy from_numpy timesteps model_outputslower_order_nums _step_index _begin_indexto)selfr9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrerrr__init__|s@$  &  zDEISMultistepScheduler.__init__cC|jS)zg The index counter for current timestep. It will increase 1 after each scheduler step. )rhrkrrr step_indexz!DEISMultistepScheduler.step_indexcCrm)zq The index for the first timestep. It should be set from pipeline with `set_begin_index` method. rirnrrr begin_indexrpz"DEISMultistepScheduler.begin_indexrrcCs ||_dS)z Sets the begin index for the scheduler. This function should be run from pipeline before the inference. Args: begin_index (`int`): The begin index for the scheduler. Nrq)rkrrrrrset_begin_indexs z&DEISMultistepScheduler.set_begin_indexradevicecs8jjdkr%tdjjd|ddddddtj}n\jjdkrUjj|d}t d|d|dddddtj}|jj 7}n,jjdkrxjj|}t jjd| tj}|d8}n t jjdt dj j d }jjrt|t|}j||d }t fd d |D}t||ddgtj}n't|t dt||}dj dj dd }t||ggtj}t|_t|j|tjd _t|_dgjj_d_d_d_ jd_dS)a Sets the discrete timesteps used for the diffusion chain (to be run before inference). Args: num_inference_steps (`int`): The number of diffusion steps used when generating samples with a pre-trained model. device (`str` or `torch.device`, *optional*): The device to which the timesteps should be moved to. If `None`, the timesteps are not moved. r8rr NrSleadingtrailingzY is not supported. Please make sure to choose one of 'linspace', 'leading' or 'trailing'.rJ) in_sigmasracg|]}|qSr) _sigma_to_t)r,sigma log_sigmasrkrrr.z8DEISMultistepScheduler.set_timesteps..rtrrT)!configrGrbr8r9roundrcastypeint64arangerHrarrayrYrFr]flip_convert_to_karras concatenater!interplenrrdr_rjrerar>rfrgrhri)rkrartre step_ratior_ sigma_lastrr{r set_timestepssR   6  $      z$DEISMultistepScheduler.set_timestepssamplereturncCs|j}|j^}}}|tjtjfvr|}|||t|}| }tj ||j j dd}tj |d|j jd}|d}t || ||}|j||g|R}||}|S)as "Dynamic thresholding: At each sampling step we set s to a certain percentile absolute pixel value in xt0 (the prediction of x_0 at timestep t), and if s > 1, then we threshold xt0 to the range [-s, s] and then divide by s. Dynamic thresholding pushes saturated pixels (those near -1 and 1) inwards, thereby actively preventing pixels from saturation at each step. We find that dynamic thresholding results in significantly better photorealism as well as better image-text alignment, especially when using very large guidance weights." https://arxiv.org/abs/2205.11487 r rL)rmax)rshaperr!float64floatreshaperbprodabsquantilerrAclamprB unsqueezerj)rkrr batch_sizechannelsremaining_dims abs_samplesrrr_threshold_samples    z(DEISMultistepScheduler._threshold_samplec Cstt|d}||ddtjf}tj|dkddjddj|jddd}|d}||}||}||||} t| dd} d| || |} | |j} | S)Ng|=r)axisr)rr ) rbr]maximumnewaxiscumsumargmaxcliprr) rkrzr| log_sigmadistslow_idxhigh_idxlowhighwrrrrry9s, z"DEISMultistepScheduler._sigma_to_tcCs$d|ddd}||}||fS)Nr rrJr)rkrzr[r\rrr_sigma_to_alpha_sigma_tQsz.DEISMultistepScheduler._sigma_to_alpha_sigma_trwc Cst|jdr |jj}nd}t|jdr|jj}nd}|dur |n|d}|dur,|n|d}d}tdd|}|d|}|d|}|||||} | S)z6Constructs the noise schedule of Karras et al. (2022). sigma_minN sigma_maxrSrg@r )hasattrrrritemrbr8) rkrwrarrrhoramp min_inv_rho max_inv_rhor_rrrrXs      z)DEISMultistepScheduler._convert_to_karrasr model_outputc Os t|dkr |dn|dd}|dur#t|dkr|d}ntd|dur-tddd|j|j}||\}}|jjd krI||||} n"|jjd krR|} n|jjd kra||||} n td |jjd |jj rt| | } |jj dkr||| |St d)a Convert the model output to the corresponding type the DEIS algorithm needs. Args: model_output (`torch.Tensor`): The direct output from the learned diffusion model. timestep (`int`): The current discrete timestep in the diffusion chain. sample (`torch.Tensor`): A current instance of a sample created by the diffusion process. Returns: `torch.Tensor`: The converted model output. rtimestepNr z/missing `sample` as a required keyward argumentre1.0.0Passing `timesteps` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`r3r v_predictionzprediction_type given as zV must be one of `epsilon`, `sample`, or `v_prediction` for the DEISMultistepScheduler.r6'only support log-rho multistep deis now) rpoprr r_rorrr?r@rrCrU) rkrrargskwargsrrzr[r\x0_predrrrconvert_model_outputqs6         z+DEISMultistepScheduler.convert_model_outputcOs2t|dkr |dn|dd}t|dkr|dn|dd}|dur3t|dkr/|d}ntd|dur=tdd d |durGtdd d |j|jd|j|j}}||\} }||\} }t| t|} t| t|} | | } |j j d kr| | ||t | d |}|St d)au One step for the first-order DEIS (equivalent to DDIM). Args: model_output (`torch.Tensor`): The direct output from the learned diffusion model. timestep (`int`): The current discrete timestep in the diffusion chain. prev_timestep (`int`): The previous discrete timestep in the diffusion chain. sample (`torch.Tensor`): A current instance of a sample created by the diffusion process. Returns: `torch.Tensor`: The sample tensor at the previous timestep. rrNr prev_timestepr0 missing `sample` as a required keyward argumentrerrPassing `prev_timestep` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`r6r5r) rrrr r_rorrr]rrCrrU)rkrrrrrrr\sigma_sr[alpha_sr^lambda_shx_trrrdeis_first_order_updates8     "z.DEISMultistepScheduler.deis_first_order_updatemodel_output_listcOst|dkr |dn|dd}t|dkr|dn|dd}|dur3t|dkr/|d}ntd|dur=tddd |durGtddd |j|jd|j|j|j|jd}}} ||\} }||\} }|| \} } |d |d } }|| || | | }}}|jjd krdd}||||||||}||||||||}| || || ||}|St d)a One step for the second-order multistep DEIS. Args: model_output_list (`List[torch.Tensor]`): The direct outputs from learned diffusion model at current and latter timesteps. sample (`torch.Tensor`): A current instance of a sample created by the diffusion process. Returns: `torch.Tensor`: The sample tensor at the previous timestep. r timestep_listNr rrrrPassing `timestep_list` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`rrSr6cSs2|t| t|dt|t|S)Nr rbr])rbcrrrind_fns2zIDEISMultistepScheduler.multistep_deis_second_order_update..ind_fnr rrrr r_rorrrCrU)rkrrrrrrr\sigma_s0sigma_s1r[alpha_s0alpha_s1m0m1rho_trho_s0rho_s1rcoef1coef2rrrr"multistep_deis_second_order_updatesD       z9DEISMultistepScheduler.multistep_deis_second_order_updatecOst|dkr |dn|dd}t|dkr|dn|dd}|dur3t|dkr/|d}ntd|dur=tddd |durGtddd |j|jd|j|j|j|jd|j|jdf\}}} } ||\} }||\} }|| \} } || \}} |d |d |d }}}|| || | | | |f\}}}}|jjdkrdd}||||||||||}||||||||||}||||||||||}| || ||||||}|St d)a One step for the third-order multistep DEIS. Args: model_output_list (`List[torch.Tensor]`): The direct outputs from learned diffusion model at current and latter timesteps. sample (`torch.Tensor`): A current instance of a sample created by diffusion process. Returns: `torch.Tensor`: The sample tensor at the previous timestep. rrNr rrz/ missing`sample` as a required keyward argumentrrrrSrr6cSs|t|t|t|dt|t|t|t|ddt|d}t|t|t|t|}||S)Nr rr)rrrd numerator denominatorrrrrjs   (zHDEISMultistepScheduler.multistep_deis_third_order_update..ind_fnrr)rkrrrrrrr\rrsigma_s2r[rralpha_s2rrm2rrrrho_s2rrrcoef3rrrr!multistep_deis_third_order_update)sR        $z8DEISMultistepScheduler.multistep_deis_third_order_updatecCsd|dur|j}||k}t|dkrt|jd}|St|dkr*|d}|S|d}|S)Nrr )renonzerorr)rkrschedule_timestepsindex_candidatesrorrrindex_for_timesteps    z)DEISMultistepScheduler.index_for_timestepcCs@|jdurt|tjr||jj}|||_dS|j |_dS)zF Initialize the step_index counter for the scheduler. N) rr isinstancerTensorrjrertrrhri)rkrrrr_init_step_indexs   z'DEISMultistepScheduler._init_step_indexr return_dictc Cs`|jdur td|jdur|||jt|jdko'|jjo't|jdk}|jt|jdko<|jjorfrgrrrrhr ) rkrrrrrElower_order_secondr&rrrrsteps2   ((  zDEISMultistepScheduler.stepcOs|S)a? Ensures interchangeability with schedulers that need to scale the denoising model input depending on the current timestep. Args: sample (`torch.Tensor`): The input sample. Returns: `torch.Tensor`: A scaled input sample. r)rkrrrrrrscale_model_inputs z(DEISMultistepScheduler.scale_model_inputoriginal_samplesnoiserec sjj|j|jd}|jjdkr)t|r)jj|jtjd|j|jtjd}n j|j||j}j durFfdd|D}nj durUj g|j d}n j g|j d}|| }t |j t |j kr}|d}t |j t |j ksn|\}}||||} | S)Nr~mpsrcrxr)r)r,rrrkrrr.r}z4DEISMultistepScheduler.add_noise..rrS)r_rjrtrtyperis_floating_pointrer!rrrorflattenrrr) rkrrrer_ step_indicesrzr[r\ noisy_samplesrrr add_noises$     z DEISMultistepScheduler.add_noisecCs|jjSN)rr9rnrrr__len__szDEISMultistepScheduler.__len__)r/r0r1r2Nrr3Fr4r5r6r7TFr8r)rr)T)+__name__ __module__ __qualname____doc__r _compatiblesorderrintrstrrrbndarrayboolrlpropertyrorrrsrrrtrrrryrrrrrrrrrr rrr IntTensorrrrrrrr*Js.      B   ?" = > I Y   A "r*)rr)rtypingrrrrnumpyrbrconfiguration_utilsrrutilsr scheduling_utilsr r r r)r*rrrrs  ,