o pi@sddlZddlmZmZmZmZddlZddlZddl m Z m Z ddl m Z mZddlmZddlmZmZmZeeZ dd d ZGd ddee ZdS)N)ListOptionalTupleUnion) ConfigMixinregister_to_config) deprecatelogging) randn_tensor)KarrasDiffusionSchedulersSchedulerMixinSchedulerOutput+?cosinecCs|dkr dd}n|dkrdd}ntd|g}t|D]}||}|d|}|td|||||qtj|tjdS) a Create a beta schedule that discretizes the given alpha_t_bar function, which defines the cumulative product of (1-beta) over time from t = [0,1]. Contains a function alpha_bar that takes an argument t and transforms it to the cumulative product of (1-beta) up to that part of the diffusion process. Args: num_diffusion_timesteps (`int`): the number of betas to produce. max_beta (`float`): the maximum beta to use; use values lower than 1 to prevent singularities. alpha_transform_type (`str`, *optional*, default to `cosine`): the type of noise schedule for alpha_bar. Choose from `cosine` or `exp` Returns: betas (`np.ndarray`): the betas used by the scheduler to step the model outputs rcSs t|ddtjddS)NgMb?gT㥛 ?r)mathcospitrr/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/schedulers/scheduling_dpmsolver_singlestep.py alpha_bar_fn:s z)betas_for_alpha_bar..alpha_bar_fnexpcSst|dS)Ng()rrrrrrr?sz"Unsupported alpha_transform_type: r dtype) ValueErrorrangeappendmintorchtensorfloat32)num_diffusion_timestepsmax_betaalpha_transform_typerbetasit1t2rrrbetas_for_alpha_bar!s    "r+c%@seZdZdZddeDZdZeddddd d d d d dddd d ded d fde dedede de e j de de dededede de ded e ed!e e d"ed#e e f"d$d%Zd&e d'ee fd(d)Zed*d+Zed,d-Zdad/e fd0d1Z dbd&e d2ee ejfd3e ee fd4d5Zd6ejd'ejfd7d8Zd9d:Zd;d<Zd=ejd'ejfd>d?Zd d@dAejd6ejd'ejfdBdCZd d dDdAejd6ejdEe ejd'ejfdFdGZ d d dDdHeejd6ejdEe ejd'ejfdIdJZ!d d@dHeejd6ejd'ejfdKdLZ"d d d dMdHeejd6ejdNe dEe ejd'ejf dOdPZ#dcdQdRZ$dSdTZ% UdddAejdVee ejfd6ejdWed'ee&e'ff dXdYZ(d6ejd'ejfdZd[Z)d\ejdEejd3ej*d'ejfd]d^Z+d_d`Z,d S)eDPMSolverSinglestepScheduleru `DPMSolverSinglestepScheduler` is a fast dedicated high-order solver for diffusion ODEs. This model inherits from [`SchedulerMixin`] and [`ConfigMixin`]. Check the superclass documentation for the generic methods the library implements for all schedulers such as loading and saving. Args: num_train_timesteps (`int`, defaults to 1000): The number of diffusion steps to train the model. beta_start (`float`, defaults to 0.0001): The starting `beta` value of inference. beta_end (`float`, defaults to 0.02): The final `beta` value. beta_schedule (`str`, defaults to `"linear"`): The beta schedule, a mapping from a beta range to a sequence of betas for stepping the model. Choose from `linear`, `scaled_linear`, or `squaredcos_cap_v2`. trained_betas (`np.ndarray`, *optional*): Pass an array of betas directly to the constructor to bypass `beta_start` and `beta_end`. solver_order (`int`, defaults to 2): The DPMSolver order which can be `1` or `2` or `3`. It is recommended to use `solver_order=2` for guided sampling, and `solver_order=3` for unconditional sampling. prediction_type (`str`, defaults to `epsilon`, *optional*): Prediction type of the scheduler function; can be `epsilon` (predicts the noise of the diffusion process), `sample` (directly predicts the noisy sample`) or `v_prediction` (see section 2.4 of [Imagen Video](https://imagen.research.google/video/paper.pdf) paper). thresholding (`bool`, defaults to `False`): Whether to use the "dynamic thresholding" method. This is unsuitable for latent-space diffusion models such as Stable Diffusion. dynamic_thresholding_ratio (`float`, defaults to 0.995): The ratio for the dynamic thresholding method. Valid only when `thresholding=True`. sample_max_value (`float`, defaults to 1.0): The threshold value for dynamic thresholding. Valid only when `thresholding=True` and `algorithm_type="dpmsolver++"`. algorithm_type (`str`, defaults to `dpmsolver++`): Algorithm type for the solver; can be `dpmsolver` or `dpmsolver++` or `sde-dpmsolver++`. The `dpmsolver` type implements the algorithms in the [DPMSolver](https://huggingface.co/papers/2206.00927) paper, and the `dpmsolver++` type implements the algorithms in the [DPMSolver++](https://huggingface.co/papers/2211.01095) paper. It is recommended to use `dpmsolver++` or `sde-dpmsolver++` with `solver_order=2` for guided sampling like in Stable Diffusion. solver_type (`str`, defaults to `midpoint`): Solver type for the second-order solver; can be `midpoint` or `heun`. The solver type slightly affects the sample quality, especially for a small number of steps. It is recommended to use `midpoint` solvers. lower_order_final (`bool`, defaults to `True`): Whether to use lower-order solvers in the final steps. Only valid for < 15 inference steps. This can stabilize the sampling of DPMSolver for steps < 15, especially for steps <= 10. use_karras_sigmas (`bool`, *optional*, defaults to `False`): Whether to use Karras sigmas for step sizes in the noise schedule during the sampling process. If `True`, the sigmas are determined according to a sequence of noise levels {σi}. final_sigmas_type (`str`, *optional*, defaults to `"zero"`): The final `sigma` value for the noise schedule during the sampling process. If `"sigma_min"`, the final sigma is the same as the last sigma in the training schedule. If `zero`, the final sigma is set to 0. lambda_min_clipped (`float`, defaults to `-inf`): Clipping threshold for the minimum value of `lambda(t)` for numerical stability. This is critical for the cosine (`squaredcos_cap_v2`) noise schedule. variance_type (`str`, *optional*): Set to "learned" or "learned_range" for diffusion models that predict variance. If set, the model's output contains the predicted Gaussian variance. cCsg|]}|jqSr)name).0errr sz'DPMSolverSinglestepScheduler.r ig-C6?g{Gz?linearNrepsilonFgףp= ?? dpmsolver++midpointzeroinfnum_train_timesteps beta_startbeta_end beta_schedule trained_betas solver_orderprediction_type thresholdingdynamic_thresholding_ratiosample_max_valuealgorithm_type solver_typelower_order_finaluse_karras_sigmasfinal_sigmas_typelambda_min_clipped variance_typecCs| dkr d}tdd||durtj|tjd|_n:|dkr*tj|||tjd|_n*|dkr@tj|d|d|tjdd |_n|d krJt||_n t|d |jd |j|_ tj |j d d|_ t |j |_ t d|j |_t|j t|j|_d|j |j d|_d |_| dvr| dkr|jddn t| d |j| dvr| dvr|jddn t| d |j| dvr|dkrtd|d| dd|_tjd |d|tjdddd}t||_dg||_d|_|||_d|_d|_|j d|_dS)N dpmsolverzalgorithm_type `dpmsolver` is deprecated and will be removed in a future version. Choose from `dpmsolver++` or `sde-dpmsolver++` insteadzalgorithm_types=dpmsolver1.0.0rr1 scaled_linear?rsquaredcos_cap_v2z is not implemented for r3rdimr )rIr4sde-dpmsolver++deisr4)rB)r5heun)logrhobh1bh2r5)rCr4rPr6z`final_sigmas_type` z' is not supported for `algorithm_type` z%. Please chooose `sigma_min` instead.cpu)!r r!r"r#r'linspacer+NotImplementedError __class__alphascumprodalphas_cumprodsqrtalpha_tsigma_tloglambda_tsigmasinit_noise_sigmarrnum_inference_stepsnpcopy from_numpy timesteps model_outputssampleget_order_list order_list _step_index _begin_indexto)selfr8r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHdeprecation_messagerjrrr__init__sP $  &   z%DPMSolverSinglestepScheduler.__init__rfreturncCsR|}|jj}|dkrtd|jjr|dkrP|ddkr/gd|ddddgdg}|S|ddkrBgd|ddg}|Sgd|dddg}|S|dkrw|ddkrjddg|ddddg}|Sddg|ddg}|S|dkrdg|}|S|dkrgd|d}|S|dkrddg|d}|S|dkrdg|}|S)z Computes the solver order at each time step. Args: num_inference_steps (`int`): The number of diffusion steps used when generating samples with a pre-trained model. z,Order > 3 is not supported by this schedulerr)r rrvr r)configr=rrD)rrrfstepsorderordersrrrrms@ "      z+DPMSolverSinglestepScheduler.get_order_listcC|jS)zg The index counter for current timestep. It will increase 1 after each scheduler step. )rorrrrr step_indexz'DPMSolverSinglestepScheduler.step_indexcCr{)zq The index for the first timestep. It should be set from pipeline with `set_begin_index` method. rpr|rrr begin_indexr~z(DPMSolverSinglestepScheduler.begin_indexrrcCs ||_dS)z Sets the begin index for the scheduler. This function should be run from pipeline before the inference. Args: begin_index (`int`): The begin index for the scheduler. Nr)rrrrrrset_begin_index s z,DPMSolverSinglestepScheduler.set_begin_indexdevicerjcs|dur |dur td|dur|durtd|dur$jjr$td|p)t|}|_|dur;t|tj}n.t t j dgjj }tdjjd||ddddddtj}tdjjd}jjrt|t |}j||d}tfd d |D}n t|tdt||}jjd krdjdjdd}njjd krd}n td jjt||ggtj}t |j|d_t |j|t jd_dgjj_d_ jj!s|jjdkrt"#dj$ddjj!s.jjd kr.t"#dj$dd%|_&d_'d_(jd_dS)a Sets the discrete timesteps used for the diffusion chain (to be run before inference). Args: num_inference_steps (`int`): The number of diffusion steps used when generating samples with a pre-trained model. device (`str` or `torch.device`, *optional*): The device to which the timesteps should be moved to. If `None`, the timesteps are not moved. timesteps (`List[int]`, *optional*): Custom timesteps used to support arbitrary spacing between timesteps. If `None`, then the default timestep spacing strategy of equal spacing between timesteps schedule is used. If `timesteps` is passed, `num_inference_steps` must be `None`. Nz?Must pass exactly one of `num_inference_steps` or `timesteps`.zDPMSolverSinglestepScheduler.set_timesteps.. sigma_minr6zC `final_sigmas_type` must be one of `sigma_min` or `zero`, but got )rrrzChanging scheduler {self.config} to have `lower_order_final` set to True to handle uneven amount of inference steps. Please make sure to always use an even number of `num_inference steps when using `lower_order_final=False`.T)rDz `last_sigmas_type='zero'` is not supported for `lower_order_final=False`. Changing scheduler {self.config} to have `lower_order_final` set to True.rX))rrwrElenrfrgarrayastypeint64r! searchsortedfliprcrGrYr8roundrhr^rb_convert_to_karrasinterparangerF concatenater#rirqrdrjr=rkrlrDloggerwarningrrmrnrorp)rrrfrrj clipped_idxrd sigma_lastrrr set_timestepssh          z*DPMSolverSinglestepScheduler.set_timestepsrlcCs|j}|j^}}}|tjtjfvr|}|||t|}| }tj ||j j dd}tj |d|j jd}|d}t || ||}|j||g|R}||}|S)as "Dynamic thresholding: At each sampling step we set s to a certain percentile absolute pixel value in xt0 (the prediction of x_0 at timestep t), and if s > 1, then we threshold xt0 to the range [-s, s] and then divide by s. Dynamic thresholding pushes saturated pixels (those near -1 and 1) inwards, thereby actively preventing pixels from saturation at each step. We find that dynamic thresholding results in significantly better photorealism as well as better image-text alignment, especially when using very large guidance weights." https://arxiv.org/abs/2205.11487 r rN)r max)rshaper!r#float64floatreshapergprodabsquantilerwr@clamprA unsqueezerq)rrrlr batch_sizechannelsremaining_dims abs_samplesrrr_threshold_samplems    z.DPMSolverSinglestepScheduler._threshold_samplec Cstt|d}||ddtjf}tj|dkddjddj|jddd}|d}||}||}||||} t| dd} d| || |} | |j} | S)Ng|=r)axisr)rr ) rgrbmaximumnewaxiscumsumargmaxcliprr) rrrr log_sigmadistslow_idxhigh_idxlowhighwrrrrrs, z(DPMSolverSinglestepScheduler._sigma_to_tcCs$d|ddd}||}||fS)Nr rrLr)rrrr`rarrr_sigma_to_alpha_sigma_tsz4DPMSolverSinglestepScheduler._sigma_to_alpha_sigma_trc Cst|jdr |jj}nd}t|jdr|jj}nd}|dur |n|d}|dur,|n|d}d}tdd|}|d|}|d|}|||||} | S)z6Constructs the noise schedule of Karras et al. (2022).rN sigma_maxrWrg@r )hasattrrwrritemrgrY) rrrrfrrrhoramp min_inv_rho max_inv_rhordrrrrs      z/DPMSolverSinglestepScheduler._convert_to_karrasrl model_outputc OsJt|dkr |dn|dd}|dur#t|dkr|d}ntd|dur-tddd|jjd vr|jjd kr_|jjd vrI|dddd f}|j|j }| |\}}||||} n/|jjd krh|} n&|jjdkr|j|j }| |\}}||||} n td|jjd|jj r| | } | S|jjdkr#|jjd kr|jjd vr|dddd f} nE|} nB|jjd kr|j|j }| |\}}||||} n&|jjdkr|j|j }| |\}}||||} n td|jjd|jj r!|j ||j|}}||| |} | | } ||| |} | SdS)a0 Convert the model output to the corresponding type the DPMSolver/DPMSolver++ algorithm needs. DPM-Solver is designed to discretize an integral of the noise prediction model, and DPM-Solver++ is designed to discretize an integral of the data prediction model. The algorithm and model type are decoupled. You can use either DPMSolver or DPMSolver++ for both noise prediction and data prediction models. Args: model_output (`torch.Tensor`): The direct output from the learned diffusion model. sample (`torch.Tensor`): A current instance of a sample created by the diffusion process. Returns: `torch.Tensor`: The converted model output. rtimestepNr z/missing `sample` as a required keyward argumentrjrJPassing `timesteps` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`rVr2)learned learned_rangervrl v_predictionzprediction_type given as z\ must be one of `epsilon`, `sample`, or `v_prediction` for the DPMSolverSinglestepScheduler.rI)rpoprr rwrBr>rHrdr}rr?rr`ra) rrrrlargskwargsrrr`rax0_predr2rrrconvert_model_outputsj                   z1DPMSolverSinglestepScheduler.convert_model_outputrlnoiserc Ost|dkr |dn|dd}t|dkr|dn|dd}|dur3t|dkr/|d}ntd|dur=tdd d |durGtdd d |j|jd|j|j}} ||\} }|| \} } t| t|} t| t| } | | }|j j d kr|| || t | d |}|S|j j dkr| | ||t |d |}|S|j j dkr|dusJ|| t | || dt d|||t d t d||}|S)az One step for the first-order DPMSolver (equivalent to DDIM). Args: model_output (`torch.Tensor`): The direct output from the learned diffusion model. timestep (`int`): The current discrete timestep in the diffusion chain. prev_timestep (`int`): The previous discrete timestep in the diffusion chain. sample (`torch.Tensor`): A current instance of a sample created by the diffusion process. Returns: `torch.Tensor`: The sample tensor at the previous timestep. rrNr prev_timestepr0 missing `sample` as a required keyward argumentrjrJrPassing `prev_timestep` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`r4r3rIrP) rrrr rdr}rr!rbrwrBrr_)rrrrlrrrrrrasigma_sr`alpha_srclambda_shx_trrrdpm_solver_first_order_update(sN     $ " z:DPMSolverSinglestepScheduler.dpm_solver_first_order_updatemodel_output_listc Ost|dkr |dn|dd}t|dkr|dn|dd}|dur3t|dkr/|d}ntd|dur=tddd |durGtddd |j|jd|j|j|j|jd}} } ||\} }|| \} } || \} } t| t|}t| t| }t| t| }|d |d }}||||}}||}|d |||}}|j j dkr |j j dkr|| || t | d |d| t | d |}|S|j j dkr || || t | d || t | d |d |}|S|j j dkrd|j j dkr;| | ||t |d |d|t |d |}|S|j j dkrb| | ||t |d ||t |d |d |}|S|j j dkr|dusrJ|j j dkr|| t | || dt d||d| dt d|||t d t d ||}|S|j j dkr|| t | || dt d||| d t d|d|d ||t d t d ||}|S)a One step for the second-order singlestep DPMSolver that computes the solution at time `prev_timestep` from the time `timestep_list[-2]`. Args: model_output_list (`List[torch.Tensor]`): The direct outputs from learned diffusion model at current and latter timesteps. timestep (`int`): The current and latter discrete timestep in the diffusion chain. prev_timestep (`int`): The previous discrete timestep in the diffusion chain. sample (`torch.Tensor`): A current instance of a sample created by the diffusion process. Returns: `torch.Tensor`: The sample tensor at the previous timestep. r timestep_listNr rrrrJPassing `timestep_list` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`rrWrr3r4r5rLrRrIrPr)rrrr rdr}rr!rbrwrBrCrr_)rrrrlrrrrrrasigma_s0sigma_s1r`alpha_s0alpha_s1rc lambda_s0 lambda_s1m0m1rh_0r0D0D1rrrr)singlestep_dpm_solver_second_order_updatehs        ) #   $zFDPMSolverSinglestepScheduler.singlestep_dpm_solver_second_order_updatec!Ost|dkr |dn|dd}t|dkr|dn|dd}|dur3t|dkr/|d}ntd|dur=tddd |durGtddd |j|jd|j|j|j|jd|j|jdf\}}} } ||\} }||\} }|| \} } || \}} t| t|}t| t|}t| t| }t|t| }|d |d |d }}}||||||}}}||||}}|}d|||d|||}}||||||}d||||}|j j dkrj|j j dkr+|| || t | d|| t | d|d|} | S|j j dkrh|| || t | d|| t | d|d|| t | d||dd|} | S|j j dkr|j j dkr| |||t |d||t |d|d|} | S|j j dkr| |||t |d||t |d|d||t |d||dd|} | S)a One step for the third-order singlestep DPMSolver that computes the solution at time `prev_timestep` from the time `timestep_list[-3]`. Args: model_output_list (`List[torch.Tensor]`): The direct outputs from learned diffusion model at current and latter timesteps. timestep (`int`): The current and latter discrete timestep in the diffusion chain. prev_timestep (`int`): The previous discrete timestep in the diffusion chain. sample (`torch.Tensor`): A current instance of a sample created by diffusion process. Returns: `torch.Tensor`: The sample tensor at the previous timestep. rrNr rr/ missing`sample` as a required keyward argumentrJrrrWrr3g@r4r5rRrLrI) rrrr rdr}rr!rbrwrBrCr)!rrrrlrrrrrarrsigma_s2r`rralpha_s2rcrr lambda_s2rrm2rrh_1rr1rD1_0D1_1rD2rrrr(singlestep_dpm_solver_third_order_updates      "  &   $zEDPMSolverSinglestepScheduler.singlestep_dpm_solver_third_order_updaterlryrryc Os t|dkr |dn|dd}t|dkr|dn|dd}|dur3t|dkr/|d}ntd|durFt|dkrB|d}ntd |durPtdd d |durZtdd d |dkrh|j|d ||dS|dkrt|j|||dS|dkr|j||dStd|)a One step for the singlestep DPMSolver. Args: model_output_list (`List[torch.Tensor]`): The direct outputs from learned diffusion model at current and latter timesteps. timestep (`int`): The current and latter discrete timestep in the diffusion chain. prev_timestep (`int`): The previous discrete timestep in the diffusion chain. sample (`torch.Tensor`): A current instance of a sample created by diffusion process. order (`int`): The solver order at this step. Returns: `torch.Tensor`: The sample tensor at the previous timestep. rrNr rrrrvz/ missing `order` as a required keyward argumentrJrrrWrrzOrder must be 1, 2, 3, got )rrrr rrr) rrrrlryrrrrrrrrsinglestep_dpm_solver_update?s:      z9DPMSolverSinglestepScheduler.singlestep_dpm_solver_updatecCsd|dur|j}||k}t|dkrt|jd}|St|dkr*|d}|S|d}|S)Nrr )rjnonzerorr)rrrschedule_timestepsindex_candidatesr}rrrindex_for_timesteps    z/DPMSolverSinglestepScheduler.index_for_timestepcCs@|jdurt|tjr||jj}|||_dS|j |_dS)zF Initialize the step_index counter for the scheduler. N) r isinstancer!Tensorrqrjrrrorp)rrrrrr_init_step_indexs   z-DPMSolverSinglestepScheduler._init_step_indexTr return_dictc Cs |jdur td|jdur|||j||d}t|jjdD] }|j|d|j|<q"||jd<|jj dkrFt |j ||j |j d}nd}|j|j}|j| durb|d8}|j| dusV|dkri||_|j|j|j||d} |jd7_|s| fSt| d S) a Predict the sample from the previous timestep by reversing the SDE. This function propagates the sample with the singlestep DPMSolver. Args: model_output (`torch.Tensor`): The direct output from learned diffusion model. timestep (`int`): The current discrete timestep in the diffusion chain. sample (`torch.Tensor`): A current instance of a sample created by the diffusion process. return_dict (`bool`): Whether or not to return a [`~schedulers.scheduling_utils.SchedulerOutput`] or `tuple`. Returns: [`~schedulers.scheduling_utils.SchedulerOutput`] or `tuple`: If return_dict is `True`, [`~schedulers.scheduling_utils.SchedulerOutput`] is returned, otherwise a tuple is returned where the first element is the sample tensor. NzaNumber of inference steps is 'None', you need to run 'set_timesteps' after creating the schedulerrr rWrP) generatorrrr) prev_sample)rfrr}rrrrwr=rkrBr rrrrnrlrror) rrrrrlrrr(rryrrrrsteps8        z!DPMSolverSinglestepScheduler.stepcOs|S)a? Ensures interchangeability with schedulers that need to scale the denoising model input depending on the current timestep. Args: sample (`torch.Tensor`): The input sample. Returns: `torch.Tensor`: A scaled input sample. r)rrrlrrrrrscale_model_inputs z.DPMSolverSinglestepScheduler.scale_model_inputoriginal_samplesc sjj|j|jd}|jjdkr)t|r)jj|jtjd|j|jtjd}n j|j||j}j durFfdd|D}nj durUj g|j d}n j g|j d}|| }t |j t |j kr}|d}t |j t |j ksn|\}}||||} | S)Nrmpsrcrr)r)r.rrrrrrr0rz:DPMSolverSinglestepScheduler.add_noise..rrW)rdrqrrtyper!is_floating_pointrjr#rr}rflattenrrr) rrrrrjrd step_indicesrr`ra noisy_samplesrrr add_noises$     z&DPMSolverSinglestepScheduler.add_noisecCs|jjSN)rwr8r|rrr__len__sz$DPMSolverSinglestepScheduler.__len__)r)NNNr )NT)-__name__ __module__ __qualname____doc__r _compatiblesryrrintstrrrgndarrayboolrtrrmpropertyr}rrrr!rrrrrrrrrrrrrrrrrr IntTensorr rrrrrr,Ms6;      L$      W" e D s l @   G "r,)rr)rtypingrrrrnumpyrgr!configuration_utilsrrutilsr r utils.torch_utilsr scheduling_utilsr rr get_loggerrrr+r,rrrrs   ,