o pi@sddlZddlmZmZmZmZmZddlZddl Z ddl m Z m Z ddl mZddlmZddlmZmZmZ dd d ZGd ddee ZdS)N)CallableListOptionalTupleUnion) ConfigMixinregister_to_config) deprecate) randn_tensor)KarrasDiffusionSchedulersSchedulerMixinSchedulerOutput+?cosinecCs|dkr dd}n|dkrdd}ntd|g}t|D]}||}|d|}|td|||||qtj|tjdS) a Create a beta schedule that discretizes the given alpha_t_bar function, which defines the cumulative product of (1-beta) over time from t = [0,1]. Contains a function alpha_bar that takes an argument t and transforms it to the cumulative product of (1-beta) up to that part of the diffusion process. Args: num_diffusion_timesteps (`int`): the number of betas to produce. max_beta (`float`): the maximum beta to use; use values lower than 1 to prevent singularities. alpha_transform_type (`str`, *optional*, default to `cosine`): the type of noise schedule for alpha_bar. Choose from `cosine` or `exp` Returns: betas (`np.ndarray`): the betas used by the scheduler to step the model outputs rcSs t|ddtjddS)NgMb?gT㥛 ?r)mathcospitrf/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/schedulers/scheduling_sasolver.py alpha_bar_fn8s z)betas_for_alpha_bar..alpha_bar_fnexpcSst|dS)Ng()rrrrrrr=sz"Unsupported alpha_transform_type: r dtype) ValueErrorrangeappendmintorchtensorfloat32)num_diffusion_timestepsmax_betaalpha_transform_typerbetasit1t2rrrbetas_for_alpha_bars    "r+c(@sleZdZdZddeDZdZeddddd d d d d d d dddd ed d ddfde dedede de e e jeefde de de de edededed e d!ed"e ed#ed$e e d%e d&e f&d'd(Zed)d*Zed+d,Zdfd-e fd.d/Zdgd0e d1e e ejffd2d3Zd4ejd5ejfd6d7Zd8d9Zd:d;ZdZd d?d@ejd4ejd5ejfdAdBZdCdDZ dEdFZ!dGdHZ"dIdJZ#d@ejd4ejdKejdLe dMejd5ejf dNdOZ$dPejdQejdRejdSejdLe dMejd5ejfdTdUZ%dhdVdWZ&dXdYZ' did@ejdZe d4ejd[ed5e e(e)ff d\d]Z*d4ejd5ejfd^d_Z+d`ejdKejdaej,d5ejfdbdcZ-dddeZ.d S)jSASolverScheduleru% `SASolverScheduler` is a fast dedicated high-order solver for diffusion SDEs. This model inherits from [`SchedulerMixin`] and [`ConfigMixin`]. Check the superclass documentation for the generic methods the library implements for all schedulers such as loading and saving. Args: num_train_timesteps (`int`, defaults to 1000): The number of diffusion steps to train the model. beta_start (`float`, defaults to 0.0001): The starting `beta` value of inference. beta_end (`float`, defaults to 0.02): The final `beta` value. beta_schedule (`str`, defaults to `"linear"`): The beta schedule, a mapping from a beta range to a sequence of betas for stepping the model. Choose from `linear`, `scaled_linear`, or `squaredcos_cap_v2`. trained_betas (`np.ndarray`, *optional*): Pass an array of betas directly to the constructor to bypass `beta_start` and `beta_end`. predictor_order (`int`, defaults to 2): The predictor order which can be `1` or `2` or `3` or '4'. It is recommended to use `predictor_order=2` for guided sampling, and `predictor_order=3` for unconditional sampling. corrector_order (`int`, defaults to 2): The corrector order which can be `1` or `2` or `3` or '4'. It is recommended to use `corrector_order=2` for guided sampling, and `corrector_order=3` for unconditional sampling. prediction_type (`str`, defaults to `epsilon`, *optional*): Prediction type of the scheduler function; can be `epsilon` (predicts the noise of the diffusion process), `sample` (directly predicts the noisy sample`) or `v_prediction` (see section 2.4 of [Imagen Video](https://imagen.research.google/video/paper.pdf) paper). tau_func (`Callable`, *optional*): Stochasticity during the sampling. Default in init is `lambda t: 1 if t >= 200 and t <= 800 else 0`. SA-Solver will sample from vanilla diffusion ODE if tau_func is set to `lambda t: 0`. SA-Solver will sample from vanilla diffusion SDE if tau_func is set to `lambda t: 1`. For more details, please check https://arxiv.org/abs/2309.05019 thresholding (`bool`, defaults to `False`): Whether to use the "dynamic thresholding" method. This is unsuitable for latent-space diffusion models such as Stable Diffusion. dynamic_thresholding_ratio (`float`, defaults to 0.995): The ratio for the dynamic thresholding method. Valid only when `thresholding=True`. sample_max_value (`float`, defaults to 1.0): The threshold value for dynamic thresholding. Valid only when `thresholding=True` and `algorithm_type="dpmsolver++"`. algorithm_type (`str`, defaults to `data_prediction`): Algorithm type for the solver; can be `data_prediction` or `noise_prediction`. It is recommended to use `data_prediction` with `solver_order=2` for guided sampling like in Stable Diffusion. lower_order_final (`bool`, defaults to `True`): Whether to use lower-order solvers in the final steps. Default = True. use_karras_sigmas (`bool`, *optional*, defaults to `False`): Whether to use Karras sigmas for step sizes in the noise schedule during the sampling process. If `True`, the sigmas are determined according to a sequence of noise levels {σi}. lambda_min_clipped (`float`, defaults to `-inf`): Clipping threshold for the minimum value of `lambda(t)` for numerical stability. This is critical for the cosine (`squaredcos_cap_v2`) noise schedule. variance_type (`str`, *optional*): Set to "learned" or "learned_range" for diffusion models that predict variance. If set, the model's output contains the predicted Gaussian variance. timestep_spacing (`str`, defaults to `"linspace"`): The way the timesteps should be scaled. Refer to Table 2 of the [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://huggingface.co/papers/2305.08891) for more information. steps_offset (`int`, defaults to 0): An offset added to the inference steps, as required by some model families. cCsg|]}|jqSr)name).0errr szSASolverScheduler.r ig-C6?g{Gz?linearNrepsilonFgףp= ??data_predictionTinflinspacernum_train_timesteps beta_startbeta_end beta_schedule trained_betaspredictor_ordercorrector_orderprediction_typetau_func thresholdingdynamic_thresholding_ratiosample_max_valuealgorithm_typelower_order_finaluse_karras_sigmaslambda_min_clipped variance_typetimestep_spacing steps_offsetcCs|durtj|tjd|_n:|dkrtj|||tjd|_n*|dkr4tj|d|d|tjdd|_n|dkr>t||_n t|d|jd|j|_tj |jd d |_ t |j |_ t d |j |_ t|j t|j |_d |j |j d|_d|_| d vrt| d|jd|_tjd |d |tjdddd }t||_dgt||d |_dgt||d |_| durdd|_n| |_| dk|_d |_d|_d|_d|_|j d|_dS)Nrr1 scaled_linear?rsquaredcos_cap_v2z is not implemented for r3rdimr )r4noise_predictioncSs|dkr |dkr dSdS)Ni r rrrrrrz,SASolverScheduler.__init__..r4cpu)!r!r"r#r'r6r+NotImplementedError __class__alphascumprodalphas_cumprodsqrtalpha_tsigma_tloglambda_tsigmasinit_noise_sigmanum_inference_stepsnpcopy from_numpy timestepsmax timestep_list model_outputsr? predict_x0lower_order_nums last_sample _step_index _begin_indexto)selfr7r8r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrIrerrr__init__sP  &   zSASolverScheduler.__init__cC|jS)zg The index counter for current timestep. It will increase 1 after each scheduler step. )rlrorrr step_indexzSASolverScheduler.step_indexcCrq)zq The index for the first timestep. It should be set from pipeline with `set_begin_index` method. rmrrrrr begin_indexrtzSASolverScheduler.begin_indexrvcCs ||_dS)z Sets the begin index for the scheduler. This function should be run from pipeline before the inference. Args: begin_index (`int`): The begin index for the scheduler. Nru)rorvrrrset_begin_indexs z!SASolverScheduler.set_begin_indexradevicec spttjdgjj}jj|}jj dkr;t d|d|d ddddd t j}nXjj dkri||d}t d|d| ddddd t j}|jj7}n*jj dkrjj|}t |d|  t j}|d8}n tjj dt djjd }jjrt |t | }j||d }t fd d |D }t ||ddgt j}n't |t dt||}djdjdd }t ||ggt j}t|_t|j|tjd _t|_ dgt!jj"jj#d_$d_%d_&d_'d_(jd_dS)a Sets the discrete timesteps used for the diffusion chain (to be run before inference). Args: num_inference_steps (`int`): The number of diffusion steps used when generating samples with a pre-trained model. device (`str` or `torch.device`, *optional*): The device to which the timesteps should be moved to. If `None`, the timesteps are not moved. rr6r NrPleadingtrailingzY is not supported. Please make sure to choose one of 'linspace', 'leading' or 'trailing'.rK) in_sigmasracsg|]}|qSr) _sigma_to_t)r.sigma log_sigmasrorrr0rSz3SASolverScheduler.set_timesteps..)rxrrT))r! searchsortedflipr^configrFr7numpyitemrHrbr6roundrcastypeint64arangerIrarrayrYrEr]_convert_to_karras concatenater#interplenrdr_rnrerarfr<r=rhrjrkrlrm) rorarx clipped_idx last_timestepre step_ratior_ sigma_lastrr~r set_timestepssL  6  6         zSASolverScheduler.set_timestepssamplereturncCs|j}|j^}}}|tjtjfvr|}|||t|}| }tj ||j j dd}tj |d|j jd}|d}t || ||}|j||g|R}||}|S)as "Dynamic thresholding: At each sampling step we set s to a certain percentile absolute pixel value in xt0 (the prediction of x_0 at timestep t), and if s > 1, then we threshold xt0 to the range [-s, s] and then divide by s. Dynamic thresholding pushes saturated pixels (those near -1 and 1) inwards, thereby actively preventing pixels from saturation at each step. We find that dynamic thresholding results in significantly better photorealism as well as better image-text alignment, especially when using very large guidance weights." https://arxiv.org/abs/2205.11487 r rM)r rf)rshaper!r#float64floatreshaperbprodabsquantilerrAclamprB unsqueezern)rorr batch_sizechannelsremaining_dims abs_samplesrrr_threshold_sample4s    z#SASolverScheduler._threshold_samplec Cstt|d}||ddtjf}tj|dkddjddj|jddd}|d}||}||}||||} t| dd} d| || |} | |j} | S)Ng|=r)axisr)rfr ) rbr]maximumnewaxiscumsumargmaxcliprr) ror}r log_sigmadistslow_idxhigh_idxlowhighwrrrrr|Vs, zSASolverScheduler._sigma_to_tcCs$d|ddd}||}||fS)Nr rrKr)ror}r[r\rrr_sigma_to_alpha_sigma_tnsz)SASolverScheduler._sigma_to_alpha_sigma_tr{c Cst|jdr |jj}nd}t|jdr|jj}nd}|dur |n|d}|dur,|n|d}d}tdd|}|d|}|d|}|||||} | S)z6Constructs the noise schedule of Karras et al. (2022). sigma_minN sigma_maxrPrg@r )hasattrrrrrrbr6) ror{rarrrhoramp min_inv_rho max_inv_rhor_rrrrus      z$SASolverScheduler._convert_to_karrasr model_outputc Ost|dkr |dn|dd}|dur#t|dkr|d}ntd|dur-tddd|j|j}||\}}|jjd vr|jj d kr_|jj d vrV|dddd f}||||} n"|jj d krh|} n|jj dkrw||||} n td|jj d|jj r| | } | S|jjdvr|jj d kr|jj d vr|dddd f} n+|} n(|jj d kr||||} n|jj dkr||||} n td|jj d|jj r|j ||j|}}||| |} | | } ||| |} | SdS)aO Convert the model output to the corresponding type the data_prediction/noise_prediction algorithm needs. Noise_prediction is designed to discretize an integral of the noise prediction model, and data_prediction is designed to discretize an integral of the data prediction model. The algorithm and model type are decoupled. You can use either data_prediction or noise_prediction for both noise prediction and data prediction models. Args: model_output (`torch.Tensor`): The direct output from the learned diffusion model. sample (`torch.Tensor`): A current instance of a sample created by the diffusion process. Returns: `torch.Tensor`: The converted model output. rtimestepNr z/missing `sample` as a required keyward argumentre1.0.0zPassing `timesteps` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`)r4r2)learned learned_ranger v_predictionzprediction_type given as zQ must be one of `epsilon`, `sample`, or `v_prediction` for the SASolverScheduler.)rO)rpoprr r_rsrrrCr>rGr@rr[r\) rorrargskwargsrr}r[r\x0_predr2rrrconvert_model_outputs^                z&SASolverScheduler.convert_model_outputcCs|dvsJd|dkrt| t||dS|dkr4t| |dt|||dS|dkrYt| |dd|dt|||dd|dS|dkrt| |dd|dd|dt|||dd|dd|dSdS) zd Calculate the integral of exp(-x) * x^order dx from interval_start to interval_end rr rr)order is only supported for 0, 1, 2 and 3rr rrNr!r)roorderinterval_start interval_endrrr%get_coefficients_exponential_negatives,     z7SASolverScheduler.get_coefficients_exponential_negativecCst|dvsJdd|d|}d|d|}|dkr1t|dt|| d|dS|dkrRt||d|dt|| d|ddS|dkrt||dd|d|dd|dt|| d|ddS|dkrt||dd|dd|d|dd|dd|dt|| d|ddSd S) zl Calculate the integral of exp(x(1+tau^2)) * x^order dx from interval_start to interval_end rrr rrrrNr)rorrrtauinterval_end_covinterval_start_covrrr%get_coefficients_exponential_positivesL( z7SASolverScheduler.get_coefficients_exponential_positivec Cs$|dvsJ|t|dksJ|dkrdggS|dkrJd|d|d|d |d|dgd|d|d|d |d|dggS|dkr|d|d|d|d}|d|d|d|d}|d|d|d|d}d||d |d||d|d|gd||d |d||d|d|gd||d |d||d|d|ggS|dkr|d|d|d|d|d|d}|d|d|d|d|d|d}|d|d|d|d|d|d}|d|d|d|d|d|d}d||d |d|d||d|d|d|d|d|d||d |d|d|gd||d |d|d||d|d|d|d|d|d||d |d|d|gd||d |d|d||d|d|d|d|d|d||d |d|d|gd||d |d|d||d|d|d|d|d|d||d |d|d|ggSdS)zB Calculate the coefficient of lagrange polynomial rr rrrN)r)ror lambda_list denominator1 denominator2 denominator3 denominator4rrrlagrange_polynomial_coefficient.s        z1SASolverScheduler.lagrange_polynomial_coefficientc Cs|dvsJ|t|ksJdg}||d|}t|D];}d} t|D]-} |jr@| ||| ||d| |||7} q&| ||| ||d| ||7} q&|| qt||ksdJd|S)N)r rrrz4the length of lambda list must be equal to the orderr rz3the length of coefficients does not match the order)rrrrirrr) rorrrrr coefficientslagrange_coefficientr( coefficientjrrrget_coefficients_fns"     z%SASolverScheduler.get_coefficients_fnnoiserrc Os t|dkr |dn|dd}|dur#t|dkr|d}ntd|dur6t|dkr2|d}ntd|durIt|dkrE|d}ntd |dur\t|d krX|d }ntd |durftdd d |j} |j|jd|j|j} } || \} } || \} } t | t | }t | t | }t |}||}g}t |D] }|j|}||j|\}}t |t |}| |q| |||||}|}|jrm|dkrm|j|jd}||\}}t |t |}|ddtd|d||dd|d|ddtd|d| d|dd||7<|ddtd|d||dd|d|ddtd|d| d|dd||8<t |D]>}|jr|d|d| t|d |||| |d 7}qq|d|d | ||| |d 7}qq|jr| tdtd|d||}n|| ttd|d|}|jrt|d || | |||}n | | |||}||j}|S)ag One step for the SA-Predictor. Args: model_output (`torch.Tensor`): The direct output from the learned diffusion model at the current timestep. prev_timestep (`int`): The previous discrete timestep in the diffusion chain. sample (`torch.Tensor`): A current instance of a sample created by the diffusion process. order (`int`): The order of SA-Predictor at this timestep. Returns: `torch.Tensor`: The sample tensor at the previous timestep. r prev_timestepNr z0 missing `sample` as a required keyward argumentrz/ missing `noise` as a required keyward argumentrz/ missing `order` as a required keyward argumentrz- missing `tau` as a required keyward argumentrzPassing `prev_timestep` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`r3rrrr rhr_rsrr!r] zeros_likerrrrirrZrnr) rorrrrrrrrmodel_output_listr\sigma_s0r[alpha_s0r^ lambda_s0 gradient_parthrr(sialpha_sisigma_si lambda_sigradient_coefficientsx temp_sigma temp_alpha_s temp_sigma_s temp_lambda_s noise_partx_trrr!stochastic_adams_bashforth_updates               FF   0*$* z3SASolverScheduler.stochastic_adams_bashforth_updatethis_model_outputrk last_noise this_samplec Ost|dkr |dn|dd} |dur#t|dkr|d}ntd|dur6t|dkr2|d}ntd|durIt|dkrE|d}ntd |dur\t|d krX|d }ntd |durot|d krk|d }ntd | durytddd|j} |j|j|j|jd} } || \} } || \}} t | t | }t |t | }t |}||}g}t |D] }|j|}||j|\}}t |t |}| |q| |g}| |||||}|}|jrd|dkrd|ddtd|d||d|d|ddtd|d| d|dd|7<|ddtd|d||d|d|ddtd|d| d|dd|8<t |D]>}|jr|d|d| t|d |||||d 7}qh|d|d | ||||d 7}qh|jr| tdtd|d||}n|| ttd|d|}|jrt|d || | |||}n | ||||}||j}|S)a One step for the SA-Corrector. Args: this_model_output (`torch.Tensor`): The model outputs at `x_t`. this_timestep (`int`): The current timestep `t`. last_sample (`torch.Tensor`): The generated sample before the last predictor `x_{t-1}`. this_sample (`torch.Tensor`): The generated sample after the last predictor `x_{t}`. order (`int`): The order of SA-Corrector at this step. Returns: `torch.Tensor`: The corrected sample tensor at the current timestep. r this_timestepNr z4 missing`last_sample` as a required keyward argumentrz3 missing`last_noise` as a required keyward argumentrz4 missing`this_sample` as a required keyward argumentrz. missing`order` as a required keyward argumentz, missing`tau` as a required keyward argumentrzPassing `this_timestep` is deprecated and has no effect as model output conversion is now handled via an internal counter `self.step_index`r3rr)rorrkrrrrrrrrr\rr[rr^rrrrr(rrrrmodel_prev_listrrrrrrrstochastic_adams_moulton_update.s                  FF   0*$* z1SASolverScheduler.stochastic_adams_moulton_updatecCsd|dur|j}||k}t|dkrt|jd}|St|dkr*|d}|S|d}|S)Nrr )renonzerorr)rorschedule_timestepsindex_candidatesrsrrrindex_for_timesteps    z$SASolverScheduler.index_for_timestepcCs@|jdurt|tjr||jj}|||_dS|j |_dS)zF Initialize the step_index counter for the scheduler. N) rv isinstancer!Tensorrnrerxrrlrm)rorrrr_init_step_indexs   z"SASolverScheduler._init_step_indexr return_dictcCs|jdur td|jdur|||jdko|jdu}|j||d}|r<||jd}|j||j|j ||j |d}t t |j j|j jddD]} |j| d|j| <|j| d|j| <qK||jd<||jd<t|j||j|jd} |j jrt|j jt|j|j} t|j jt|j|jd} n|j j} |j j} t| |jd|_t| |jd |_ |jdksJ|j dksJ||_| |_ ||jd}|j||| |j|d } |jt |j j|j jdkr|jd7_|jd7_|s| fSt| d S) a Predict the sample from the previous timestep by reversing the SDE. This function propagates the sample with the SA-Solver. Args: model_output (`torch.Tensor`): The direct output from learned diffusion model. timestep (`int`): The current discrete timestep in the diffusion chain. sample (`torch.Tensor`): A current instance of a sample created by the diffusion process. generator (`torch.Generator`, *optional*): A random number generator. return_dict (`bool`): Whether or not to return a [`~schedulers.scheduling_utils.SchedulerOutput`] or `tuple`. Returns: [`~schedulers.scheduling_utils.SchedulerOutput`] or `tuple`: If return_dict is `True`, [`~schedulers.scheduling_utils.SchedulerOutput`] is returned, otherwise a tuple is returned where the first element is the sample tensor. NzaNumber of inference steps is 'None', you need to run 'set_timesteps' after creating the schedulerrrrP)rrkrrrrr ) generatorrxrr)rrrrr) prev_sample)rarrsrrkrr?rgrrthis_corrector_orderrrfrr<r=rhr rrxrrDr rrerjthis_predictor_orderrrlr)rorrrrr use_correctormodel_output_convert current_taur(rrrrrrrstepsl   "     zSASolverScheduler.stepcOs|S)a? Ensures interchangeability with schedulers that need to scale the denoising model input depending on the current timestep. Args: sample (`torch.Tensor`): The input sample. Returns: `torch.Tensor`: A scaled input sample. r)rorrrrrrscale_model_input:s z#SASolverScheduler.scale_model_inputoriginal_samplesrecCs|jj|jd|_|jj|jd}||j}||d}|}t|jt|jkr:|d}t|jt|jks+d||d}|}t|jt|jkr_|d}t|jt|jksP||||}|S)N)rxrrKrPr )rYrnrxrflattenrrr)rorrrerYsqrt_alpha_prodsqrt_one_minus_alpha_prod noisy_samplesrrr add_noiseJs     zSASolverScheduler.add_noisecCs|jjSN)rr7rrrrr__len__dszSASolverScheduler.__len__)r)NNr)NT)/__name__ __module__ __qualname____doc__r _compatiblesrr rintstrrrrbndarrayrrboolrppropertyrsrvrwr!rxrrrr|rrrrrrrrrrrrrr r  IntTensorrrrrrrr,Ks>      K   B" \,m     f r,)rr)rtypingrrrrrrrbr!configuration_utilsrr utilsr utils.torch_utilsr scheduling_utilsr rrr+r,rrrrs   ,