o ߥi6@sddlZddlZddlmmZgdZddZGdddZdidddd difd d Z id d ddfddZ GdddZ ddZ ddZ dS)N)NoiseScheduleVP model_wrappermodel_wrapper_guided_diffusion DPM_Solverinterpolate_fncCs:|dfd|jd}|j}|||||S)z?Index tensor using t and format the output according to x. rr)sizendimdevicetoview)tensortxshaper rj/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/multi_modal/videocomposer/dpm_solver.py_isrc@sLeZdZdddddejfddZddZd d Zd d Zd dZ ddZ dS)rdiscreteNg?g4@cCsV|dvr td|||_|dkr[|dur$dtd|jdd}n |dus*Jdt|}t||_d |_t d d |jddd d j |d |_ | d j |d |_ dSd |_||_||_d|_d|_t|jd |jtjdd |jtj|j|_tt|jd |jtjd|_||_|dkrd|_dSd |_dS)aCreate a wrapper class for the forward SDE (VP type). *** Update: We support discrete-time diffusion models by implementing a picewise linear interpolation for log_alpha_t. We recommend to use schedule='discrete' for the discrete-time diffusion models, especially for high-resolution images. *** The forward SDE ensures that the condition distribution q_{t|0}(x_t | x_0) = N ( alpha_t * x_0, sigma_t^2 * I ). We further define lambda_t = log(alpha_t) - log(sigma_t), which is the half-logSNR (described in the DPM-Solver paper). Therefore, we implement the functions for computing alpha_t, sigma_t and lambda_t. For t in [0, T], we have: log_alpha_t = self.marginal_log_mean_coeff(t) sigma_t = self.marginal_std(t) lambda_t = self.marginal_lambda(t) Moreover, as lambda(t) is an invertible function, we also support its inverse function: t = self.inverse_lambda(lambda_t) =============================================================== We support both discrete-time DPMs (trained on n = 0, 1, ..., N-1) and continuous-time DPMs (trained on t in [t_0, T]). 1. For discrete-time DPMs: For discrete-time DPMs trained on n = 0, 1, ..., N-1, we convert the discrete steps to continuous time steps by: t_i = (i + 1) / N e.g. for N = 1000, we have t_0 = 1e-3 and T = t_{N-1} = 1. We solve the corresponding diffusion ODE from time T = 1 to time t_0 = 1e-3. Args: betas: A `torch.Tensor`. The beta array for the discrete-time DPM. (See the original DDPM paper for details) alphas_cumprod: A `torch.Tensor`. The cumprod alphas for the discrete-time DPM. (See the original DDPM paper for details) Note that we always have alphas_cumprod = cumprod(1 - betas). Therefore, we only need to set one of `betas` and `alphas_cumprod`. **Important**: Please pay special attention for the args for `alphas_cumprod`: The `alphas_cumprod` is the \hat{alpha_n} arrays in the notations of DDPM. Specifically, DDPMs assume that q_{t_n | 0}(x_{t_n} | x_0) = N ( \sqrt{\hat{alpha_n}} * x_0, (1 - \hat{alpha_n}) * I ). Therefore, the notation \hat{alpha_n} is different from the notation alpha_t in DPM-Solver. In fact, we have alpha_{t_n} = \sqrt{\hat{alpha_n}}, and log(alpha_{t_n}) = 0.5 * log(\hat{alpha_n}). 2. For continuous-time DPMs: We support two types of VPSDEs: linear (DDPM) and cosine (improved-DDPM). The hyperparameters for the noise schedule are the default settings in DDPM and improved-DDPM: Args: beta_min: A `float` number. The smallest beta for the linear schedule. beta_max: A `float` number. The largest beta for the linear schedule. cosine_s: A `float` number. The hyperparameter in the cosine schedule. cosine_beta_max: A `float` number. The hyperparameter in the cosine schedule. T: A `float` number. The ending time of the forward process. =============================================================== Args: schedule: A `str`. The noise schedule of the forward SDE. 'discrete' for discrete-time DPMs, 'linear' or 'cosine' for continuous-time DPMs. Returns: A wrapper object of the forward SDE (VP type). =============================================================== Example: # For discrete-time DPMs, given betas (the beta array for n = 0, 1, ..., N - 1): >>> ns = NoiseScheduleVP('discrete', betas=betas) # For discrete-time DPMs, given alphas_cumprod (the \hat{alpha_n} # array for n = 0, 1, ..., N - 1): >>> ns = NoiseScheduleVP('discrete', alphas_cumprod=alphas_cumprod) # For continuous-time DPMs (VPSDE), linear schedule: >>> ns = NoiseScheduleVP('linear', continuous_beta_0=0.1, continuous_beta_1=20.) )rlinearcosinezZUnsupported noise schedule {}. The schedule needs to be 'discrete' or 'linear' or 'cosine'rN?rrdim?g)r)dtypegMb?g8@@rgO@a?) ValueErrorformatscheduletorchlogcumsumlentotal_NTlinspacereshaper t_arraylog_alpha_arraybeta_0beta_1cosine_scosine_beta_maxmathatanpi cosine_t_maxcoscosine_log_alpha_0)selfr"betasalphas_cumprodcontinuous_beta_0continuous_beta_1r log_alphasrrr__init__sbc     zNoiseScheduleVP.__init__csjdkrt|dj|jj|jdSjdkr3d|djjd|jSjdkrGfd d }||j }|Sd S) zT Compute log(alpha_t) of a given continuous-time label t in [0, T]. rrrrrgпrrcs*tt|jdjtjdS)Nrr)r#r$r5r/r1r3)sr7rrs z9NoiseScheduleVP.marginal_log_mean_coeff..N) r"rr*r+r r r,r.r-r6)r7r log_alpha_fn log_alpha_trrArmarginal_log_mean_coeffs$      z'NoiseScheduleVP.marginal_log_mean_coeffcCst||S)zO Compute alpha_t of a given continuous-time label t in [0, T]. )r#exprEr7rrrrmarginal_alphaszNoiseScheduleVP.marginal_alphac Cstdtd||S)zO Compute sigma_t of a given continuous-time label t in [0, T]. rr)r#sqrtrFrErGrrr marginal_stdszNoiseScheduleVP.marginal_stdcCs.||}dtdtd|}||S)zn Compute lambda_t = log(alpha_t) - log(sigma_t) of a given continuous-time label t in [0, T]. rrr)rEr#r$rF)r7rlog_mean_coefflog_stdrrrmarginal_lambdas zNoiseScheduleVP.marginal_lambdacs jdkr2djjtd|td|}jd|}|t|jjjSjdkrjdttd|jd|}t | dt j |jd gt j |jd g}| d Sdtd|td|}fd d }||}|S) z` Compute the continuous-time label t in [0, T] of a given half-logSNR lambda_t. rrgrr?rgr>rrcs0tt|jddjtjjS)Nrr)r#arccosrFr6r/r1r3)rDrArrrBsz0NoiseScheduleVP.inverse_lambda..)r"r.r-r# logaddexpzerosr rIr rr*flipr,r+)r7lambtmpDelta log_alphart_fnrrArinverse_lambdas2      zNoiseScheduleVP.inverse_lambda) __name__ __module__ __qualname__r#float32r=rErHrJrMrXrrrrrs   rnoiseuncondrc sj fddd  fdd fdd f dd } d vs-Jd vs3J| S) a]Create a wrapper function for the noise prediction model. DPM-Solver needs to solve the continuous-time diffusion ODEs. For DPMs trained on discrete-time labels, we need to firstly wrap the model function to a noise prediction model that accepts the continuous time as the input. We support four types of the diffusion model by setting `model_type`: 1. "noise": noise prediction model. (Trained by predicting noise). 2. "x_start": data prediction model. (Trained by predicting the data x_0 at time 0). 3. "v": velocity prediction model. (Trained by predicting the velocity). The "v" prediction is derivation detailed in Appendix D of [1], and is used in Imagen-Video [2]. [1] Salimans, Tim, and Jonathan Ho. "Progressive distillation for fast sampling of diffusion models." arXiv preprint arXiv:2202.00512 (2022). [2] Ho, Jonathan, et al. "Imagen Video: High Definition Video Generation with Diffusion Models." arXiv preprint arXiv:2210.02303 (2022). 4. "score": marginal score function. (Trained by denoising score matching). Note that the score function and the noise prediction model follows a simple relationship: ``` noise(x_t, t) = -sigma_t * score(x_t, t) ``` We support three types of guided sampling by DPMs by setting `guidance_type`: 1. "uncond": unconditional sampling by DPMs. The input `model` has the following format: `` model(x, t_input, **model_kwargs) -> noise | x_start | v | score `` 2. "classifier": classifier guidance sampling [3] by DPMs and another classifier. The input `model` has the following format: `` model(x, t_input, **model_kwargs) -> noise | x_start | v | score `` The input `classifier_fn` has the following format: `` classifier_fn(x, t_input, cond, **classifier_kwargs) -> logits(x, t_input, cond) `` [3] P. Dhariwal and A. Q. Nichol, "Diffusion models beat GANs on image synthesis," in Advances in Neural Information Processing Systems, vol. 34, 2021, pp. 8780-8794. 3. "classifier-free": classifier-free guidance sampling by conditional DPMs. The input `model` has the following format: `` model(x, t_input, cond, **model_kwargs) -> noise | x_start | v | score `` And if cond == `unconditional_condition`, the model output is the unconditional DPM output. [4] Ho, Jonathan, and Tim Salimans. "Classifier-free diffusion guidance." arXiv preprint arXiv:2207.12598 (2022). The `t_input` is the time label of the model, which may be discrete-time labels (i.e. 0 to 999) or continuous-time labels (i.e. epsilon to T). We wrap the model function to accept only `x` and `t_continuous` as inputs, and outputs the predicted noise: `` def model_fn(x, t_continuous) -> noise: t_input = get_model_input_time(t_continuous) return noise_pred(model, x, t_input, **model_kwargs) `` where `t_continuous` is the continuous time labels (i.e. epsilon to T). And we use `model_fn` for DPM-Solver. =============================================================== Args: model: A diffusion model with the corresponding format described above. noise_schedule: A noise schedule object, such as NoiseScheduleVP. model_type: A `str`. The parameterization type of the diffusion model. "noise" or "x_start" or "v" or "score". model_kwargs: A `dict`. A dict for the other inputs of the model function. guidance_type: A `str`. The type of the guidance for sampling. "uncond" or "classifier" or "classifier-free". condition: A pytorch tensor. The condition for the guided sampling. Only used for "classifier" or "classifier-free" guidance type. unconditional_condition: A pytorch tensor. The condition for the unconditional sampling. Only used for "classifier-free" guidance type. guidance_scale: A `float`. The scale for the guided sampling. classifier_fn: A classifier function. Only used for the classifier guidance. classifier_kwargs: A `dict`. A dict for the other inputs of the classifier function. Returns: A noise prediction model that accepts the noised data and the continuous time as the inputs. c jdkr|djdS|Sa Convert the continuous-time `t_continuous` (in [epsilon, T]) to the model input time. For discrete-time DPMs, we convert `t_continuous` in [1 / N, 1] to `t_input` in [0, 1000 * (N - 1) / N]. For continuous-time DPMs, we just use `t_continuous`. rr@@r"r' t_continuousnoise_schedulerrget_model_input_timeQ z+model_wrapper..get_model_input_timeNcs|}|dur||fi}n |||fi}dkr"|Sdkr9||}}||||SdkrP||}}||||Sdkr^|}| |SdS)Nr]x_startvscore)rHrJ)rrdcondt_inputoutputalpha_tsigma_t)rgmodel model_kwargs model_typerfrr noise_pred_fn\s0  z$model_wrapper..noise_pred_fncsdt$|d}||fi}tj||dWdS1s+wYdS)z] Compute the gradient of the classifier, i.e. nabla_{x} log p_t(cond | x_t). TrN)r# enable_graddetachrequires_grad_autogradgradsum)rrmx_inlog_prob) classifier_fnclassifier_kwargs conditionrr cond_grad_fnps  $z#model_wrapper..cond_grad_fnc sdkr ||Sdkr.dusJ|}||}|}||}|||Sdkrldks:durA||dSt|gd}t|gd}tg}|||dd\} }| || SdS)S The noise predicition model function that is used for DPM-Solver. r^ classifierNclassifier-freer)rlr?)rJr#catchunk) rrdrm cond_gradrpr]r{t_inc_in noise_uncond) r}rrrgguidance_scale guidance_typertrfunconditional_conditionrrmodel_fnzs,     zmodel_wrapper..model_fn)r]rirj)r^rrNr) rqrfrsrrrrrrr}r~rr) r}r~rrrgrrrqrrrsrtrfrrrs j   rFrc sH fddfdd fdd fdd} | S) a Create a wrapper function for the noise prediction model guided diffusion. DPM-Solver needs to solve the continuous-time diffusion ODEs. For DPMs trained on discrete-time labels, we need to firstly wrap the model function to a noise prediction model that accepts the continuous time as the input. We support three types of the diffusion model by setting `mean_type`: 1. "x_{t-1}": previous data prediction model. (Trained by predicting the data x_{t-1} at time t-1). 2. "x0": data prediction model. (Trained by predicting the data x0 at time 0). 3. "eps": noise prediction model. (Trained by predicting the noise). We support three types of guided sampling by DPMs: 1. unconditional sampling by DPMs. The input `model` has the following format: `` model(xt, t_input, **model_kwargs) -> x_{t-1} | x0 | eps `` 2. classifier guidance sampling [3] by DPMs and another classifier. The input `model` has the following format: `` model(xt, t_input, **model_kwargs) -> x_{t-1} | x0 | eps `` The input `condition_fn` has the following format: `` condition_fn(xt, t_input, **model_kwargs) -> logits(xt, t_input) `` [3] P. Dhariwal and A. Q. Nichol, "Diffusion models beat GANs on image synthesis," in Advances in Neural Information Processing Systems, vol. 34, 2021, pp. 8780-8794. 3. classifier-free guidance sampling by conditional DPMs. The input `model` has the following format: `` model(xt, t_input, **model_kwargs) -> x_{t-1} | x0 | eps `` [4] Ho, Jonathan, and Tim Salimans. "Classifier-free diffusion guidance." arXiv preprint arXiv:2207.12598 (2022). The `t_input` is the time label of the model, which may be discrete-time labels or continuous-time labels (i.e. epsilon to T). We wrap the model function to accept only `x` and `t_continuous` as inputs, and outputs the predicted noise: `` def model_fn(x, t_continuous) -> noise: t_input = get_model_input_time(t_continuous) return noise_pred_fn(model, x, t_input, **model_kwargs) `` where `t_continuous` is the continuous time labels (i.e. epsilon to T). And we use `model_fn` for DPM-Solver. =============================================================== Args: model: A diffusion model with the corresponding format described above. noise_schedule: A noise schedule object, such as NoiseScheduleVP. var_type: A `str`. The variance type of the diffusion model. "learned" or "learned_range" or "fixed_large" or "fixed_small". mean_type: A `str`. The prediction type of the diffusion model. "x_{t-1}" or "x0" or "eps". model_kwargs: A `dict`. A dict for the other inputs of the model function. clamp: A `float`. The range to clamp the data. rescale_timesteps: A `bool`. Whether to rescale the timesteps. num_timesteps: An `int`. The number of the total diffusion steps. guide_scale: A `float`. The strength of the classifier-free guidance. condition_fn: A function. The function of the classifier guidance. Returns: A noise prediction model that accepts the noised data and the continuous time as the inputs. csr |dS|S)Nra)float)r) num_timestepsrescale_timestepsrr_scale_timestepssz8model_wrapper_guided_diffusion.._scale_timestepscr_r`rbrcrerrrgrhz.get_model_input_timecs|djddkr||jd}|}||fi|}dkr/|jddd\}}n dkr<|jddd\}}dkrC|}n'd krItd krj||} } | } |t| | |t| | }dur||} } |d|  ||fi|}|S) NrNrrlearnedr?r learned_rangeepszx_{t-1}rrx0) r*rexpandrNotImplementedErrorr" expand_dimsrHrJrrI)xtrdrrrmout_rmuposterior_mean_coef1posterior_mean_coef2dimsrrorp)r condition_fnrg mean_typerqrfvar_typerrrtsD   z5model_wrapper_guided_diffusion..noise_pred_fncs|djddkr||jd}dur!||fi}|Sttr,tdks.J||fid}||fid}drN|dn|dd}|ddd|f|ddd|f|ddd|f}|S)rrNrrNr?fixed)r*rr isinstancelistr& startswithr )rrdry_outu_outr) guide_scalerrrtrrrr&s"  &z0model_wrapper_guided_diffusion..model_fnr) rqrfrrrrrrrrrr) rrrgrrrqrrrtrfrrrrrs W  (rc@seZdZ    d6ddZddZdd Zd d Zd d ZddZddZ  d7ddZ    d8ddZ      d9ddZ d:ddZ d:dd Z    d;d!d"Z d:d#d$Z % & % ' ( dd2d3Z -   . / 0 1   & %d?d4d5ZdS)@r dpmsolver++NrcCs(||_||_||_||_||_||_dS)aConstruct a DPM-Solver. We support both the noise prediction model ("predicting epsilon") and the data prediction model ("predicting x0"). If `predict_x0` is False, we use the solver for the noise prediction model (DPM-Solver). If `predict_x0` is True, we use the solver for the data prediction model (DPM-Solver++). In such case, we further support the "dynamic thresholding" in [1] when `thresholding` is True. The "dynamic thresholding" can greatly improve the sample quality for pixel-space DPMs with large guidance scales. Args: model_fn: A noise prediction model function which accepts the continuous-time input (t in [epsilon, T]): `` def model_fn(x, t_continuous): return noise `` noise_schedule: A noise schedule object, such as NoiseScheduleVP. predict_x0: A `bool`. If true, use the data prediction model; else, use the noise prediction model. thresholding: A `bool`. Valid when `predict_x0` is True. Whether to use the "dynamic thresholding" in [1]. max_val: A `float`. Valid when both `predict_x0` and `thresholding` are True. The max value for thresholding. [1] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022b. N)rqrfalgorithm_type percentilethresholding_max_valclamp)r7rrfrrrrrrrr=@s $ zDPM_Solver.__init__cC |||S)z4 Return the noise prediction model. )rqr7rrrrrnoise_prediction_fnks zDPM_Solver.noise_prediction_fnc Cs|||}|}|j||j|}}|t|||t||}|jdur]tjt | |j ddf|jdd}tt ||j t||j|}t|| ||}|S|jduri|t t}|S)zG Return the data prediction model (with thresholding). Nrrrr)rrrfrHrJrrr#quantileabsr*rmaximumr ones_liker r r) r7rrr]rrorprr@rrrdata_prediction_fnqs<    zDPM_Solver.data_prediction_fncCs"|jdkr |||S|||S)z_ Convert the model to the noise prediction model or the data prediction model. r)rrrrrrrrs   zDPM_Solver.model_fnc Cs|dkr6|jt||}|jt||}t|||d|}|j|S|dkrFt|||d|S|dkred} t|d| |d| |d | |} | St d |)a7Compute the intermediate time steps for sampling. Args: skip_type: A `str`. The type for the spacing of the time steps. We support three types: - 'logSNR': uniform logSNR for the time steps. - 'time_uniform': uniform time for the time steps. (**Recommended for high-resolutional data**.) - 'time_quadratic': quadratic time for the time steps. (Used in DDIM for low-resolutional data.) t_T: A `float`. The starting time of the sampling (default is T). t_0: A `float`. The ending time of the sampling (default is epsilon). N: A `int`. The total number of the spacing of the time steps. device: A torch device. Returns: A pytorch tensor of the time steps, with the shape (N + 1,). logSNRr time_uniformtime_quadraticr?rzSUnsupported skip_type {}, need to be 'logSNR' or 'time_uniform' or 'time_quadratic') rfrMr#rr r)cpuitemrXpowr r!) r7 skip_typet_Tt_0Nr lambda_Tlambda_0 logSNR_stepst_orderrrrrget_time_stepss8   zDPM_Solver.get_time_stepsc Cs4|dkr8|dd}|ddkrdg|dddg}nQ|ddkr-dg|ddg}n@dg|ddg}n5|dkr]|ddkrL|d}dg|}n!|dd}dg|ddg}n|dkrid}dg|}ntd|dkr~||||||} | |fS||||||ttdg|d|} | |fS)a Get the order of each step for sampling by the singlestep DPM-Solver. We combine both DPM-Solver-1,2,3 to use all the function evaluations, which is named as "DPM-Solver-fast". Given a fixed number of function evaluations by `steps`, the sampling procedure by DPM-Solver-fast is: - If order == 1: We take `steps` of DPM-Solver-1 (i.e. DDIM). - If order == 2: - Denote K = (steps // 2). We take K or (K + 1) intermediate time steps for sampling. - If steps % 2 == 0, we use K steps of DPM-Solver-2. - If steps % 2 == 1, we use K steps of DPM-Solver-2 and 1 step of DPM-Solver-1. - If order == 3: - Denote K = (steps // 3 + 1). We take K intermediate time steps for sampling. - If steps % 3 == 0, we use (K - 2) steps of DPM-Solver-3, and 1 step of DPM-Solver-2 and 1 step of DPM-Solver-1. - If steps % 3 == 1, we use (K - 1) steps of DPM-Solver-3 and 1 step of DPM-Solver-1. - If steps % 3 == 2, we use (K - 1) steps of DPM-Solver-3 and 1 step of DPM-Solver-2. ============================================ Args: order: A `int`. The max order for the solver (2 or 3). steps: A `int`. The total number of function evaluations (NFE). skip_type: A `str`. The type for the spacing of the time steps. We support three types: - 'logSNR': uniform logSNR for the time steps. - 'time_uniform': uniform time for the time steps. (**Recommended for high-resolutional data**.) - 'time_quadratic': quadratic time for the time steps. (Used in DDIM for low-resolutional data.) t_T: A `float`. The starting time of the sampling (default is T). t_0: A `float`. The ending time of the sampling (default is epsilon). device: A torch device. Returns: orders: A list of the solver order of each step. rrr?z"'order' must be '1' or '2' or '3'.r)r rr#r%rr ) r7stepsorderrrrr Korderstimesteps_outerrrr.get_orders_and_timesteps_for_singlestep_solvers|#       z9DPM_Solver.get_orders_and_timesteps_for_singlestep_solvercCr)z Denoise at the final step, which is equivalent to solve the ODE from lambda_s to infty by first-order discretization. )r)r7rr@rrrdenoise_to_zero_fns zDPM_Solver.denoise_to_zero_fnFcCs|j}||||}}||} ||||} } ||||} } t| }|jdkrXt| }|durB|||}| | ||||}|rV|d|ifS|St| }|durg|||}t| | || ||}|r~|d|ifS|S)a DPM-Solver-1 (equivalent to DDIM) from time `s` to time `t`. Args: x: A pytorch tensor. The initial value at time `s`. s: A pytorch tensor. The starting time, with the shape (1,). t: A pytorch tensor. The ending time, with the shape (1,). model_s: A pytorch tensor. The model function evaluated at time `s`. If `model_s` is None, we evaluate the model by `x` and `s`; otherwise we directly use it. return_intermediate: A `bool`. If true, also return the model value at time `s`. Returns: x_t: A pytorch tensor. The approximated solution at time `t`. rNmodel_s) rfrMrErJr#rFrexpm1r)r7rr@rrreturn_intermediatenslambda_slambda_th log_alpha_srDsigma_srprophi_1x_trrrdpm_solver_first_updates8         z"DPM_Solver.dpm_solver_first_updater dpmsolvercCs|dvr td||durd}|j}||||} } | | } | || } || } |||| ||}}}|||| ||}}}t|t|}}|j dkrt | | }t | }|durx| ||}||||||}| || }|dkr||||||d|||||}n|dkr||||||d|||| d||}npt || }t | }|dur| ||}t||||||}| || }|dkrt||||||d|||||}n$|dkr8t||||||d|||| d||}|rB|||d fS|S) a Singlestep solver DPM-Solver-2 from time `s` to time `t`. Args: x: A pytorch tensor. The initial value at time `s`. s: A pytorch tensor. The starting time, with the shape (1,). t: A pytorch tensor. The ending time, with the shape (1,). r1: A `float`. The hyperparameter of the second-order solver. model_s: A pytorch tensor. The model function evaluated at time `s`. If `model_s` is None, we evaluate the model by `x` and `s`; otherwise we directly use it. return_intermediate: A `bool`. If true, also return the model value at time `s` and `s1` (the intermediate time). solver_type: either 'dpmsolver' or 'taylor'. The type for the high-order solvers. The type slightly impacts the performance. We recommend to use 'dpmsolver' type. Returns: x_t: A pytorch tensor. The approximated solution at time `t`. rtaylor<'solver_type' must be either 'dpmsolver' or 'taylor', got {}Nrrrrr)rmodel_s1 r r!rfrMrXrErJr#rFrrr)r7rr@rr1rr solver_typerrrr lambda_s1s1r log_alpha_s1rDrsigma_s1rpalpha_s1rophi_11rx_s1rrrrr#singlestep_dpm_solver_second_update>s                     z.DPM_Solver.singlestep_dpm_solver_second_updateUUUUUU?UUUUUU?c +Csr| dvr td| |durd}|durd}|j} | || |} } | | } | || }| || }| |}| |}| || || || |f\}}}}| || || || |f\}}}}t|t|t|}}}|j dkrSt | | }t | | }t | }t | | || d} || d}!|!| d}"|dur| ||}|dur||||||}#| |#|}|||||||||| ||}$| |$|}%| d kr||||||d|||!|%|}&n| d krRd|||}'d||%|}(||'||(||})d |(|'||}*||||||||!|)||"|*}&nt || }t || }t | }t || || d} || d}!|!| d}"|dur| ||}|durt||||||}#| |#|}t|||||||||| ||}$| |$|}%| d krt||||||d|||!|%|}&nF| d kr,d|||}'d||%|}(||'||(||})d |(|'||}*t||||||||!|)||"|*}&|r7|&|||%d fS|&S) a Singlestep solver DPM-Solver-3 from time `s` to time `t`. Args: x: A pytorch tensor. The initial value at time `s`. s: A pytorch tensor. The starting time, with the shape (1,). t: A pytorch tensor. The ending time, with the shape (1,). r1: A `float`. The hyperparameter of the third-order solver. r2: A `float`. The hyperparameter of the third-order solver. model_s: A pytorch tensor. The model function evaluated at time `s`. If `model_s` is None, we evaluate the model by `x` and `s`; otherwise we directly use it. model_s1: A pytorch tensor. The model function evaluated at time `s1` (the intermediate time given by `r1`). If `model_s1` is None, we evaluate the model at `s1`; otherwise we directly use it. return_intermediate: A `bool`. If true, also return the model value at time `s`, `s1` and `s2` (the intermediate times). solver_type: either 'dpmsolver' or 'taylor'. The type for the high-order solvers. The type slightly impacts the performance. We recommend to use 'dpmsolver' type. Returns: x_t: A pytorch tensor. The approximated solution at time `t`. rrNrrrrrrrr)rrmodel_s2r)+r7rr@rrr2rrrrrrrrr lambda_s2rs2rr log_alpha_s2rDrrsigma_s2rpralpha_s2rorphi_12rphi_22phi_2phi_3rx_s2rrD1_0D1_1D1D2rrr"singlestep_dpm_solver_third_updates                                          z-DPM_Solver.singlestep_dpm_solver_third_updatecCs|dvr td||j}|d|d}}|d|d} } || || ||} } } || ||}}|| ||}}t|}| | }| | }||}d|||}|jdkrt | }|dkr||||||d|||}|S|d kr|||||||||d|}|St |}|dkrt||||||d|||}|S|d krt|||||||||d|}|S) a Multistep solver DPM-Solver-2 from time `t_prev_list[-1]` to time `t`. Args: x: A pytorch tensor. The initial value at time `s`. model_prev_list: A list of pytorch tensor. The previous computed model values. t_prev_list: A list of pytorch tensor. The previous times, each time has the shape (1,) t: A pytorch tensor. The ending time, with the shape (1,). solver_type: either 'dpmsolver' or 'taylor'. The type for the high-order solvers. The type slightly impacts the performance. We recommend to use 'dpmsolver' type. Returns: x_t: A pytorch tensor. The approximated solution at time `t`. rrrrrrrr) r r!rfrMrErJr#rFrr)r7rmodel_prev_list t_prev_listrrr model_prev_1 model_prev_0t_prev_1t_prev_0 lambda_prev_1 lambda_prev_0rlog_alpha_prev_0rD sigma_prev_0rproh_0rr0rrrrrr"multistep_dpm_solver_second_updates~           z-DPM_Solver.multistep_dpm_solver_second_updatec#Cs|j}|\}}} |\} } } || || || ||f\} }}}|| ||}}|| ||}}t|}|| }||}||}||||}}d|| |}d|||}||||||}d||||}|jdkrt| }||d} | |d}!|||||| || |||!|}"|"St|}||d} | |d}!t|||||| || |||!|}"|"S)a Multistep solver DPM-Solver-3 from time `t_prev_list[-1]` to time `t`. Args: x: A pytorch tensor. The initial value at time `s`. model_prev_list: A list of pytorch tensor. The previous computed model values. t_prev_list: A list of pytorch tensor. The previous times, each time has the shape (1,) t: A pytorch tensor. The ending time, with the shape (1,). solver_type: either 'dpmsolver' or 'taylor'. The type for the high-order solvers. The type slightly impacts the performance. We recommend to use 'dpmsolver' type. Returns: x_t: A pytorch tensor. The approximated solution at time `t`. rrr)rfrMrErJr#rFrr)#r7rrrrrr model_prev_2rrt_prev_2rr lambda_prev_2rrrrrDrrproh_1r rr rrrrrrrrrrrr!multistep_dpm_solver_third_updateJsb                  z,DPM_Solver.multistep_dpm_solver_third_updatec Csf|dkr |j||||dS|dkr|j||||||dS|dkr,|j|||||||dStd|)a Singlestep DPM-Solver with the order `order` from time `s` to time `t`. Args: x: A pytorch tensor. The initial value at time `s`. s: A pytorch tensor. The starting time, with the shape (1,). t: A pytorch tensor. The ending time, with the shape (1,). order: A `int`. The order of DPM-Solver. We only support order == 1 or 2 or 3. return_intermediate: A `bool`. If true, also return the model value at time `s`, `s1` and `s2` (the intermediate times). solver_type: either 'dpmsolver' or 'taylor'. The type for the high-order solvers. The type slightly impacts the performance. We recommend to use 'dpmsolver' type. r1: A `float`. The hyperparameter of the second-order or third-order solver. r2: A `float`. The hyperparameter of the third-order solver. Returns: x_t: A pytorch tensor. The approximated solution at time `t`. rrr?)rrrr)rrrr(Solver order must be 1 or 2 or 3, got {})rrrr r!) r7rr@rrrrrrrrrsinglestep_dpm_solver_updates4 z'DPM_Solver.singlestep_dpm_solver_updatecCsh|dkr|j||d||ddS|dkr|j|||||dS|dkr-|j|||||dStd|)a0 Multistep DPM-Solver with the order `order` from time `t_prev_list[-1]` to time `t`. Args: x: A pytorch tensor. The initial value at time `s`. model_prev_list: A list of pytorch tensor. The previous computed model values. t_prev_list: A list of pytorch tensor. The previous times, each time has the shape (1,) t: A pytorch tensor. The ending time, with the shape (1,). order: A `int`. The order of DPM-Solver. We only support order == 1 or 2 or 3. solver_type: either 'dpmsolver' or 'taylor'. The type for the high-order solvers. The type slightly impacts the performance. We recommend to use 'dpmsolver' type. Returns: x_t: A pytorch tensor. The approximated solution at time `t`. rr)rr?rrr)rr rr r!)r7rrrrrrrrrmultistep_dpm_solver_updates  z&DPM_Solver.multistep_dpm_solver_update皙?q??h㈵>c  sj} |td|} | | } | |t| |}|t| |}|}d}|dkrBdfdd}fdd}n!|dkr\d \fd d}fd d}ntd |t| | | kr| | |}||| |\}}||| |fi|}t t||||t t|t|}d d}|||| }t |dkr|}|} |}| | } t ||t|d||| }||7}t| | | ksntd||S)uZ The adaptive step size solver based on singlestep DPM-Solver. Args: x: A pytorch tensor. The initial value at time `t_T`. order: A `int`. The (higher) order of the solver. We only support order == 2 or 3. t_T: A `float`. The starting time of the sampling (default is T). t_0: A `float`. The ending time of the sampling (default is epsilon). h_init: A `float`. The initial step size (for logSNR). atol: A `float`. The absolute tolerance of the solver. For image data, the default setting is 0.0078, followed [1]. rtol: A `float`. The relative tolerance of the solver. The default setting is 0.05. theta: A `float`. The safety hyperparameter for adapting the step size. The default setting is 0.9, followed [1]. t_err: A `float`. The tolerance for the time. We solve the diffusion ODE until the absolute error between the current time and `t_0` is less than `t_err`. The default setting is 1e-5. solver_type: either 'dpmsolver' or 'taylor'. The type for the high-order solvers. The type slightly impacts the performance. We recommend to use 'dpmsolver' type. Returns: x_0: A pytorch tensor. The approximated solution at time `t_0`. [1] A. Jolicoeur-Martineau, K. Li, R. Piché-Taillefer, T. Kachman, and I. Mitliagkas, "Gotta go fast when generating data with score-based models," arXiv preprint arXiv:2105.14080, 2021. rrr?rcsj|||ddS)NTr)rrr@rrArrrBsz0DPM_Solver.dpm_solver_adaptive..csj|||fd|S)N)rrrrr@rkwargsrr7rrrrBsr)rrcsj|||ddS)NT)rrrrrrrrrB scsj|||fd|S)N)rrr)rrrrr7rrrrBsz;For adaptive step size solver, order must be 2 or 3, got {}cSs*tt||jddfjdddS)NrrT)rkeepdim)r#rIsquarer*rmean)rjrrrrB%s rgzadaptive solver nfe)rfr#onesr rMrr r!rr"rXmaxallmin float_powerrprint)r7rrrrh_initatolrtolthetat_errrrr@rrrx_prevnfe lower_update higher_updaterx_lowerlower_noise_kwargsx_higherdeltanorm_fnErrrdpm_solver_adaptivesT&     zDPM_Solver.dpm_solver_adaptivecCs|j||j|}}|dur!tj|jdg|jR|jd}|dg|jR}t|| |t|| |}|jddkrI| dS|S)a# Compute the noised input xt = alpha_t * x + sigma_t * noise. Args: x: A `torch.Tensor` with shape `(batch_size, *shape)`. t: A `torch.Tensor` with shape `(t_size,)`. Returns: xt with shape `(t_size, batch_size, *shape)`. Nrr rr) rfrHrJr#randnrr r*rrsqueeze)r7rrr]rorprrrr add_noise5s     zDPM_Solver.add_noiser?r multistepTcCsh|dur d|jjn|}|dur|jjn|}|dkr|dks"Jd|j||||||||| | | | | d S)z Inverse the sample `x` from time `t_start` to `t_end` by DPM-Solver. For discrete-time DPMs, we use `t_start=1/N`, where `N` is the total time steps during training. NrrzTime range needs to be greater than 0. For discrete-time DPMs, it needs to be in [1 / N, 1], where N is the length of betas array) rt_startt_endrrmethodlower_order_finaldenoise_to_zerorr*r+r)rfr'r(sample)r7rrr?r@rrrArBrCrr*r+rrrrrrinverseKs$zDPM_Solver.inversec CsT|dur d|jjn|} |dur|jjn|}| dkr|dks"Jd|j}tu|dkr=|j|||| | | | d}n?|dkr||ksGJ|j||| ||d}|jdd |ks\Jd}||}|g}| ||g}t d |D]}||}|j |||||| d }| || | ||qqt ||d D]L}||}|r|d krt ||d |}n|}|j |||||| d }t |d D]}||d ||<||d ||<q||d <||kr| |||d <qn|d vru|dkr|j||||| |d\}}n|dkr||}|g|}|j||| ||d}t|D]Z\}}||||d }}|j|||||d}|j|}|d |d}|d krKdn |d |d|}|dkr\dn |d|d|}|j||||| ||d}qntd|| rtd|| }|||}Wd|SWd|S1swY|S)a Compute the sample at time `t_end` by DPM-Solver, given the initial `x` at time `t_start`. ===================================================== We support the following algorithms for both noise prediction model and data prediction model: - 'singlestep': Singlestep DPM-Solver (i.e. "DPM-Solver-fast" in the paper), which combines different orders of singlestep DPM-Solver. We combine all the singlestep solvers with order <= `order` to use up all the function evaluations (steps). The total number of function evaluations (NFE) == `steps`. Given a fixed NFE == `steps`, the sampling procedure is: - If `order` == 1: - Denote K = steps. We use K steps of DPM-Solver-1 (i.e. DDIM). - If `order` == 2: - Denote K = (steps // 2) + (steps % 2). We take K intermediate time steps for sampling. - If steps % 2 == 0, we use K steps of singlestep DPM-Solver-2. - If steps % 2 == 1, we use (K - 1) steps of singlestep DPM-Solver-2 and 1 step of DPM-Solver-1. - If `order` == 3: - Denote K = (steps // 3 + 1). We take K intermediate time steps for sampling. - If steps % 3 == 0, we use (K - 2) steps of singlestep DPM-Solver-3, and 1 step of singlestep DPM-Solver-2 and 1 step of DPM-Solver-1. - If steps % 3 == 1, we use (K - 1) steps of singlestep DPM-Solver-3 and 1 step of DPM-Solver-1. - If steps % 3 == 2, we use (K - 1) steps of singlestep DPM-Solver-3 and 1 step of singlestep DPM-Solver-2. - 'multistep': Multistep DPM-Solver with the order of `order`. The total number of function evaluations (NFE) == `steps`. We initialize the first `order` values by lower order multistep solvers. Given a fixed NFE == `steps`, the sampling procedure is: Denote K = steps. - If `order` == 1: - We use K steps of DPM-Solver-1 (i.e. DDIM). - If `order` == 2: - We firstly use 1 step of DPM-Solver-1, then use (K - 1) step of multistep DPM-Solver-2. - If `order` == 3: - We firstly use 1 step of DPM-Solver-1, then 1 step of multistep DPM-Solver-2, then (K - 2) step of multistep DPM-Solver-3. - 'singlestep_fixed': Fixed order singlestep DPM-Solver (i.e. DPM-Solver-1 or singlestep DPM-Solver-2 or singlestep DPM-Solver-3). We use singlestep DPM-Solver-`order` for `order`=1 or 2 or 3, with total [`steps` // `order`] * `order` NFE. - 'adaptive': Adaptive step size DPM-Solver (i.e. "DPM-Solver-12" and "DPM-Solver-23" in the paper). We ignore `steps` and use adaptive step size DPM-Solver with a higher order of `order`. You can adjust the absolute tolerance `atol` and the relative tolerance `rtol` to balance the computatation costs (NFE) and the sample quality. - If `order` == 2, we use DPM-Solver-12 which combines DPM-Solver-1 and singlestep DPM-Solver-2. - If `order` == 3, we use DPM-Solver-23 which combines singlestep DPM-Solver-2 and singlestep DPM-Solver-3. ===================================================== Some advices for choosing the algorithm: - For **unconditional sampling** or **guided sampling with small guidance scale** by DPMs: Use singlestep DPM-Solver ("DPM-Solver-fast" in the paper) with `order = 3`. e.g. >>> dpm_solver = DPM_Solver(model_fn, noise_schedule, predict_x0=False) >>> x_sample = dpm_solver.sample(x, steps=steps, t_start=t_start, t_end=t_end, order=3, skip_type='time_uniform', method='singlestep') - For **guided sampling with large guidance scale** by DPMs: Use multistep DPM-Solver with `predict_x0 = True` and `order = 2`. e.g. >>> dpm_solver = DPM_Solver(model_fn, noise_schedule, predict_x0=True) >>> x_sample = dpm_solver.sample(x, steps=steps, t_start=t_start, t_end=t_end, order=2, skip_type='time_uniform', method='multistep') We support three types of `skip_type`: - 'logSNR': uniform logSNR for the time steps. **Recommended for low-resolutional images** - 'time_uniform': uniform time for the time steps. **Recommended for high-resolutional images**. - 'time_quadratic': quadratic time for the time steps. ===================================================== Args: x: A pytorch tensor. The initial value at time `t_start` e.g. if `t_start` == T, then `x` is a sample from the standard normal distribution. steps: A `int`. The total number of function evaluations (NFE). t_start: A `float`. The starting time of the sampling. If `T` is None, we use self.noise_schedule.T (default is 1.0). t_end: A `float`. The ending time of the sampling. If `t_end` is None, we use 1. / self.noise_schedule.total_N. e.g. if total_N == 1000, we have `t_end` == 1e-3. For discrete-time DPMs: - We recommend `t_end` == 1. / self.noise_schedule.total_N. For continuous-time DPMs: - We recommend `t_end` == 1e-3 when `steps` <= 15; and `t_end` == 1e-4 when `steps` > 15. order: A `int`. The order of DPM-Solver. skip_type: A `str`. The type for the spacing of the time steps. 'time_uniform' or 'logSNR' or 'time_quadratic'. method: A `str`. The method for sampling. 'singlestep' or 'multistep' or 'singlestep_fixed' or 'adaptive'. denoise_to_zero: A `bool`. Whether to denoise to time 0 at the final step. Default is `False`. If `denoise_to_zero` is `True`, the total NFE is (`steps` + 1). This trick is firstly proposed by DDPM (https://arxiv.org/abs/2006.11239) and score_sde (https://arxiv.org/abs/2011.13456). Such trick can improve the FID for diffusion models sampling by diffusion SDEs for low-resolutional images (such as CIFAR-10). However, we observed that such trick does not matter for high-resolutional images. As it needs an additional NFE, we do not recommend it for high-resolutional images. lower_order_final: A `bool`. Whether to use lower order solvers at the final steps. Only valid for `method=multistep` and `steps < 15`. We empirically find that this trick is a key to stabilizing the sampling by DPM-Solver with very few steps (especially for steps <= 10). So we recommend to set it to be `True`. solver_type: A `str`. The taylor expansion type for the solver. `dpmsolver` or `taylor`. We recommend `dpmsolver`. atol: A `float`. The absolute tolerance of the adaptive step size solver. Valid when `method` == 'adaptive'. rtol: A `float`. The relative tolerance of the adaptive step size solver. Valid when `method` == 'adaptive'. Returns: x_end: A pytorch tensor. The approximated solution at time `t_end`. NrrzTime range needs to be greater than 0. For discrete-time DPMs, it needs to be in [1 / N, 1], where N is the length of betas arrayadaptive)rrrr*r+rr>)rrrrr rr r) singlestepsinglestep_fixedrH)rrrrrr rIr?)rrrzGot wrong method {}r)rfr'r(r r#no_gradr8rrrrangerappendr&r enumeraterrMrr r!r#r r) r7rrr?r@rrrArBrCrr*r+rrr timestepssteprrr step_orderirrrr@timesteps_inner lambda_innerrrrrrrrDqs         e eezDPM_Solver.sample)rNrN)NF)rNFr)rrNNFr)r)FrNN)rrrrrrr) r=NNr?rr>TFrrrF) r=NNr?rr>TFrrr)rYrZr[r=rrrrrrrrrr rrrr8r<rErDrrrrr>s + $P  4 Y  @ = 8 ' _ (rc Cs|jd|jd}}tj|d|d|ddfgdd}tj|dd\}}tj|dd}|d} tt|dtj d|j dtt||tj |d|j d| } tt| | | d| d} tj |d| dd d} tj |d| dd d} tt|dtj d|j dtt||tj |d|j d| }|d |dd}tj |d|dd d}tj |d|ddd d}||| ||| | }|S)a A piecewise linear function y = f(x), using xp and yp as keypoints. We implement f(x) in a differentiable way (i.e. applicable for autograd). The function f(x) is well-defined for all x-axis. (For x beyond the bounds of xp, we use the outmost points of xp to define the linear function.) Args: x: PyTorch tensor with shape [N, C], where N is the batch size, C is the number of channels (we use C = 1 for DPM-Solver). xp: PyTorch tensor with shape [C, K], where K is the number of keypoints. yp: PyTorch tensor with shape [C, K]. Returns: The function values f(x), with shape [N, C]. rrr?rr9)rindexr)rr#r unsqueezerepeatsortargminwhereeqrr gatherr;r)rxpyprrall_x sorted_all_x x_indicesx_idxcand_start_idx start_idxend_idxstart_xend_x start_idx2y_positions_expandedstart_yend_ycandrrrr`sh           rcCs|dd|dS)z Expand the tensor `v` to the dim `dims`. Args: `v`: a PyTorch tensor with shape [N]. `dim`: a `int`. Returns: a PyTorch tensor with shape [N, 1, 1, ..., 1] and the total dimension is `dims`. ).rrr)rjrrrrrs r)r1r#torch.nn.functionalnn functionalF__all__rrrrrrrrrrrsFT 5 (* 7