o Gir @sddlZddlZddlmZddlmZddlZddlZddl m Z m Z ddl m Z mZddlmZmZerAddlZdd lmZeGd d d e Z ddedededdejfddZGdddee ZdS)N) dataclass)Literal) ConfigMixinregister_to_config) BaseOutputis_scipy_available)KarrasDiffusionSchedulersSchedulerMixin) integratec@s.eZdZUdZejed<dZejdBed<dS)LMSDiscreteSchedulerOutputaq Output class for the scheduler's `step` function output. Args: prev_sample (`torch.Tensor` of shape `(batch_size, num_channels, height, width)` for images): Computed sample `(x_{t-1})` of previous timestep. `prev_sample` should be used as next model input in the denoising loop. pred_original_sample (`torch.Tensor` of shape `(batch_size, num_channels, height, width)` for images): The predicted denoised sample `(x_{0})` based on the model output from the current timestep. `pred_original_sample` can be used to preview progress or for guidance. prev_sampleNpred_original_sample)__name__ __module__ __qualname____doc__torchTensor__annotations__rrr`/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/schedulers/scheduling_lms_discrete.pyr s  r +?cosinenum_diffusion_timestepsmax_betaalpha_transform_type)rexplaplacereturncCs|dkr dd}n|dkrdd}n|dkrdd}ntd|g}t|D]}||}|d |}|td |||||q(tj|tjd S) a> Create a beta schedule that discretizes the given alpha_t_bar function, which defines the cumulative product of (1-beta) over time from t = [0,1]. Contains a function alpha_bar that takes an argument t and transforms it to the cumulative product of (1-beta) up to that part of the diffusion process. Args: num_diffusion_timesteps (`int`): The number of betas to produce. max_beta (`float`, defaults to `0.999`): The maximum beta to use; use values lower than 1 to avoid numerical instability. alpha_transform_type (`str`, defaults to `"cosine"`): The type of noise schedule for `alpha_bar`. Choose from `cosine`, `exp`, or `laplace`. Returns: `torch.Tensor`: The betas used by the scheduler to step the model outputs. rcSs t|ddtjddS)NgMb?gT㥛 ?r)mathcospitrrr alpha_bar_fnNs z)betas_for_alpha_bar..alpha_bar_fnrc SsPdtdd|tddtd|d}t|}t|d|S)Ngr ?rgư>)r!copysignlogfabsrsqrt)r%lmbsnrrrrr&Ss4 rcSst|dS)Ng()r!rr$rrrr&Zsz"Unsupported alpha_transform_type: r dtype) ValueErrorrangeappendminrtensorfloat32)rrrr&betasit1t2rrrbetas_for_alpha_bar4s     "r:c@sDeZdZdZddeDZdZe     dTdede de de de j e e Bd BdededededdeddefddZede ejBfddZedefd d!Zedefd"d#ZdUd$edd fd%d&Zd'ejd(e ejBdejfd)d*Zd+ed,ed-ede fd.d/ZdVd0ed1e ejBfd2d3Z dVd(e ejBd4ejd Bdefd5d6Zd(e ejBdd fd7d8Zd9e j d:e j de j fd;d<Zd=ejdejfd>d?Zd=ejd0edejfd@dAZ BdWd=ejd0edCe dDe dejf dEdFZ! G HdXdIejd(e ejBd'ejd+edJede"e#Bf dKdLZ$dMejdNejdOejdejfdPdQZ%defdRdSZ&d S)YLMSDiscreteScheduleru A linear multistep scheduler for discrete beta schedules. This model inherits from [`SchedulerMixin`] and [`ConfigMixin`]. Check the superclass documentation for the generic methods the library implements for all schedulers such as loading and saving. Args: num_train_timesteps (`int`, defaults to `1000`): The number of diffusion steps to train the model. beta_start (`float`, defaults to `0.0001`): The starting `beta` value of inference. beta_end (`float`, defaults to `0.02`): The final `beta` value. beta_schedule (`"linear"`, `"scaled_linear"`, or `"squaredcos_cap_v2"`, defaults to `"linear"`): The beta schedule, a mapping from a beta range to a sequence of betas for stepping the model. trained_betas (`np.ndarray`, *optional*): Pass an array of betas directly to the constructor to bypass `beta_start` and `beta_end`. use_karras_sigmas (`bool`, *optional*, defaults to `False`): Whether to use Karras sigmas for step sizes in the noise schedule during the sampling process. If `True`, the sigmas are determined according to a sequence of noise levels {σi}. use_exponential_sigmas (`bool`, *optional*, defaults to `False`): Whether to use exponential sigmas for step sizes in the noise schedule during the sampling process. use_beta_sigmas (`bool`, *optional*, defaults to `False`): Whether to use beta sigmas for step sizes in the noise schedule during the sampling process. Refer to [Beta Sampling is All You Need](https://huggingface.co/papers/2407.12173) for more information. prediction_type (`"epsilon"`, `"sample"`, or `"v_prediction"`, defaults to `"epsilon"`): Prediction type of the scheduler function; can be `epsilon` (predicts the noise of the diffusion process), `sample` (directly predicts the noisy sample`) or `v_prediction` (see section 2.4 of [Imagen Video](https://huggingface.co/papers/2210.02303) paper). timestep_spacing (`"linspace"`, `"leading"`, or `"trailing"`, defaults to `"linspace"`): The way the timesteps should be scaled. Refer to Table 2 of the [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://huggingface.co/papers/2305.08891) for more information. steps_offset (`int`, defaults to `0`): An offset added to the inference steps, as required by some model families. cCsg|]}|jqSr)name).0errr szLMSDiscreteScheduler.r -C6?{Gz?linearNFepsilonlinspacernum_train_timesteps beta_startbeta_end beta_schedule trained_betasuse_karras_sigmasuse_exponential_sigmasuse_beta_sigmasprediction_type)rDsample v_predictiontimestep_spacing)rEleadingtrailing steps_offsetc Cs^t|jj|jj|jjgdkrtd|dur!tj|tjd|_ n:|dkr1tj |||tjd|_ n*|dkrGtj |d|d|tjdd|_ n|dkrQt ||_ n t |d |j d |j |_tj|jd d |_td|j|jd} t| ddd dggtj} t| |_d|_||_||dg|_d|_d|_d|_|jd|_dS)Nr znOnly one of `config.use_beta_sigmas`, `config.use_exponential_sigmas`, `config.use_karras_sigmas` can be used.r.rC scaled_linearr'rsquaredcos_cap_v2z is not implemented for ?r)dimFcpu)sumconfigrMrLrKr0rr4r5r6rEr:NotImplementedError __class__alphascumprodalphas_cumprodnparray concatenateastype from_numpysigmasnum_inference_steps set_timesteps derivativesis_scale_input_called _step_index _begin_indexto) selfrFrGrHrIrJrKrLrMrNrQrTrhrrr__init__s4$  "  zLMSDiscreteScheduler.__init__r cCs,|jjdvr |jS|jdddS)a0 The standard deviation of the initial noise distribution. Returns: `float` or `torch.Tensor`: The standard deviation of the initial noise distribution, computed based on the maximum sigma value and the timestep spacing configuration. )rErSrr r')r]rQrhmaxrprrrinit_noise_sigmas z%LMSDiscreteScheduler.init_noise_sigmacC|jS)z The index counter for current timestep. It will increase by 1 after each scheduler step. Returns: `int` or `None`: The current step index, or `None` if not initialized. )rmrsrrr step_index zLMSDiscreteScheduler.step_indexcCru)z The index for the first timestep. It should be set from pipeline with `set_begin_index` method. Returns: `int` or `None`: The begin index for the scheduler, or `None` if not set. rnrsrrr begin_indexrwz LMSDiscreteScheduler.begin_indexrycCs ||_dS)z Sets the begin index for the scheduler. This function should be run from pipeline before the inference. Args: begin_index (`int`, defaults to `0`): The begin index for the scheduler. Nrx)rpryrrrset_begin_indexs z$LMSDiscreteScheduler.set_begin_indexrOtimestepcCs>|jdur |||j|j}||ddd}d|_|S)a Ensures interchangeability with schedulers that need to scale the denoising model input depending on the current timestep. Args: sample (`torch.Tensor`): The input sample. timestep (`float` or `torch.Tensor`): The current timestep in the diffusion chain. Returns: `torch.Tensor`: A scaled input sample. Nrr r'T)rv_init_step_indexrhrl)rprOr{sigmarrrscale_model_inputs   z&LMSDiscreteScheduler.scale_model_inputorderr% current_ordercs<fdd}tj|jjdddd}|S)a Compute the linear multistep coefficient. Args: order (`int`): The order of the linear multistep method. t (`int`): The current timestep index. current_order (`int`): The current order for which to compute the coefficient. Returns: `float`: The computed linear multistep coefficient. csRd}tD] }|kr q||j|jj|9}q|S)NrW)r1rh)tauprodkrrrpr%rrlms_derivatives  4z@LMSDiscreteScheduler.get_lms_coefficient..lms_derivativer rA)epsrelr)r quadrh)rprr%rrintegrated_coeffrrrget_lms_coefficient s&z(LMSDiscreteScheduler.get_lms_coefficientridevicecs"|_jjdkrtjdjjd|tjdddd}nVjjdkrHjjj}td|| ddd tj}|jj 7}n-jjdkrljjj}tjjd|   tj}|d8}n t jjd t djjd }t|t|tdt||}jjrj|d }t fd d |D}n1jjrňj||d}t fdd |D}njjr݈j||d}t fdd |D}t|dgg tj}t|j|d_t|j|tjd_d_d_jd_g_ dS)a Sets the discrete timesteps used for the diffusion chain (to be run before inference). Args: num_inference_steps (`int`): The number of diffusion steps used when generating samples with a pre-trained model. device (`str` or `torch.device`, *optional*): The device to which the timesteps should be moved to. If `None`, the timesteps are not moved. rErr r.NrYrRrSzY is not supported. Please make sure to choose one of 'linspace', 'leading' or 'trailing'.r') in_sigmascg|]}|qSr _sigma_to_tr=r} log_sigmasrprrr?Pz6LMSDiscreteScheduler.set_timesteps..)rricrrrrrrrr?Srcrrrrrrrr?VrrZ)rrr/r[)!rir]rQrcrErFr5copyarangeroundrfrTr0rdrbr)interplenrK_convert_to_karrasrL_convert_to_exponentialrM_convert_to_betarerrgrorh timestepsrmrnrk)rprirr step_ratiorhrrrrj(sJ  * $     z"LMSDiscreteScheduler.set_timestepsschedule_timestepscCs:|dur|j}||k}t|dkrdnd}||S)ak Find the index of a given timestep in the timestep schedule. Args: timestep (`float` or `torch.Tensor`): The timestep value to find in the schedule. schedule_timesteps (`torch.Tensor`, *optional*): The timestep schedule to search in. If `None`, uses `self.timesteps`. Returns: `int`: The index of the timestep in the schedule. For the very first step, returns the second index if multiple matches exist to avoid skipping a sigma when starting mid-schedule (e.g., for image-to-image). Nr r)rnonzeroritem)rpr{rindicesposrrrindex_for_timestepcs   z'LMSDiscreteScheduler.index_for_timestepcCs@|jdurt|tjr||jj}|||_dS|j |_dS)z Initialize the step index for the scheduler based on the given timestep. Args: timestep (`float` or `torch.Tensor`): The current timestep to initialize the step index from. N) ry isinstancerrrorrrrmrn)rpr{rrrr|s   z%LMSDiscreteScheduler._init_step_indexr}rc Cstt|d}||ddtjf}tj|dkddjddj|jddd}|d}||}||}||||} t| dd} d| || |} | |j} | S)a Convert sigma values to corresponding timestep values through interpolation. Args: sigma (`np.ndarray`): The sigma value(s) to convert to timestep(s). log_sigmas (`np.ndarray`): The logarithm of the sigma schedule used for interpolation. Returns: `np.ndarray`: The interpolated timestep value(s) corresponding to the input sigma(s). g|=Nr)axisr)rrr ) rcr)maximumnewaxiscumsumargmaxclipshapereshape) rpr}r log_sigmadistslow_idxhigh_idxlowhighwr%rrrrs, z LMSDiscreteScheduler._sigma_to_trc Cs\|d}|d}d}tdd|j}|d|}|d|}|||||}|S)a Construct the noise schedule as proposed in [Elucidating the Design Space of Diffusion-Based Generative Models](https://huggingface.co/papers/2206.00364). Args: in_sigmas (`torch.Tensor`): The input sigma values to be converted. Returns: `torch.Tensor`: The converted sigma values following the Karras noise schedule. rYrg@r )rrcrEri) rpr sigma_min sigma_maxrhoramp min_inv_rho max_inv_rhorhrrrrs    z'LMSDiscreteScheduler._convert_to_karrascCst|jdr |jj}nd}t|jdr|jj}nd}|dur |n|d}|dur,|n|d}ttt |t ||}|S)a Construct an exponential noise schedule. Args: in_sigmas (`torch.Tensor`): The input sigma values to be converted. num_inference_steps (`int`): The number of inference steps to generate the noise schedule for. Returns: `torch.Tensor`: The converted sigma values following an exponential schedule. rNrrYr) hasattrr]rrrrcrrEr!r))rprrirrrhrrrrs     z,LMSDiscreteScheduler._convert_to_exponential333333?alphabetac st|jdr |jjndt|jdr|jjnddur n|ddur,n|dtfddfddd tdd |DD}|S) a Construct a beta noise schedule as proposed in [Beta Sampling is All You Need](https://huggingface.co/papers/2407.12173). Args: in_sigmas (`torch.Tensor`): The input sigma values to be converted. num_inference_steps (`int`): The number of inference steps to generate the noise schedule for. alpha (`float`, *optional*, defaults to `0.6`): The alpha parameter for the beta distribution. beta (`float`, *optional*, defaults to `0.6`): The beta parameter for the beta distribution. Returns: `torch.Tensor`: The converted sigma values following a beta distribution schedule. rNrrYrcsg|] }|qSrr)r=ppf)rrrrr?sz9LMSDiscreteScheduler._convert_to_beta..csg|] }tjj|qSr)scipystatsrr)r=r{)rrrrr?sr )rr]rrrrcrdrE)rprrirrrhr)rrrrrrs       z%LMSDiscreteScheduler._convert_to_betaT model_output return_dictc sJjstdjdur|jj}jjdkr%|||}n/jjdkrA|| |ddd||dd}njjdkrJ|}n td jjd |||}j |t j krmj d t jdfd d tD} |tddt| tj D} jd7_|s| |fSt| |dS)aa Predict the sample from the previous timestep by reversing the SDE. This function propagates the diffusion process from the learned model outputs (most often the predicted noise). Args: model_output (`torch.Tensor`): The direct output from learned diffusion model. timestep (`float` or `torch.Tensor`): The current discrete timestep in the diffusion chain. sample (`torch.Tensor`): A current instance of a sample created by the diffusion process. order (`int`, defaults to 4): The order of the linear multistep method. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~schedulers.scheduling_utils.SchedulerOutput`] or tuple. Returns: [`~schedulers.scheduling_utils.SchedulerOutput`] or `tuple`: If return_dict is `True`, [`~schedulers.scheduling_utils.SchedulerOutput`] is returned, otherwise a tuple is returned where the first element is the sample tensor. zThe `scale_model_input` function should be called before `step` to ensure correct denoising. See `StableDiffusionPipeline` for a usage example.NrDrPrr r'rOzprediction_type given as z, must be one of `epsilon`, or `v_prediction`rcsg|] }j|qSr)rrv)r= curr_orderrrprrr?_sz-LMSDiscreteScheduler.step..css|] \}}||VqdSNr)r=coeff derivativerrr bs  z,LMSDiscreteScheduler.step..)rr)rlwarningswarnrvr|rhr]rNr0rkr2rpopr3r1r\zipreversedrmr ) rprr{rOrrr}rr lms_coeffsrrrrstep!s>     ,       zLMSDiscreteScheduler.steporiginal_samplesnoisercs jj|j|jd}|jjdkr)t|r)jj|jtjd|j|jtjd}n j|j||j}j durFfdd|D}nj durUj g|j d}n j g|j d}|| }t |j t |j kr}|d}t |j t |j ksn|||}|S) am Add noise to the original samples according to the noise schedule at the specified timesteps. Args: original_samples (`torch.Tensor`): The original samples to which noise will be added. noise (`torch.Tensor`): The noise tensor to add to the original samples. timesteps (`torch.Tensor`): The timesteps at which to add noise, determining the noise level from the schedule. Returns: `torch.Tensor`: The noisy samples with added noise scaled according to the timestep schedule. rmpsr.Ncrr)r)r=r%rrprrr?rz2LMSDiscreteScheduler.add_noise..rrY)rhrorr/typeris_floating_pointrr5ryrvrflattenr unsqueeze)rprrrrh step_indicesr} noisy_samplesrrr add_noisers"      zLMSDiscreteScheduler.add_noisecCs|jjSr)r]rFrsrrr__len__szLMSDiscreteScheduler.__len__) r@rArBrCNFFFrDrEr)rr)rr)rT)'rrrrr _compatiblesrrintfloatstrrcndarraylistboolrrqpropertyrrrtrvryrzr~rrrjrr|rrrrr tuplerrrrrrrr;hs$    1  < $# 5 Q 0r;)rr)r!r dataclassesrtypingrnumpyrcrconfiguration_utilsrrutilsrrscheduling_utilsr r scipy.statsrr r rrrr:r;rrrrs4     4