o ۷i{T@sddlmZddlZddlZddlmZmZddlm Z m Z ddl m Z ddl mZe eZeGd d d e ZGd d d eeZdS) ) dataclassN) ConfigMixinregister_to_config) BaseOutputlogging) randn_tensor)SchedulerMixinc@seZdZUdZejed<dS)$CMStochasticIterativeSchedulerOutputa> Output class for the scheduler's `step` function. Args: prev_sample (`torch.Tensor` of shape `(batch_size, num_channels, height, width)` for images): Computed sample `(x_{t-1})` of previous timestep. `prev_sample` should be used as next model input in the denoising loop. prev_sampleN)__name__ __module__ __qualname____doc__torchTensor__annotations__rrh/home/ubuntu/vllm_env/lib/python3.10/site-packages/diffusers/schedulers/scheduling_consistency_models.pyr s  r c@seZdZdZdZe       dAd ed ed ed ededededdfddZ e defddZ e defddZ dBdeddfddZ dejdeejBdejfdd Zd!eejBfd"d#Z   dCd$edBd%eejBd&eedBfd'd(Zd)ejdejfd*d+Zd,ejdeejejffd-d.Zd,ejdeejejffd/d0Z dDdeejBd1ejdBdefd2d3ZdeejBddfd4d5Z  dEd6ejdeejBdejd7ejdBd8edeeBf d9d:Z d;ejdZ!defd?d@Z"dS)FCMStochasticIterativeSchedulera Multistep and onestep sampling for consistency models. This model inherits from [`SchedulerMixin`] and [`ConfigMixin`]. Check the superclass documentation for the generic methods the library implements for all schedulers such as loading and saving. Args: num_train_timesteps (`int`, defaults to 40): The number of diffusion steps to train the model. sigma_min (`float`, defaults to 0.002): Minimum noise magnitude in the sigma schedule. Defaults to 0.002 from the original implementation. sigma_max (`float`, defaults to 80.0): Maximum noise magnitude in the sigma schedule. Defaults to 80.0 from the original implementation. sigma_data (`float`, defaults to 0.5): The standard deviation of the data distribution from the EDM [paper](https://huggingface.co/papers/2206.00364). Defaults to 0.5 from the original implementation. s_noise (`float`, defaults to 1.0): The amount of additional noise to counteract loss of detail during sampling. A reasonable range is [1.000, 1.011]. Defaults to 1.0 from the original implementation. rho (`float`, defaults to 7.0): The parameter for calculating the Karras sigma schedule from the EDM [paper](https://huggingface.co/papers/2206.00364). Defaults to 7.0 from the original implementation. clip_denoised (`bool`, defaults to `True`): Whether to clip the denoised outputs to `(-1, 1)`. timesteps (`list` or `np.ndarray` or `torch.Tensor`, *optional*): An explicit timestep schedule that can be optionally specified. The timesteps are expected to be in increasing order. r (Mb`?T@??@Tnum_train_timesteps sigma_min sigma_max sigma_datas_noiserho clip_denoisedreturnNc Csp||_tdd|}||} || } d|_t| |_t| |_ d|_ d|_ d|_ d|_ |jd|_dS)Nrr Fcpu)init_noise_sigmanplinspace_convert_to_karras sigma_to_tnum_inference_stepsr from_numpysigmas timestepscustom_timestepsis_scale_input_called _step_index _begin_indexto) selfrrrr r!r"r#rampr-r.rrr__init__Ks     z'CMStochasticIterativeScheduler.__init__cC|jS)z The index counter for current timestep. It will increase 1 after each scheduler step. Returns: `int` or `None`: The current step index, or `None` if not yet initialized. )r1r4rrr step_indexg z)CMStochasticIterativeScheduler.step_indexcCr7)z The index for the first timestep. It should be set from pipeline with `set_begin_index` method. Returns: `int` or `None`: The begin index, or `None` if not yet set. r2r8rrr begin_indexrr:z*CMStochasticIterativeScheduler.begin_indexrr<cCs ||_dS)z Sets the begin index for the scheduler. This function should be run from pipeline before the inference. Args: begin_index (`int`, defaults to `0`): The begin index for the scheduler. Nr;)r4r<rrrset_begin_index~s z.CMStochasticIterativeScheduler.set_begin_indexsampletimestepcCsF|jdur |||j|j}||d|jjdd}d|_|S)av Scales the consistency model input by `(sigma**2 + sigma_data**2) ** 0.5`. Args: sample (`torch.Tensor`): The input sample. timestep (`float` or `torch.Tensor`): The current timestep in the diffusion chain. Returns: `torch.Tensor`: A scaled input sample. NrrT)r9_init_step_indexr-configr r0)r4r>r?sigmarrrscale_model_inputs   z0CMStochasticIterativeScheduler.scale_model_inputr-cCs2t|tjstj|tjd}dt|d}|S)a> Gets scaled timesteps from the Karras sigmas for input to the consistency model. Args: sigmas (`float` or `np.ndarray`): A single Karras sigma or an array of Karras sigmas. Returns: `np.ndarray`: A scaled input timestep array. dtypeg@o@ggE6) isinstancer'ndarrayarrayfloat64log)r4r-r.rrrr*s z)CMStochasticIterativeScheduler.sigma_to_tr+devicer.c Cs|dur |dur td|dur|durtd|durRtdt|D]}||||dkr3tdq#|d|jjkrFtd|jjdtj|tjd }d |_n:||jjkrjtd |d |jjd |jjd||_ |jj|j }t d|| ddd tj}d|_|jj}|ddd }||d}||}||}t||jjgg tj}t|j|d|_t|drt|j|tjd |_n t|j|d|_d|_d|_|jd|_dS)a Sets the timesteps used for the diffusion chain (to be run before inference). Args: num_inference_steps (`int`, *optional*): The number of diffusion steps used when generating samples with a pre-trained model. device (`str` or `torch.device`, *optional*): The device to which the timesteps should be moved to. If `None`, the timesteps are not moved. timesteps (`list[int]`, *optional*): Custom timesteps used to support arbitrary spacing between timesteps. If `None`, then the default timestep spacing strategy of equal spacing between timesteps is used. If `timesteps` is passed, `num_inference_steps` must be `None`. NzEExactly one of `num_inference_steps` or `timesteps` must be supplied.z:Can only pass one of `num_inference_steps` or `timesteps`.r z(`timesteps` must be in descending order.rz=`timesteps` must start before `self.config.train_timesteps`: .rDTz`num_inference_steps`: z6 cannot be larger than `self.config.train_timesteps`: zG as the unet model trained with this scheduler can only handle maximal z timesteps.F)rKmpsr%) ValueErrorrangelenrArr'rHint64r/r+arangeroundcopyastyper)r* concatenaterfloat32rr,r3r-str startswithr.r1r2) r4r+rKr.i step_ratiorr5r-rrr set_timestepssP *   z,CMStochasticIterativeScheduler.set_timestepsr5cCsH|jj}|jj}|jj}|d|}|d|}|||||}|S)a Construct the noise schedule as proposed in [Elucidating the Design Space of Diffusion-Based Generative Models](https://huggingface.co/papers/2206.00364). Args: ramp (`np.ndarray`): A ramp array of values between 0 and 1 used to interpolate between sigma_min and sigma_max. Returns: `np.ndarray`: The Karras sigma schedule array. r )rArrr")r4r5rrr" min_inv_rho max_inv_rhor-rrrr)s   z1CMStochasticIterativeScheduler._convert_to_karrasrBcCsD|jj}|d|d|d}|||d|dd}||fS)a| Computes the scaling factors for the consistency model output. Args: sigma (`torch.Tensor`): The current sigma value in the noise schedule. Returns: `tuple[torch.Tensor, torch.Tensor]`: A tuple containing `c_skip` (scaling for the input sample) and `c_out` (scaling for the model output). rr)rAr )r4rBr c_skipc_outrrr get_scalingss z+CMStochasticIterativeScheduler.get_scalingscCsT|jj}|jj}|d||d|d}||||d|dd}||fS)a Gets the scalings used in the consistency model parameterization (from Appendix C of the [paper](https://huggingface.co/papers/2303.01469)) to enforce boundary condition. > [!TIP] > `epsilon` in the equations for `c_skip` and `c_out` is set to `sigma_min`. Args: sigma (`torch.Tensor`): The current sigma in the Karras sigma schedule. Returns: `tuple[torch.Tensor, torch.Tensor]`: A two-element tuple where `c_skip` (which weights the current sample) is the first element and `c_out` (which weights the consistency model output) is the second element. rr)rArr )r4rBrr r`rarrr#get_scalings_for_boundary_condition&s  zBCMStochasticIterativeScheduler.get_scalings_for_boundary_conditionschedule_timestepscCs:|dur|j}||k}t|dkrdnd}||S)ak Find the index of a given timestep in the timestep schedule. Args: timestep (`float` or `torch.Tensor`): The timestep value to find in the schedule. schedule_timesteps (`torch.Tensor`, *optional*): The timestep schedule to search in. If `None`, uses `self.timesteps`. Returns: `int`: The index of the timestep in the schedule. For the very first step, returns the second index if multiple matches exist to avoid skipping a sigma when starting mid-schedule (e.g., for image-to-image). Nr r)r.nonzerorQitem)r4r?rdindicesposrrrindex_for_timestep>s   z1CMStochasticIterativeScheduler.index_for_timestepcCs@|jdurt|tjr||jj}|||_dS|j |_dS)z Initialize the step index for the scheduler based on the given timestep. Args: timestep (`float` or `torch.Tensor`): The current timestep to initialize the step index from. N) r<rFrrr3r.rKrir1r2)r4r?rrrr@]s   z/CMStochasticIterativeScheduler._init_step_index model_output generator return_dictcCsNt|ttjtjfrtd|jd|jst d|j j }|j j }|j dur-|||j|j }|j d|j jkrE|j|j d} n|jd} ||\} } | || |} |j jrc| dd} t|jdkrvt|j|j|j|d} nt|} | |j j}| j||d}| ||d |d d }|jd7_|s|fSt|d S) a Predict the sample from the previous timestep by reversing the SDE. This function propagates the diffusion process from the learned model outputs (most often the predicted noise). Args: model_output (`torch.Tensor`): The direct output from the learned diffusion model. timestep (`float` or `torch.Tensor`): The current timestep in the diffusion chain. sample (`torch.Tensor`): A current instance of a sample created by the diffusion process. generator (`torch.Generator`, *optional*): A random number generator. return_dict (`bool`, defaults to `True`): Whether or not to return a [`~schedulers.scheduling_consistency_models.CMStochasticIterativeSchedulerOutput`] or `tuple`. Returns: [`~schedulers.scheduling_consistency_models.CMStochasticIterativeSchedulerOutput`] or `tuple`: If return_dict is `True`, [`~schedulers.scheduling_consistency_models.CMStochasticIterativeSchedulerOutput`] is returned, otherwise a tuple is returned where the first element is the sample tensor. zLPassing integer indices (e.g. from `enumerate(timesteps)`) as timesteps to `z\.step()` is not supported. Make sure to pass one of the `scheduler.timesteps` as a timestep.zThe `scale_model_input` function should be called before `step` to ensure correct denoising. See `StableDiffusionPipeline` for a usage example.Nr rM)rErKrk)minmaxrr)r )rFintr IntTensor LongTensorrO __class__r0loggerwarningrArrr9r@r-rrcr#clamprQr.rshaperErK zeros_liker!r1r )r4rjr?r>rkrlrrrB sigma_nextr`radenoisednoisez sigma_hatr rrrsteplsJ         z#CMStochasticIterativeScheduler.steporiginal_samplesrzcs jj|j|jd}|jjdkr)t|r)jj|jtjd|j|jtjd}n j|j||j}j durFfdd|D}nj durUj g|j d}n j g|j d}|| }t |j t |j kr}|d}t |j t |j ksn|||}|S) am Add noise to the original samples according to the noise schedule at the specified timesteps. Args: original_samples (`torch.Tensor`): The original samples to which noise will be added. noise (`torch.Tensor`): The noise tensor to add to the original samples. timesteps (`torch.Tensor`): The timesteps at which to add noise, determining the noise level from the schedule. Returns: `torch.Tensor`: The noisy samples with added noise scaled according to the timestep schedule. )rKrErNrDNcsg|]}|qSr)ri).0trdr4rr sz.rrM)r-r3rKrEtyperis_floating_pointr.rXr<r9rvflattenrQ unsqueeze)r4r~rzr.r- step_indicesrB noisy_samplesrrr add_noises"      z(CMStochasticIterativeScheduler.add_noisecCs|jjS)z Returns the number of training timesteps. Returns: `int`: The number of training timesteps configured for the scheduler. )rArr8rrr__len__sz&CMStochasticIterativeScheduler.__len__)rrrrrrT)r)NNN)N)NT)#r rrrorderrrofloatboolr6propertyr9r<r=rrrCr'rGr*rYrKlistr]r)tuplerbrcrir@ Generatorr r}rrrrrrr+s     J    a 0r) dataclassesrnumpyr'rconfiguration_utilsrrutilsrrutils.torch_utilsrscheduling_utilsr get_loggerr rsr rrrrrs