o }oipt@sddlmZmZddlmZmZmZddlZddl Z ddl m Z ddl m Z mZddlmZmZmZmZmZddlmZGdd d e eZGd d d eZGd d d eZGddde ZGddde jjeZGdddeZGddde eZGdddeZ dS))ABCabstractmethod)OptionalTupleTypeN)mask_sequence_tensor) NeuralModule typecheck) FloatType LengthsType NeuralTypeSpectrogramTypeVoidType)loggingc seZdZdZdededeffdd Zedefdd Zedefd d Z d ed e j de j fddZ ede j de j dee j e j ffddZededeidedeidede j de j fddZddde j de j dee j dee j e j ffddZedd Zd!d"ZZS)#StochasticDifferentialEquationz1Base class for stochastic differential equations.time_mintime_max num_stepscsht|dkrtd|||krtd|d|||_||_|dkr/td|||_dS)Nrz+time_min should be positive, current value z5time_max should be larger than time_min, current max and min z.num_steps needs to be positive: current value )super__init__ ValueErrorrrr)selfrrr __class__e/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/audio/parts/submodules/diffusion.pyrs  z'StochasticDifferentialEquation.__init__returncCs |j|jS)zTime step for this SDE. This denotes the step size between `0` and `self.time_max` when using `self.num_steps`. )rrrrrrdt1s z!StochasticDifferentialEquation.dtcCs |j|jS)zTime range for this SDE.)rrrrrr time_delta8s z)StochasticDifferentialEquation.time_deltasizedevicecCstj||d|j|j}|S)aGenerate random time steps in the valid range. Time steps are generated between `self.time_min` and `self.time_max`. Args: size: number of samples device: device to use Returns: A tensor of floats with shape (size,) r")torchrandr r)rr!r"timerrr generate_time=s z,StochasticDifferentialEquation.generate_timestater&cKdS)z Args: state: tensor of shape (B, C, D, T) time: tensor of shape (B,) Returns: Tuple with drift and diffusion coefficients. Nr)rr(r&kwargsrrr coefficientsLs z+StochasticDifferentialEquation.coefficients prior_meanBCDTsample input_types output_typescCr))zGenerate a sample from the prior distribution p_T. Args: prior_mean: Mean of the prior distribution Returns: A sample from the prior distribution. Nr)rr,rrrprior_samplingXsz-StochasticDifferentialEquation.prior_samplingN) state_lengthr7c Ks>|jd|||d|\}}||j}|t|j}||fS)aAssume we have the following SDE: dx = drift(x, t) * dt + diffusion(x, t) * dwt where `wt` is the standard Wiener process. We assume the following discretization: new_state = current_state + total_drift + total_diffusion * z_norm where `z_norm` is sampled from normal distribution with zero mean and unit variance. Args: state: current state of the process, shape (B, C, D, T) time: current time of the process, shape (B,) state_length: length of the valid time steps for each example in the batch, shape (B,) **kwargs: other parameters Returns: Drift and diffusion. )r(r&r7Nr)r+rnpsqrt) rr(r&r7r*drift_coefficientdiffusion_coefficientdrift diffusionrrr discretizels z)StochasticDifferentialEquation.discretizecCr))zCreate a copy of this SDE.Nrrrrrcopysz#StochasticDifferentialEquation.copycCsJ|jjd|jd|jd|jd}|d|j7}|d|j7}|S)Nz (time_min= , time_max= , num_steps=) dt:  time_delta: )r__name__rrrrr rdescrrr__repr__s&z'StochasticDifferentialEquation.__repr__)rE __module__ __qualname____doc__floatintrpropertyrr r$r"Tensorr'rrr+r r rr6rr>r?rH __classcell__rrrrrs>(    ' rcs,eZdZdZ    d2dededed ed ed ed effd d ZedefddZedefddZ e e de e de e e dedde deiddejdejdejdejfddZe de e deide e deiddejdejfdd Ze e de e de e e dede dee ded!ddejdejdejdejfd"d#Ze e de e e de e de e e ded$d%d&e dee ded'd (d3dejdejdejd)eejdeejejff d*d+Zdejdejfd,d-Zd.d/Zd0d1ZZS)4%OrnsteinUhlenbeckVarianceExplodingSDEaThis class implements the Ornstein-Uhlenbeck SDE with variance exploding noise schedule. The SDE is given by: dx = theta * (y - x) dt + g(t) dw where `theta` is the stiffness parameter and `g(t)` is the diffusion coefficient: g(t) = std_min * (std_max/std_min)^t * sqrt(2 * log(std_max/std_min)) References: Richter et al., Speech Enhancement and Dereverberation with Diffusion-based Generative Models, Tr. ASLP 2023 dQ??:0yE> stiffnessstd_minstd_maxrrrepscstj|||d|dkrtd|||_||_|dkr%td|||kr3td|d|||_||_td|j j td|jtd |jtd |jtd |j td |j td |j td|jdS)Nrrrrz&eps should be positive, current value z*std_min should be positive, current value z3std_max should be larger than std_min, current max rInitialized %s withz stiffness: %sz std_min: %sz std_max: %sz num_steps: %sz time_min: %sz time_max: %sz eps: %s)rrrrYrVrWrXrdebugrrErrr)rrVrWrXrrrrYrrrrs& z.OrnsteinUhlenbeckVarianceExplodingSDE.__init__rcCs|j|j|jSN)rXrWrYrrrr std_ratioz/OrnsteinUhlenbeckVarianceExplodingSDE.std_ratiocCst|j|jSr])r8logr^rYrrrr log_std_ratior_z3OrnsteinUhlenbeckVarianceExplodingSDE.log_std_ratior-r.r(r,r&meanr3r(r,r&cCs:t|j |}|dddd}||d||}|S)aBReturn the mean of the perturbation kernel for this SDE. Args: state: current state of the process, shape (B, C, D, T) prior_mean: mean of the prior distribution time: current time of the process, shape (B,) Returns: A tensor of shape (B, C, D, T) )r$exprVview)rr(r,r&weightrcrrrperturb_kernel_meansz9OrnsteinUhlenbeckVarianceExplodingSDE.perturb_kernel_meanstdcCsX|jd|j}|t|jd|td|j|9}||j|j}t|}|S)a|Return the standard deviation of the perturbation kernel for this SDE. Note that the standard deviation depends on the time and the noise schedule, which is parametrized using `self.stiffness`, `self.std_min` and `self.std_max`. Args: time: current time of the process, shape (B,) Returns: A tensor of shape (B,) )rWrar$powr^rfrVr9)rr&varrjrrrperturb_kernel_stds * z8OrnsteinUhlenbeckVarianceExplodingSDE.perturb_kernel_std)rcrjcCs\t||jks Jt||jksJ|j|||d}|j|d}|dddd}||fS)aReturn the mean and standard deviation of the perturbation kernel for this SDE. Args: state: current state of the process, shape (B, C, D, T) prior_mean: mean of the prior distribution time: current time of the process, shape (B,) rbr&rdre)r$allrrrirorg)rr(r,r&rcrjrrrperturb_kernel_paramss  z;OrnsteinUhlenbeckVarianceExplodingSDE.perturb_kernel_paramsToptional)r(r&r,r7)r:r;Nr7cCsv|j||}|jt|j|td|j}|jdgdg| dR}|dur7t ||}t ||}||fS)aCompute drift and diffusion coefficients for this SDE. Args: state: current state of the process, shape (B, C, D, T) time: current time of the process, shape (B,) prior_mean: mean of the prior distribution state_length: length of the valid time steps for each example in the batch Returns: Drift and diffusion coefficients. rkrdreN) rVrWr$rmr^r8r9rargdimr)rr(r&r,r7r:r;rrrr+7s$   z2OrnsteinUhlenbeckVarianceExplodingSDE.coefficientscCsN|jtj|jd|jd}|j|d}|dddd}|t||}|S)zGenerate a sample from the prior distribution p_T. Args: prior_mean: Mean of the prior distribution rr#rprdre)rr$onesshaper"rorg randn_like)rr,r&rjr2rrrr6cs  z4OrnsteinUhlenbeckVarianceExplodingSDE.prior_samplingc Cs$t|j|j|j|j|j|j|jdS)N)rVrWrXrrrrY)rQrVrWrXrrrrYrrrrr?wsz*OrnsteinUhlenbeckVarianceExplodingSDE.copycCs|jjd|jd|jd|jd|jd|jd|jd|jd}|d |j 7}|d |j 7}|d |j 7}|d |j 7}|S) Nz (stiffness=z , std_min=z , std_max=rAz , time_min=r@z, eps=rBrCrDz std_ratio: z log_std_ratio: ) rrErVrWrXrrrrYrr r^rarFrrrrHs Fz.OrnsteinUhlenbeckVarianceExplodingSDE.__repr__)rRrSrTrUr])rErIrJrKrLrMrrNr^rar r rtupler r$rOrirorrr rrr+r6r?rHrPrrrrrQs'   $     $        rQc seZdZdeedeeffdd Z  ddejdejde ejd e ejd e ejejff d d Z d ej dej d ejfddZddddejdejde ejd e ejd e ejejff ddZddZddZZS)%ReverseStochasticDifferentialEquationsdescore_estimatorcs8tj|j|j|jd||_||_td|j j dS)zUse the forward SDE and a score estimator to define the reverse SDE. Args: sde: forward SDE score_estimator: neural score estimator rZzInitialized %sN) rrrrrr| forward_sderr\rrErr{r|rrrrsz.ReverseStochasticDifferentialEquation.__init__Nr(r&score_conditionr7rcKtd)zCompute drift and diffusion coefficients for the reverse SDE. Args: state: current state of the process, shape (B, C, D, T) time: current time of the process, shape (B,) z/Coefficients not necessary for the reverse SDE.NotImplementedErrorrr(r&rr7r*rrrr+sz2ReverseStochasticDifferentialEquation.coefficientsrwr"cCr)z4Prior sampling is not necessary for the reverse SDE.z1Prior sampling not necessary for the reverse SDE.r)rrwr"rrrr6sz4ReverseStochasticDifferentialEquation.prior_samplingrr7c Ks|jjd||d|\}}|dur|ntj||gdd}|j|||d\} } ||d| } |} |dur@t| |} t| |} | | fS)aDiscretize the reverse SDE. Args: state: current state of the process, shape (B, C, D, T) time: current time of the process, shape (B,) score_condition: condition for the score estimator state_length: length of the valid time steps for each example in the batch **kwargs: other parameters for discretization of the forward SDE )r(r&Nreruinput input_length conditionrkr)r}r>r$catr|rmr) rr(r&rr7r* forward_driftforward_diffusion score_inputscore_r<r=rrrr>s  z0ReverseStochasticDifferentialEquation.discretizecCst|j|jdSNr{r|)rzr}r?r|rrrrr?z*ReverseStochasticDifferentialEquation.copycCs"|jjd|jd|jd}|S)Nz(sde=z, score_estimator=rB)rrEr}r|rFrrrrHsz.ReverseStochasticDifferentialEquation.__repr__NN)rErIrJrrrrr$rOrrr+Sizer"r6r>r?rHrPrrrrrzs>   )rzcseZdZdZ        d d ed ed ed ed eedeededeffdd Ze e de e de dde e de ddde de e e de dddde d!dejdejdeejdejfddZZS)"PredictorCorrectorSampleraPredictor-Corrector sampler for the reverse SDE. Args: sde: forward SDE score_estimator: neural score estimator predictor: predictor for the reverse process corrector: corrector for the reverse process num_steps: number of time steps for the reverse process num_corrector_steps: number of corrector steps time_max: maximum time time_min: minimum time snr: SNR for Annealed Langevin Dynamics output_type: type of the output ('state' for the final state, or 'mean' for the mean of the final state) References: - Song et al., Score-based generative modeling through stochastic differential equations, 2021 reverse_diffusionannealed_langevin_dynamics2reN?rc predictor correctorrnum_corrector_stepsrrsnr output_typec svt||_|dur||j_td|jj|dur*||j_td|jj||j_td|jj|jj|_|jj|_|jj|_|dkrRt |j|d|_ nt d||dkrht |j|| |d|_ nt d || d vrztd | | |_td |jjtd |td|td|jtd|jtd|jtd|td| td|jdS)Nzsde.time_max set to: %szsde.time_min set to: %szsde.num_steps set to: %srrzUnexpected predictor: r)r{r|rrzUnexpected corrector: )rcr(Unexpected output type: r[z predictor: %sz corrector: %sz num_steps: %sz time_min: %sz time_max: %sz num_corrector_steps: %sz snr: %sz output_type: %s)rrr?r{rrinforrReverseDiffusionPredictorr RuntimeErrorAnnealedLangevinDynamicsrrrr\rrE) rr{r|rrrrrrrrrrrrsB          z"PredictorCorrectorSampler.__init__r-Trsr.)r,rr7)r2r7r3r,rr7rc Cs|jj|d}|durt||}tj|j|j|j|jd}|D]&}|tj |j d|jd}|j ||||d\}}|j |||||d\}} q|j dkrN|} n|j dkrV| } ntd |j |durgt| |} | |fS) aTakes prior (noisy) mean and generates a sample by solving the reverse SDE. Args: prior_mean: mean for the prior distribution, e.g., noisy observation score_condition: conditioning for the score estimator state_length: length of the valid time steps for each example in the batch Returns: Generated `sample` and the corresponding `sample_length`. )r,Nr#rr(r&rr7)r(r&rr,r7r(rcr)r{r6rr$linspacerrrr"rvrwrrrr) rr,rr7r( time_stepstr&r state_meanr2rrrforward0s0     z!PredictorCorrectorSampler.forward)rrrreNNrrcr])rErIrJrKstrrMrrLrr r r ryr r$inference_moderOrrPrrrrrs^   >   rc s^eZdZdZfddZeeddddejdejde ejd e ejfd d Z Z S) Predictorz{Predictor for the reverse process. Args: sde: forward SDE score_estimator: neural score estimator cstt||d|_dSr)rrrz reverse_sder~rrrr{s zPredictor.__init__Nrr(r&rr7cKr))aPredict the next state of the reverse process. Args: state: current state of the process, shape (B, C, D, T) time: current time of the process, shape (B,) score_condition: conditioning for the score estimator state_length: length of the valid time steps for each example in the batch Returns: New state and mean. NrrrrrrszPredictor.forward) rErIrJrKrrr$rrOrrrPrrrrrss  rcs8eZdZdZfddZedddddZZS)rzPredict the next state of the reverse process using the reverse diffusion process. Args: sde: forward SDE score_estimator: neural score estimator cstj||ddSr)rrr~rrrrrz"ReverseDiffusionPredictor.__init__Nrc Ksb|jjd||||d|\}}t|}||} | ||} |dur-t| |} t| |} | | fS)aPredict the next state of the reverse process using the reverse diffusion process. Args: state: current state of the process, shape (B, C, D, T) time: current time of the process, shape (B,) score_condition: conditioning for the score estimator state_length: length of the valid time steps for each example in the batch Returns: New state and mean of the diffusion process. rNr)rr>r$rxr) rr(r&rr7r*r<r=z_normrc new_staterrrrs     z!ReverseDiffusionPredictor.forward rErIrJrKrr$rrrPrrrrrs  rc seZdZdZdeedeededeffdd Z e e e de e ed ee de d d e ed ed d d d e de idedddZZS) CorrectorzCorrector for the reverse process. Args: sde: forward SDE score_estimator: neural score estimator snr: SNR for Annealed Langevin Dynamics num_steps: number of steps for the corrector r{r|rrcsNt||_||_||_||_td|jj td|td|dS)Nr[z snr: %sz num_steps: %s) rrr{r|rrrr\rrE)rr{r|rrrrrrs  zCorrector.__init__r-r.Trsrr(r3NcCr))a` Args: state: current state of the process, shape (B, C, D, T) time: current time of the process, shape (B,) score_condition: conditioning for the score estimator state_length: length of the valid time steps for each example in the batch Returns: New state and mean. Nr)rr(r&rr7rrrrszCorrector.forwardr)rErIrJrKrrrrLrMrrr r rryr r r$rrrPrrrrrs.    rcs2eZdZdZfddZedddZZS)rzAnnealed Langevin Dynamics for the reverse process. References: - Song et al., Score-based generative modeling through stochastic differential equations, 2021 c s6t|tstdt|tjdd|i|dS)NzCExpected an instance of OrnsteinUhlenbeckVarianceExplodingSDE, got r{r) isinstancerQrtyperr)rr{r*rrrrs z!AnnealedLangevinDynamics.__init__Nc Cs|jj|d}|jdgdg|dR}t|jD];}|dur$|ntj||gdd}|j|||d\}} t |} d|j | d} || |} | | t | d}q|durft ||}t | |} || fS)aCorrect the state using Annealed Langevin Dynamics. Args: state: current state of the process, shape (B, C, D, T) time: current time of the process, shape (B,) score_condition: conditioning for the score estimator state_length: length of the valid time steps for each example in the batch Returns: New state and mean of the diffusion process. References: Alg. 4 in http://arxiv.org/abs/2011.13456 rprdreNrrrk)r{rorgrurangerr$rr|rxrrmr9r) rr(r&rr7rjirrrr step_sizercrrrrs     z AnnealedLangevinDynamics.forwardrrrrrrrs  r)!abcrrtypingrrrnumpyr8r$#nemo.collections.common.parts.utilsrnemo.core.classesrr nemo.core.neural_typesr r r r r nemo.utilsrrrQrzrnnModulerrrrrrrrs&  nS%,6