o %ݫiV@sdZddlmZddlZddlmZddlmmZddl m Z ddl m Z ddl mZddlmZe jZejZejjdd Zd d ZGd d d ejZGdddejZGdddejZGdddejZGdddeZdS)z Neural network modules for DIFFWAVE: A VERSATILE DIFFUSION MODEL FOR AUDIO SYNTHESIS For more details: https://arxiv.org/pdf/2009.09761.pdf Authors * Yingzhi WANG 2022 )sqrtN) transforms)linear)Conv1d)DenoisingDiffusioncCs|t|S)z'sigmoid linear unit activation function)torchsigmoid)xr U/home/ubuntu/.local/lib/python3.10/site-packages/speechbrain/lobes/models/DiffWave.pysilu.sr c  Csptj|||||||||| | d | j} | t| dd} dttj| ddd} t| dddd} | S) ascalculates MelSpectrogram for a raw audio signal and preprocesses it for diffwave training Arguments --------- sample_rate : int Sample rate of audio signal. hop_length : int Length of hop between STFT windows. win_length : int Window size. n_fft : int Size of FFT. n_mels : int Number of mel filterbanks. f_min : float Minimum frequency. f_max : float Maximum frequency. power : float Exponent for the magnitude spectrogram. normalized : bool Whether to normalize by magnitude after stft. norm : str or None If "slaney", divide the triangular mel weights by the width of the mel band mel_scale : str Scale to use: "htk" or "slaney". audio : torch.tensor input audio signal Returns ------- mel : torch.Tensor ) sample_rate hop_length win_lengthn_fftn_melsf_minf_maxpower normalizednorm mel_scale?gh㈵>)mindg)rMelSpectrogramtodevicerclamplog10)r rrrrrrrrrraudio audio_to_melmelr r r diffwave_mel_spectogram4s&0 r%cs8eZdZdZfddZddZddZdd ZZS) DiffusionEmbeddingaEmbeds the diffusion step into an input vector of DiffWave Arguments --------- max_steps: int total diffusion steps Example ------- >>> from speechbrain.lobes.models.DiffWave import DiffusionEmbedding >>> diffusion_embedding = DiffusionEmbedding(max_steps=50) >>> time_step = torch.randint(50, (1,)) >>> step_embedding = diffusion_embedding(time_step) >>> step_embedding.shape torch.Size([1, 512]) cs@t|jd||ddtddd|_tddd|_dS)N embeddingF) persistent input_size n_neurons)super__init__register_buffer_build_embeddingLinear projection1 projection2)self max_steps __class__r r r/s  zDiffusionEmbedding.__init__cCsP|jtjtjfvr|j|}n||}||}t|}||}t|}|S)aforward function of diffusion step embedding Arguments --------- diffusion_step: torch.Tensor which step of diffusion to execute Returns ------- diffusion step embedding: tensor [bs, 512] ) dtyperint32int64r'_lerp_embeddingr3r r4)r5diffusion_stepr r r r forwards    zDiffusionEmbedding.forwardcCsDt|}t|}|j|}|j|}|||||S)zDeals with the cases where diffusion_step is not int Arguments --------- t: torch.Tensor which step of diffusion to execute Returns ------- embedding : torch.Tensor )rfloorlongceilr')r5tlow_idxhigh_idxlowhighr r r r<s    z"DiffusionEmbedding._lerp_embeddingcCsVt|d}tdd}|d|dd}tjt|t|gdd}|S)zBuild embeddings in a designed way Arguments --------- max_steps: int total diffusion steps Returns ------- table: torch.Tensor @rg$@g@gO@dim)rarange unsqueezecatsincos)r5r6stepsdimstabler r r r1s  z#DiffusionEmbedding._build_embedding) __name__ __module__ __qualname____doc__r/r>r<r1 __classcell__r r r7r r&xs  r&cs(eZdZdZfddZddZZS)SpectrogramUpsampleruUpsampler for spectrograms with Transposed Conv Only the upsampling is done here, the layer-specific Conv can be found in residual block to map the mel bands into 2× residual channels Example ------- >>> from speechbrain.lobes.models.DiffWave import SpectrogramUpsampler >>> spec_upsampler = SpectrogramUpsampler() >>> mel_input = torch.rand(3, 80, 100) >>> upsampled_mel = spec_upsampler(mel_input) >>> upsampled_mel.shape torch.Size([3, 80, 25600]) csNttddddgddgddgd|_tddddgddgddgd|_dS)NrG )stridepadding)r.r/ConvTranspose2dconv1conv2r5r7r r r/s  zSpectrogramUpsampler.__init__cCsHt|d}||}t|d}||}t|d}t|d}|S)aOUpsamples spectrograms 256 times to match the length of audios Hop length should be 256 when extracting mel spectrograms Arguments --------- x: torch.Tensor input mel spectrogram [bs, 80, mel_len] Returns ------- upsampled spectrogram [bs, 80, mel_len*256] rGg?)rrLr`F leaky_relurasqueeze)r5r r r r r>s     zSpectrogramUpsampler.forwardrSrTrUrVr/r>rWr r r7r rXs  rXcs,eZdZdZdfdd Zd ddZZS) ResidualBlocka Residual Block with dilated convolution Arguments --------- n_mels: int input mel channels of conv1x1 for conditional vocoding task residual_channels: int channels of audio convolution dilation: int dilation cycles of audio convolution uncond: bool conditional/unconditional generation Example ------- >>> from speechbrain.lobes.models.DiffWave import ResidualBlock >>> res_block = ResidualBlock(n_mels=80, residual_channels=64, dilation=3) >>> noisy_audio = torch.randn(1, 1, 22050) >>> timestep_embedding = torch.rand(1, 512) >>> upsampled_mel = torch.rand(1, 80, 22050) >>> output = res_block(noisy_audio, timestep_embedding, upsampled_mel) >>> output[0].shape torch.Size([1, 64, 22050]) Fc sxtt|d|d|dddd|_td|d|_|s*t|d|d dddd |_nd|_t|d|d dddd |_dS) NrYTsamekaiming) in_channels out_channels kernel_sizedilationskip_transposer^ conv_initr*r+rGrkrlrmror^rp)r.r/r dilated_convr2diffusion_projectionconditioner_projectionoutput_projection)r5rresidual_channelsrnuncondr7r r r/s>     zResidualBlock.__init__Nc Cs|dur |jdus|dur|jdusJ||d}||}|jdur+||}n ||}|||}tj|ddd\}}t|t|}||}tj|ddd\}}||t d|fS)a forward function of Residual Block Arguments --------- x: torch.Tensor input sample [bs, 1, time] diffusion_step: torch.Tensor the embedding of which step of diffusion to execute conditioner: torch.Tensor the condition used for conditional generation Returns ------- residual output [bs, residual_channels, time] a skip of residual branch [bs, residual_channels, time] NrhrGrIg@) rtrsrLrrrchunkrtanhrur) r5r r= conditionerygatefilterresidualskipr r r r><s    zResidualBlock.forwardFNrfr r r7r rgs&rgcs@eZdZdZ d fdd Zd ddZ    d dd ZZS) DiffWavea DiffWave Model with dilated residual blocks Arguments --------- input_channels: int input mel channels of conv1x1 for conditional vocoding task residual_layers: int number of residual blocks residual_channels: int channels of audio convolution dilation_cycle_length: int dilation cycles of audio convolution total_steps: int total steps of diffusion unconditional: bool conditional/unconditional generation Example ------- >>> from speechbrain.lobes.models.DiffWave import DiffWave >>> diffwave = DiffWave( ... input_channels=80, ... residual_layers=30, ... residual_channels=64, ... dilation_cycle_length=10, ... total_steps=50, ... ) >>> noisy_audio = torch.randn(1, 1, 25600) >>> timestep = torch.randint(50, (1,)) >>> input_mel = torch.rand(1, 80, 100) >>> predicted_noise = diffwave(noisy_audio, timestep, input_mel) >>> predicted_noise.shape torch.Size([1, 1, 25600]) Fcst|_|_|_|_|_|_tdjddddd_ t j_ jr0d_ nt _ tfddtjD_tjjddddd_tjdddddd_dS) NrGTrirjrqcs,g|]}tjjd|jjdqS)rh)rw)rginput_channelsrvdilation_cycle_length unconditional).0irbr r s z%DiffWave.__init__..zero)r.r/rresidual_layersrvrr total_stepsrinput_projectionr&diffusion_embeddingspectrogram_upsamplerrXnn ModuleListrangeskip_projectionru)r5rrrvrrrr7rbr r/sP     zDiffWave.__init__Nc Cs|dur |jdus|dur|jdusJ||}t|}||}|jr+||}d}|jD]}||||\}}|dur@|n||}q0|tt|j}||}t|}| |}|S)a DiffWave forward function Arguments --------- audio: torch.Tensor input gaussian sample [bs, 1, time] diffusion_step: torch.Tensor which timestep of diffusion to execute [bs, 1] spectrogram: torch.Tensor spectrogram data [bs, 80, mel_len] length: torch.Tensor sample lengths - not used - provided for compatibility only Returns ------- predicted noise [bs, 1, time] N) rrrcrelurrrlenrru) r5r"r= spectrogramlengthr rlayerskip_connectionr r r r>s         zDiffWave.forwardcCs|||||dS)zForward function suitable for wrapping by diffusion. For this model, `out_mask_value`/`latent_mask_value` are unused and discarded. See :meth:`~DiffWave.forward` for details.)rrr )r5r timestepscond_embrout_mask_valuelatent_mask_valuer r r diffusion_forwardszDiffWave.diffusion_forwardr)NN)NNNN)rSrTrUrVr/r>rrWr r r7r res+ ;-rcsJeZdZdZ       dfdd Ze    d ddZZS) DiffWaveDiffusionaqAn enhanced diffusion implementation with DiffWave-specific inference Arguments --------- model: nn.Module the underlying model timesteps: int the total number of timesteps noise: str|nn.Module the type of noise being used "gaussian" will produce standard Gaussian noise beta_start: float the value of the "beta" parameter at the beginning of the process (see DiffWave paper) beta_end: float the value of the "beta" parameter at the end of the process sample_min: float sample_max: float Used to clip the output. show_progress: bool whether to show progress during inference Example ------- >>> from speechbrain.lobes.models.DiffWave import DiffWave >>> diffwave = DiffWave( ... input_channels=80, ... residual_layers=30, ... residual_channels=64, ... dilation_cycle_length=10, ... total_steps=50, ... ) >>> from speechbrain.lobes.models.DiffWave import DiffWaveDiffusion >>> from speechbrain.nnet.diffusion import GaussianNoise >>> diffusion = DiffWaveDiffusion( ... model=diffwave, ... beta_start=0.0001, ... beta_end=0.05, ... timesteps=50, ... noise=GaussianNoise, ... ) >>> input_mel = torch.rand(1, 80, 100) >>> output = diffusion.inference( ... unconditional=False, ... scale=256, ... condition=input_mel, ... fast_sampling=True, ... fast_sampling_noise_schedule=[0.0001, 0.001, 0.01, 0.05, 0.2, 0.5], ... ) >>> output.shape torch.Size([1, 25600]) NFc st||||||||dSr)r.r/) r5modelrnoise beta_startbeta_end sample_min sample_max show_progressr7r r r/5s zDiffWaveDiffusion.__init__cCsD|dur td}|r|dusJn |dusJ|j}|r#|dus#J|r9|dur9|}dt|}|jdd} n |j}|j}|j} g} tt|D]H} t|j dD]>} |j| d| | kri|j| krnqS|j| d| | d|j| d|j| dd} | | | nqSqJ|st|j dkr| d}tj |j d||j d|d }ntj d||d }tt|dddD][}d||d}||d| |d}||tj| |g|d |d}||||}|dkrt|}d | |dd | |||d}|||7}t|d d }q|S) aProcesses the inference for diffwave One inference function for all the locally/globally conditional generation and unconditional generation tasks Arguments --------- unconditional: bool do unconditional generation if True, else do conditional generation scale: int scale to get the final output wave length for conditional generation, the output wave length is scale * condition.shape[-1] for example, if the condition is spectrogram (bs, n_mel, time), scale should be hop length for unconditional generation, scale should be the desired audio length condition: torch.Tensor input spectrogram for vocoding or other conditions for other conditional generation, should be None for unconditional generation fast_sampling: bool whether to do fast sampling fast_sampling_noise_schedule: list the noise schedules used for fast sampling device: str|torch.device inference device Returns ------- predicted_sample: torch.Tensor the predicted audio (bs, 1, t) NcudarGrrIg?rhrx)rrr)rrtensorcumprodbetasalphasalphas_cumprodrrrappendshaperLrandnrre randn_liker )r5rscale condition fast_samplingfast_sampling_noise_schedulerinference_noise_scheduleinference_alphasinference_alpha_cuminference_stepssrBtwiddler"nc1c2 noise_predrsigmar r r inferenceKs&             zDiffWaveDiffusion.inference)NNNNNNF)NFNN) rSrTrUrVr/rno_gradrrWr r r7r rs 8r)rVmathrrtorch.nnrtorch.nn.functional functionalrc torchaudiorspeechbrain.nnetrspeechbrain.nnet.CNNrspeechbrain.nnet.diffusionrr2r_jitscriptr r%Moduler&rXrgrrr r r r s(       DU.j