o -iV<@sddlZddlmZmZmZddlZddlmmZ ddlmZm Z gdZ Gdddej Z Gdddej ZGd d d ej ZGd d d ej ZGd ddej ZdS)N)ListOptionalTuple)nnTensor)ResBlock MelResNet Stretch2dUpsampleNetworkWaveRNNcs>eZdZdZd deddffdd Zdedefd d ZZS) rafResNet block based on *Efficient Neural Audio Synthesis* :cite:`kalchbrenner2018efficient`. Args: n_freq: the number of bins in a spectrogram. (Default: ``128``) Examples >>> resblock = ResBlock() >>> input = torch.rand(10, 128, 512) # a random spectrogram >>> output = resblock(input) # shape: (10, 128, 512) n_freqreturnNc sRtttj||dddt|tjddtj||dddt||_dS)NF in_channels out_channels kernel_sizebiasTinplace)super__init__r SequentialConv1d BatchNorm1dReLUresblock_model)selfr  __class__R/home/ubuntu/LTX-2/.venv/lib/python3.10/site-packages/torchaudio/models/wavernn.pyrs   zResBlock.__init__specgramcCs|||S)zPass the input through the ResBlock layer. Args: specgram (Tensor): the input sequence to the ResBlock layer (n_batch, n_freq, n_time). Return: Tensor shape: (n_batch, n_freq, n_time) )rrr#r!r!r"forward(s zResBlock.forward)r __name__ __module__ __qualname____doc__intrrr% __classcell__r!r!rr"rs  rc sPeZdZdZ ddedededed ed d f fd d Zded efddZZS)raMelResNet layer uses a stack of ResBlocks on spectrogram. Args: n_res_block: the number of ResBlock in stack. (Default: ``10``) n_freq: the number of bins in a spectrogram. (Default: ``128``) n_hidden: the number of hidden dimensions of resblock. (Default: ``128``) n_output: the number of output dimensions of melresnet. (Default: ``128``) kernel_size: the number of kernel size in the first Conv1d layer. (Default: ``5``) Examples >>> melresnet = MelResNet() >>> input = torch.rand(10, 128, 512) # a random spectrogram >>> output = melresnet(input) # shape: (10, 128, 508) r  n_res_blockr n_hiddenn_outputrrNcshtfddt|D}tjtj||ddttjddg|tj|ddR|_dS) Ncsg|]}tqSr!)r).0_r0r!r" Isz&MelResNet.__init__..FrTrr)rrr) rrrangerrrrrmelresnet_model)rr/r r0r1r ResBlocksrr4r"rDs  zMelResNet.__init__r#cCs ||S)zPass the input through the MelResNet layer. Args: specgram (Tensor): the input sequence to the MelResNet layer (n_batch, n_freq, n_time). Return: Tensor shape: (n_batch, n_output, n_time - kernel_size + 1) )r7r$r!r!r"r%Ss zMelResNet.forwardr-r r r r.r&r!r!rr"r4s"rcs@eZdZdZdededdffdd Zdedefd d ZZS) r aUpscale the frequency and time dimensions of a spectrogram. Args: time_scale: the scale factor in time dimension freq_scale: the scale factor in frequency dimension Examples >>> stretch2d = Stretch2d(time_scale=10, freq_scale=5) >>> input = torch.rand(10, 100, 512) # a random spectrogram >>> output = stretch2d(input) # shape: (10, 500, 5120) time_scale freq_scalerNcst||_||_dSN)rrr;r:)rr:r;rr!r"rms  zStretch2d.__init__r#cCs||jd|jdS)zPass the input through the Stretch2d layer. Args: specgram (Tensor): the input sequence to the Stretch2d layer (..., n_freq, n_time). Return: Tensor shape: (..., n_freq * freq_scale, n_time * time_scale) )repeat_interleaver;r:r$r!r!r"r%ss zStretch2d.forwardr&r!r!rr"r _s r csheZdZdZ     ddeedededed ed ed d ffd d Zded eeeffddZ Z S)r aUpscale the dimensions of a spectrogram. Args: upsample_scales: the list of upsample scales. n_res_block: the number of ResBlock in stack. (Default: ``10``) n_freq: the number of bins in a spectrogram. (Default: ``128``) n_hidden: the number of hidden dimensions of resblock. (Default: ``128``) n_output: the number of output dimensions of melresnet. (Default: ``128``) kernel_size: the number of kernel size in the first Conv1d layer. (Default: ``5``) Examples >>> upsamplenetwork = UpsampleNetwork(upsample_scales=[4, 4, 16]) >>> input = torch.rand(10, 128, 10) # a random spectrogram >>> output = upsamplenetwork(input) # shape: (10, 128, 1536), (10, 128, 1536) r-r r.upsample_scalesr/r r0r1rrNc std}|D]}||9}q ||_|dd||_t||||||_t|d|_g} |D]2} t| d} tj ddd| ddfd| fdd} t jj | j d| dd| | | | q/tj| |_dS)NrrF)rrrpaddingr?)rr total_scaleindentrresnetr resnet_stretchrConv2dtorchinit constant_weightappendrupsample_layers) rr@r/r r0r1rrDupsample_scale up_layersscalestretchconvrr!r"rs$      zUpsampleNetwork.__init__r#cCsf||d}||}|d}|d}||}|ddddd|j|j f}||fS)aPass the input through the UpsampleNetwork layer. Args: specgram (Tensor): the input sequence to the UpsampleNetwork layer (n_batch, n_freq, n_time) Return: Tensor shape: (n_batch, n_freq, (n_time - kernel_size + 1) * total_scale), (n_batch, n_output, (n_time - kernel_size + 1) * total_scale) where total_scale is the product of all elements in upsample_scales. rN)rF unsqueezerGsqueezerNrE)rr# resnet_outputupsampling_outputr!r!r"r%s    &zUpsampleNetwork.forwardr9) r'r(r)r*rr+rrrr%r,r!r!rr"r s."r cseZdZdZ       ddeededed ed ed ed ed edededdffdd ZdededefddZe j j ddede ede ee effddZZS)r aWWaveRNN model from *Efficient Neural Audio Synthesis* :cite:`wavernn` based on the implementation from `fatchord/WaveRNN `_. The original implementation was introduced in *Efficient Neural Audio Synthesis* :cite:`kalchbrenner2018efficient`. The input channels of waveform and spectrogram have to be 1. The product of `upsample_scales` must equal `hop_length`. See Also: * `Training example `__ * :class:`torchaudio.pipelines.Tacotron2TTSBundle`: TTS pipeline with pretrained model. Args: upsample_scales: the list of upsample scales. n_classes: the number of output classes. hop_length: the number of samples between the starts of consecutive frames. n_res_block: the number of ResBlock in stack. (Default: ``10``) n_rnn: the dimension of RNN layer. (Default: ``512``) n_fc: the dimension of fully connected layer. (Default: ``512``) kernel_size: the number of kernel size in the first Conv1d layer. (Default: ``5``) n_freq: the number of bins in a spectrogram. (Default: ``128``) n_hidden: the number of hidden dimensions of resblock. (Default: ``128``) n_output: the number of output dimensions of melresnet. (Default: ``128``) Example >>> wavernn = WaveRNN(upsample_scales=[5,5,8], n_classes=512, hop_length=200) >>> waveform, sample_rate = torchaudio.load(file) >>> # waveform shape: (n_batch, n_channel, (n_time - kernel_size + 1) * hop_length) >>> specgram = MelSpectrogram(sample_rate)(waveform) # shape: (n_batch, n_channel, n_freq, n_time) >>> output = wavernn(waveform, specgram) >>> # output shape: (n_batch, n_channel, (n_time - kernel_size + 1) * hop_length, n_classes) r-r.r r@ n_classes hop_lengthr/n_rnnn_fcrr r0r1rNc s:t||_|dr|dn|d|_||_| d|_||_||_tt |j|_ d} |D]} | | 9} q0| |jkrFt d| d|t |||| | ||_t||jd||_tj||dd|_tj||j|dd|_tjdd|_tjdd|_t||j||_t||j||_t||j|_dS) NrArz/Expected: total_scale == hop_length, but found z != T) batch_firstr)rrr_padr[n_auxrZrYr+mathlog2n_bits ValueErrorr upsamplerLinearfcGRUrnn1rnn2rrelu1relu2fc1fc2fc3) rr@rYrZr/r[r\rr r0r1rDrOrr!r"rs,    zWaveRNN.__init__waveformr#cs|ddkr td|ddkrtd|d|d}}|d}tjd|j|j|jd}tjd|j|j|jd}|\}}| dd}| dd}fddt d D}|d d d d |d|df}|d d d d |d|df} |d d d d |d|d f} |d d d d |d |d f} tj | d ||gd d} | } | } | |\} }| | } | } tj | | gd d} | |\} }| | } tj | | gd d} | } | } tj | | gd d} | } | } | } | dS)aPass the input through the WaveRNN model. Args: waveform: the input waveform to the WaveRNN layer (n_batch, 1, (n_time - kernel_size + 1) * hop_length) specgram: the input spectrogram to the WaveRNN layer (n_batch, 1, n_freq, n_time) Return: Tensor: shape (n_batch, 1, (n_time - kernel_size + 1) * hop_length, n_classes) rz*Require the input channel of waveform is 1z*Require the input channel of specgram is 1r)dtypedevicerAcsg|]}j|qSr!r`r2irr!r"r5.sz#WaveRNN.forward..r.Nr]r>dim)sizerdrUrIzerosr[rqrrre transposer6catrTrgrirjrmrkrnrlro)rrpr# batch_sizeh1h2auxaux_idxa1a2a3a4xresr3r!rvr"r%sB    """"       zWaveRNN.forwardlengthscs|j}|j}tjj|jjf}|\}|dur#|jj}g}| \}}}tj d|j f||d} tj d|j f||d} tj |df||d} fddt dD} t |D]|ddddf} fdd| D\}}}}tj | | |gdd} | } | d| \}} | | d } tj | |gdd}|d| \}} | | d } tj | |gdd} t| } tj | |gdd} t| } | }tj|dd}t|d} d | d jd d } || q^t|dd d |fS) aInference method of WaveRNN. This function currently only supports multinomial sampling, which assumes the network is trained on cross entropy loss. Args: specgram (Tensor): Batch of spectrograms. Shape: `(n_batch, n_freq, n_time)`. lengths (Tensor or None, optional): Indicates the valid length of each audio in the batch. Shape: `(batch, )`. When the ``specgram`` contains spectrograms with different durations, by providing ``lengths`` argument, the model will compute the corresponding valid output lengths. If ``None``, it is assumed that all the audio in ``waveforms`` have valid length. Default: ``None``. Returns: (Tensor, Optional[Tensor]): Tensor The inferred waveform of size `(n_batch, 1, n_time)`. 1 stands for a single channel. Tensor or None If ``lengths`` argument was provided, a Tensor of shape `(batch, )` is returned. It indicates the valid length in time axis of the output Tensor. Nr)rrrqcs6g|]}ddj|j|dddfqS)Nrrsrt)rrr!r"r5xs6z!WaveRNN.infer..r]cs"g|] }|ddddfqSr<r!)r2a)rur!r"r5~s"rxrrArC)rrrqrIr functionalpadr_rerDrzr{r[r6r}rgrirTrjFrelurmrnrosoftmax multinomialfloatrcrMstackpermute)rr#rrrrqoutputb_sizer3seq_lenrrr aux_splitm_ta1_ta2_ta3_ta4_tinplogits posteriorr!)rrurr"inferKs@       z WaveRNN.infer)r-rXrXr.r r r r<)r'r(r)r*rr+rrr%rIjitexportrrrr,r!r!rr"r sF%    *92r )ratypingrrrrItorch.nn.functionalrrrr__all__Modulerrr r r r!r!r!r"s #+!G