o i7@sddlZddlmZddlmZddlmZddlmZddl m Z Gdddej Z Gd d d ej Z Gd d d e Zd dZdddZGdddej ZGdddej ZGdddeZddZGdddej ZdS)N)List)RotaryPositionalEmbeddings) ResidualFSQTransformerBlockc sLeZdZdZ d dedededeffdd Zd ejd ejfd d Z Z S)ISTFTa Custom implementation of ISTFT since torch.istft doesn't allow custom padding (other than `center=True`) with windowing. This is because the NOLA (Nonzero Overlap Add) check fails at the edges. See issue: https://github.com/pytorch/pytorch/issues/62323 Specifically, in the context of neural vocoding we are interested in "same" padding analogous to CNNs. The NOLA constraint is met as we trim padded samples anyway. Args: n_fft (int): Size of Fourier transform. hop_length (int): The distance between neighboring sliding window frames. win_length (int): The size of window frame and STFT filter. padding (str, optional): Type of padding. Options are "center" or "same". Defaults to "same". samen_fft hop_length win_lengthpaddingcsLt|dvr td||_||_||_||_t|}| d|dS)N)centerr #Padding must be 'center' or 'same'.window) super__init__ ValueErrorr r r r torch hann_windowregister_buffer)selfr r r r r __class__P/home/ubuntu/.local/lib/python3.10/site-packages/neucodec/codec_decoder_vocos.pyrs  zISTFT.__init__specreturnc CsV|jdkrtj||j|j|j|jddS|jdkr"|j|jd}ntd|dks0Jd|j \}}}tj j ||jd d d }||jd d d d f}|d |j|j}tj j j|d |fd |jfd |jfd d d dd|| f}|jd |dd d} tj j j| d |fd |jfd |jfd || } | dksJ|| }|S)a Compute the Inverse Short Time Fourier Transform (ISTFT) of a complex spectrogram. Args: spec (Tensor): Input complex spectrogram of shape (B, N, T), where B is the batch size, N is the number of frequency bins, and T is the number of time frames. Returns: Tensor: Reconstructed time-domain signal of shape (B, L), where L is the length of the output signal. rT)rr rzExpected a 3D tensor as inputrbackward)dimnormN) output_size kernel_sizestridergdy=)r ristftr r r rrr!shapefftirfftnn functionalfoldsquareexpand transposesqueezeall) rrpadBNTifftr#y window_sqwindow_enveloperrrforward'sJ    z ISTFT.forwardr __name__ __module__ __qualname____doc__intstrrrTensorr; __classcell__rrrrr s rc@&eZdZdZdejdejfddZdS) FourierHeadz'Base class for inverse fourier modules.xrcCtd)aJ Args: x (Tensor): Input tensor of shape (B, L, H), where B is the batch size, L is the sequence length, and H denotes the model dimension. Returns: Tensor: Reconstructed time-domain audio signal of shape (B, T), where T is the length of the output signal. -Subclasses must implement the forward method.NotImplementedErrorrrHrrrr;ds zFourierHead.forwardNr>r?r@rArrDr;rrrrrGarGc sJeZdZdZd dedededeffdd Zd ejd ejfd d Z Z S) ISTFTHeada ISTFT Head module for predicting STFT complex coefficients. Args: dim (int): Hidden dimension of the model. n_fft (int): Size of Fourier transform. hop_length (int): The distance between neighboring sliding window frames, which should align with the resolution of the input features. padding (str, optional): Type of padding. Options are "center" or "same". Defaults to "same". r r!r r r cs8t|d}tj|||_t||||d|_dS)Nr)r r r r )rrrr+Linearoutrr')rr!r r r out_dimrrrr|s  zISTFTHead.__init__rHrcCs|||}|dd}|jddd\}}t|}tj|dd}t|}t|}||d|}||}| d|fS)ay Forward pass of the ISTFTHead module. Args: x (Tensor): Input tensor of shape (B, L, H), where B is the batch size, L is the sequence length, and H denotes the model dimension. Returns: Tensor: Reconstructed time-domain audio signal of shape (B, T), where T is the length of the output signal. rrr!gY@)maxy?) rRr0chunkrexpclipcossinr' unsqueeze)rrHx_predmagpr8Saudiorrrr;s     zISTFTHead.forwardr<r=rrrrrPps rPcCs|t|SN)rsigmoid)rHrrr nonlinearitysrc cCstjj||dddS)Nư>T) num_groups num_channelsepsaffine)rr+ GroupNorm) in_channelsrfrrr Normalizesrlcs0eZdZddddfdd Zd ddZZS) ResnetBlockNFi) out_channels conv_shortcut temb_channelscst||_|dur|n|}||_||_t||_tjj ||dddd|_ |dkr3tj |||_ t||_ tj||_tjj ||dddd|_|j|jkrp|jrbtjj ||dddd|_dStjj ||dddd|_dSdS)Nrr)r$r%r r)rrrkrnuse_conv_shortcutrlnorm1rr+Conv1dconv1rQ temb_projnorm2Dropoutdropoutconv2ro nin_shortcut)rrkrnrorxrprrrrs2          zResnetBlock.__init__cCs|}||}t|}||}|dur'||t|ddddddf}||}t|}||}||}|j|jkrQ|j rL| |}||S| |}||Sra) rrrcrtrurvrxryrkrnrqrorz)rrHtembhrrrr;s   &      zResnetBlock.forwardra)r>r?r@rr;rErrrrrms $rmc@rF)BackbonezeBase class for the generator's backbone. It preserves the same temporal resolution across all layers.rHrcKrI)ai Args: x (Tensor): Input tensor of shape (B, C, L), where B is the batch size, C denotes output features, and L is the sequence length. Returns: Tensor: Output of shape (B, L, H), where B is the batch size, L is the sequence length, and H denotes the model dimension. rJrK)rrHkwargsrrrr;s zBackbone.forwardNrNrrrrr}rOr}cs8eZdZdZd fdd Zdejd ejfd d ZZS) VocosBackbonea Vocos backbone module built with ConvNeXt blocks. Supports additional conditioning with Adaptive Layer Normalization Args: input_channels (int): Number of input features channels. dim (int): Hidden dimension of the model. intermediate_dim (int): Intermediate dimension used in ConvNeXtBlock. num_layers (int): Number of ConvNeXtBlock layers. layer_scale_init_value (float, optional): Initial value for layer scaling. Defaults to `1 / num_layers`. adanorm_num_embeddings (int, optional): Number of embeddings for AdaLayerNorm. None means non-conditional model. Defaults to None.  @c sttjddd|_d|_}d}t|||j|dt|||j|dg}tj||_|}t |dfdd t |D}tj||_ tj d d |_ t|||j|dt|||j|dg} tj| |_dS) Nr)r$r rg?)rkrnrprxrTcsg|] }tdqS))r!n_heads rotary_embedr).0_heads hidden_dimtime_rotary_embedrr %s z*VocosBackbone.__init__..re)rh)rrr+rsembedtemb_chrm Sequential prior_netrrange transformers LayerNormfinal_layer_normpost_net) rrdepthr pos_meb_dimblock_inrxrtransformer_blocksrrrrr sP    zVocosBackbone.__init__rHrcCsf|dd}||}||}|dd}||}|dd}||}|dd}||}|S)Nrr)r0rrrrrrMrrrr;>s         zVocosBackbone.forward)rrrr) r>r?r@rArrrDr;rErrrrrs 5rcCs6t|tjrtjj|jddtj|jddSdS)Ng{Gz?)stdr) isinstancer+rsinit trunc_normal_weight constant_biasmrrr init_weightsKs rcseZdZ         d fd d Zd!ddZddZddZddZddZddZ ddZ ddZ ddZ Z S)"CodecDecoderVocosrrrr@r?F@c s\t||_t|gddd|_t||||d|_t||jd|jdd|_| dS)N)rrrrrrrr)r!levelsnum_quantizers)rrrrrr )r!r r r ) rrr r quantizerrbackbonerPheadreset_parameters) rrrrrr vq_num_quantizersvq_dimvq_commit_weightvq_weight_initvq_full_commit_loss codebook_size codebook_dimrrrrRs   zCodecDecoderVocos.__init__TcCsj|dur%|ddd}||\}}|ddd}|ddd}||dfS||}||\}}||fS)NTrrr)permuterrr)rrHvqqrrrrr;us  zCodecDecoderVocos.forwardcCs|j|_|j|}|Sra)revalvq2embrrrHrrrrs  zCodecDecoderVocos.vq2embcCs|j|_|j}|Sra)rrget_emb)rembsrrrrs  zCodecDecoderVocos.get_embcCs$|dddddf}||}|Sramodelrrrr inference_vqs zCodecDecoderVocos.inference_vqcCs$||\}}}}||}|dfSra)rr)rrHrlossperprrr inference_0s zCodecDecoderVocos.inference_0cCs||}|dfSrarrMrrr inferences zCodecDecoderVocos.inferencecCdd}||dS)z:Remove weight normalization module from all of the layers.cSs*z tjj|WdStyYdSwra)rr+utilsremove_weight_normrrrrr_remove_weight_norms  zACodecDecoderVocos.remove_weight_norm.._remove_weight_normNapply)rrrrrrsz$CodecDecoderVocos.remove_weight_normcCr)z9Apply weight normalization module from all of the layers.cSs.t|tjs t|tjrtjj|dSdSra)rr+rsConvTranspose1drr weight_normrrrr_apply_weight_normsz?CodecDecoderVocos.apply_weight_norm.._apply_weight_normNr)rrrrrapply_weight_normsz#CodecDecoderVocos.apply_weight_normcCs|tdSra)rr)rrrrrsz"CodecDecoderVocos.reset_parameters) rrrrrrrrFFrr)T)r>r?r@rr;rrrrrrrrrErrrrrQs. #   r)rd)rtorch.nnr+typingrtorchtune.modulesrvector_quantize_pytorchr bs_roformer5rModulerrGrPrcrlrmr}rrrrrrrs      V4 <P