o ̳iV@sddlZddlZddlZddlmZddlmZddlm Z m Z m Z ddl Tddl mZddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZmZGd ddejZGdddejZGdddeZddZd,ddZGdddejZ GdddejZ!d-ddZ"Gdd d ejZ#Gd!d"d"e#Z$d#d$Z%Gd%d&d&ejZ&Gd'd(d(ejZ'd)d*Z(e)d+kre(dSdS).N) ResidualVQ)WNConv1d DecoderBlockResLSTM)*) activations)Optional) ConvNeXtBlock AdaLayerNormTransformerBlock)RotaryPositionalEmbeddings) ResidualFSQ)Module ModuleListc JeZdZdZd dedededeffdd Zd ejd ejfd d Z Z S)ISTFTa Custom implementation of ISTFT since torch.istft doesn't allow custom padding (other than `center=True`) with windowing. This is because the NOLA (Nonzero Overlap Add) check fails at the edges. See issue: https://github.com/pytorch/pytorch/issues/62323 Specifically, in the context of neural vocoding we are interested in "same" padding analogous to CNNs. The NOLA constraint is met as we trim padded samples anyway. Args: n_fft (int): Size of Fourier transform. hop_length (int): The distance between neighboring sliding window frames. win_length (int): The size of window frame and STFT filter. padding (str, optional): Type of padding. Options are "center" or "same". Defaults to "same". samen_fft hop_length win_lengthpaddingcsLt|dvr td||_||_||_||_t|}| d|dS)N)centerr#Padding must be 'center' or 'same'.window) super__init__ ValueErrorrrrrtorch hann_windowregister_buffer)selfrrrrr __class__R/home/ubuntu/.local/lib/python3.10/site-packages/xcodec2/vq/codec_decoder_vocos.pyr s  zISTFT.__init__specreturnc CsV|jdkrtj||j|j|j|jddS|jdkr"|j|jd}ntd|dks0Jd|j \}}}tj j ||jd d d }||jd d d d f}|d |j|j}tj j j|d |fd |jfd |jfd d d dd|| f}|jd |dd d} tj j j| d |fd |jfd |jfd || } | dksJ|| }|S)a Compute the Inverse Short Time Fourier Transform (ISTFT) of a complex spectrogram. Args: spec (Tensor): Input complex spectrogram of shape (B, N, T), where B is the batch size, N is the number of frequency bins, and T is the number of time frames. Returns: Tensor: Reconstructed time-domain signal of shape (B, L), where L is the length of the output signal. rT)rrrzExpected a 3D tensor as inputrbackward)dimnormN) output_size kernel_sizestridergdy=)rristftrrrrrr,shapefftirfftnn functionalfoldsquareexpand transposesqueezeall) r"r'padBNTifftr.y window_sqwindow_enveloper%r%r&forward+s0    z ISTFT.forwardr __name__ __module__ __qualname____doc__intstrrrTensorrF __classcell__r%r%r#r&rs  rc@&eZdZdZdejdejfddZdS) FourierHeadz'Base class for inverse fourier modules.xr(cCtd)aJ Args: x (Tensor): Input tensor of shape (B, L, H), where B is the batch size, L is the sequence length, and H denotes the model dimension. Returns: Tensor: Reconstructed time-domain audio signal of shape (B, T), where T is the length of the output signal. -Subclasses must implement the forward method.NotImplementedErrorr"rSr%r%r&rF\s zFourierHead.forwardNrIrJrKrLrrOrFr%r%r%r&rRYrRc r) ISTFTHeada ISTFT Head module for predicting STFT complex coefficients. Args: dim (int): Hidden dimension of the model. n_fft (int): Size of Fourier transform. hop_length (int): The distance between neighboring sliding window frames, which should align with the resolution of the input features. padding (str, optional): Type of padding. Options are "center" or "same". Defaults to "same". rr,rrrcs8t|d}tj|||_t||||d|_dS)Nr))rrrr)rrrr6Linearoutrr2)r"r,rrrout_dimr#r%r&rts zISTFTHead.__init__rSr(cCs|||}|dd}|jddd\}}t|}tj|dd}t|}t|}||d|}||}| d|fS)ay Forward pass of the ISTFTHead module. Args: x (Tensor): Input tensor of shape (B, L, H), where B is the batch size, L is the sequence length, and H denotes the model dimension. Returns: Tensor: Reconstructed time-domain audio signal of shape (B, T), where T is the length of the output signal. rr)r,gY@)maxy?) r]r;chunkrexpclipcossinr2 unsqueeze)r"rSx_predmagprCSaudior%r%r&rFzs     zISTFTHead.forwardrGrHr%r%r#r&r[hs r[cCs|t|SN)rsigmoid)rSr%r%r& nonlinearitysrn cCstjj||dddS)Nư>T) num_groups num_channelsepsaffine)rr6 GroupNorm) in_channelsrqr%r%r& Normalizesrwcs0eZdZddddfdd Zd ddZZS) ResnetBlockNFi) out_channels conv_shortcut temb_channelscst||_|dur|n|}||_||_t||_tjj ||dddd|_ |dkr3tj |||_ t||_ tj||_tjj ||dddd|_|j|jkrp|jrbtjj ||dddd|_dStjj ||dddd|_dSdS)Nr*rr/r0rr)rrrvryuse_conv_shortcutrwnorm1rr6Conv1dconv1r\ temb_projnorm2Dropoutdropoutconv2rz nin_shortcut)r"rvryrzrr{r#r%r&rsN      zResnetBlock.__init__cCs|}||}t|}||}|dur'||t|ddddddf}||}t|}||}||}|j|jkrQ|j rL| |}||S| |}||Srl) r~rnrrrrrrvryr}rzr)r"rStembhr%r%r&rFs   &      zResnetBlock.forwardrlrIrJrKrrFrPr%r%r#r&rxs &rxcs$eZdZfddZddZZS) AttnBlockcs~t||_t||_tjj||dddd|_tjj||dddd|_ tjj||dddd|_ tjj||dddd|_ dS)Nrrr|) rrrvrwr-rr6rqkvproj_out)r"rvr#r%r&rs6   zAttnBlock.__init__c Cs|}||}||}||}||}|j\}}}|ddd}t||} | t|d} tj j j | dd} | ddd} t|| }| |}||S)Nrr)rgr_) r-rrrr3permuterbmmrMr6r7softmaxr) r"rSh_rrrbcrw_r%r%r&rFs        zAttnBlock.forwardrr%r%r#r&rs rvanillacCsB|dvs Jd|dtd|d|d|dkrt|SdS)N)rlinearnonez attn_type z unknownzmaking attention of type 'z' with z in_channelsr)printr)rv attn_typer%r%r& make_attns rc@rQ)BackbonezeBase class for the generator's backbone. It preserves the same temporal resolution across all layers.rSr(cKrT)ai Args: x (Tensor): Input tensor of shape (B, C, L), where B is the batch size, C denotes output features, and L is the sequence length. Returns: Tensor: Output of shape (B, L, H), where B is the batch size, L is the sequence length, and H denotes the model dimension. rUrV)r"rSkwargsr%r%r&rFs zBackbone.forwardNrYr%r%r%r&rrZrcs:eZdZdZ d fdd Zdejd ejfd d ZZS) VocosBackbonea Vocos backbone module built with ConvNeXt blocks. Supports additional conditioning with Adaptive Layer Normalization Args: input_channels (int): Number of input features channels. dim (int): Hidden dimension of the model. intermediate_dim (int): Intermediate dimension used in ConvNeXtBlock. num_layers (int): Number of ConvNeXtBlock layers. layer_scale_init_value (float, optional): Initial value for layer scaling. Defaults to `1 / num_layers`. adanorm_num_embeddings (int, optional): Number of embeddings for AdaLayerNorm. None means non-conditional model. Defaults to None.  @c sttjddd|_d|_}d}t|||j|dt|||j|dg}tj||_|}t |dfdd t |D}tj||_ tj d d |_ t|||j|dt|||j|dg} tj| |_dS) Nr*)r/rrg?)rvryr{rr_csg|] }tdqS))r,n_heads rotary_embedr ).0_heads hidden_dimtime_rotary_embedr%r& Ls z*VocosBackbone.__init__..rp)rs)rrr6rembedtemb_chrx Sequential prior_netrrange transformers LayerNormfinal_layer_normpost_net) r"rdepthr pos_meb_dimblock_inrrtransformer_blocksrr#rr&r4s8    zVocosBackbone.__init__rSr(cCsf|dd}||}||}|dd}||}|dd}||}|dd}||}|S)Nrr))r;rrrrrrXr%r%r&rF\s         zVocosBackbone.forward)rrrr) rIrJrKrLrrrOrFrPr%r%r#r&r&s (rcCs6t|tjrtjj|jddtj|jddSdS)Ng{Gz?)stdr) isinstancer6rinit trunc_normal_weight constant_biasmr%r%r& init_weightsns rcseZdZ         d fd d Zd!ddZddZddZddZddZddZ ddZ ddZ ddZ Z S)"CodecDecoderVocosrrrr@r?F@c s\t||_t|gddd|_t||||d|_t||jd|jdd|_| dS)N)rrrrrrrr)r,levelsnum_quantizersrrrrrrr,rrr) rrrr quantizerrbackboner[headreset_parameters r"rrrrrvq_num_quantizersvq_dimvq_commit_weightvq_weight_initvq_full_commit_loss codebook_size codebook_dimr#r%r&rts  zCodecDecoderVocos.__init__TcCsj|dur%|ddd}||\}}|ddd}|ddd}||dfS||}||\}}||fS)NTrr)r)rrrr)r"rSvqrrr%r%r&rFs  zCodecDecoderVocos.forwardcC|j|_|j|}|Srlrevalvq2embr"rrSr%r%r&r  zCodecDecoderVocos.vq2embcC|j|_|j}|Srlrrget_embr"embsr%r%r&r  zCodecDecoderVocos.get_embcC$|dddddf}||}|Srlmodelrr%r%r& inference_vq zCodecDecoderVocos.inference_vqcC$||\}}}}||}|dfSrlrrr"rSrlossperpr%r%r& inference_0 zCodecDecoderVocos.inference_0cC||}|dfSrlrrXr%r%r& inference zCodecDecoderVocos.inferencecCdd}||dS):Remove weight normalization module from all of the layers.cS*z tjj|WdStyYdSwrlrr6utilsremove_weight_normrrr%r%r&_remove_weight_norm  zACodecDecoderVocos.remove_weight_norm.._remove_weight_normNapplyr"rr%r%r&rz$CodecDecoderVocos.remove_weight_normcCr)9Apply weight normalization module from all of the layers.cS.t|tjs t|tjrtjj|dSdSrlrr6rConvTranspose1drr weight_normrr%r%r&_apply_weight_normz?CodecDecoderVocos.apply_weight_norm.._apply_weight_normNrr"rr%r%r&apply_weight_normz#CodecDecoderVocos.apply_weight_normcC|tdSrlrrr"r%r%r&rz"CodecDecoderVocos.reset_parameters) rrrrrrrrFFrrTrIrJrKrrFrrrrrrr rrPr%r%r#r&rss. )   rcseZdZ           dfd d Zd d dZddZddZddZddZddZ ddZ ddZ ddZ Z S)!CodecDecoderVocos_transposerrrrrrrFrc st||_t||| | d|| | d|_t||||d|_tt tj ||dddddt tj ||ddd|_ t ||jd|jd d |_ |dS) Nr))rr,rrthreshold_ema_dead_code commitment weight_initfull_commit_lossrr*r)rvryr/r0routput_padding)rvryr/rrrr)rrrrrrrr6rGELUrinverse_mel_convr[rrrr#r%r&rsB   z$CodecDecoderVocos_transpose.__init__TcCsB|dur||\}}}|||fS||}||\}}||fS)NT)rrr)r"rSrr commit_lossrr%r%r&rFs   z#CodecDecoderVocos_transpose.forwardcCrrlrrr%r%r&rrz"CodecDecoderVocos_transpose.vq2embcCrrlrrr%r%r&r"rz#CodecDecoderVocos_transpose.get_embcCrrlrrr%r%r&r'rz(CodecDecoderVocos_transpose.inference_vqcCrrlrrr%r%r&r,rz'CodecDecoderVocos_transpose.inference_0cCrrlrrXr%r%r&r1rz%CodecDecoderVocos_transpose.inferencecCr)rcSrrlrrr%r%r&r9rzKCodecDecoderVocos_transpose.remove_weight_norm.._remove_weight_normNrrr%r%r&r6rz.CodecDecoderVocos_transpose.remove_weight_normcCr)rcSrrlrrr%r%r&rDrzICodecDecoderVocos_transpose.apply_weight_norm.._apply_weight_normNrr r%r%r&r Ar z-CodecDecoderVocos_transpose.apply_weight_normcCr rlr rr%r%r&rJrz,CodecDecoderVocos_transpose.reset_parameters) rrrrrrrrFFrrrrr%r%r#r&rs. 7   rcCsttjr dnd}td|t|}tdd}d}d}t||||}td|j| t ||d d }td td |jd }WddS1s]wYdS)NcudacpuzUsing device: zModel initialized.r)r2zDummy input shape: F)rz Forward pass without VQ:zOutput shape: r) rdevicer is_availablerrtorandnr3rno_grad)rr batch_sizervsequence_length dummy_input output_no_vqrr%r%r&mainPs    "r'__main__)ro)r)*sysnumpynprtorch.nnr6 residual_vqrmodulerrralias_free_torchrtypingr r r bs_roformer5r torchtune.modulesrvector_quantize_pytorchrrrrrRr[rnrwrxrrrrrrrr'rIr%r%r%r&s>       H0 = 1Hit-