o ii@sdZddlZddlZddlmZmZmZmZddlZ ddl Z ddl m m ZddlmZGddde j jZGddde j jZGd d d e j jZGd d d e j jZGd dde j jZGddde j jZdS)z_HiFi-GAN Modules. This code is modified from https://github.com/kan-bayashi/ParallelWaveGAN. N)AnyDictListOptional) ResidualBlockcseZdZdZdddddgdgdgd gd gd gd gd d d d did fdedededededeedeedeedeeededededeee fdeffdd Z d-d e j d!e e j d"e j fd#d$Zd%d&Zd'd(Zd)d*Z d-d e j d!e e j d"e j fd+d,ZZS).HiFiGANGeneratorzHiFiGAN generator module.Pi)r r )rr)r )r rT LeakyReLUnegative_slope皙? in_channels out_channelschannelsglobal_channels kernel_sizeupsample_scalesupsample_kernel_sizesresblock_kernel_sizesresblock_dilationsuse_additional_convsbiasnonlinear_activationnonlinear_activation_paramsuse_weight_normcst|ddksJdt|t|ksJt| t|ks#Jtt|||_t||_t||_t j j |||d|ddd|_ t j |_t j |_tt|D]n}||d||ksgJ|jt j tt j | d i| t j j|d||d|d||||||d||d||ddg7_tt|D]}|jt|||d|d| || | | | dg7_qqYt j t j t j j |d|d||d|dddt j |_|dkrt j ||d|_|r||dS) aInitialize HiFiGANGenerator module. Args: in_channels (int): Number of input channels. out_channels (int): Number of output channels. channels (int): Number of hidden representation channels. global_channels (int): Number of global conditioning channels. kernel_size (int): Kernel size of initial and final conv layer. upsample_scales (List[int]): List of upsampling scales. upsample_kernel_sizes (List[int]): List of kernel sizes for upsample layers. resblock_kernel_sizes (List[int]): List of kernel sizes for residual blocks. resblock_dilations (List[List[int]]): List of list of dilations for residual blocks. use_additional_convs (bool): Whether to use additional conv layers in residual blocks. bias (bool): Whether to add bias parameter in convolution layers. nonlinear_activation (str): Activation function module name. nonlinear_activation_params (Dict[str, Any]): Hyperparameters for activation function. use_weight_norm (bool): Whether to use weight norm. If set to true, it will be applied to all of the conv layers. r r Kernel size must be odd number.padding)r&output_padding)rr dilationsr rr!r"rN)super__init__lenintnpprodupsample_factor num_upsamples num_blockstorchnnConv1d input_conv ModuleList upsamplesblocksrange SequentialgetattrConvTranspose1drrTanh output_conv global_convapply_weight_normreset_parameters)selfrrrrrrrrrrr r!r"r#ij __class__r)S/home/ubuntu/.local/lib/python3.10/site-packages/espnet2/gan_tts/hifigan/hifigan.pyr+s| (            zHiFiGANGenerator.__init__NcgreturncCs||}|dur|||}t|jD]&}|j||}d}t|jD]}||j||j||7}q%||j}q||}|S)aCalculate forward propagation. Args: c (Tensor): Input tensor (B, in_channels, T). g (Optional[Tensor]): Global conditioning tensor (B, global_channels, 1). Returns: Tensor: Output tensor (B, out_channels, T). N)r6r@r:r1r8r2r9r?)rCrIrJrDcsrEr)r)rHforwards   zHiFiGANGenerator.forwardcC dtjjfdd}||dS)zReset parameters. This initialization follows the official implementation manner. https://github.com/jik876/hifi-gan/blob/master/models.py mcSs@t|tjjtjjfr|jjddt d|ddSdS)NrLg{Gz?zReset parameters in .) isinstancer3r4r5r=weightdatanormal_loggingdebugrPr)r)rH_reset_parameterssz._reset_parametersNr3r4Moduleapply)rCrYr)r)rHrBsz!HiFiGANGenerator.reset_parameterscCrO)z:Remove weight normalization module from all of the layers.rPcSs<ztd|dtjj|WdStyYdSw)NzWeight norm is removed from rQ)rVrWr3r4utilsremove_weight_norm ValueErrorrXr)r)rH_remove_weight_norms  z@HiFiGANGenerator.remove_weight_norm.._remove_weight_normNrZ)rCr`r)r)rHr^z#HiFiGANGenerator.remove_weight_normcCrO)9Apply weight normalization module from all of the layers.rPcSsDt|tjjst|tjjr tjj|td|ddSdSNzWeight norm is applied to rQ) rRr3r4r5r=r] weight_normrVrWrXr)r)rH_apply_weight_norms z>HiFiGANGenerator.apply_weight_norm.._apply_weight_normNrZrCrer)r)rHrAraz"HiFiGANGenerator.apply_weight_normcCs@|dur |d}|j|ddd|d}|dddS)aPerform inference. Args: c (torch.Tensor): Input tensor (T, in_channels). g (Optional[Tensor]): Global conditioning tensor (global_channels, 1). Returns: Tensor: Output tensor (T ** upsample_factor, out_channels). Nrr )rJ) unsqueezerN transposesqueeze)rCrIrJr)r)rH inferences zHiFiGANGenerator.inference)N)__name__ __module__ __qualname____doc__r-rboolstrrrr+r3TensorrrNrBr^rArj __classcell__r)r)rFrHrs      q   rcseZdZdZdddddgdgdddd d d idd f d edededeededeededededeee fdedeffdd Z de j de j fddZ dd Zd!d"ZZS)#HiFiGANPeriodDiscriminatorz$HiFiGAN period discriminator module.r rr rrrrr TrrrFrrperiod kernel_sizesrdownsample_scalesmax_downsample_channelsr r!r"r#use_spectral_normc  s>tt|dks J|dddksJd|dddks%Jd||_tj|_|} |}|D]8}|jtjtjj | ||ddf|df|ddddfdt tj| d i| g7_|} t |d|}q4tjj |||dddfd|ddddfd|_ | r| rt d| r|| r|dSdS) aInitialize HiFiGANPeriodDiscriminator module. Args: in_channels (int): Number of input channels. out_channels (int): Number of output channels. period (int): Period. kernel_sizes (list): Kernel sizes of initial conv layers and the final conv layer. channels (int): Number of initial channels. downsample_scales (List[int]): List of downsampling scales. max_downsample_channels (int): Number of maximum downsampling channels. use_additional_convs (bool): Whether to use additional conv layers in residual blocks. bias (bool): Whether to add bias parameter in convolution layers. nonlinear_activation (str): Activation function module name. nonlinear_activation_params (Dict[str, Any]): Hyperparameters for activation function. use_weight_norm (bool): Whether to use weight norm. If set to true, it will be applied to all of the conv layers. use_spectral_norm (bool): Whether to use spectral norm. If set to true, it will be applied to all of the conv layers. r rr r$r%r0Either use use_weight_norm or use_spectral_norm.Nr))r*r+r,rwr3r4r7convsr;Conv2dr<minr?r_rAapply_spectral_norm)rCrrrwrxrryrzr r!r"r#r{in_chsout_chsdownsample_scalerFr)rHr+sN &   z#HiFiGANPeriodDiscriminator.__init__xrKcCs|j\}}}||jdkr"|j||j}t|d|fd}||7}|||||j|j}g}|jD] }||}||g7}q3||}t|dd}||g7}|S)zCalculate forward propagation. Args: c (Tensor): Input tensor (B, in_channels, T). Returns: list: List of each layer's tensors. rreflectr r ) shaperwFpadviewr}r?r3flatten)rCrbrItn_padoutslayerr)r)rHrN3s     z"HiFiGANPeriodDiscriminator.forwardcCrO)rbrPcS6t|tjjrtjj|td|ddSdSrcrRr3r4r~r]rdrVrWrXr)r)rHreSzHHiFiGANPeriodDiscriminator.apply_weight_norm.._apply_weight_normNrZrfr)r)rHrAPz,HiFiGANPeriodDiscriminator.apply_weight_normcCrO);Apply spectral normalization module from all of the layers.rPcSrNzSpectral norm is applied to rQrRr3r4r~r] spectral_normrVrWrXr)r)rH_apply_spectral_norm]rzLHiFiGANPeriodDiscriminator.apply_spectral_norm.._apply_spectral_normNrZrCrr)r)rHrZrz.HiFiGANPeriodDiscriminator.apply_spectral_norm)rkrlrmrnr-rrorprrr+r3rqrNrArrrr)r)rFrHrssT     T rsc szeZdZdZgdddddgdgddd d d d id d d fdeedeeefffdd Z de j de j fddZ Z S)HiFiGANMultiPeriodDiscriminatorz*HiFiGAN multi-period discriminator module.r rrr rr rrrtrurvTrrrF rrrxrryrzr r!r"r#r{periodsdiscriminator_paramscsPttj|_|D]}t|}||d<|jtdi|g7_q dS)aInitialize HiFiGANMultiPeriodDiscriminator module. Args: periods (List[int]): List of periods. discriminator_params (Dict[str, Any]): Parameters for hifi-gan period discriminator module. The period parameter will be overwritten. rwNr)) r*r+r3r4r7discriminatorscopydeepcopyrs)rCrrrwparamsrFr)rHr+hs   z(HiFiGANMultiPeriodDiscriminator.__init__rrKcCs"g}|jD] }|||g7}q|S)zCalculate forward propagation. Args: x (Tensor): Input noise signal (B, 1, T). Returns: List: List of list of each discriminator outputs, which consists of each layer output tensors. )rrCrrfr)r)rHrNs z'HiFiGANMultiPeriodDiscriminator.forward)rkrlrmrnrr-rrprr+r3rqrNrrr)r)rFrHres*  rcseZdZdZddgdddddgdd d d idd f d ededeedededededeededeeefde de ffdd Z de j dee j fddZ dd Zd!d"ZZS)#HiFiGANScaleDiscriminatorz$HiFi-GAN scale discriminator module.r )rrrvrTr r rrr rrrFrrrxrrz max_groupsr ryr!r"r#r{c sttj|_t|dksJ|D] } | ddksJq|jtjtjj|||d||ddddt tj| d i| g7_|}|}d}|D];}|jtjtjj|||d||ddd||dt tj| d i| g7_|}t |d|}t |d|}qOt |d|}|jtjtjj|||dd|ddd|dt tj| d i| g7_|jtjj|||dd|ddd|dg7_| r| rt d | r| | r| d Sd S) aInitilize HiFiGAN scale discriminator module. Args: in_channels (int): Number of input channels. out_channels (int): Number of output channels. kernel_sizes (List[int]): List of four kernel sizes. The first will be used for the first conv layer, and the second is for downsampling part, and the remaining two are for the last two output layers. channels (int): Initial number of channels for conv layer. max_downsample_channels (int): Maximum number of channels for downsampling layers. bias (bool): Whether to add bias parameter in convolution layers. downsample_scales (List[int]): List of downsampling scales. nonlinear_activation (str): Activation function module name. nonlinear_activation_params (Dict[str, Any]): Hyperparameters for activation function. use_weight_norm (bool): Whether to use weight norm. If set to true, it will be applied to all of the conv layers. use_spectral_norm (bool): Whether to use spectral norm. If set to true, it will be applied to all of the conv layers. rr r r)r r&)rstrider&groupsr )rrr&r rr|Nr))r*r+r3r4r7layersr,r;r5r<rr_rAr)rCrrrxrrzrr ryr!r"r#r{ksrrrrrFr)rHr+s %     z"HiFiGANScaleDiscriminator.__init__rrKcCs&g}|jD] }||}||g7}q|S)zCalculate forward propagation. Args: x (Tensor): Input noise signal (B, 1, T). Returns: List[Tensor]: List of output tensors of each layer. )rrr)r)rHrNs   z!HiFiGANScaleDiscriminator.forwardcCrO)rbrPcSrrcrrXr)r)rHre.rzGHiFiGANScaleDiscriminator.apply_weight_norm.._apply_weight_normNrZrfr)r)rHrA+rz+HiFiGANScaleDiscriminator.apply_weight_normcCrO)rrPcSrrrrXr)r)rHr8rzKHiFiGANScaleDiscriminator.apply_spectral_norm.._apply_spectral_normNrZrr)r)rHr5rz-HiFiGANScaleDiscriminator.apply_spectral_norm)rkrlrmrnr-rrprrror+r3rqrNrArrrr)r)rFrHrsT     } rcseZdZdZddddddddgdd d d d gd dddid dfdededeeefdeeefdef fdd Z de j de e e j fddZ ZS)HiFiGANMultiScaleDiscriminatorz*HiFi-GAN multi-scale discriminator module.r AvgPool1drr rrr&r rrrvrTrrrr rrrxrrzrr ryr!r"Fscalesdownsample_poolingdownsample_pooling_paramsrfollow_official_normcsttj|_t|D]+}t|}|r-|dkr%d|d<d|d<nd|d<d|d<|jt di|g7_qd|_ |dkrPt tj|di||_ dSdS) aInitilize HiFiGAN multi-scale discriminator module. Args: scales (int): Number of multi-scales. downsample_pooling (str): Pooling module name for downsampling of the inputs. downsample_pooling_params (Dict[str, Any]): Parameters for the above pooling module. discriminator_params (Dict[str, Any]): Parameters for hifi-gan scale discriminator module. follow_official_norm (bool): Whether to follow the norm setting of the official implementaion. The first discriminator uses spectral norm and the other discriminators use weight norm. rFr#Tr{Nr r)) r*r+r3r4r7rr:rrrpoolingr<)rCrrrrrrDrrFr)rHr+Cs" '     z'HiFiGANMultiScaleDiscriminator.__init__rrKcCs6g}|jD]}|||g7}|jdur||}q|S)a Calculate forward propagation. Args: x (Tensor): Input noise signal (B, 1, T). Returns: List[List[torch.Tensor]]: List of list of each discriminator outputs, which consists of eachlayer output tensors. N)rrrr)r)rHrN~s   z&HiFiGANMultiScaleDiscriminator.forward)rkrlrmrnr-rprrror+r3rqrrNrrr)r)rFrHr@s@   &;rcseZdZdZddddddddgdd d d d gd dddid d gdddddgdgdd d dddid dd fdededeeefdeeefdede edeeefffdd Z d!e j d"e e e j fd#d$Z ZS)%)HiFiGANMultiScaleMultiPeriodDiscriminatorz9HiFi-GAN multi-scale + multi-period discriminator module.rrrr rr rrrvrTrrrrrrrrtruFrrscale_downsample_poolingscale_downsample_pooling_paramsscale_discriminator_paramsrrperiod_discriminator_paramscs0tt|||||d|_t||d|_dS)aInitilize HiFiGAN multi-scale + multi-period discriminator module. Args: scales (int): Number of multi-scales. scale_downsample_pooling (str): Pooling module name for downsampling of the inputs. scale_downsample_pooling_params (dict): Parameters for the above pooling module. scale_discriminator_params (dict): Parameters for hifi-gan scale discriminator module. follow_official_norm (bool): Whether to follow the norm setting of the official implementaion. The first discriminator uses spectral norm and the other discriminators use weight norm. periods (list): List of periods. period_discriminator_params (dict): Parameters for hifi-gan period discriminator module. The period parameter will be overwritten. )rrrrr)rrN)r*r+rmsdrmpd)rCrrrrrrrrFr)rHr+s 9 z2HiFiGANMultiScaleMultiPeriodDiscriminator.__init__rrKcCs||}||}||S)aLCalculate forward propagation. Args: x (Tensor): Input noise signal (B, 1, T). Returns: List[List[Tensor]]: List of list of each discriminator outputs, which consists of each layer output tensors. Multi scale and multi period ones are concatenated. )rr)rCrmsd_outsmpd_outsr)r)rHrNs z1HiFiGANMultiScaleMultiPeriodDiscriminator.forward)rkrlrmrnr-rprrrorr+r3rqrNrrr)r)rFrHrsb    &Fr)rnrrVtypingrrrrnumpyr.r3torch.nn.functionalr4 functionalr&espnet2.gan_tts.hifigan.residual_blockrr[rrsrrrrr)r)r)rHs" H 5'R