o }oin@sddlmZddlZddlmZddlmZddlmZddl m Z m Z ddl m Z mZddlmZmZmZmZdd lmZdd lmZgd Zd d Z dinputmaskweightbiasmomentumepsreturnc Cs|d}||d|}||d|dd|}||dt|d|} | |dddfd|dddfd} | S)zApplies Masked Instance Normalization for each channel in each data sample in a batch. See :class:`~MaskedInstanceNorm1d` for details. ).NN)sumtorchsqrt) rrrrrr lengthsmeanvaroutrrrmasked_instance_norm&s ,r,c sZeZdZdZ    ddedededed ed d f fd d Zdeded efddZ Z S)MaskedInstanceNorm1daRApplies Instance Normalization over a masked 3D input (a mini-batch of 1D inputs with additional channel dimension).. See documentation of :class:`~torch.nn.InstanceNorm1d` for details. Shape: - Input: :math:`(N, C, L)` - Mask: :math:`(N, 1, L)` - Output: :math:`(N, C, L)` (same shape as input) r皙?F num_featuresr raffinetrack_running_statsr!Ncstt||||||dSN)superr-__init__)selfr/r rr0r1 __class__rrr4BszMaskedInstanceNorm1d.__init__rrcCst|||j|j|j|jSr2)r,rrrr )r5rrrrrforwardLszMaskedInstanceNorm1d.forward)rr.FF) __name__ __module__ __qualname____doc__intfloatboolr4rr8 __classcell__rrr6rr-6s( r-cs8eZdZUdZdgZeed<fddZddZZ S) PartialConv1da Zero padding creates a unique identifier for where the edge of the data is, such that the model can almost always identify exactly where it is relative to either edge given a sufficient receptive field. Partial padding goes to some lengths to remove this affect. slide_winsizecsXtt|j|i|tdd|jd}|jd|dd|jjd|jjd|_ dS)Nrweight_maskUpdaterF) persistentr$) r3rAr4r&ones kernel_sizeregister_bufferrDshaperB)r5argskwargsrDr6rrr4ZszPartialConv1d.__init__c Cs|durtjdd|jd|j|jd}n|}t||}t4tj||j d|j |j |j dd}t ||dk|j}|j|}t|dd}t||}Wdn1sWwY|||j|j}|jdur|jd|jd}t||||} t| |} | St||} | S)NrCr$)dtypedevice)rstridepaddingdilationgroupsr)r&rFrIrLrMmulno_gradFconv1drDrNrOrP masked_fillrBclamp _conv_forwardrrview out_channels) r5rmask_inr update_maskupdate_mask_filled mask_ratioraw_out bias_viewoutputrrrr8`s6       zPartialConv1d.forward) r9r:r;r< __constants__r>__annotations__r4r8r@rrr6rrAPs  rAcs&eZdZdfdd ZddZZS) LinearNormTlinearcsBttjj|||d|_tjjj|jjtjj |ddS)Nrgain) r3r4r&nnLinear linear_layerinitxavier_uniform_rcalculate_gain)r5in_dimout_dimr w_init_gainr6rrr4s $zLinearNorm.__init__cC ||Sr2)rk)r5xrrrr8 zLinearNorm.forward)Trer9r:r;r4r8r@rrr6rrdsrdcsJeZdZUdgZeed<         d fdd Zd d d ZZS) ConvNormuse_partial_paddingrCNTreFc stt||dur|ddksJt||dd}| |_tjj} | r(t} | |||||||d|_ tjj j |j j tjj |d| rOtjj|j |_ | dur\| |dd|_dSd|_dS)Nr$rC)rGrNrOrPrrgT)r0)r3rvr4r=rwr&riConv1drAconvrlrmrrnutils weight_normnorm) r5 in_channelsrZrGrNrOrPrrqrwuse_weight_normnorm_fnconv_fnr6rrr4s.  zConvNorm.__init__cCs|jr|||}|jdur|||}n|dur||}||}|jdur-||}|r>||dddd}|SNrCr$)rwryr|rRis_adapter_availableforward_enabled_adapters transpose)r5signalrretrrrr8s       zConvNorm.forward) rCrCNrCTreFFNr2) r9r:r;rbr?rcr4r8r@rrr6rrvs 'rvcs$eZdZfddZddZZS) LocationLayerc sHtt|dd}td|||dddd|_t||ddd|_dS)NrCr$F)rGrOrrNrPtanhrrq)r3r4r=rv location_convrdlocation_dense)r5attention_n_filtersattention_kernel_size attention_dimrOr6rrr4s  zLocationLayer.__init__cCs$||}|dd}||}|Sr)rrr)r5attention_weights_catprocessed_attentionrrrr8s   zLocationLayer.forwardrurrr6rrs rcs,eZdZfddZddZddZZS) Attentioncs\tt||ddd|_t||ddd|_t|ddd|_t||||_td |_ dS)NFrrrCrfinf) r3r4rd query_layer memory_layervrlocation_layerr>score_mask_value)r5attention_rnn_dim embedding_dimrattention_location_n_filtersattention_location_kernel_sizer6rrr4s zAttention.__init__cCs@||d}||}|t|||}|d}|S)aS PARAMS ------ query: decoder output (batch, n_mel_channels * n_frames_per_step) processed_memory: processed encoder outputs (B, T_in, attention_dim) attention_weights_cat: cumulative and prev. att weights (B, 2, max_time) RETURNS ------- alignment (batch, max_time) rCr")r unsqueezerrr&rsqueeze)r5queryprocessed_memoryrprocessed_queryprocessed_attention_weightsenergiesrrrget_alignment_energiess   z Attention.get_alignment_energiesc CsX||||}|dur|j||jtj|dd}t|d|}| d}||fS)a) PARAMS ------ attention_hidden_state: attention rnn last output memory: encoder outputs processed_memory: processed encoder outputs attention_weights_cat: previous and cummulative attention weights mask: binary mask for padded data NrCdim) rdata masked_fill_rrTsoftmaxr&bmmrr) r5attention_hidden_statememoryrrr alignmentattention_weightsattention_contextrrrr8s  zAttention.forward)r9r:r;r4rr8r@rrr6rrs rcs(eZdZdfdd ZdddZZS) Prenet?csDt|g|dd}||_tjddt||D|_dS)Nr"cSsg|] \}}t||ddqS)Frf)rd).0in_sizeout_sizerrr "sz#Prenet.__init__..)r3r4 p_dropoutr&ri ModuleListziplayers)r5rosizesrin_sizesr6rrr4s  zPrenet.__init__Fc Cs|rE|jD]=}t||}|dd}tjt|j |j  d|j }| | d| d}||dd|j }q|S|jD]}tjt|||j dd}qH|S)NrrCT)ptraining)rrTrelurr&autogradr bernoullirnewsizefill_rexpanddropout)r5rs inferencerex0rrrrr8%s , zPrenet.forward)rFrurrr6rrrcCsT||}t|ddd|ddf}t|dd|dddf}||}|Sr2)r&rsigmoid)input_ainput_bn_channels_intin_actt_acts_actactsrrrfused_add_tanh_sigmoid_multiply3s   rcs0eZdZdZfddZddefddZZS) Invertible1x1Convz The layer outputs both the convolution, and the log determinant of its weight matrix. If reverse=True it does convolution with inverse csttjj||ddddd|_tjt|| d}t |dkr7d|dddf|dddf<| ||d}||jj _ d|_dS)NrCrFrGrNrOrr")r3r4r&rirxrylinalgqr FloatTensornormal_detrYrrinv_conv)r5cWr6rrr4Bs    zInvertible1x1Conv.__init__Freversec Cs|rA|jdurinversertorMrLrlogdet) r5zr W_inverser batch_size group_size n_of_groups log_det_Wrrrr8Ps"      zInvertible1x1Conv.forwardr)r9r:r;r<r4r?r8r@rrr6rr;s rcs:eZdZdZfddZdeejejffddZZ S)WaveNetz This is the WaveNet like layer for the affine coupling. The primary difference from WaveNet is the convolutions need not be causal. There is also no dilation size reset. The dilation only doubles on each layer cst|ddks J|ddksJ||_||_tj|_tj|_tj ||d}tjj j |dd}||_ tj |d|d}|j j|jj||_tj |d||d}tjj j |dd|_t|D]P} d| } t|| | d} tjj |d||| | d} tjj j | dd} |j| | |dkrd|} n|} tj || d}tjj j |dd}|j|qndS)Nr$rCrr)name)rPrO)r3r4n_layers n_channelsr&rir in_layersres_skip_layersrxrzr{startrrzero_rend cond_layerranger=append)r5 n_in_channelsn_mel_channelsrrrGrrrirPrOin_layerres_skip_channelsres_skip_layerr6rrr4rs:         zWaveNet.__init__ forward_inputc Cs|d|d}}||}t|}||}t|jD]X}|d|j}t|j|||dd||d|jddf|j}|j ||}||jdkrq||ddd|jddf}||dd|jdddf}q||}q| |S)NrrCr$) rr& zeros_likerrrrrrrr) r5raudiospectrar spect_offsetr res_skip_actsrrrr8s"    " "  zWaveNet.forward) r9r:r;r<r4rr&rr8r@rrr6rrks "'rcs<eZdZdZdgffdd ZddZd fdd ZZS) ConditionalLayerNormz This module is used to condition torch.nn.LayerNorm. If we don't have any conditions, this will be a normal LayerNorm. Ncs\t|d|v|_tj||j d|jr,tj|||_tj|||_| dSdS)Nr)elementwise_affine) r conditionr3r4r&rirj cond_weight cond_biasinit_parametersr5 hidden_dim condition_dimrr6rrr4s  zConditionalLayerNorm.__init__cCsTtjj|jjdtjj|jjdtjj|jjdtjj|jjddS)Ng?)r&rirl constant_rrrrr5rrrrsz$ConditionalLayerNorm.init_parameterscsBt|}|jr|durtd|||}|||}|S)NYou should add additional data types as conditions (e.g. speaker id or reference audio) and define speaker_encoder in your config.)r3r8rrrrr5inputs conditioningr6rrr8s zConditionalLayerNorm.forwardr2)r9r:r;r<r4rr8r@rrr6rrs  rcs.eZdZdZgffdd ZdddZZS)ConditionalInputz} This module is used to condition any model inputs. If we don't have any conditions, this will be a normal pass. cst|tddg_fdd|D_|_|_djvr/||kr/tj ||_ djvr@tj |||_ dSdS)Nrrcsg|] }|jvr|qSr) support_types)rrr rrrsz-ConditionalInput.__init__..) rr3r4rrrrr&rirjadd_proj concat_projrr6r rr4s   zConditionalInput.__init__NcCst|jdkr@|durtdd|jvr#|j|jkr||}||}d|jvr@|d|jdd}tj ||gdd}| |}|S) z Args: inputs (torch.tensor): B x T x H tensor. conditioning (torch.tensor): B x 1 x C conditioning embedding. rNr rrrCr"r) lenrrrrrrepeatrIr&catrr rrrr8s     zConditionalInput.forwardr2r9r:r;r<r4r8r@rrr6rrsrcs>eZdZd fdd ZeddZedd Zd d ZZS) StyleAttention c s\tt|||}tjt|||_tjj||dd||dd|_ tjj |jdS)NrT) embed_dim num_headsrrkdimvdim batch_first) r3rr4r&ri ParameterrtokensMultiheadAttentionmharlr)r5gst_size n_style_tokenn_style_attn_head token_sizer6rrr4s zStyleAttention.__init__cCstdttdtdddS)NBDr(Toptional)r token_id)rr r r rrr input_typess zStyleAttention.input_typescCdtdtiS)N style_embr'rr r rrr output_types zStyleAttention.output_typescCsR|d}|d}t|jd|dd}|j|||d\}}|d}|S)NrrCr")rkeyvalue)rrrTrr rr"r)r5r rrr r/_rrrr8%s   zStyleAttention.forwardrrr) r9r:r;r4propertyr-r1r8r@rrr6rrs  rcs(eZdZd fdd Zd d d ZZS) Conv2DReLUNormr$rCTrcsHtt|tjj||||||d|_tj||_tj ||_ dS)Nr) r3r8r4r&riConv2dry LayerNormr|Dropoutr)r5r}rZrGrNrOrrr6rrr40s  zConv2DReLUNorm.__init__NcCs`|dur||}|dddd}t||}|dddd}||}||}|S)Nrr9rCr$) contiguouspermuterTrryr|r)r5rsx_maskrrrr88s  zConv2DReLUNorm.forward)r9r$rCTrr2rurrr6rr8/rr8csZeZdZdZfddZeddZeddZdd Ze dd dZ e ddZ Z S)ReferenceEncoderz? Encode mel-spectrograms to an utterance level feature c s~ttdgt|_tjfddtt |D_ j |t |d} tjj |d| |dd_ dS)NrCc s:g|]}ttj|tj|ddqS)rC)r}rZrGrNrOrr)r8r= filter_size)rrrrrGrOr5rNrrrOs  z-ReferenceEncoder.__init__..)n_convsr"T) input_size hidden_sizer)r3r@r4listrAr&rirrrrcalculate_post_conv_lengthsGRUgru) r5n_mels cnn_filtersr gru_hiddenrGrNrOrpost_conv_heightr6rBrr4Ks  zReferenceEncoder.__init__cCtdttdtdS)Nr(r)T_specr()r inputs_lengthsrr r r rrrr-a  zReferenceEncoder.input_typescCr.)Nr+r'r0r rrrr1hr2zReferenceEncoder.output_typesc Cs|ddd}|}||dd}|jD]}|||}||}||dd}q||jd|jdd}|j t || ddd}||\}}t |dd \}}|t t||dddf}|S) NrCr$r9rr"TF)renforce_sorted)r)rrlengths_to_masksrrGr=rYrIrIflatten_parametersrcpurr&aranger) r5r rQrsx_lensx_maskslayerpacked_xr5rrrr8ns     zReferenceEncoder.forwardrCr9r$cCs*t|D]}||d||d}q|S)z?Batch lengths after n convolution with fixed kernel/stride/pad.r$rC)r)r(rCrGrNpadr5rrrrGs z,ReferenceEncoder.calculate_post_conv_lengthscCs6t||j|jd||dk}|S)z"Batch of lengths to batch of masksrrC)r&rXmaxrrMrrIr)r(masksrrrrUs z!ReferenceEncoder.lengths_to_masks)rCr9r$rC) r9r:r;r<r4r7r-r1r8 staticmethodrGrUr@rrr6rr@Fs    r@csDeZdZdZ d fdd ZeddZed d Zd d ZZ S)GlobalStyleTokenz4 Global Style Token based Speaker Embedding rrrcs(tt|||_t|||d|_dS)N)r#r$r%)r3rar4reference_encoderrstyle_attention)r5rbr#r$r%r6rrr4s  zGlobalStyleToken.__init__cCrN)NrOr()inp inp_lengthsrRr rrrr-rSzGlobalStyleToken.input_typescCr.)Ngstr'r0r rrrr1r2zGlobalStyleToken.output_typescCs|||}||}|Sr2)rbrc)r5rdrestyle_embeddingrfrrrr8s  zGlobalStyleToken.forwardr6) r9r:r;r<r4r7r-r1r8r@rrr6rras  racs(eZdZdZfddZddZZS)SpeakerLookupTablez- LookupTable based Speaker Embedding cs"tt|tj|||_dSr2)r3rhr4r&ri Embeddingtable)r5 n_speakersrr6rrr4szSpeakerLookupTable.__init__cCrrr2)rj)r5speakerrrrr8rtzSpeakerLookupTable.forwardrrrr6rrhs rhcsLeZdZdZd fdd ZeddZeddZd d Zdd d Z Z S)SpeakerEncoderz class SpeakerEncoder represents speakers representation. This module can combine GST (global style token) based speaker embeddings and lookup table speaker embeddings. NcsDtt|||_||_|durtjt||_ dSd|_ dS)a  lookup_module: Torch module to get lookup based speaker embedding gst_module: Neural module to get GST based speaker embedding precomputed_embedding_dim: Give precomputed speaker embedding dimension to use precompute speaker embedding N) r3rmr4 lookup_module gst_moduler&riremptyprecomputed_emb)r5rnroprecomputed_embedding_dimr6rrr4s  zSpeakerEncoder.__init__cCs8tddtdtddtdtddtdtdddS)NTr*r(rO)rrlreference_specreference_spec_lens)rr r r r rrrr-s zSpeakerEncoder.input_typescCr.)Nembsr'r0r rrrr1r2zSpeakerEncoder.output_typescCstj||_dSr2)r&rirrq)r5embrrroverwrite_precomputed_embsz(SpeakerEncoder.overwrite_precomputed_embcCsd}|jdur|jd|dS|jdur|dur||}|durE|durE|jdur@|||}|dur:|}|S||}|Std|S)NrrCzCYou may add `gst_module` in speaker_encoder to use reference_audio.)rqrrrnrorwarning)r5rrlrsrtrur+rrrr8s      zSpeakerEncoder.forward)NNN)NNNN) r9r:r;r<r4r7r-r1rwr8r@rrr6rrms  rm)r)4typingrr&rtorch.autogradrtorch.nnrrTtorch.nn.utils.rnnrrnemo.core.classesrr nemo.core.neural_types.elementsr r r r "nemo.core.neural_types.neural_typer nemo.utilsrrrr>r,riInstanceNorm1dr-rxrAModulerdAdapterModuleMixinrvrrrrrrr;rrrr8r@rarhrmrrrrs\       1 ==0G'.(N!