o wi%@sTddlZddlmZddlmZddlmZmZddlmZGdddejj Z dS)N) rearrange)nn)ConditionalInputConvNorm)binarize_attention_parallelcseZdZdZddddgdffdd Zedd Zdd d Zed dZeddZ eddZ edddZ eddZ dddZ ZS)AlignmentEncodera; Module for alignment text and mel spectrogram. Args: n_mel_channels: Dimension of mel spectrogram. n_text_channels: Dimension of text embeddings. n_att_channels: Dimension of model temperature: Temperature to scale distance by. Suggested to be 0.0005 when using dist_type "l2" and 15.0 when using "cosine". condition_types: List of types for nemo.collections.tts.modules.submodules.ConditionalInput. dist_type: Distance type to use for similarity measurement. Supports "l2" and "cosine" distance. PigMb@?l2c st||_t||||_tjjdd|_tjj dd|_ t t ||dddddtj t |d|ddd|_t t ||dddddtj t |d|dddtj t ||ddd|_|d krl|j|_dS|d krv|j|_dStd |d ) NdimTrelu) kernel_sizebias w_init_gain)rrr cosinezUnknown distance type '')super__init__ temperaturer cond_inputtorchrSoftmaxsoftmax LogSoftmax log_softmax SequentialrReLUkey_proj query_projget_euclidean_distdist_fnget_cosine_dist ValueError)selfn_mel_channelsn_text_channelsn_att_channelsrcondition_types dist_type __class__a/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/tts/modules/aligner.pyr&s,   zAlignmentEncoder.__init__cCs(|durdSt|d}|j||dS)NzB T2 1 -> B 1 1 T2)rdata masked_fill_)inputsmask mask_valuer.r.r/ _apply_maskIs zAlignmentEncoder._apply_maskNcCs>||}||}|j||d}|||td|dS)aCalculation of distance matrix. Args: queries (torch.tensor): B x C1 x T1 tensor (probably going to be mel data). keys (torch.tensor): B x C2 x T2 tensor (text data). mask (torch.tensor): B x T2 x 1 tensor, binary mask for variable length entries and also can be used for ignoring unnecessary elements from keys in the resulting distance matrix (True = mask element, False = leave unchanged). Output: dist (torch.tensor): B x T1 x T2 tensor.  queries_enckeys_encinfr)r!r r#r5floatsqueeze)r&keysqueriesr3r7r8distr.r.r/get_distQs  zAlignmentEncoder.get_distcCs2t|d}t|d}||d}|jddd}|S)NB C T1 -> B C T1 1B C T2 -> B C 1 T2r rT)axiskeepdim)rsum)r7r8distancel2_distr.r.r/r"gs   z#AlignmentEncoder.get_euclidean_distcCs8t|d}t|d}tjjj||dd }t|d}|S)Nr@rArr zB T1 T2 -> B 1 T1 T2)rrr functionalcosine_similarity)r7r8 cosine_distr.r.r/r$qs   z AlignmentEncoder.get_cosine_distcCsJt|||}|ddddddf}tt|jdd|s#J|S)zCalculation of durations. Args: attn_soft (torch.tensor): B x 1 x T1 x T2 tensor. text_len (torch.tensor): B tensor, lengths of text. spect_len (torch.tensor): B tensor, lengths of mel spectrogram. r Nrrr )rrDralleq) attn_softtext_len spect_len attn_hard durationsr.r.r/ get_durationsys zAlignmentEncoder.get_durationsc Cs|\}}}tt|jdd|sJt||dg}t|D]}|t ||t |tj t |||dfq#tj ||j |jdS)aSelect elements from the distance matrix for the given durations and mask and return mean distance. Args: dist (torch.tensor): B x T1 x T2 tensor. durations (torch.tensor): B x T2 tensor. Dim T2 should sum to T1. mask (torch.tensor): B x T2 x 1 binary mask for variable length entries and also can be used for ignoring unnecessary elements in dist by T2 dim (True = mask element, False = leave unchanged). Output: mean_dist (torch.tensor): B x 1 tensor. rr r)repeats)dtypedevice)sizerrJrKrDrr5rangeappendmeanarangerepeat_interleavetensorrSrT)r>rPr3 batch_sizet1_sizet2_sizemean_dist_by_durationsdist_idxr.r.r/get_mean_dist_by_durationss    z+AlignmentEncoder.get_mean_dist_by_durationsc Csxt|d|j}d}d}t|||D]"}t||||D] }||||f7}q |||7}|||7}q||S)apCalculates the mean distance between text and audio embeddings given a range of text tokens. Args: l2_dists (torch.tensor): L2 distance matrix from Aligner inference. T1 x T2 tensor. durs (torch.tensor): List of durations corresponding to each text token. T2 tensor. Should sum to T1. start_token (int): Index of the starting token for the word of interest. num_tokens (int): Length (in tokens) of the word of interest. Output: mean_dist_for_word (float): Mean embedding distance between the word indicated and its predicted audio frames. Nr)rrDr0rV) l2_distsdurs start_token num_tokens start_frame total_framesdist_sum token_ind frame_indr.r.r/get_mean_distance_for_words  z+AlignmentEncoder.get_mean_distance_for_wordc Cs||dd|dd}||}||}|j||d}|j |} |dur;|| t|dddfd} | } | | |t d | | } | | fS)aForward pass of the aligner encoder. Args: queries (torch.tensor): B x C1 x T1 tensor (probably going to be mel data). keys (torch.tensor): B x C2 x T2 tensor (text data). mask (torch.tensor): B x T2 x 1 tensor, binary mask for variable length entries (True = mask element, False = leave unchanged). attn_prior (torch.tensor): prior for attention matrix. conditioning (torch.tensor): B x 1 x C2 conditioning embedding Output: attn (torch.tensor): B x 1 x T1 x T2 attention mask. Final dim T2 should sum to 1. attn_logprob (torch.tensor): B x 1 x T1 x T2 log-prob attention mask. rr r6Ng:0yE>r9) r transposer!r r#rrrlogcloner5r:r) r&r=r<r3 attn_prior conditioningr7r8rEattn attn_logprobr.r.r/forwards   $ zAlignmentEncoder.forward)N)NNN)__name__ __module__ __qualname____doc__r staticmethodr5r?r"r$rQrarkrs __classcell__r.r.r,r/rs.#       r) reinopsrr'nemo.collections.tts.modules.submodulesrr(nemo.collections.tts.parts.utils.helpersrModulerr.r.r.r/s