o }oiDN@sddlmZddlZddlmZddlmmZddlm Z ddl m Z ddl m Z ddlmZmZmZmZddlmZdgZGd d d ejZGd dde e ZdS) ) OrderedDictN) typecheck) Exportable) NeuralModule)EncodedRepresentation LengthsType NeuralTypeSpectrogramType) ProbsType MSDD_modulecs&eZdZdfdd ZddZZS) ConvLayerr r r c s@tt|ttj||||dttj|ddd|_dS)N in_channels out_channels kernel_sizestridegMbP?gGz?)epsmomentum) superr __init__nn SequentialConv2dReLU BatchNorm2dcnn)selfrrrr __class__^/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/modules/msdd_diarizer.pyrs  zConvLayer.__init__cCs||}|SN)r)r featurer#r#r$forward's zConvLayer.forward)r r rr)__name__ __module__ __qualname__rr' __classcell__r#r#r!r$r sr cseZdZdZeddZeddZddZ      d0dededede dededede dede de ffdd Z dd Z d!d"Z d#d$Zd%d&Zd'd(Zd1d*d+Zed,d-Zd.d/ZZS)2r a Multi-scale Diarization Decoder (MSDD) for overlap-aware diarization and improved diarization accuracy from clustering diarizer. Based on the paper: Taejin Park et. al, "Multi-scale Speaker Diarization with Dynamic Scale Weighting", Interspeech 2022. Arxiv version: https://arxiv.org/pdf/2203.15974.pdf Args: num_spks (int): Max number of speakers that are processed by the model. In `MSDD_module`, `num_spks=2` for pairwise inference. hidden_size (int): Number of hidden units in sequence models and intermediate layers. num_lstm_layers (int): Number of the stacked LSTM layers. dropout_rate (float): Dropout rate for linear layers, CNN and LSTM. cnn_output_ch (int): Number of channels per each CNN layer. emb_dim (int): Dimension of the embedding vectors. scale_n (int): Number of scales in multi-scale system. clamp_max (float): Maximum value for limiting the scale weight values. conv_repeat (int): Number of CNN layers after the first CNN layer. weighting_scheme (str): Name of the methods for estimating the scale weights. context_vector_type (str): If 'cos_sim', cosine similarity values are used for the input of the sequence models. If 'elem_prod', element-wise product values are used for the input of the sequence models. cCsttdttdtdS)z< Return definitions of module output ports. BTCr-r.r/D)probs scale_weights)rrr r r#r#r$ output_typesLs   zMSDD_module.output_typescCs6ttdtttdttdttdtdS)z< Return definitions of module input ports. r0r-)r-r/r1r/r,) ms_emb_seqlength ms_avg_embstargets)rrr tuplerrr r4r#r#r$ input_typesXs   zMSDD_module.input_typescCst|tjkrtjj|j|jj ddSt|tj tj tj fvrQ| D]*\}}d|vr9tjj|jq(d|vrFtjj|jq(d|vrP|j dq(dSdS)Ng{Gz? weight_ih weight_hhbias)typerLineartorchinitxavier_uniform_weightr>datafill_GRULSTMRNNnamed_parameters orthogonal_)r mnameparamr#r#r$ init_weightsfs zMSDD_module.init_weights??r conv_scale_weightcos_simnum_spks hidden_sizenum_lstm_layers dropout_rate cnn_output_chemb_dimscale_n clamp_max conv_repeatweighting_schemecontext_vector_typec s<td|_d|_d|_||_||_||_||_| |_ d|_ d|_ ||_ | |_ | |_tjjdd|_tjjd|j d|_tj|||j dd|d |_|j d krttd||j|j|dfd d g|_td| dD]} |jtd||jdfd d qlt|_t|j dD]} |jtj|jd dqt||||_t||j|_ n|j dkrtj||d d|_!tj"#|j!j$nt%d|j td||j|_&|jdkrt|j|j||_'|j'(|j)n|jdkrt|j|j||_*nt%d|jt+||_,|j&(|j)|j(|j)||_-dS)Nr 2rPgư>dimr)rfrT) num_layers batch_first bidirectionaldropoutrWrrF)affineattn_scale_weight)r>No such weighting scheme as rX elem_prodNo such context vector type as ).rr_speaker_model batch_sizer7r^rYr_r]rachanrr[rbrcrArSoftmaxsoftmaxCosineSimilaritycos_distrHlstm ModuleListr convrangeappendconv_bnrr@conv_to_linearlinear_to_weightsW_arBeye_rD ValueErrorhidden_to_spks dist_to_embapplyrOproduct_to_embDropoutrjr`) r rYrZr[r\r]r^r_r`rarbrcconv_idxr!r#r$rssx         zMSDD_module.__init__cCs\|jd|_|jd|_|jd|_|ddddd|j}|}|dd|jddd}|ddddd|j|jd|j}|j dkrL| ||}n|j dkrX| ||}nt d |j | |j}|jd krs||||} n|jd kr||||} nt d |j|t| } || } |t| d} || } t| } | |fS) a) Core model that accepts multi-scale cosine similarity values and estimates per-speaker binary label. Args: ms_emb_seq (Tensor): Multiscale input embedding sequence Shape: (batch_size, length, scale_n, emb_dim) length (Tensor): The actual length of embedding sequences without zero padding Shape: (batch_size,) ms_avg_embs (Tensor): Cluster-average speaker embedding vectors. Shape: (batch_size, scale_n, self.emb_dim, max_spks) targets (Tensor): Ground-truth labels for the finest segment. Shape: (batch_size, feats_len, max_spks) Returns: preds (Tensor): Predicted binary speaker label for each speaker. Shape: (batch_size, feats_len, max_spks) scale_weights (Tensor): Multiscale weights per each base-scale segment. Shape: (batch_size, length, scale_n, max_spks) rr rPrrWrlrmrXrnro)shaperqr7r^ unsqueezeexpandrYpermutereshaperbconv_scale_weightsattention_scale_weightsrtodevicerccosine_similarityelement_wise_productrjFrelurwrrSigmoid)r r6r7r8r9 _ms_emb_seqms_emb_seq_singlems_avg_embs_permr3 context_emb lstm_outputlstm_hidden_out spk_predspredsr#r#r$ core_models0   $        zMSDD_module.core_modelc Cs||j|j|j|j}||j|j|j|j|j}|d|j|j}|dd|j|j|j}||}t||j|j|jd|j||j|j|j|j|j} | |j|j|j|j} | | } | S)a3 Calculate element wise product values among cluster-average embedding vectors and input embedding vector sequences. This function is selected by assigning `self.context_vector_type = "elem_prod"`. `elem_prod` method usually takes more time to converge compared to `cos_sim` method. Args: scale_weights (Tensor): Multiscale weight vector. Shape: (batch_size, feats_len, scale_n, max_spks) ms_avg_embs_perm (Tensor): Tensor containing cluster-average speaker embeddings for each scale. Shape: (batch_size, length, scale_n, emb_dim) ms_emb_seq (Tensor): Tensor containing multi-scale speaker embedding sequences. `ms_emb_seq` is a single channel input from the given audio stream input. Shape: (batch_size, length, num_spks, emb_dim) Returns: context_emb (Tensor): Output of `dist_to_emb` linear layer containing context for speaker label estimation. rrr ) rrqr7rYr_r^rrAbmmr) r r3r8r6scale_weight_flattenms_avg_embs_flattenms_emb_seq_flattenms_emb_seq_flatten_repelemwise_productcontext_vectorsrr#r#r$rs z MSDD_module.element_wise_productcCs8|||}t||}||j|jd}||}|S)a Calculate cosine similarity values among cluster-average embedding vectors and input embedding vector sequences. This function is selected by assigning self.context_vector_type = "cos_sim". Args: scale_weights (Tensor): Multiscale weight vector. Shape: (batch_size, feats_len, scale_n, max_spks) ms_avg_embs_perm (Tensor): Tensor containing cluster-average speaker embeddings for each scale. Shape: (batch_size, length, scale_n, emb_dim) _ms_emb_seq (Tensor): Tensor containing multi-scale speaker embedding sequences. `ms_emb_seq` is a single channel input from the given audio stream input. Shape: (batch_size, length, num_spks, emb_dim) Returns: context_emb (Tensor): Output of `dist_to_emb` linear layer containing context for speaker label estimation. r)rvrAmulviewrqr7r)r r3r8r cos_dist_seqrrr#r#r$r&s   zMSDD_module.cosine_similaritycCs||dd||dd}|ddddd}t||d|j|j|j}ttj |ddd}||j |j |j|j}| |}|S)a Use weighted inner product for calculating each scale weight. W_a matrix has (emb_dim * emb_dim) learnable parameters and W_a matrix is initialized with an identity matrix. Compared to "conv_scale_weight" method, this method shows more evenly distributed scale weights. Args: ms_avg_embs_perm (Tensor): Tensor containing cluster-average speaker embeddings for each scale. Shape: (batch_size, length, scale_n, emb_dim) ms_emb_seq (Tensor): Tensor containing multi-scale speaker embedding sequences. `ms_emb_seq` is input from the given audio stream input. Shape: (batch_size, length, num_spks, emb_dim) Returns: scale_weights (Tensor): Weight vectors that determine the weight of each scale. Shape: (batch_size, length, num_spks, emb_dim) rr rPr)dim1dim2) rflattenrrAmatmulrr_rYsigmoiddiagonalrqr7rt)r rr6mat_amat_b weighted_corrr3r#r#r$rAs z#MSDD_module.attention_scale_weightsc Cstj||gdd}|ddd}|j||jd|jddd}td|jdD]}|j||j||j|dd}q)| |j |j |j |j }||}|t|}|||}|d d d d |j}|S) a Use multiple Convnet layers to estimate the scale weights based on the cluster-average embedding and input embedding sequence. Args: ms_avg_embs_perm (Tensor): Tensor containing cluster-average speaker embeddings for each scale. Shape: (batch_size, length, scale_n, emb_dim) ms_emb_seq_single (Tensor): Tensor containing multi-scale speaker embedding sequences. ms_emb_seq_single is input from the given audio stream input. Shape: (batch_size, length, num_spks, emb_dim) Returns: scale_weights (Tensor): Weight vectors that determine the weight of each scale. Shape: (batch_size, length, num_spks, emb_dim) rPrerr T) conv_module bn_module first_layerF) conv_inputrrrrr)rAcatrr conv_forwardryr|rzrarrqr7r]r^r}rjr leaky_relurtr~rrY) r rrms_cnn_input_seqconv_outr lin_input_seq hidden_seqr3r#r#r$r_s$ zMSDD_module.conv_scale_weightsFcCs~||}|s|ddddn|}||j|j|j|j}|ddd}||dddddddd}|t |}|S)a A module for convolutional neural networks with 1-D filters. As a unit layer batch normalization, non-linear layer and dropout modules are included. Note: If `first_layer=True`, the input shape is set for processing embedding input. If `first_layer=False`, then the input shape is set for processing the output from another `conv_forward` module. Args: conv_input (Tensor): Reshaped tensor containing cluster-average embeddings and multi-scale embedding sequences. Shape: (batch_size*length, 1, scale_n*(num_spks+1), emb_dim) conv_module (ConvLayer): ConvLayer instance containing torch.nn.modules.conv modules. bn_module (torch.nn.modules.batchnorm.BatchNorm2d): Predefined Batchnorm module. first_layer (bool): Boolean for switching between the first layer and the others. Default: `False` Returns: conv_out (Tensor): Convnet output that can be fed to another ConvLayer module or linear layer. Shape: (batch_size*length, 1, cnn_output_ch, emb_dim) rrPr r) rrrqr7r]r^rrrjrr)r rrrrrr#r#r$rs zMSDD_module.conv_forwardcCs|||||\}}||fSr%)r)r r6r7r8r9rr3r#r#r$r'szMSDD_module.forwardcCst|j}tj|jdfd|d}tjd||j|j|d}tjd|j|j|j |d}td||j  }t ||||gS)z{ Generate input examples for tracing etc. Returns (tuple): A tuple of input examples. r{)size fill_valuerr )r) next parametersrrAfullrrandnr_r^rYroundfloatr:)r r input_examplelensavg_embsr9r#r#r$rs zMSDD_module.input_example) rPrQrPrRrSrTrUrVr rWrX)F)r(r)r*__doc__propertyr5r;rOintrstrrrrrrrrrr'rr+r#r#r!r$r ,sd       S;% (" ) collectionsrrAtorch.nnrtorch.nn.functional functionalrnemo.core.classes.commonrnemo.core.classes.exportablernemo.core.classes.modulernemo.core.neural_typesrrrr nemo.core.neural_types.elementsr __all__Moduler r r#r#r#r$s