o }oi?@sddlmZmZmZddlZddlmZmZedfdej dej dej fdd Z d ej dej fd d Z e d fdej dej dej dej fddZ de d fd ej dedededej f ddZddde d fd ej dededededej dej fddZd ej d!edej dej fd"d#Zd ej dej dej fd$d%Zdgd ej d'ed(edej fd)d*Zd+ej d'edej fd,d-Zd.ej d/ej d0ej dej deej ej ff d1d2Zd3ej d4ej dej fd5d6Zd7eej deej fd8d9Zd:ej dej fd;d<Ze d fd=ej d>eej d7eej dej deej eej ff d?d@Ze d fd=ej d>eej d7eej dej dej f dAdBZd ej dej fdCdDZdEej dFedej deej ej ffdGdHZdEej dFedej dej fdIdJZ dKej dej fdLdMZ!d:ej dNedOedPedej f dQdRZ" S T U V Wdhd:ej dXedOedNedPedFedej fdYdZZ#d>ej d7ej d[ej$deeej eej ffd\d]Z% Wdid ej d^edFedeej ej ej ffd_d`Z&GdadbdbZ'GdcddddZ(Gdedfdfej)j*Z+dS)j)DictListTupleN)eigheigvalshgǺ6?emb_aemb_breturncCs|jddks|jddkrtd|jd|j|tj|ddd|}|tj|ddd|}t||dd}|d|S)a Calculate cosine similarities of the given two set of tensors. The output is an N by N matrix where N is the number of feature vectors. Args: a (Tensor): Matrix containing speaker representation vectors. (N x embedding_dim) b (Tensor): Matrix containing speaker representation vectors. (N x embedding_dim) Returns: res (Tensor): N by N matrix containing the cosine similarities of the values. rz;Number of feature vectors should be greater than 1 but got z and dim)shape ValueErrortorchnorm unsqueezemm transposefill_diagonal_)rrepsa_normb_normresrg/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/asr/parts/utils/offline_clustering.pycos_similarity(s rXcCs&||}}||||}|S)a* Min-max scale the input affinity matrix X, which will lead to a dynamic range of [0, 1]. Args: X (Tensor): Matrix containing cosine similarity values among embedding vectors (N x N) Returns: v_norm (Tensor): Min-max normalized value of X. )minmax)rv_minv_maxv_normrrr ScalerMinMaxAs r"cpuspecEmbAspecEmbBdevicecCsP||||}}|jdd|jdd}}||d}|jdd}|S)a Calculate Euclidean distances from the given feature tensors. Args: specEmbA (Tensor): Matrix containing spectral embedding vectors from eigenvalue decomposition (N x embedding_dim). specEmbB (Tensor): Matrix containing spectral embedding vectors from eigenvalue decomposition (N x embedding_dim). Returns: dis (Tensor): Euclidean distance values of the two sets of spectral embedding vectors. r r rg@)torsumsqueeze)r$r%r&ABdisrrrgetEuclideanDistanceRs  r. n_clusters random_staten_local_trialscCst|||}|j\}}tj|||jd}td|d}tj|gdtj d} || d|d<| d| d<||}|d d|jd |jdd|} | djddjdd} | } td|D]} t|| }t| jdkrtj| ddd}ntj| dd}t|||}|jd}|| d|jd |jd|d| |d}| djdd |d}t| |}|jdd}t|}||} ||} ||}|||| <|| | <qj|| fS) a Choose initial centroids for initializing k-means algorithm. The performance of k-means algorithm can vary significantly by the initial centroids. To alleviate this problem, k-means++ algorithm chooses initial centroids based on the probability proportional to the distance from the formally chosen centroids. The centroids selected by k-means++ algorithm improve the chance of getting more accurate and stable clustering results. The overall implementation of k-means++ algorithm is inspired by the numpy based k-means++ implementation in: https://github.com/scikit-learn/scikit-learn Originally, the implementation of the k-means++ algorithm in scikit-learn is based on the following research article: Arthur, David, and Sergei Vassilvitskii. k-means++: The advantages of careful seeding. Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, Society for Industrial and Applied Mathematics (2007) Args: X (Tensor): Matrix containing cosine similarity values among embedding vectors (N x N) n_clusters (int): Maximum number of speakers for estimating number of speakers. Shows stable performance under 20. random_state (int): Seed variable for setting up a random state. n_local_trials (int): Number of trials for creating initial values of the center points. device (torch.device) Torch device variable. Returns: centers (Tensor): The coordinates for center points that are used for initializing k-means algorithm. indices (Tensor): The indices of the best candidate center points. dtyperr r')rNr r )r manual_seedr(r zerosr4randintlongfullintr*repeatviewpowr)rrangeranditemlencumsum searchsortedminimumargmin)rr0r1r2r& n_samples n_featurescenters center_idindicesclosest_dist_diffclosest_dist_sq current_potc rand_vals torch_cumsum candidate_idsN_ci distance_diffdistancedistance_to_candidatescandidates_potbest_candidaterrrkmeans_plusplus_torchis< *   ( 4     rZg-C6? num_clusters threshold iter_limitc Cs.||}|jd}t||||d}|d}t|} t|D]q} t|||d} t | jdkr6| Stj | dd} | } t|D].} t | | k |}t|d|}|jddkrk|tt |d}|jdd|| <qEt|| d}ttttj|ddd}||kr| Sq#| S)a Run k-means algorithm on the given set of spectral embeddings in X. The threshold and iter_limit variables are set to show the best performance on speaker diarization tasks. The overall implementation of k-means algorithm is inspired by the k-means algorithm implemented in https://github.com/scikit-learn/scikit-learn. References: Arthur, David, and Sergei Vassilvitskii. k-means++: The advantages of careful seeding. Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, Society for Industrial and Applied Mathematics (2007). Args: X (Tensor): Cosine similarity matrix calculated from speaker embeddings num_clusters (int): The estimated number of speakers. threshold (float): This threshold limits the change of center values. If the square of the center shift values are bigger than this threshold, the iteration stops. iter_limit (int): The maximum number of iterations that is allowed by the k-means algorithm. device (torch.device): Torch device variable Returns: selected_cluster_indices (Tensor): The assigned cluster labels from the k-means clustering. r)r0r1r&r&r r r5r6)floatr(r rZrr8r:r@r.rCrGclonenonzeror* index_selectr9meanr?r)sqrt)rr\r]r^r1r& input_sizeplusplus_init_statesrJselected_cluster_indices iter_counteuc_dist center_initsindexselected_clusterchosen_indicescenter_delta_powcenter_shift_powrrr kmeans_torchs2%   "rq affinity_mat seg_indexc Cs|jd}tj|tjd|}tj|tjd|}d||<||}t|D]H}|}tj|||d||kr?|S|tdk }t | dkrY|d}|D]} || |} tj|| d|dq[q(|S)a Find the largest affinity_mat connected components for each given node. This is for checking whether the affinity_mat is fully connected. Args: affinity_mat (Tensor): A square matrix (tensor) containing normalized cosine distance values seg_index (int): The segment index that is targeted to be explored. Returns: connected_nodes (Tensor): A tensor containing booleans that indicate whether the node is connected. rr3T)out)r rr8boolr(r@r) logical_ortensorrbtr*rCsizer) rrrsr&num_of_segmentsconnected_nodesnodes_to_exploreklast_num_componentrLi neighborsrrrgetTheLargestComponent s&     rcCst|d||jdkS)zM Check whether the given affinity matrix is a fully connected graph. r)rr)r )rrr&rrrisGraphFullyConnected.srbinaryp_value mask_methodcCs(|j}t|}tj|dddddd|f}td|j||jt |jdf<|ddd|f }t |d |dj }|dksR|durft|jd|j|||f<|S|dkrx|||f|||f<|S|dkrt |||f|||f<|St d |) a( Binarize top-p values for each row from the given affinity matrix. Args: affinity_mat (Tensor): A square matrix (tensor) containing normalized cosine similarity values p_value (int): The number of top values that are selected from each row. mask_method (str): The method that is used to manipulate the affinity matrix. The default method is 'binary'. Returns: binarized_affinity_mat (Tensor): A binarized affinity matrix based on the given mask method. r T)r descendingNrrdropsigmoidzUnknown mask method: )r r zeros_likehalfargsortonesr(r&Tarangeflattenr=rr)rrrrr binarized_affinity_mat sorted_matrix indices_row indices_colrrrgetKneighborsConnections5s$   raffinity_mat_rawcCs(|dkr|nt||}d||j}|S)z[ Calculate a binarized graph matrix and symmetrize the binarized graph matrix. rg?)rr)rrrsymm_affinity_matrrrgetAffinityGraphMatZsrmatmax_Nn_listcCsXtd}t||}t|D]\}}t||}t||}|s"||kr'||fSq||fS)z Generate connections until fully connect all the nodes in the graph. If the graph is not fully connected, it might generate inaccurate results. r )rrwr enumerater)rrrr&rrrrfully_connectedrrrgetMinimumConnectionds     rmapping_argmatscore_mat_sizecCsBtj|tjd|j}tj|dd\}}||j||<|S)z Count the numbers in the mapping dictionary and create lists that contain repeated indices that will be used for creating a repeated affinity matrix. This repeated matrix is then used for fusing multiple affinity values. r3T) return_counts)rr8int32r(r&uniquer<)rr repeat_listidxscountsrrrgetRepeatedListvsrtimestamps_in_scalesc Csttt|}g}|D]}||}|tj|ddq t|}||}g}|D]/}||}t||jddf} t||jddf } tj t | | dd} || q)|S)a Calculate the mapping between the base scale and other scales. A segment from a longer scale is repeatedly mapped to a segment from a shorter scale or the base scale. Args: timestamps_in_scales (list): List containing timestamp tensors for each scale. Each tensor has dimensions of (Number of base segments) x 2. Returns: session_scale_mapping_list (list): List containing argmin arrays indexed by scale index. r r r) listr@rCappendrrdrtiler rxrGabs) r scale_listsegment_anchor_list scale_idxtime_stamps_floatbase_scale_idxbase_scale_anchorsession_scale_mapping_listcurr_scale_anchorcurr_matbase_mat argmin_matrrrget_argmin_mats rembcCsF|jddkrtdgg|j}|S|}t||}t|}|S)ar Calculate cosine similarity values among speaker embeddings then min-max normalize the affinity matrix. Args: emb (Tensor): Matrix containing embedding vectors. emb variable should be float(FP32) type to make the data-type compatible with torch.mm operation for both CPU and GPU(CUDA). dimension: (Number of embedding vectors) x (embedding dimension) Returns: sim_d (Tensor): Matrix containing cosine similarity values among the given embedding vectors. dimension: (Number of embedding vectors) x (Number of embedding vectors) rr )r rrwr(r&r`rr")rsim_drrrgetCosAffinityMatrixs rmultiscale_weightsembeddings_in_scalescCsg}||}t|}ttt|}|D]-}||}|||} ||}t|t| jd|} tj | | dd} | | qt |} t | ddd|} t| jdkre| d} | |} | |fS)a Generate a scale-interpolated single embedding vector by calculating the weighted sum of the multiple embedding vectors from different scales. The output is a set of embedding vectors corresponding to the base-scale segments. Args: multiscale_weights (Tensor): Tensor containing Multiscale weights Dimensions: (Number of scales) x 1 embeddings_in_scales (list): List containing split embedding tensors by each scale timestamps_in_scales (list): List containing split timestamps tensors by each scale device (torch.device): Torch device variable Returns: context_emb (Tensor): A set of scale-interpolated embedding vectors. Dimensions: (Number of base-scale segments) x (Dimensions of embedding vector) session_scale_mapping_list (list): List containing argmin arrays indexed by scale index. rrepeatsr r6r )r(rrr@rCrrrwr repeat_interleaverstackmatmulpermuterxr*r)rrrr& rep_mat_listrrrremb_tr rep_emb_tstacked_scale_embs context_embrrrget_scale_interpolated_embss"    "  rcCstj|dd|}t|}ttt|}tt|dt|d|}|D]?}||}|||} t | } t |t | j d|} tj | | dd|} tj | | dd|} |||| 7}q)|S)at Calculate cosine similarity values among speaker embeddings for each scale then apply multiscale weights to calculate the fused similarity matrix. NOTE: Due to CUDA memory limit, the embedding vectors in embeddings_in_scales are stored in `cpu` device. Args: multiscale_weights (Tensor): Tensor containing multiscale weights Dimensions: (Number of scales) x 1 embeddings_in_scales (list): List containing split embedding tensors by each scale timestamps_in_scales (list): List containing split timestamps tensors by each scale device (torch.device): Torch device variable Returns: fused_sim_d (Tensor): An affinity matrix that is obtained by calculating the weighted sum of the multiple affinity matrices from the different scales. rr r'rr )rr*r(rrr@rCr8rrrrwr r)rrrr&rr fused_sim_drrrscore_mat_torchrrepeated_tensor_0repeated_tensor_1rrrgetMultiScaleCosAffinityMatrixs"rcCs4|dtjt|dd}t|}||}|S)zA Calculate a laplacian matrix from an affinity matrix X. rr r )rrr)r diag_embed)rDLrrr getLaplacians  r laplaciancudacCsN|r|dur tj}||}n |td}t|\}}||fS)zK Calculate eigenvalues and eigenvectors from the Laplacian matrix. Nr#)rrcurrent_devicer`r(r&r)rrr&lambdas diffusion_maprrr eigDecomposes  rcCsF|r|dur tj}||}n |td}t|}|S)z? Calculate only eigenvalues from the Laplacian matrix. Nr#)rrrr`r(r&r)rrr&rrrr eigValueSh-s rrcCs,t|r t|}|dd|ddS)z3 Calculate the gaps between lambda values. r Nr')r is_complexreal)rrrrgetLamdaGaplist;s  ranchor_sample_n anchor_spk_nsigmac Cs|jd}tj|dd}t||j}g}t|D]A}ttd||df|j}t||j |j} t t || tj t | ddddj } ||| } || q||t|} | S)a Add randomly generated synthetic embeddings to make eigenanalysis more stable. We refer to these embeddings as anchor embeddings. emb (Tensor): The input embedding from the embedding extractor. anchor_sample_n (int): Number of embedding samples per speaker. anchor_sample_n = 10 is recommended. anchor_spk_n (int): Number of speakers for synthetic embedding. anchor_spk_n = 3 is recommended. sigma (int): The amplitude of synthetic noise for each embedding vector. If the sigma value is too small, under-counting could happen. If the sigma value is too large, over-counting could happen. sigma = 50 is recommended. r rr )r rstdrwr(r&r@rrandnrrdiagrrrrvstack) rrrremb_dimstd_org new_emb_list_emb_m emb_noiseemb_gen new_emb_nprrr addAnchorEmbDs"   (    r 2Frandom_test_countc Csg}t|D].}t|t||||}t|} t| |jdddddd|d} | \} } || qt t t t |d |d} | S) aS Calculate the number of speakers using NME analysis with anchor embeddings. Add dummy speaker embedding vectors and run speaker counting multiple times to enhance the speaker counting accuracy for the short audio samples. Args: emb (Tensor): The input embedding from the embedding extractor. cuda (bool): Use cuda for the operations if cuda==True. random_test_count (int): Number of trials of the enhanced counting with randomness. The higher the count, the more accurate the enhanced counting is. anchor_spk_n (int): Number of speakers for synthetic embedding. anchor_spk_n = 3 is recommended. anchor_sample_n (int): Number of embedding samples per speaker. anchor_sample_n = 10 is recommended. sigma (float): The amplitude of synthetic noise for each embedding vector. If the sigma value is too small, under-counting could happen. If the sigma value is too large, over-counting could happen. sigma = 50 is recommended. Returns: comp_est_num_of_spk (Tensor): The estimated number of speakers. `anchor_spk_n` is subtracted from the estimated number of speakers to factor out the dummy speaker embedding vectors. r333333?Tri,)max_num_speakersmax_rp_threshold sparse_searchsparse_search_volume fixed_thres nme_mat_sizerr ) r@rr7rrNMESCr forwardrrBrwrmode)rrrrrrest_num_of_spk_listseedemb_augrnmescest_num_of_spkrcomp_est_num_of_spkrrrgetEnhancedSpeakerCountis&&   (rmultiscale_segment_countscCst|jdkrtdt|jdt|jdkr$tdt|jdt||jdkr7|jdksOntd|jdd|jdd|jdd |}tj||dd }tj||dd }t|t|}}||fS) a Split multiscale embeddings and multiscale timestamps and put split scale-wise data into python lists. This formatting function is needed to make the input type as `torch.Tensor`. Args: embeddings_in_scales (Tensor): Concatenated Torch tensor containing embeddings in multiple scales timestamps_in_scales (Tensor): Concatenated Torch tensor containing timestamps in multiple scales multiscale_segment_counts (LongTensor): Concatenated Torch LongTensor containing number of segments per each scale Returns: embeddings_in_scales (list): List containing split embedding tensors by each scale timestamps_in_scales (list): List containing split timestamps tensors by each scale r6z>embeddings_in_scales Tensor should have 2 dimensions, but got .z>timestamps_in_scales Tensor should have 2 dimensions, but got rzmultiscale_segment_counts, embeddings_in_scales, and timestamps_in_scales should have the same length, but got z, z, and z respectively.r )rCr rrr)tolistsplitr)rrr split_indexrrrsplit_input_datas.(rrcCsZt|}t|||jd}t|d}t|}t|dt||jdd}|||fS)ar Estimate the number of speakers using eigendecomposition on the Laplacian Matrix. Args: affinity_mat (Tensor): N by N affinity matrix max_num_speakers (int): Maximum number of clusters to consider for each session cuda (bool): If cuda available eigendecomposition is computed on GPUs. Returns: num_of_spk (Tensor): The estimated number of speakers lambdas (Tensor): The lambda values from eigendecomposition lambda_gap (Tensor): The gap between the lambda values from eigendecomposition rr&rNr ) rrr&rsortrargmaxrr )rrrrrr lambda_gap num_of_spkrrrestimateNumofSpeakerss " rc @seZdZdZddddedfdeded ed ed ejf d d Zdej fddZ dedfdej d ed ejdej fddZ ddej ded edej fddZ dS)SpectralClusteringz Perform spectral clustering by calculating spectral embeddings then run k-means clustering algorithm on the spectral embeddings. rr Fr#r0r1n_random_trialsrr&cCs(||_||_t|d|_||_||_dS)a Initialize the variables needed for spectral clustering and k-means++. Args: n_clusters (int): Number of the estimated (or oracle) number of speakers random_state (int): Random seed that determines a random state of k-means initialization. n_random_trials (int): Number of trials with different random seeds for k-means initialization. k-means++ algorithm is executed for multiple times then the final result is obtained by taking a majority vote. cuda (bool): if cuda=True, spectral clustering is done on GPU. device (torch.device): Torch device variable r N)r0r1rrrr&)selfr0r1rrr&rrr__init__s   zSpectralClustering.__init__r cCs4|jd|jdkrtd|j||j|jd}|S)z Call self.clusterSpectralEmbeddings() function to predict cluster labels. Args: X (Tensor): Affinity matrix input Returns: labels (Tensor): Clustering label output rr z+The affinity matrix is not a square matrix.r )r rclusterSpectralEmbeddingsrr&)rrlabelsrrrrs zSpectralClustering.forwardaffinityc Cs||j||j|d}g}t|j|j|jD]}t||j||d}||qt|}t t |ddd} || } | S)a Perform k-means clustering on spectral embeddings. To alleviate the effect of randomness, k-means clustering is performed for (self.n_random_trials) times then the final labels are obtained by taking a majority vote. If speed is the major concern, self.n_random_trials should be set to 1. n_random_trials=30 is recommended to see an improved result. Args: affinity (Tensor): Affinity matrix input cuda (torch.bool): Use cuda for spectral clustering if cuda=True device (torch.device): Torch device variable Returns: labels (Tensor): clustering label output )n_spksr)rr\r1r&rr ) getSpectralEmbeddingsr0r@r1rrqrrrr) rrrr& spectral_emb labels_setrandom_state_seed_labelsstacked_labels label_indexrrrrr#s   z,SpectralClustering.clusterSpectralEmbeddingsrrrc Cslt|}t|||jd\}}|ddd|f}t|dddd}|j|ddf} | d|jS)a Calculate eigenvalues and eigenvectors to extract spectral embeddings. Args: affinity (Tensor): Affinity matrix input cuda (torch.bool): Use cuda for spectral clustering if cuda=True device (torch.device): Torch device variable Returns: labels (Tensor): clustering label output r Nr r')rrr&rrryr:r) rrrrrrrdiffusion_map_rinv_idx embeddingrrrrFs z(SpectralClustering.getSpectralEmbeddingsN)rF) __name__ __module__ __qualname____doc__rr&r<rurTensorrrrrrrrrs<   $#rc@seZdZdZdddddddddded f d ejd ed ed e dedede dede de de dejfddZ de ejejffddZ dedejfddZ dedejfddZdejfd d!Zd"S)#ra Normalized Maximum Eigengap based Spectral Clustering (NME-SC) uses Eigengap analysis to get an estimated p-value for affinity binarization and an estimated number of speakers. p_value (also referred to as p_neighbors) is for taking top p number of affinity values and convert those to 1 while convert the rest of values to 0. p_value can be also tuned on a development set without performing NME-analysis. Fixing p_value brings about significantly faster clustering speed, but the performance is limited to the development set. References: Tae Jin Park et al., Auto-Tuning Spectral Clustering for Speaker Diarization Using Normalized Maximum Eigengap, IEEE Signal Processing Letters 27 (2019), https://arxiv.org/abs/2003.02405 Args: Please refer to def __init__(). Methods: NMEanalysis(): Performs NME-analysis to estimate p_value and the number of speakers subsampleAffinityMat(nme_mat_size): Subsamples the number of speakers to reduce the computational load getPvalueList(): Generates a list containing p-values that need to be examined. getEigRatio(p_neighbors): Calculates g_p, which is a ratio between p_neighbors and the maximum eigengap getLamdaGaplist(lambdas): Calculates lambda gap values from an array contains lambda values estimateNumofSpeakers(affinity_mat): Estimates the number of speakers using lambda gap list rrTr/rFr#rrrrrruse_subsampling_for_nmermaj_vote_spk_count parallelismrr&c Csx||_||_||_||_||_||_td|_||_ d|_ td|_ ||_ |j d|_| |_| |_| |_| |_dS)a Args: mat (Tensor): Cosine similarity matrix calculated from the provided speaker embeddings. max_num_speakers (int): Maximum number of speakers for estimating number of speakers. Shows stable performance under 20. max_rp_threshold (float): Limits the range of parameter search. Clustering performance can vary depending on this range. Default is 0.25. sparse_search (bool): To increase the speed of parameter estimation, sparse_search=True limits the number of p_values we search. sparse_search_volume (int): Number of p_values we search during NME analysis. Default is 30. The lower the value, the faster NME-analysis becomes. However, a value lower than 20 might cause a poor parameter estimation. nme_mat_size (int): Targeted size of matrix for NME analysis. use_subsampling_for_nme (bool): Use subsampling to reduce the calculational complexity. Default is True. fixed_thres (float or None): A fixed threshold which can be used instead of estimating the threshold with NME analysis. If fixed_thres is float, it skips the NME analysis part. maj_vote_spk_count (bool): If True, take a majority vote on all p-values in the given range to estimate the number of speakers. The majority voting may contribute to surpress overcounting of the speakers and improve speaker counting accuracy. parallelism (bool): If True, turn on parallelism based on torch.jit.script library. cuda (bool): Use cuda for Eigen decomposition if cuda=True. device (torch.device): Torch device variable r6g|=rN)rrr(rrrrrw min_p_valuerrrrr p_value_listrr&r)r*) rrrrrrrr(rr)r*rr&rrrrs 5   zNMESC.__init__r cCs|jr ||j}ntd}g}i}||_|jjd}t|}t|}|j rQg}t |jD]\}} | tj |j| q2|D] } | tj | qDnt |jD] \}} | || qVt |jD]!\}} ||} | d| d} } | ||<| || <| ||<qht|}|j|}t|j|}t||jdst|j|j|j|jd\}}||tj}|jrtt|d} | |fS||} | |fS)aB Subsample the input matrix to reduce the computational load. Returns: est_num_of_spk (Tensor): Estimated number of speakers from NMESC approach p_hat_value (Tensor): Estimated p-value (determines how many neighboring values to be selected) r rr_)r(subsampleAffinityMatrrrw getPvalueListr,r r8r*rrjitfork getEigRatiowaitr<rBrGrrrr&rrtyper)r)rsubsample_ratioresultsest_spk_n_dictp_volumeeig_ratio_listrfuturesp_idxrfutureoutputg_prindex_nn rp_p_valuerr p_hat_valuerrrrsJ             z NMESC.forwardcCsTttdt|jjd|tj}|jdd|dd|f|_|S)a Perform subsampling of affinity matrix. This subsampling is for calculational complexity, not for performance. The smaller nme_mat_size is, - the bigger the chance of missing a speaker. - the faster p-value estimation speed (based on eigen decomposition). The recommended nme_mat_size is 250~750. However, if there are speakers who speak for very short period of time in the recording, this subsampling might make the system miss underrepresented speakers. Use this variable with caution. Args: nme_mat_size (int): The targeted matrix size Returns: subsample_ratio (float): The ratio between nme_mat_size and the original matrix size r rN)rrrwrr r3r<rB)rrr4rrrr-s,$zNMESC.subsampleAffinityMat p_neighborsc Cst|j|}t||j|j\}}}tj|d|jdd}|d}||t||j }||jj d||j } t | |gS)a For a given p_neighbors value, calculate g_p, which is a ratio between p_neighbors and the maximum eigengap values. References: Tae Jin Park et al., Auto-Tuning Spectral Clustering for Speaker Diarization Using Normalized Maximum Eigengap, IEEE Signal Processing Letters 27 (2019), https://arxiv.org/abs/2003.02405 Args: p_neighbors (int): Determines how many binary graph connections we want to keep for each row. Returns: est_num_of_spk (int): Estimated number of speakers g_p (float): The ratio between p_neighbors value and the maximum eigen gap value. NT)rr) rrrrrrrrrBrr r) rrArrrrlambda_gap_listarg_sorted_idxmax_key max_eig_gapr=rrrr1s   zNMESC.getEigRatiocCs|jdur,|jdkr,ttt|jjd|jtj|j |_ |j d}nQttt|jjd|j tj|j |_ |j rtt|j t|jtj}t|td}t|j |}tjd|j |dtj}n td|j d}|jddkrtd|S)an Generates a p-value (p_neighbour) list for searching. p_value_list must include 2 (min_p_value) since at least one neighboring segment should be selected other than itself. If fixed_thres value is specified, then only one p-value is specified. If fixed_thres is not provided, multiple p-values are searched. If sparse_search is True: - Limit the number of p-values to be searched to sparse_search_volume. - N should be at least 2 to include a number greater than 1. If sparse_search is False: - Scan all the p_values from 1 to max_N - If sparse_search is False, NMESC analysis could take more time compared to sparse_search = True. Returns: p_value_list (Tensor): Tensor containing the p_values to be searched. Ngrr6r )startendstepsz!p_value_list should not be empty.)rrrfloorrwrr r3r<r+rrrrrrlinspacerr)rr, search_volumeNrHrrrr.;s"(( zNMESC.getPvalueListN)r"r#r$r%rr&r&r<r`rurrrr-r1r.rrrrr^sT'     F<rcseZdZ      d)dedededed ed ef fd d Zd ddded ddfdejdedede dedejde dedej fddZ de e ejfdej fdd Z   !   d*d"ejd#ejd$ej d%ejdedede d&edede dedej fd'd(ZZS)+SpeakerClusteringr'TFmin_samples_for_nmescrrr)r*rcspt||_||_||_||_||_||_t dg|_ t dg|_ |jr0t d|_ dSt d|_ dS)a Clustering method for speaker diarization based on cosine similarity. NME-SC part is converted to torch.tensor based operations in NeMo 1.9. Args: min_samples_for_nmesc (int): The minimum number of samples required for NME clustering. This avoids zero p_neighbour_lists. If the input has fewer segments than min_samples, it is directed to the enhanced speaker counting mode. nme_mat_size (int): The targeted matrix size for NME analysis. sparse_search (bool): Toggle sparse search mode. If True, limit the size of p_value_list to sparse_search_volume. maj_vote_spk_count (bool): If True, take a majority vote on all p-values in the given range to estimate the number of speakers. The majority voting may contribute to surpress overcounting of the speakers and improve speaker counting accuracy. parallelism (bool): Use dynamic parallelism feature in torch.jit compiler to accelerate the p-value search. cuda (bool): Boolean variable for toggling cuda availability. rrr#N) superrrOrrr*rr)rr&rrr&)rrOrrr)r*r __class__rrres &zSpeakerClustering.__init__r'rrr/rr roracle_num_speakersrrrest_num_of_spk_enhancedrkmeans_random_trialsr c  Cst||||j|||j|j|j|j|jd } |jd|jkr)| \} } t || } n || _ | \} } |} |dkr=t |} n|dkrHt | } nt | } t| ||j|jd}| | }|S)a This function takes a cosine similarity matrix `mat` and returns the speaker labels for the segments in the given input embeddings. Args: mat (Tensor): Cosine similarity matrix (affinity matrix) calculated from the provided speaker embeddings. oracle_num_speakers (int): The number of speakers in a session, as specified by the reference transcript. Can be used as `chunk_cluster_count` in long-form clustering mode. max_num_speakers (int): The upper bound for the number of speakers in each session. max_rp_threshold (float): Limits the range of parameter search. The clustering performance can vary based on this range. The default value is 0.15. sparse_search_volume (int): The number of p_values considered during NME analysis. The default is 30. Lower values speed up the NME-analysis but might lead to poorer parameter estimations. Values below 20 are not recommended. est_num_of_spk_enhanced (int): The number of speakers estimated from enhanced speaker counting. If the value is -1, the enhanced speaker counting is skipped. fixed_thres (float): If a `fixed_thres` value is provided, the NME-analysis process will be skipped. This value should be optimized on a development set for best results. By default, it is set to -1.0, and the function performs NME-analysis to estimate the threshold. kmeans_random_trials (int): The number of random trials for initializing k-means clustering. More trials can result in more stable clustering. The default is 1. Returns: Y (LongTensor): Speaker labels (clustering output) in integer format for the segments in the given input embeddings. ) rrrrrrr)r*rr&r)r0rrr&)rrrr)r*rr&r rOrrrr<rBr)rrrSrrrrTrrUrrr@rrr0spectral_modelYrrrforward_unit_infers:,       z$SpeakerClustering.forward_unit_infer param_dictc Cs|d}|d}|d}|d}t|d}t|d}t|d}t|d} t|d } t|d } |j|||||| ||| | d S) a A function wrapper designed for inference in exported script format. Note: Dict is used to allow easy inference of the exported jit model in Triton server using easy to understand naming convention. See https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#special-conventions-for-pytorch-backend Args: param_dict (dict): Dictionary containing the arguments for speaker clustering. See `forward_infer` function for the argument information. Returns: (LongTensor): Speaker labels for the segments in the given input embeddings. embeddings timestampsrrrSrenhanced_count_thresrrr) rrrrrSrrr\rr)r<rBr` forward_infer) rrYrrrrrSrr\rrrrrrrs,zSpeakerClustering.forward(rrrrr\c  Cst|||\|_|_|jd} | jddkrtjdtjdS| jdt||jkr5|dkr5t | |j d} nt d} |dkr@|}t ||j|j|j d}|j||||| | | | dS) a Calculate the affinity matrix using timestamps and speaker embeddings, run NME analysis to estimate the best p-value, and perform spectral clustering based on the estimated p-value and the calculated affinity matrix. Caution: For compatibility with libtorch, python boolean `False` has been replaced with `torch.LongTensor(-1)`. Args: embeddings_in_scales (Tensor): List containing concatenated Torch tensor embeddings across multiple scales. The length of the list is equal to the number of scales. Each tensor has dimensions of (Number of base segments) x (Embedding Dimension). timestamps_in_scales (Tensor): List containing concatenated Torch tensor timestamps across multiple scales. The length of the list is equal to the number of scales. Each tensor has dimensions of (Total number of segments across all scales) x 2. Example: >>> timestamps_in_scales[0] = torch.Tensor([[0.4, 1.4], [0.9, 1.9], [1.4, 2.4], ... [121.2, 122.2]]) multiscale_segment_counts (LongTensor): A Torch tensor containing the number of segments for each scale. The tensor has dimensions of (Number of scales). Example: >>> multiscale_segment_counts = torch.LongTensor([31, 52, 84, 105, 120]) multiscale_weights (Tensor): Multi-scale weights used when merging affinity scores. Example: >>> multiscale_weights = torch.tensor([1.4, 1.3, 1.2, 1.1, 1.0]) oracle_num_speakers (int): The number of speakers in a session as given by the reference transcript. max_num_speakers (int): The upper bound for the number of speakers in each session. max_rp_threshold (float): Limits the range of parameter search. The clustering performance can vary based on this range. The default value is 0.15. enhanced_count_thres (int): For shorter audio recordings, the clustering algorithm might not accumulate enough speaker profiles for each cluster. Thus, the function `getEnhancedSpeakerCount` uses anchor embeddings (dummy representations) to mitigate the effects of cluster sparsity. A value of 80 is recommended for `enhanced_count_thres`. sparse_search_volume (int): The number of p_values considered during NME analysis. The default is 30. Lower values speed up the NME-analysis but might lead to poorer parameter estimations. Values below 20 are not recommended. fixed_thres (float): If a `fixed_thres` value is provided, the NME-analysis process will be skipped. This value should be optimized on a development set for best results. By default, it is set to -1.0, and the function performs NME-analysis to estimate the threshold. kmeans_random_trials (int): The number of random trials for initializing k-means clustering. More trials can result in more stable clustering. The default is 1. Returns: (LongTensor): Speaker labels for the segments in the provided input embeddings. r'rr r5r3)rr)rrrr&)rrSrrrrTrUr)rrrr rr8int64rrOrrrwrr&rX)rrrrrrSrrr\rrrUrrTrrrrr]s6C  zSpeakerClustering.forward_infer)rNr'TFFF)r'rrr^r/rr )r"r#r$r<rurrrwr&r` LongTensorrXrstrrr] __classcell__rrrQrrMds-  P.     rM)r)rrrrF)F),typingrrrr torch.linalgrrrwr&rr"r&r.r<rZr`rqrrrarrrrrrrrrrurrrrrr`rrrrnnModulerMrrrrs!&  W L% %    4 ** " ' ; , q