o wiT2@sJddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z mZmZmZmZmZddlZddlZddlmZddlmZmZddlmZdd lmZmZdd l m!Z!dd l"m#Z#dd l$m$Z$dd l%m&Z&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1m2Z2m3Z3m4Z4ddl5m6Z6ddl7m8Z8ddl9m:Z:ddl;mZ>m?Z?m@Z@mAZAmBZBmCZCmDZDddlEmFZFddlGmHZHmIZIddlJmKZKmLZLmMZMddlNmOZOddlPmQZQzddlRmSZSWneTyddlUmVZVeVd'ddZSYnwgd ZWGd!d"d"eFe/ZXGd#d$d$ejYjZZ[Gd%d&d&eZ\dS)(N) OrderedDict)Path)mode)AnyDictListOptionalTupleUnion) instantiate)LightningModuleTrainer)rank_zero_only) DictConfig open_dict) Annotation)DiarizationErrorRate)tqdm)AudioToSpeechMSDDInferDatasetAudioToSpeechMSDDTrainDataset) score_labels)MultiBinaryAccuracy)ClusteringDiarizer)ExportableEncDecModel)_MODEL_CONFIG_YAML_SPEAKER_MODEL _VAD_MODELget_available_model_names)NeuralDiarizerInferenceConfig)EncDecSpeakerLabelModel)WaveformFeaturizer) audio_rttm_mapget_embs_and_timestampsget_id_tup_dictget_scale_mapping_argmatget_uniq_id_list_from_manifestlabels_to_pyannote_objectmake_rttm_with_overlapparse_scale_configsrttm_to_labels)ModelPT)PretrainedModelInfo typecheck) AudioSignal LengthsType NeuralType) ProbsType)logging)autocast)contextmanagerccs dVdS)zauto-casting context managerN)enabledr4r4d/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/asr/models/msdd_models.pyr2Gs r2)EncDecDiarLabelModelClusterEmbeddingNeuralDiarizercs,eZdZdZedeefddZdJdede ffdd Z d d Z d d Z ddZ ddZ dKdedededefddZdeeeeffddZdeeeeffddZdeeeeffd d!Zd"d#Zd$d%Zedeeeeffd&d'Zedeeeffd(d)Zd*ejd+ejd,ejdejfd-d.Z e!d*ejd/ejd,ejdejfd0d1Z"e!d2ejd3ejd4ejd,ejde#ejejeje#ejejfff d5d6Z$d7d8Z%e&d9d:Z'd;e(dZ*dLd;e(d| dr>t j ||d|j_td|n7| drTt j||d|_td|n!|tt vrdtd|d}td |t j||d |j_|jjjj|_dS) a< Initialize speaker embedding model with model name or path passed through config. Note that speaker embedding model is loaded to `self.msdd` to enable multi-gpu and multi-node training. In addition, speaker embedding model is also saved with msdd model when `.ckpt` files are saved. cpuN.nemo) map_locationz&Speaker Model restored locally from {}.ckptzCrequested {} model name not available in pretrained models, instead titanet_large$Loading pretrained {} model from NGC model_namery)rIrRrS model_pathrrtorchcuda is_availabledevicerH global_rankendswithr restore_fromrarer1infoformatload_from_checkpointrwarningfrom_pretrainedrT_speaker_params)rkrrank_idr4r4r6rbs.       z(EncDecDiarLabelModel._init_speaker_modelc Cst|d|dddd}d|vr!|ddur!td|dSt|j|j|j|j||j j j |j d|j jd }|j|_|}|j}|d}tjjj||||d dd|d d |d dd S)N sample_rate int_valuesF)rr augmentormanifest_filepathJCould not load dataset as `manifest_filepath` was None. Provided config : ) remb_dirrusoft_label_thres featurizerr_rNrOr batch_size drop_last num_workersr pin_memorydatasetr collate_fnrshufflerr)r getr1rrrrrurrIr]r_rNrHr collectiondata_collectionmsdd_train_collate_fnrutilsdata DataLoader)rkconfigrr collate_dsrrr4r4r6__setup_dataloader_from_configs<    z3EncDecDiarLabelModel.__setup_dataloader_from_configFremb_dictemb_seqclus_label_dictc Cs|dd}d|vr|ddurtd|dSt|d||||j|j|jjjd|d }|j |_ |}|j } |d} t j jj|| | |dd||dd |d dd S) NrFrr) rrrrr seq_eval_moder_use_single_scale_clusrOrrrrrr)rr1rrrrr^r]r_rrmsdd_infer_collate_fnrrrr) rkrrrrrOrrrrrr4r4r6$__setup_dataloader_from_config_infers8     z9EncDecDiarLabelModel.__setup_dataloader_from_config_infertrain_data_configcC|j|d|_dSN)r)3_EncDecDiarLabelModel__setup_dataloader_from_config _train_dl)rkrr4r4r6setup_training_data z(EncDecDiarLabelModel.setup_training_dataval_data_layer_configcCrr)r_validation_dl)rkrr4r4r6setup_validation_datarz*EncDecDiarLabelModel.setup_validation_datatest_data_configcCs,|jr|j||j|j|j|jd|_dSdS)N)rrrrrO)rO9_EncDecDiarLabelModel__setup_dataloader_from_config_inferemb_sess_test_dict emb_seq_testclus_test_label_dict_test_dlrkrr4r4r6setup_test_data"s z$EncDecDiarLabelModel.setup_test_datacCsdS)zu MSDD does not use multiple_test_data template. This function is a placeholder for preventing error. Nr4rr4r4r6setup_multiple_test_data,sz-EncDecDiarLabelModel.setup_multiple_test_datacCs|jdur|jSdSN)rrvr4r4r6test_dataloader2s z$EncDecDiarLabelModel.test_dataloaderc Cslt|jdrt|jjd}nt}td|tdttdttdttdttdttdtd S) N _sample_rate)freq)BT)r)rCrD)rr)rrrrrrfeaturesfeature_lengthms_seg_timestamps ms_seg_countsclus_label_index scale_mappingtargets)hasattrr]r-rr/r.r0)rk audio_eltyper4r4r6 input_types6s       z EncDecDiarLabelModel.input_typescCsttdttdtdS)Nr)rrrr)probs scale_weights)rr/r0rvr4r4r6 output_typesFs   z!EncDecDiarLabelModel.output_typesembsrrc s|djd|jd}tj||dddfddtdtD}g}t|D]/}g}tD]} ||| } |||| | ddfq8t| ddd} || q0t|} | S) a/ Reshape the given tensor and organize the embedding sequence based on the original sequence counts. Repeat the embeddings according to the scale_mapping information so that the final embedding sequence has the identical length for all scales. Args: embs (Tensor): Merged embeddings without zero-padding in the batch. See `ms_seg_counts` for details. Shape: (Total number of segments in the batch, emb_dim) scale_mapping (Tensor): The element at the m-th row and the n-th column of the scale mapping matrix indicates the (m+1)-th scale segment index which has the closest center distance with (n+1)-th segment in the base scale. Example: scale_mapping_argmat[2][101] = 85 In the above example, it means that 86-th segment in the 3rd scale (python index is 2) is mapped with 102-th segment in the base scale. Thus, the longer segments bound to have more repeating numbers since multiple base scale segments (since the base scale has the shortest length) fall into the range of the longer segments. At the same time, each row contains N numbers of indices where N is number of segments in the base-scale (i.e., the finest scale). Shape: (batch_size, scale_n, self.diar_window_length) ms_seg_counts (Tensor): Cumulative sum of the number of segments in each scale. This information is needed to reconstruct the multi-scale input matrix during forward propagating. Example: `batch_size=3, scale_n=6, emb_dim=192` ms_seg_counts = [[8, 9, 12, 16, 25, 51], [11, 13, 14, 17, 25, 51], [ 9, 9, 11, 16, 23, 50]] In this function, `ms_seg_counts` is used to get the actual length of each embedding sequence without zero-padding. Returns: ms_emb_seq (Tensor): Multi-scale embedding sequence that is mapped, matched and repeated. The longer scales are less repeated, while shorter scales are more frequently repeated following the scale mapping tensor. rdimcg|] }||qSr4r4.0irZ split_emb_tupr4r6 |z7EncDecDiarLabelModel.get_ms_emb_seq..NrG) shapersplitviewtolistrangerXr@stackpermute) rkrrrrbatch_emb_listms_emb_seq_list batch_idx feats_list scale_index repeat_matrepp ms_emb_seqr4rr6get_ms_emb_seqOs+       z#EncDecDiarLabelModel.get_ms_emb_seqrcs~|djd|jd}tj||dddfddtdtD}g}t|D]p}||} t||} t| d| ||} g} tD]E} g}t|jj D]2}| | |k }t |szt |jjjjj|j}n tj||| |dd}||qZ| t|qPt| }||q0t|dddd }| |j}|jrJd |S) aA Calculate the cluster-average speaker embedding based on the ground-truth speaker labels (i.e., cluster labels). Args: embs (Tensor): Merged embeddings without zero-padding in the batch. See `ms_seg_counts` for details. Shape: (Total number of segments in the batch, emb_dim) clus_label_index (Tensor): Merged ground-truth cluster labels from all scales with zero-padding. Each scale's index can be retrieved by using segment index in `ms_seg_counts`. Shape: (batch_size, maximum total segment count among the samples in the batch) ms_seg_counts (Tensor): Cumulative sum of the number of segments in each scale. This information is needed to reconstruct multi-scale input tensors during forward propagating. Example: `batch_size=3, scale_n=6, emb_dim=192` .. code:: python ms_seg_counts = [ [ 8, 9, 12, 16, 25, 51], [11, 13, 14, 17, 25, 51], [ 9, 9, 11, 16, 23, 50] ] Counts of merged segments: (121, 131, 118) embs has shape of (370, 192) clus_label_index has shape of (3, 131) Shape: (batch_size, scale_n) Returns: ms_avg_embs (Tensor): Multi-scale cluster-average speaker embedding vectors. These embedding vectors are used as reference for each speaker to predict the speaker label for the given multi-scale embedding sequences. Shape: (batch_size, scale_n, emb_dim, self.num_spks_per_model) rrrcrr4r4rrr4r6rrzCEncDecDiarLabelModel.get_cluster_avg_embs_model..NrGrzVms_avg_embs.requires_grad = True. ms_avg_embs should be detached from the torch graph.)rrrrrrrXsumrImax_num_of_spksclonedetachanyzerosrarer^decoder emb_sizestormeanr@rrfloat requires_grad)rkrrrrrrms_avg_embs_listroracle_clus_idx max_seq_lenclus_label_index_batchsession_avg_emb_set_listr spk_set_listidx_whereavg_embsession_avg_emb_set ms_avg_embsr4rr6get_cluster_avg_embs_models6+        z/EncDecDiarLabelModel.get_cluster_avg_embs_modelprocessed_signalprocessed_signal_lenrc Cs|j}t|j|}|jjj}t|j ddd|j }ggg} } } t |} |j d} t | D]s}t |jD]b}|||}t|||d|D]M\}\}}t|t|}}t|||}t ||t j|}||dd||f|ddd||f<| || ||qQq=| ||dq6t | |}t | |}t | |}|dkrt gd}}t | }nt |jjt | d|}t | |d}||f}||||fS)aw Load acoustic feature from audio segments for each scale and save it into a torch.tensor matrix. In addition, create variables containing the information of the multiscale subsegmentation information. Note: `self.emb_batch_size` determines the number of embedding tensors attached to the computational graph. If `self.emb_batch_size` is greater than 0, speaker embedding models are simultaneosly trained. Due to the constrant of GPU memory size, only a subset of embedding tensors can be attached to the computational graph. By default, the graph-attached embeddings are selected randomly by `torch.randperm`. Default value of `self.emb_batch_size` is 0. Args: processed_signal (Tensor): Zero-padded Feature input. Shape: (batch_size, feat_dim, the longest feature sequence length) processed_signal_len (Tensor): The actual legnth of feature input without zero-padding. Shape: (batch_size,) ms_seg_timestamps (Tensor): Timestamps of the base-scale segments. Shape: (batch_size, scale_n, number of base-scale segments, self.num_spks_per_model) ms_seg_counts (Tensor): Cumulative sum of the number of segments in each scale. This information is needed to reconstruct the multi-scale input matrix during forward propagating. Shape: (batch_size, scale_n) Returns: ms_mel_feat (Tensor): Feature input stream split into the same length. Shape: (total number of segments, feat_dim, self.frame_per_sec * the-longest-scale-length) ms_mel_feat_len (Tensor): The actual length of feature without zero-padding. Shape: (total number of segments,) seq_len (Tensor): The length of the input embedding sequences. Shape: (total number of segments,) detach_ids (tuple): Tuple containing both detached embeding indices and attached embedding indices scale_dictrNr)rminrNritemr]r^rrWrur`rrrrZ enumeraterrrfloat32r@rtensorarange manual_seedrH current_epochrandperm)rkrrrrr_emb_batch_sizefeat_dimmax_sample_countms_mel_feat_len_listsequence_lengths_listms_mel_feat_listtotal_seg_countrr scale_idx scale_seg_numksttend _features ms_mel_featms_mel_feat_lenseq_lenattacheddetached detach_idsr4r4r6get_ms_mel_feats<.     $",   z$EncDecDiarLabelModel.get_ms_mel_featcCs|j||||d\}}||fS)z Wrapper function for inference case. This `forward_infer` is only used during inference, where `forward` is used for training and validation. rlengthr r)ra)rk input_signalinput_signal_length emb_vectorsrpredsrr4r4r6 forward_infers z"EncDecDiarLabelModel.forward_infercCsR|jjj||d\}} ||| ||\} } } } t=|jj|jjj| | d| | dd\}}t| j d|j d |j }| || dddf<Wdn1s[wY|jj t| ddkr|jjj| | d| | dd\}}||| dddf<||||}|||||}|j|| ||d\}}||fS)z9Function to compute forward pass for training/validation.)r1r0rG) audio_signalr0rNr/)rarer]r.rno_gradevalforward_for_exportrrrrrtrainrXrr)rkrrrrrrrrrr6audio_signal_lensequence_lengthsr-logitsembs_drembs_arr r4rr4r4r6forward)s4       zEncDecDiarLabelModel.forwardbatchrc Cs|\}}}}}}} tdd|D} |j||||||| d\} } |j| | | d} || | | tj|j\}} } |j d| dd|j d|j j d d dd|j d |dd|j d| iS) z"Function to compute training step.cSg|]}|dqSrr4rxr4r4r6rPz6EncDecDiarLabelModel.training_step..rrlabels target_lensrgT sync_dist learning_raterlr train_f1_acc) rrrr@rgrir empty_cachecomputelog _optimizer param_groupsreset)rkrArrrrrrrrr<r4_rgf1_accr4r4r6 training_stepMs(    z"EncDecDiarLabelModel.training_steprdataloader_idxc Cs|\}}}}}} } tdd|D} |j|||||| | d\} } |j| | | d}|| | | |j\}} } |jd|dd|jd|dd||d S) z$Function to compute validation step.cSrBrCr4rDr4r4r6rhrFz8EncDecDiarLabelModel.validation_step..rrGval_lossTrJ val_f1_accrYrZ)rrr@rgrjrPrQ)rkrArrXrrrrrrrr<r4rUrgrVr4r4r6validation_stepes&  z$EncDecDiarLabelModel.validation_stepoutputscCs\tdd|D}|j\}}}|j|jd|dd|jd|dd||dS)NcSrB)rYr4rDr4r4r6r~rFzCEncDecDiarLabelModel.multi_validation_epoch_end..rYTrJrZr[)rrrrjrPrTrQ)rkr]rX val_loss_meanrVrUr4r4r6multi_validation_epoch_end}s z/EncDecDiarLabelModel.multi_validation_epoch_endcCsLtdd|D}|j\}}}|j|jd|dd||dS)NcSrB) test_lossr4rDr4r4r6rrFz=EncDecDiarLabelModel.multi_test_epoch_end.. test_f1_accTrJ)r`ra)rrrrhrPrTrQ)rkr]rXtest_loss_meanrVrUr4r4r6multi_test_epoch_ends z)EncDecDiarLabelModel.multi_test_epoch_endcCsH|j\}}}t|jj}tt|jjj }||}||fS)aL Calculate F1 score and accuracy of the predicted sigmoid values. Returns: f1_score (float): F1 score of the estimated diarized speaker label sequences. simple_acc (float): Accuracy of predicted speaker labels: (total # of correct labels)/(total # of sigmoid values) ) rhrPrrtrueboolprodrrr)rkf1_scorerU num_correct total_count simple_accr4r4r6compute_accuraciess  z'EncDecDiarLabelModel.compute_accuraciesr)F)r)0__name__ __module__ __qualname____doc__ classmethodrr+rDrr rQrcrJrbrdictrrr rrrrrrpropertystrr/rrrTensorrr7rr r.r5r,r@listrWrWr\r_rcrk __classcell__r4r4rlr6r7Ps , "# "  9 G  N #"  r7cseZdZdZdededeeffdd ZddZd e d e fd d Z d e e de e effddZde d e de d e fddZde de fddZddZddZddZddZZS) r8a This class is built for calculating cluster-average embeddings, segmentation and load/save of the estimated cluster labels. The methods in this class is used for the inference of MSDD models. Args: cfg_diar_infer (DictConfig): Config dictionary from diarization inference YAML file cfg_msdd_model (DictConfig): Config dictionary from MSDD model checkpoint file Class Variables: self.cfg_diar_infer (DictConfig): Config dictionary from diarization inference YAML file cfg_msdd_model (DictConfig): Config dictionary from MSDD model checkpoint file self._speaker_model (class `EncDecSpeakerLabelModel`): This is a placeholder for class instance of `EncDecSpeakerLabelModel` self.scale_window_length_list (list): List containing the window lengths (i.e., scale length) of each scale. self.scale_n (int): Number of scales for multi-scale clustering diarizer self.base_scale_index (int): The index of the base-scale which is the shortest scale among the given multiple scales cfg_diar_inferrI speaker_modelcsbt||_||_||_t|jjjjj |_ t |j |_ t |j d|_ t|j|jd|_dS)NrG)rErx)rPrQrw _cfg_msddrerurRrSrTrUscale_window_length_listrXrZbase_scale_indexrclus_diar_model)rkrwrIrxrlr4r6rQs   zClusterEmbedding.__init__cCs:|jjjjj|_||jjj|jjj \|_ |_ |_ }dS)za Launch clustering diarizer to prepare embedding vectors and clustering results. N) rwrR clusteringrTmax_num_speakersrun_clustering_diarizerrytest_dsrrrrr)rkrUr4r4r6prepare_cluster_embs_infersz+ClusterEmbedding.prepare_cluster_embs_inferbase_clus_label_dictsession_scale_mapping_dictc Csddt|jD}|D]j\}}tdd||D}|||j|<t|jdD]K}g}||jd|jdks@Jdt||} t| dD]$} | ||vr_t|||| k} n t |dkrgdn|d} | | qL||||<q,q|S) a In multi-scale speaker diarization system, clustering result is solely based on the base-scale (the shortest scale). To calculate cluster-average speaker embeddings for each scale that are longer than the base-scale, this function assigns clustering results for the base-scale to the longer scales by measuring the distance between subsegment timestamps in the base-scale and non-base-scales. Args: base_clus_label_dict (dict): Dictionary containing clustering results for base-scale segments. Indexed by `uniq_id` string. session_scale_mapping_dict (dict): Dictionary containing multiscale mapping information for each session. Indexed by `uniq_id` string. Returns: all_scale_clus_label_dict (dict): Dictionary containing clustering labels of all scales. Indexed by scale_index in integer format. cSsi|]}|iqSr4r4rrr4r4r6 zAClusterEmbedding.assign_labels_to_longer_segs..cSrBrCr4rDr4r4r6rrFzAClusterEmbedding.assign_labels_to_longer_segs..rGrz]The number of base scale labels does not match the segment numbers in uniq_scale_mapping_dictr) rrZitemsnparrayr{rmaxrrXr@) rkrrall_scale_clus_label_dictuniq_iduniq_scale_mapping_dictbase_scale_clus_labelrnew_clus_label max_indexseg_idxseg_clus_labelr4r4r6assign_labels_to_longer_segss$    z-ClusterEmbedding.assign_labels_to_longer_segs clus_labelsemb_scale_seq_dictc Csdd||jD}|D].}|d}t|ddd}dd|dd D\}}|||||gq|d|djd} || fS) aS Retrieve base scale clustering labels from `emb_scale_seq_dict`. Args: clus_labels (list): List containing cluster results generated by clustering diarizer. emb_scale_seq_dict (dict): Dictionary containing multiscale embedding input sequences. Returns: base_clus_label_dict (dict): Dictionary containing start and end of base scale segments and its cluster label. Indexed by `uniq_id`. emb_dim (int): Embedding dimension in integer. cSi|]}|gqSr4r4rkeyr4r4r6r rz=ClusterEmbedding.get_base_clus_label_dict..rrrUcSsg|] }tt|dqS)r)roundrrDr4r4r6rz=ClusterEmbedding.get_base_clus_label_dict..rGr)r{keysrrWr@r) rkrrrlinerlabelr%r&emb_dimr4r4r6get_base_clus_label_dicts z)ClusterEmbedding.get_base_clus_label_dictspeaker_mapping_dictcs"t_fddD}|\}}||}D]g} | D]^\} } t| tkr@t t | } n| } || | } t | }t | }t|j}|D]}| ||k}tj|dd|dd|f<qZ|durdd|| D}nd}||d|| | <q-q%||fS)a MSDD requires cluster-average speaker embedding vectors for each scale. This function calculates an average embedding vector for each cluster (speaker) and each scale. Args: emb_scale_seq_dict (dict): Dictionary containing embedding sequence for each scale. Keys are scale index in integer. clus_labels (list): Clustering results from clustering diarizer including all the sessions provided in input manifest files. speaker_mapping_dict (dict): Speaker mapping dictionary in case RTTM files are provided. This is mapping between integer based speaker index and speaker ID tokens in RTTM files. Example: {'en_0638': {'speaker_0': 'en_0638_A', 'speaker_1': 'en_0638_B'}, 'en_4065': {'speaker_0': 'en_4065_B', 'speaker_1': 'en_4065_A'}, ...,} session_scale_mapping_dict (dict): Dictionary containing multiscale mapping information for each session. Indexed by `uniq_id` string. Returns: emb_sess_avg_dict (dict): Dictionary containing speaker mapping information and cluster-average speaker embedding vector. Each session-level dictionary is indexed by scale index in integer. output_clus_label_dict (dict): Subegmentation timestamps in float type and Clustering result in integer type. Indexed by `uniq_id` keys. cs*i|]}|ddjdDqS)cSrr4r4rr4r4r6r4rzDClusterEmbedding.get_cluster_avg_embs...rG)rZrrrrkr4r6r3sz9ClusterEmbedding.get_cluster_avg_embs..rrNcSsi|]\}}||qSr4r4)rrttm_keyclus_keyr4r4r6rLs)mappingavg_embs)rXrrZrrrtyperurrrrsetrtrr~r)rkrrrremb_sess_avg_dictoutput_clus_label_dictrrrr _emb_tensor emb_tensorclus_label_listspk_set label_arrayrspk_idx selected_embsinv_mapr4rr6get_cluster_avg_embss4      z%ClusterEmbedding.get_cluster_avg_embsrrcCs^||jj_||jj_|jjj|_tj |jd|_ tj |j dd|jjj j |j_|jjjj j|jjd<|jjjj |jjj_ |jj}t|trKt|n|}tj|dd}td|jjdtd||jj|jjd }|d ur}|\}}}nd \}}t|jj|jj|_||j} ||} | |} |!| | || \} } | | d <| | | |fS) a If no pre-existing data is provided, run clustering diarizer from scratch. This will create scale-wise speaker embedding sequence, cluster-average embeddings, scale mapping and base scale clustering labels. Note that speaker embedding `state_dict` is loaded from the `state_dict` in the provided MSDD checkpoint. Args: manifest_filepath (str): Input manifest file for creating audio-to-RTTM mapping. emb_dir (str): Output directory where embedding files and timestamp files are saved. Returns: emb_sess_avg_dict (dict): Dictionary containing cluster-average embeddings for each session. emb_scale_seq_dict (dict): Dictionary containing embedding tensors which are indexed by scale numbers. base_clus_label_dict (dict): Dictionary containing clustering results. Clustering results are cluster labels for the base scale segments. pred_rttmsTexist_okrt)indentzMultiscale Weights: zClustering Parameters: )rN)NNsession_scale_mapping)"rwrRrout_dirr|rr_out_dirospathjoin out_rttm_dirmakedirsr}rT_cluster_paramsrSrtrurVrrqjsondumpsr1rdiarizerr"$multiscale_embeddings_and_timestamps_embs_and_timestamps get_scale_mapload_emb_scale_seq_dictload_clustering_labelsr)rkrrcluster_paramsclustering_params_strscoresmetricrrUrrrrrr4r4r6rSs<            z(ClusterEmbedding.run_clustering_diarizercCs*i}|D] \}}t|}|||<q|S)a Save multiscale mapping data into dictionary format. Args: embs_and_timestamps (dict): Dictionary containing embedding tensors and timestamp tensors. Indexed by `uniq_id` string. Returns: session_scale_mapping_dict (dict): Dictionary containing multiscale mapping information for each session. Indexed by `uniq_id` string. )rr$)rkembs_and_timestampsrruniq_embs_and_timestampsscale_mapping_dictr4r4r6rs   zClusterEmbedding.get_scale_mapcCsDtj|dd|jd}tj|}|std|d||fS)a Check whether the laoded clustering label file is including clustering results for all sessions. This function is used for inference mode of MSDD. Args: out_dir (str): Path to the directory where clustering result files are saved. Returns: file_exists (bool): Boolean that indicates whether clustering result file exists. clus_label_path (str): Path to the clustering label output file. speaker_outputssubsegments_scalez_cluster.labelzClustering label file z does not exist.)rrrr{existsr1r)rkrclus_label_path file_existsr4r4r6check_clustering_labelss z(ClusterEmbedding.check_clustering_labelscCsT||\}}td|t| }|}Wd|S1s#wY|S)ar Load clustering labels generated by clustering diarizer. This function is used for inference mode of MSDD. Args: out_dir (str): Path to the directory where clustering result files are saved. Returns: emb_scale_seq_dict (dict): List containing clustering results in string format. z Loading cluster label file from N)rr1ropen readlines)rkrrrfrr4r4r6rs    z'ClusterEmbedding.load_clustering_labelsc Cst|jjjjj}ddtt|D}tt|D]E}tj |ddd|d}t d|d|t |d  }t|}Wd n1sIwY|D]\}} | ||<qR|||<q|S) a Load saved embeddings generated by clustering diarizer. This function is used for inference mode of MSDD. Args: out_dir (str): Path to the directory where embedding pickle files are saved. Returns: emb_scale_seq_dict (dict): Dictionary containing embedding tensors which are indexed by scale numbers. cSsi|]}|dqSrr4rr4r4r6rrz.r embeddingsrz_embeddings.pklz'Loading embedding pickle file of scale:z at rbN)rurwrRrSrTrUrrXrrrr1rrpklloadr) rkrwindow_len_listrr pickle_path input_filerrvalr4r4r6rs     z(ClusterEmbedding.load_emb_scale_seq_dict)rlrmrnrorrrrQrrrrrsrWrqrrrrrrrrvr4r4rlr6r8s4 % ?@r8cseZdZdZdeeefffdd ZddZe de fdd Z dHd e d e fd dZ deeeffddZdeeeeeejfd ejfddZdee deedeejd eejfddZddZed eeeeeeeffddZdejdejdedeed eejejeff d d!Zd"eejd#eed$ed%d eejejejffd&d'Zd"eejdeed eejejejffd(d)Z ed eeejeejeejffd*d+Z!deejd,e"d eeeeeffd-d.Z#e$ / 0 1dId2e d3e d4ee d5e%fd6d7Z& 8 9 0 0 0 1dJd:e d;edeed?ee d5e%d ee'ee'ffd@dAZ(dBe d=eed>eedCe)j*d;ed.cg|]}|qSr4r4rD digit_mapr4r6rrFrNrG)tuplerXrrqzipsortedrrrrrsqueezerrwr) rkrall_tupsr n_est_spks total_lensum_pred_dim_tuppred_matdim_tup_endr4rr6 get_pred_maths"      " zNeuralDiarizer.get_pred_mat uniq_id_listtest_data_collection preds_listc sZt|||}dd|D|D]\}}||}|d|<qfdd|D}|S)a} Merge multiple sequence inference outputs into a session level result. Args: uniq_id_list (list): List containing `uniq_id` values. test_data_collection (collections.DiarizationLabelEntity): Class instance that is containing session information such as targeted speaker indices, audio filepaths and RTTM filepaths. preds_list (list): List containing tensors filled with sigmoid values. Returns: output_list (list): List containing session-level estimated prediction matrix. cSrr4r4rrr4r4r6rrz.rcrr4r4r& output_dictr4r6rrFz.)r#rr" unsqueeze) rkr#r$r% session_dictrrr output_listr4r'r6get_integrated_preds_lists  z(NeuralDiarizer.get_integrated_preds_listcCs"|j|j_|j|j_|j|j_dS)zcAssign dictionaries containing the clustering results from the class instance `cluster_embeddings`.N)rrrr)rkcluster_embeddingsr4r4r6get_emb_clus_infers  z!NeuralDiarizer.get_emb_clus_infercsRjdj_j\}}tjjjj j }fdd|DS)a Launch diarization pipeline which starts from VAD (or a oracle VAD stamp generation), initialization clustering and multiscale diarization decoder (MSDD). Note that the result of MSDD can include multiple speakers at the same time. Therefore, RTTM output of MSDD needs to be based on `make_rttm_with_overlap()` function that can generate overlapping timestamps. `self.run_overlap_aware_eval()` function performs DER evaluation. Tcsg|]}|qSr4)run_overlap_aware_eval)r thresholdr%rkr4r6rsz*NeuralDiarizer.diarize..) rrrrOr.run_pairwise_diarizationrur^rRrTsigmoid_threshold)rk targets_listsignal_lengths_list thresholdsr4r1r6rs  zNeuralDiarizer.diarizesignalsr3diar_window_indexc Cst|}tjtj|jd}tdd|jj |D}t t |j D]}||j t|d|j |jd} } | | } | |jdkr|| | } || | ddddft|j |jd||j } }| |j |k}t|rtj| |dd|dddd|f<| |j krtj| t|j | | jd| jd|jgdd} q't|j |jd|jd|j} d} q'|| | fS)a$ This function is only used when `split_infer=True`. This module calculates cluster-average embeddings for the given short range. The range length is set by `self.diar_window_length`, and each cluster-average is only calculated for the specified range. Note that if the specified range does not contain some speakers (e.g. the range contains speaker 1, 3) compared to the global speaker sets (e.g. speaker 1, 2, 3, 4) then the missing speakers (e.g. speakers 2, 4) are assigned with zero-filled cluster-average speaker embedding. Args: signals (Tensor): Zero-padded Input multi-scale embedding sequences. Shape: (length, scale_n, emb_vectors, emb_dim) emb_vectors (Tensor): Cluster-average multi-scale embedding vectors. Shape: (length, scale_n, emb_vectors, emb_dim) diar_window_index (int): Index of split diarization wondows. test_data_collection (collections.DiarizationLabelEntity) Class instance that is containing session information such as targeted speaker indices, audio filepath and RTTM filepath. Returns: return emb_vectors_split (Tensor): Cluster-average speaker embedding vectors for each scale. emb_seq (Tensor): Zero-padded multi-scale embedding sequences. seq_len (int): Length of the sequence determined by `self.diar_window_length` variable. rcSrBrCr4rDr4r4r6rrFz4NeuralDiarizer.get_range_average..rGNrr)r zeros_likerrsplitextbasename audio_filerrrrrX target_spksrrrrrcatrrr)rkr7r3r8r$emb_vectors_splitrclus_label_tensorrr%r&r*target_clus_label_tensorr seg_lengthtarget_clus_label_boolr4r4r6get_range_averagesD  $    z NeuralDiarizer.get_range_average test_batch_test_data_collectionrrwcCs|\}}}}ggg}} } tt|jd|j} t|j|j| |_t|jdD]1} || || || } }}t| D]}| | |||\}}}| || || |qCq/t | |}t |  |} t|  |} || | fS)a This function is only used when `get_range_average` function is called. This module calculates cluster-average embeddings for the given short range. The range length is set by `self.diar_window_length`, and each cluster-average is only calculated for the specified range. Args: test_batch: (list) List containing embedding sequences, length of embedding sequences, ground truth labels (if exists) and initializing embedding vectors. test_data_collection: (list) List containing test-set dataloader contents. test_data_collection includes wav file path, RTTM file path, clustered speaker indices. Returns: sess_emb_vectors (Tensor): Tensor of cluster-average speaker embedding vectors. Shape: (batch_size, scale_n, emb_dim, 2*num_of_spks) sess_emb_seq (Tensor): Tensor of input multi-scale embedding sequences. Shape: (batch_size, length, scale_n, emb_dim) sess_sig_lengths (Tensor): Tensor of the actucal sequence length without zero-padding. Shape: (batch_size) rGr) rceilrrrrWrrrrDr@rr)rkrErFr_signalssignal_lengths_targets _emb_vectorssess_emb_vectors sess_emb_seqsess_sig_lengths split_countr$r7r3r$r8r?rr*r4r4r6get_range_clus_avg_emb s$       z%NeuralDiarizer.get_range_clus_avg_embcCs|\}}}}|jjjjjrftt|jd|j  }|j |||jj d\}} } t |jj| | |dd\} } Wdn1sDwY| t|||j d} | ddd|jdddf} nt |jj|||dd\} } Wdn1swYt| jd|j|_t| jd|j| jd} t| jd|j| jd}| | ddd| jdddf<| ||fS)a Launch forward_infer() function by feeding the session-wise embedding sequences to get pairwise speaker prediction values. If split_infer is True, the input audio clips are broken into short sequences then cluster average embeddings are calculated for inference. Split-infer might result in an improved results if calculating clustering average on the shorter tim-espan can help speaker assignment. Args: test_batch: (list) List containing embedding sequences, length of embedding sequences, ground truth labels (if exists) and initializing embedding vectors. test_data_collection: (list) List containing test-set dataloader contents. test_data_collection includes wav file path, RTTM file path, clustered speaker indices. Returns: preds (Tensor): Tensor containing predicted values which are generated from MSDD model. targets (Tensor): Tensor containing binary ground-truth values. signal_lengths (Tensor): The actual Session length (number of steps = number of base-scale segments) without zero padding. rG)rN)r1r2r3rrrr)r^rRrrT split_inferrrGrrrrWrPrr2r5reshaperXrrr)rkrEr$r7rIrJr3rOrLrMrN_predsrr4rr4r4r6 diar_infer:s4     "   zNeuralDiarizer.diar_infercCsn|jj|_|j|jjj|jdg}ggg}}}t|jjjj}dd|jj D}t t |j D]W\}}|\} } } } | |d| jd||||d|d\} }} |jjjjjrn|j| || |tt| d|tt|d|tt| dq8|jjjjjr|j\}}td|dd |d||||}|||fS) a Setup the parameters needed for batch inference and run batch inference. Note that each sample is pairwise speaker input. The pairwise inference results are reconstructed to make session-wise prediction results. Returns: integrated_preds_list: (list) List containing the session-wise speaker predictions in torch.tensor format. targets_list: (list) List containing the ground-truth labels in matrix format filled with 0 or 1. signal_lengths_list: (list) List containing the actual length of each sequence in session. rcSsg|]}|qSr4r4)rdr4r4r6rsz;NeuralDiarizer.run_pairwise_diarization..rrGzTest Inference F1 score. .4fz, simple Acc. )rrrrrErr8r%rrrrrr@rrTr^rRrTrrhextendrurrrkr1rr,)rkcumul_sample_countr%r4r5r#r$sidxrEr7rIrJr3r4rrgrjintegrated_preds_listr4r4r6r2ms.     z'NeuralDiarizer.run_pairwise_diarizationr0c Cstd|dd|jd|jdg}|jjjj}t|}t |j D].\}\}}t ||jj ||d|j|j |j|j|jd \} } t|| | |||jjd} || q#td |S) a5 Based on the predicted sigmoid values, render RTTM files then evaluate the overlap-aware diarization results. Args: preds_list: (list) List containing predicted pairwise speaker labels. threshold: (float) A floating-point threshold value that determines overlapped speech detection. - If threshold is 1.0, no overlap speech is detected and only detect major speaker. - If threshold is 0.0, all speakers are considered active at any time step. z [Threshold: rWz] [use_clus_as_main=z] [diar_window=]T)r0 infer_overlaprrrrr)collarignore_overlapverbosez )r1rrrrrErrr!rrr'rrrrrrr^r`r@) rkr%r0r]rrttm_mapr$r^r_ all_referenceall_hypothesisoutputr4r4r6r/s@     z%NeuralDiarizer.run_overlap_aware_evalvad_multilingual_marblenetNFr~vad_model_nameryr`cCs0t|rtjntjtj||||d}||S)aa Instantiate a `NeuralDiarizer` to run Speaker Diarization. Args: model_name (str): Path/Name of the neural diarization model to load. vad_model_name (str): Path/Name of the voice activity detection (VAD) model to load. map_location (str): Optional str to map the instantiated model to a device (cpu, cuda). By default, (None), it will select a GPU if available, falling back to CPU otherwise. verbose (bool): Enable verbose logging when loading models/running diarization. Returns: `NeuralDiarizer` )diar_model_pathvad_model_pathryr`)r1setLevelINFOWARNINGr init_config)rAr~rfryr`rEr4r4r6rszNeuralDiarizer.from_pretrained@rGaudio_filepathrr max_speakers num_speakersrc Cs|r tj|ddtj|dc}tj|d} |dddd|ddd g} t| d } | d d d | DWdn1s@wY|j| ||||||d| |j j j _ | t|dt|jd} Wdt| S1suwYt| S)a Run the `NeuralDiarizer` inference pipeline. Args: audio_filepath (str, list): Audio path to run speaker diarization on. max_speakers (int): If known, the max number of speakers in the file(s). num_speakers (int): If known, the exact number of speakers in the file(s). batch_size (int): Batch size when running inference. num_workers (int): Number of workers to use in data-loading. out_dir (str): Path to store intermediate files during inference (default temp directory). Returns: `pyannote.Annotation` for each audio path, containing speaker labels and segment timestamps. Tr)dirz manifest.jsonrNinfer-)rnoffsetdurationrtextrp rttm_filepath uem_filepathw css|]}t|VqdSr)rrrDr4r4r6 sz*NeuralDiarizer.__call__..) manifest_pathrorprrrr`z /pred_rttms/z.rttm)rrrrrrrwrite_initialize_configsrrErrrr)rstemr&) rkrnrrrorprr`rr|metarpred_labels_clusr4r4r6__call__sB     zNeuralDiarizer.__call__r|rcCsd||j_||j_||jj_||jj_||j_|du|jjjj_ |r(||jjjj_ | |j |jdSr) r^rrrRrrr`r}rToracle_num_speakersr~rr)rkr|rorprrrr`r4r4r6r~s   z"NeuralDiarizer._initialize_configscCstS)r;)r7rD)rAr4r4r6rD0sz$NeuralDiarizer.list_available_models)r)reNF)rmrGNNNF)/rlrmrnror rrrQrrrsrrrrrr rWrrtr"rr,r.r7rrrrrDrrPrTr2rr/rprerrrrrr~r+rDrvr4r4rlr6r9s ** ( J - 3,( .  ; r9r)]rnrrpicklerr collectionsrpathlibr statisticsrtypingrrrrr r numpyrr hydra.utilsr lightning.pytorchr r lightning.pytorch.utilitiesr omegaconfrr pyannote.corerpyannote.metrics.diarizationrr-nemo.collections.asr.data.audio_to_diar_labelrr nemo.collections.asr.metrics.derr-nemo.collections.asr.metrics.multi_binary_accrnemo.collections.asr.modelsr%nemo.collections.asr.models.asr_modelr/nemo.collections.asr.models.clustering_diarizerrrrr3nemo.collections.asr.models.configs.diarizer_configr(nemo.collections.asr.models.label_modelsr1nemo.collections.asr.parts.preprocessing.featuresr .nemo.collections.asr.parts.utils.speaker_utilsr!r"r#r$r%r&r'r(r)nemo.core.classesr*nemo.core.classes.commonr+r,nemo.core.neural_typesr-r.r/nemo.core.neural_types.elementsr0 nemo.utilsr1torch.cuda.ampr2 ImportError contextlibr3__all__r7nnModuler8r9r4r4r4r6sd                ,     WD