o piU@sdZddlZddlZddlZddlmZejZGdddZddZ dd Z Gd d d Z Gd d d Z ddZ ddZ   dddZGdddZGdddZdS)uA popular speaker recognition/diarization model (LDA and PLDA). Authors * Anthony Larcher 2020 * Nauman Dawalatabad 2020 Relevant Papers - This implementation of PLDA is based on the following papers. - PLDA model Training * Ye Jiang et. al, "PLDA Modeling in I-Vector and Supervector Space for Speaker Verification," in Interspeech, 2012. * Patrick Kenny et. al, "PLDA for speaker verification with utterances of arbitrary duration," in ICASSP, 2013. - PLDA scoring (fast scoring) * Daniel Garcia-Romero et. al, “Analysis of i-vector length normalization in speaker recognition systems,” in Interspeech, 2011. * Weiwei-LIN et. al, "Fast Scoring for PLDA with Uncertainty Propagation," in Odyssey, 2016. * Kong Aik Lee et. al, "Multi-session PLDA Scoring of I-vector for Partially Open-Set Speaker Detection," in Interspeech 2013. Credits This code is adapted from: https://projets-lium.univ-lemans.fr/sidekit/ N)linalgc@seZdZdZ      d(ddZddZddZd d Zd d Zd dZ ddZ ddZ ddZ ddZ ddZddZddZddZd)d d!Zd"d#Zd$d%Zd&d'ZdS)* StatObject_SBaA utility class for PLDA class used for statistics calculations. This is also used to pack deep embeddings and meta-information in one object. Arguments --------- modelset : list List of model IDs for each session as an array of strings. segset : list List of session IDs as an array of strings. start : int Index of the first frame of the segment. stop : int Index of the last frame of the segment. stat0 : torch.Tensor An ndarray of float64. Each line contains 0-th order statistics from the corresponding session. stat1 : torch.Tensor An ndarray of float64. Each line contains 1-st order statistics from the corresponding session. NcCs|dur6tjddd|_tjddd|_tjddd|_tjddd|_tjgtd|_tjgtd|_ dS||_||_||_||_||_||_ dS)Nr|Odtype) numpyemptymodelsetsegsetstartstoparray STAT_TYPEstat0stat1)selfr r r r rrr]/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/speechbrain/processing/PLDA_LDA.py__init__7s  zStatObject_SB.__init__cCsd}|d|jd7}|d|jd7}|d|jd7}|d|jd7}|d|jd7}|d|jd7}|d7}|S) Nz------------------------------ z modelset:  zsegset: z seg start:z seg stop:zstat0:zstat1:)r __repr__r r r rrrchrrrrPszStatObject_SB.__repr__cC@t|d}t||tjWddS1swYdS)zSaves stats in pickle format. Arguments --------- filename : path Path where the pickle file will be stored. wbNopenpickledumpHIGHEST_PROTOCOL)rfilenameoutputrrrsave_stat_object[s "zStatObject_SB.save_stat_objectcC|j|j|kS)zReturn segments of a given model. Arguments --------- mod_id : str ID of the model for which segments will be returned. Returns ------- segments )r r rmod_idrrrget_model_segsetsf zStatObject_SB.get_model_segsetscCr#)zReturn start of segment of a given model. Arguments --------- mod_id : str ID of the model for which start will be returned. Returns ------- start of segment )r r r$rrrget_model_starttr'zStatObject_SB.get_model_startcCr#)zReturn stop of segment of a given model. Arguments --------- mod_id : str ID of the model which stop will be returned. Returns ------- stop of segment )r r r$rrrget_model_stopr'zStatObject_SB.get_model_stopcCstj|jdd}|S)z*Return the mean of first order statistics.raxis)rmeanr)rmurrrget_mean_stat1szStatObject_SB.get_mean_stat1cCs0|j|jjdd}t|||jjdS)z^Compute and return the total covariance matrix of the first-order statistics. rr*)rr,rdot transposeshape)rCrrrget_total_covariance_stat1sz(StatObject_SB.get_total_covariance_stat1cCs|j|j|kddf}|S)zReturn zero-order statistics of a given model Arguments --------- mod_id : str ID of the model which stat0 will be returned. Returns ------- Zero-order statistics. N)rr )rr%Srrrget_model_stat0s zStatObject_SB.get_model_stat0cCs|j|j|kddfS)zReturn first-order statistics of a given model. Arguments --------- mod_id : str ID of the model which stat1 will be returned. Returns ------- First-order statistics. N)rr r$rrrget_model_stat1s zStatObject_SB.get_model_stat1cCst}t|j|_t|j|_tj|jjd|j jdft d|_ tj|jjd|j jdft d|_ tt|jjd}t |jD]2\}}| |jdd|j |ddf<||jdd|j |ddf<||||jd7<qF||fS)aSum the zero- and first-order statistics per model and store them in a new StatObject_SB. Returns ------- a StatObject_SB object with the statistics summed per model and a numpy array with session_per_model. rrr*N)rruniquer copydeepcopyr zerosr1rrr enumerater5sumr6)r sts_per_modelsession_per_modelidxmodelrrrsum_stat_per_models.    z StatObject_SB.sum_stat_per_modelcCsZ|jjd|jjd}tt|jjd|}|j|jdd|f|t|_dS)zCenter first order statistics. Arguments --------- mu : array Array to center on. r7N)rr1rrrepeatarangeastyper)rr-dim index_maprrr center_stat1s  zStatObject_SB.center_stat1cCs6ttjj|jdddtj}|j||_dS)z:Divide all first-order statistics by their Euclidean norm.r7r*g:0yE>N)rcliprnormrinfr0)r vect_normrrr norm_stat1szStatObject_SB.norm_stat1cCst|j||_dS)zRotate first-order statistics by a right-product. Arguments --------- R : ndarray Matrix to use for right product on the first order statistics. N)rr/r)rRrrr rotate_stat1szStatObject_SB.rotate_stat1Fc Cs*|jdkr|||jt|t|_dS|jdkr_|}|sRt|\}}|j ddd}|j |}|j dd|f}dt|j }t |t |}n ||| |dS|jdkr|jjd|jjd} |jjd} ||td|jjd| | |d| j|_dStd) a2Whiten first-order statistics If sigma.ndim == 1, case of a diagonal covariance. If sigma.ndim == 2, case of a single Gaussian with full covariance. If sigma.ndim == 3, case of a full covariance UBM. Arguments --------- mu : array Mean vector to be subtracted from the statistics. sigma : narray Co-variance matrix or covariance super-vector. isSqrInvSigma : bool True if the input Sigma matrix is the inverse of the square root of a covariance matrix. r7Nrz ikj,ikl->iljz(Wrong dimension of Sigma, must be 1 or 2)ndimrHrrsqrtrErreighrealargsortr/diagrOr1reinsumTreshape Exception) rr-sigma isSqrInvSigma sqr_inv_sigma eigen_values eigen_vectorsindsqr_inv_eval_sigmansess_nbrrr whiten_stat1s8          zStatObject_SB.whiten_stat1cttfdd|D}j|_j|_j|_j|_j|ddf_j|ddf_dS)a?Align models of the current StatServer to match a list of models provided as input parameter. The size of the StatServer might be reduced to match the input list of models. Arguments --------- model_list : ndarray of strings List of models to match. c$g|]}tj|kddqSr)rargwherer .0vrrr G$z.StatObject_SB.align_models..Nrr r r r r rr)r model_listindxrrnr align_models<     zStatObject_SB.align_modelscrg)aFAlign segments of the current StatServer to match a list of segment provided as input parameter. The size of the StatServer might be reduced to match the input list of segments. Arguments --------- segment_list: ndarray of strings list of segments to match crhri)rrjr rkrnrrro[rpz0StatObject_SB.align_segments..Nrq)r segment_listrsrrnralign_segmentsPruzStatObject_SB.align_segmentscCs.|jjd}t|j}|}t|jd|f}t||f}d}|D]4}||tj||dd} |t | | | jd7}tj||dd||ddf<|d7}q%||}t | |} t | t | } t | \} } | j} | j} | j| dddd}| dd|f}|S)azCompute and return the Linear Discriminant Analysis matrix on the first-order statistics. Columns of the LDA matrix are ordered according to the corresponding eigenvalues in descending order. Arguments --------- rank : int Rank of the LDA matrix to return. Returns ------- L : matrix r7rr*NrQ)rr1rr8r r.r;r6r,r/r0rinvrUrVrW)rrank vect_sizeunique_speakerr- class_meansSwspk_idx speaker_id spk_sessionsSbDiscriminationMatrixr`rar@Lrrrget_lda_matrix_stat1ds8      z"StatObject_SB.get_lda_matrix_stat1)NNNNNN)F)__name__ __module__ __qualname____doc__rrr"r&r(r)r.r3r5r6rBrHrMrOrfrtrwrrrrrr s4   * : rcsfdd|D}||S)zDifference between lists.csg|]}|vr|qSrrrlitemlist2rrroszdiff..)sortlist1rcrrrdiffsrcsfdd|D}|S)z7Checks if the elements if list1 are contained in list2.csg|]}|vqSrrrrrrroszismember..rrrrrismembersrc@sDeZdZdZdegegfddZddZddZd d Z d S) NdxaA class that encodes trial index information. It has a list of model names and a list of test segment names and a matrix indicating which combinations of model and test segment are trials of interest. Arguments --------- ndx_file_name : str Name of the file to load. models : list List of unique models in a ndarray. testsegs : list List of unique test segments in a ndarray. c Csttjddd|_tjddd|_tjgdd|_|dkr|jd|jd}|dkr]|dkrDt|d}t|g|}t||f}nt |}t|d}t|g|}t||f}t |}t |}tj |jd|jdfdd} t |jdD]} |tt ||| } t || | | f<q}||_||_| |_|sJddSt|} | j|_| j|_| j|_dS)NrrrboolrrQWrong Ndx format)rrr r r trialmaskr1strhstackabsr8r;rangervalidaterread) r ndx_file_namemodelstestsegsdlastpadr r rmsegsndxrrrrs<      z Ndx.__init__cCr)z!Saves the object in pickle formatrNr)routput_file_namer!rrrsave_ndx_objects "zNdx.save_ndx_objectc Cs|r|}|}n t|j|}t|j|}tt|j|}tt|j|}t}|j||_|j||_|jt|ddf} | ddt|f|_|jsSJd|jj d|jj dkrnt d|jj d|jj d|jj d|jj dkrt d|jj d|jj d|S)aRemoves some of the information in an Ndx. Useful for creating a gender specific Ndx from a pooled gender Ndx. Depending on the value of 'keep', the two input lists indicate the strings to retain or the strings to discard. Arguments --------- modlist : array A cell array of strings which will be compared with the modelset of 'inNdx'. seglist : array A cell array of strings which will be compared with the segset of 'inNdx'. keep : bool Indicating whether modlist and seglist are the models to keep or discard. Returns ------- outNdx : Ndx Nrrz&Number of models reduced from %d to %dz-Number of test segments reduced from %d to %d) rr r rr rrrrr1print) rmodlistseglistkeepkeepmodskeepsegs keepmodidx keepsegidxoutNdxtmprrrfilters6        z Ndx.filtercCst|jtj}|t|jtjM}|t|jtjM}||jjdkM}||jjdkM}||jjdkM}||jj|jjd|jjdfkM}|S)zChecks that an object of type Ndx obeys certain rules that must always be true. Returns a boolean value indicating whether the object is valid r7rPr) isinstancer rndarrayr rrSr1)rokrrrrs  z Ndx.validateN) rrrrrr rrrrrrrrrs . 3rc@s"eZdZdZdddZddZdS) ScoresaA class for storing scores for trials. The modelset and segset fields are lists of model and test segment names respectively. The element i,j of scoremat and scoremask corresponds to the trial involving model i and test segment j. Arguments --------- scores_file_name : str Name of a HDF5 file containing the following fields modelset : list list of unique models in a ndarray. segset : list list of unique test segments in a ndarray. scoremask : 2d ndarray of bool indicates the trials of interest, i.e., the entry i,j in scoremat should be ignored if scoremask[i,j] is false. scoremat : 2d ndarray scores matrix. rcCsvtjddd|_tjddd|_tjgdd|_tg|_|dkr$dSt|}|j|_|j|_|j|_|j|_dS)Nrrrrr) rrr r r scoremaskscorematrr)rscores_file_namerrrrrEs   zScores.__init__cCs`d}||jd7}|d7}||jd7}|d7}||jd7}|d7}||jd7}dS)Nz modelset: rzsegset: z scoremask: z scoremat: )r r rrrrrrrrTszScores.__repr__N)r)rrrrrrrrrrr/s  rc CsD|jjd}|jjdkr6|jj|j}t} t|dddfD]} t | |t |jd| | <q"tj |jjd|jjdftj d} |D]U} |jjdkrmt t ||jj|| |ddf|j} n | || |df} |jj|| |ddf}tj|| || d| t|| || | || <qJdS)aA function for PLDA estimation. Arguments --------- batch_start : int Index to start at in the list. mini_batch_indices : list Indices of the elements in the list (should start at zero). factor_analyser : instance of PLDA class PLDA class object. stat0 : torch.Tensor Matrix of zero-order statistics. stat1: torch.Tensor Matrix of first-order statistics. e_h : torch.Tensor An accumulator matrix. e_hh: torch.Tensor An accumulator matrix. r7rPNrr)out)Fr1SigmarSrZr/dictrr8rrxeyer;float64outer) batch_startmini_batch_indicesfactor_analyserrre_he_hhryAinv_lambda_uniquesessrr@ inv_lambdaauxrrr fa_model_loopbs4     rcCs.||j|jd}||j||j|S)NT)rr r rtrw)enrolltestr clean_ndxrrr_check_missing_models  r?Tc ! Cst|} t|} t| jj| jjks| } |r"t| | |} n|} | || |t| jj| jjks=| } t |} tj |jddd} |j | | |}t || }t d|| }tj |d}tj |d}|d|}t ||j }||}t |}t || | |}||}| | |}dtd| j || jj }dtd| j || jj }t}| j|_| j|_| j|_|ddtjf|||_|j| j | | jj 7_|j|9_|d krE|jjd }t|jj}t|j}t|D]-} |j| ddft||t|| kjd d |dd||| ddf<q||_|S) aCompute the PLDA scores between to sets of vectors. The list of trials to perform is given in an Ndx object. PLDA matrices have to be pre-computed. i-vectors/x-vectors are supposed to be whitened before. Arguments --------- enroll : speechbrain.utils.Xvector_PLDA_sp.StatObject_SB A StatServer in which stat1 are xvectors. test : speechbrain.utils.Xvector_PLDA_sp.StatObject_SB A StatServer in which stat1 are xvectors. ndx : speechbrain.utils.Xvector_PLDA_sp.Ndx An Ndx object defining the list of trials to perform. mu : double The mean vector of the PLDA gaussian. F : torch.Tensor The between-class co-variance matrix of the PLDA. Sigma : torch.Tensor The residual covariance matrix. p_known : float Probability of having a known speaker for open-set identification case (=1 for the verification task and =0 for the closed-set case). scaling_factor : float Factor to multiply statistics. check_missing : bool If True, check that all models and segments exist. Returns ------- scores : Scores r7floatrrPg@g?z ij, ji->iNrr*)r9r:rr8r r1mean_stat_per_modelrrHrrxrrZr/slogdetrYrrr rrnewaxisrrexprlogrDr=)!rrrr-rrp_knownscaling_factor check_missing enroll_ctrtest_ctrrinvSigmaI_spkKK1K2alpha1alpha2plda_cstSigma_ac Sigma_tot Sigma_tot_invTmpPhiPsi model_partseg_partscoreNopen_set_scoresriirrrfast_PLDA_scoringsd *           &rc@s"eZdZdZddZdddZdS) LDAztA class to perform Linear Discriminant Analysis. It returns the low dimensional representation as per LDA. cCs d|_dSN) transform_matrnrrrr(s z LDA.__init__NrPcCs6|dur |||_n||_t|}||j|S)aPerforms LDA and projects the vectors onto lower dimension space. Arguments --------- stat_server : object of speechbrain.processing.PLDA_LDA.StatObject_SB. Contains vectors and meta-information to perform LDA. reduced_dim : int Dimension of the reduced space. transform_mat : matrix Transformation matrix. Returns ------- new_train_obj : speechbrain.processing.PLDA_LDA.StatObject_SB N)rrr9r:rO)r stat_server reduced_dimr new_train_objrrrdo_lda+s   z LDA.do_lda)NrPN)rrrrrrrrrrr"src@s8eZdZdZ      d ddZ    d d d ZdS) PLDAa A class to train PLDA model from embeddings. The input is in speechbrain.utils.StatObject_SB format. Trains a simplified PLDA model no within-class covariance matrix but full residual covariance matrix. Arguments --------- mean : torch.Tensor Mean of the vectors. F : torch.Tensor Eigenvoice matrix. Sigma : torch.Tensor Residual matrix. rank_f : int Rank (default 100). nb_iter : int Number of iterations (default 10). scaling_factor : int Factor to use for scaling statistics (default 1.0). Example ------- >>> from speechbrain.processing.PLDA_LDA import * >>> import random, numpy >>> dim, N = 10, 100 >>> n_spkrs = 10 >>> train_xv = numpy.random.rand(N, dim) >>> md = ['md'+str(random.randrange(1,n_spkrs,1)) for i in range(N)] >>> modelset = numpy.array(md, dtype="|O") >>> sg = ['sg'+str(i) for i in range(N)] >>> segset = numpy.array(sg, dtype="|O") >>> s = numpy.array([None] * N) >>> stat0 = numpy.array([[1.0]]* N) >>> xvectors_stat = StatObject_SB(modelset=modelset, segset=segset, start=s, stop=s, stat0=stat0, stat1=train_xv) >>> # Training PLDA model: M ~ (mean, F, Sigma) >>> plda = PLDA(rank_f=5) >>> plda.plda(xvectors_stat) >>> print (plda.mean.shape) (10,) >>> print (plda.F.shape) (10, 5) >>> print (plda.Sigma.shape) (10, 10) >>> # Enrollment (20 utts), Test (30 utts) >>> en_N = 20 >>> en_xv = numpy.random.rand(en_N, dim) >>> en_sgs = ['en'+str(i) for i in range(en_N)] >>> en_sets = numpy.array(en_sgs, dtype="|O") >>> en_s = numpy.array([None] * en_N) >>> en_stat0 = numpy.array([[1.0]]* en_N) >>> en_stat = StatObject_SB(modelset=en_sets, segset=en_sets, start=en_s, stop=en_s, stat0=en_stat0, stat1=en_xv) >>> te_N = 30 >>> te_xv = numpy.random.rand(te_N, dim) >>> te_sgs = ['te'+str(i) for i in range(te_N)] >>> te_sets = numpy.array(te_sgs, dtype="|O") >>> te_s = numpy.array([None] * te_N) >>> te_stat0 = numpy.array([[1.0]]* te_N) >>> te_stat = StatObject_SB(modelset=te_sets, segset=te_sets, start=te_s, stop=te_s, stat0=te_stat0, stat1=te_xv) >>> ndx = Ndx(models=en_sets, testsegs=te_sets) >>> # PLDA Scoring >>> scores_plda = fast_PLDA_scoring(en_stat, te_stat, ndx, plda.mean, plda.F, plda.Sigma) >>> print (scores_plda.scoremat.shape) (20, 30) Nd rcCsVd|_d|_d|_||_||_||_|dur||_|dur ||_|dur)||_dSdSr)r,rrrank_fnb_iterr)rr,rrrrrrrrrs  z PLDA.__init__Fc Csj|jjd}|dur|}|}|||||_||_|\}} |jjd} |j |j 9_ |j|j 9_| |j 9} |} t | \} } t | ddd}| jdd|d|jf} | ddd|jf|_t|jD]}t|}||j|jt |j\}}|jddd}|j|}|jdd|f}dt |j}t |t |}|j|j|_t j|td}|j dd|f}t | |jf}t | |j|jf}tdt | |||j||dt j|dd| jd}|j|jt |}t d ||j !}t "||j|_| |j|| |_|jt #||_qwdS) a2Trains PLDA model with no within class covariance matrix but full residual covariance matrix. Arguments --------- stat_server : speechbrain.processing.PLDA_LDA.StatObject_SB Contains vectors and meta-information to perform PLDA output_file_name : str Name of the output file where to store PLDA model. whiten : bool Whether to perform whitening. w_stat_server : speechbrain.processing.PLDA_LDA.StatObject_SB Contains whitening vectors and meta-information. r7TrNrQr)rrrrrrrr*z ijk,i->jk)$rr1r.r3rfr,rrBr rrrrUrrWrVrrrrr9r:rTr/rXrZr;intrrDr=rxrYsqueezesolvecholesky)rrrwhiten w_stat_serverrzw_meanw_Sigmamodel_shifted_statr?class_nb sigma_obsevalsevecsr@it local_statr`rarbrcr_rG_stat0rr_R_C_Arrrpldasb           z PLDA.plda)NNNrrr)NNFN)rrrrrrrrrrrHsC r)rrT)rr9rrscipyrrrrrrrrrrrrrrrrrs, { 3: {&