o 9wiq@sdZddlZddlZddlZddlZddlZddlmZmZm Z m Z m Z ddl Z ddlZddlmZddlmZmZmZddlmZddlmZmZmZddlmZmZmZmZdd l m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'm(Z(m)Z)dd l*m+Z+dde,fddZ-Gddde(eZ.dS)zSpeech separation pipelinesN)CallableOptionalTextTupleUnion) rearrange) Annotation SlidingWindowSlidingWindowFeature)GreedyDiarizationErrorRate) Categorical ParamDictUniform)Audio InferenceModelPipeline) AudioFile) Clustering)PretrainedSpeakerEmbedding) PipelineModelSpeakerDiarizationMixin get_model)binarize batch_sizecCst|g|}tj|d|iS)zBatchify iterable fillvalue)iter itertools zip_longest)iterablerrargsr"g/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/pyannote/audio/pipelines/speech_separation.pybatchify3sr$csNeZdZdZ         d3ded ed ed ed ed edede e de e dfffdd Z edefddZejdefddZddZddZeddZ d4deeeffddZ  d5d ed!ed"e efd#d$Zd%ed&ejd'edefd(d)Z     d6d*ed+e ed,e ed-e ed.ed"e edefd/d0Zdefd1d2Z Z!S)7SpeechSeparationuJ Speech separation pipeline Parameters ---------- segmentation : Model, str, or dict, optional Pretrained segmentation model and separation model. See pyannote.audio.pipelines.utils.get_model for supported format. segmentation_step: float, optional The segmentation model is applied on a window sliding over the whole audio file. `segmentation_step` controls the step of this window, provided as a ratio of its duration. Defaults to 0.1 (i.e. 90% overlap between two consecuive windows). embedding : Model, str, or dict, optional Pretrained embedding model. Defaults to "pyannote/embedding@2022.07". See pyannote.audio.pipelines.utils.get_model for supported format. embedding_exclude_overlap : bool, optional Exclude overlapping speech regions when extracting embeddings. Defaults (False) to use the whole speech. clustering : str, optional Clustering algorithm. See pyannote.audio.pipelines.clustering.Clustering for available options. Defaults to "AgglomerativeClustering". segmentation_batch_size : int, optional Batch size used for speaker segmentation. Defaults to 1. embedding_batch_size : int, optional Batch size used for speaker embedding. Defaults to 1. der_variant : dict, optional Optimize for a variant of diarization error rate. Defaults to {"collar": 0.0, "skip_overlap": False}. This is used in `get_metric` when instantiating the metric: GreedyDiarizationErrorRate(**der_variant). use_auth_token : str, optional When loading private huggingface.co models, set `use_auth_token` to True or to a string containing your hugginface.co authentication token that can be obtained by running `huggingface-cli login` Usage ----- >>> pipeline = SpeakerDiarization() >>> diarization, separation = pipeline("/path/to/audio.wav") >>> diarization, separation = pipeline("/path/to/audio.wav", num_speakers=4) >>> diarization, separation = pipeline("/path/to/audio.wav", min_speakers=2, max_speakers=10) Hyper-parameters ---------------- segmentation.min_duration_off : float Fill intra-speaker gaps shorter than that many seconds. segmentation.threshold : float Mark speaker has active when their probability is higher than this. clustering.method : {'centroid', 'average', ...} Linkage used for agglomerative clustering clustering.min_cluster_size : int Minium cluster size. clustering.threshold : float Clustering threshold used to stop merging clusters. separation.leakage_removal : bool Zero-out sources when speaker is inactive. separation.asr_collar When using leakage removal, keep that many seconds before and after each speaker turn References ---------- Joonas Kalda, Clément Pagés, Ricard Marxer, Tanel Alumäe, and Hervé Bredin. "PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordings" Odyssey 2024. https://arxiv.org/abs/2403.02288 N皙?Jspeechbrain/spkrec-ecapa-voxceleb@5c0be3875fda05e81f3c004ed8c7c06be308de1eFAgglomerativeClustering segmentationsegmentation_step embeddingembedding_exclude_overlap clusteringembedding_batch_sizesegmentation_batch_size der_variantuse_auth_tokenc  sNt||_t|| d} ||_||_||_||_||_|p#ddd|_ | j dj } t | | |j| d|d|_ |j jj djrJttddd |_n ttd d tddd |_|jd kr_d} nt|j| d|_t|jjdd|_|jj} zt|} Wntytddttjdw| j| d|_ tt!ddgtddd|_"dS)N)r2F)collar skip_overlaprT)durationstepskip_aggregationr?)min_duration_offr&g?) thresholdr:OracleClusteringnot_applicabledownmix) sample_ratemonozclustering must be one of [, ])metric)leakage_removal asr_collar)#super__init__segmentation_modelrr+r,r/r- klusteringr1specificationsr6r _segmentationmodelpowersetr rr*r _embeddingrr?_audiorCrKeyError ValueErrorjoinlist __members__valuer.r separation)selfr*r+r,r-r.r/r0r1r2rLsegmentation_durationrC Klustering __class__r"r#rG|sX         zSpeechSeparation.__init__returncCs|jjSNrKrrWr"r"r#r0sz(SpeechSeparation.segmentation_batch_sizercCs ||j_dSr]r^)rWrr"r"r#r0s cCstr])NotImplementedErrorr_r"r"r#default_parameterssz#SpeechSeparation.default_parametersccs d} d|dV|d7}q)NrTSPEAKER_02dr)r")rWspeakerr"r"r#classess zSpeechSeparation.classescCsdS)Nztraining_cache/segmentationr"r_r"r"r#CACHED_SEGMENTATIONsz$SpeechSeparation.CACHED_SEGMENTATIONcCsz|dur t|dd}|jr0|j|vr||j\}}||fS|j||d\}}|||j<||fS|j||d\}}||fS)aIApply segmentation model Parameter --------- file : AudioFile hook : Optional[Callable] Returns ------- segmentations : (num_chunks, num_frames, num_speakers) SlidingWindowFeature separations : (num_chunks, num_samples, num_speakers) SlidingWindowFeature Nr*hook) functoolspartialtrainingrfrK)rWfilerh segmentations separationsr"r"r#get_segmentationss  z"SpeechSeparation.get_segmentationsbinary_segmentationsexclude_overlaprhcsjr"dt}d|vr"jjjdjs|djjkr"|dSj j j j \}}}|rYj j} j j} t|| | dtjj ddddk} tj | j n d tj j fd d } t| jd d } t||j}g}|dur|dd|ddt| dD]1\}}ttdd|\}}t|}t|}j ||d}|||dur|d|||dqt|}t|d|d}jrjjjdjrd|id<|Sjj|dd<|S)aExtract embeddings for each (chunk, speaker) pair Parameters ---------- file : AudioFile binary_segmentations : (num_chunks, num_frames, num_speakers) SlidingWindowFeature Binarized segmentation. exclude_overlap : bool, optional Exclude overlapping speech regions when extracting embeddings. In case non-overlapping speech is too short, use the whole speech. hook: Optional[Callable] Called during embeddings after every batch to report the progress Returns ------- embeddings : (num_chunks, num_speakers, dimension) array ztraining_cache/embeddings embeddingsrsegmentation.thresholdr9T)axiskeepdimsc3stD]N\\}}\}}jj|dd\}}tj|ddtj}tj|ddtj}t|j|jD]\}}t|krE|}n|}|dt |dfVq7qdS)Npad)r6moder3)nan) ziprOcropnp nan_to_numastypefloat32Tsumtorch from_numpy)chunkmasks_ clean_maskswaveformspeaker_activation_with_context clean_mask used_maskrpclean_segmentationsr6rlmin_num_framesrWr"r#iter_waveform_and_mask8s*   z?SpeechSeparation.get_embeddings..iter_waveform_and_mask)NN)rrN)total completedr)cSs |dduS)Nrr")br"r"r#gs z1SpeechSeparation.get_embeddings..)rz(c s) d -> c s d)c)rsrr)rkgetdictrKrLrJrMr*r;sliding_windowr6datashaperNmin_num_samplesr?mathceilr}rr r$r/ enumerater{filterrvstackappendr)rWrlrprqrhcache num_chunks num_frames num_speakersr num_samples clean_framesrbatches batch_countembedding_batchesibatch waveformsrwaveform_batch mask_batchembedding_batchrrr"rr#get_embeddingssp !      zSpeechSeparation.get_embeddingsrm hard_clusterscountcCs|jj\}}}t|d}tjt|||f}tt||D]+\} \} \} } t| D]} | dkr4q-tj| dd| | kfdd|| dd| f<q-q t ||j }|S)a;Build final discrete diarization out of clustered segmentation Parameters ---------- segmentations : (num_chunks, num_frames, num_speakers) SlidingWindowFeature Raw speaker segmentation. hard_clusters : (num_chunks, num_speakers) array Output of clustering step. count : (total_num_frames, 1) SlidingWindowFeature Instantaneous number of active speakers. Returns ------- discrete_diarization : SlidingWindowFeature Discrete (0s and 1s) diarization. r)Nru) rrr}maxrzzerosrr{uniquer rto_diarization)rWrmrrrrlocal_num_speakers num_clustersclustered_segmentationsrclusterrr*kr"r"r# reconstructs& zSpeechSeparation.reconstructrlr min_speakers max_speakersreturn_embeddingsc s|j||d}|j|||d\}}}|j||d\}}|d||d||jjjdjr0|} n t||jj dd} |j | |jjj dd } |d | t | jd krjt|d d } |rf| dt d|jjffS| dfS|jdkrt|std} n|j|| |j|d} |d| |j| | |||||jjj d\} }}t | d}||ks||krttd|d|d|d|d t | j|t j| _t j| jdddk}d| |<| || | }|!|| }|d|| || | }|j"j#|jj$d}t%|d|d}t&j'||dd dd}|j(j)rt*|jj+|j(j,|j-j.dkrt/|jj$dD]p}|jj0|}t 1|dkdt 1t 2dkd}fd d!|D}dkrft 3ddg|}d"|j$dkr|t 3d"|j$dg}t j4t5|t6d#}d |t 7|<||jj0|<q)|jj$d}|j|8|jddd|f|_|j9|d |jj:d$} |d | _;d%|vr|d%r|j<|d%| dd&\}fd'd(| =Dnd)d(t>| =|?D| j@d*} |s| |fS|dur| |dfSt5| =|j$dkr't A|dt5| =|j$dfd+f}d,d(BD|fd-d!| =D}| ||fS).aApply speaker diarization Parameters ---------- file : AudioFile Processed file. num_speakers : int, optional Number of speakers, when known. min_speakers : int, optional Minimum number of speakers. Has no effect when `num_speakers` is provided. max_speakers : int, optional Maximum number of speakers. Has no effect when `num_speakers` is provided. return_embeddings : bool, optional Return representative speaker embeddings. hook : callable, optional Callback called after each major steps of the pipeline as follows: hook(step_name, # human-readable name of current step step_artefact, # artifact generated by current step file=file) # file being processed Time-consuming steps call `hook` multiple times with the same `step_name` and additional `completed` and `total` keyword arguments usable to track progress of current step. Returns ------- diarization : Annotation Speaker diarization sources : SlidingWindowFeature Separated sources embeddings : np.array, optional Representative speaker embeddings such that `embeddings[i]` is the speaker embedding for i-th speaker in diarization.labels(). Only returned when `return_embeddings` is True. rg)rrrr*rnrF)onset initial_state)r3r3)warm_upspeaker_countingr3uri)rNr<)rqrhrr)rrrmr min_clusters max_clustersrlframesr)z2 The detected number of speakers (z/) is outside the given bounds [rAzS]. This can happen if the given audio file is too short to contain zh or more speakers. Try to lower the desired minimal number of speakers. rrdiscrete_diarizationrt)r7r6T)rhammingmissing skip_averagecs,g|]}t||dqS)r))r}arange).0gap)asr_collar_frames non_silentr"r# os z*SpeechSeparation.apply..rw)dtype)min_duration_onr: annotation)return_mappingcsi|] }|||qSr")r)rkeymappingr"r# sz*SpeechSeparation.apply..cSsi|]\}}||qSr"r")rlabelexpected_labelr"r"r#rsr)rrcSsi|]\}}||qSr"r")rindexrr"r"r#rscsg|]}|qSr"r")rr)inverse_mappingr"r#rs)C setup_hookset_num_speakersrorKrLrJrMrr*r; speaker_countreceptive_fieldr}nanmaxrrrrN dimensionrIrr-r.rwarningswarntextwrapdedentminimumrint8rrrrr6rr r aggregaterVrDintrrErOr?rangerwherediffroneslenfloat concatenatealign to_annotationr:roptimal_mappinglabelsr{re rename_labelsrxitems)rWrlrrrrrhrmrnbinarized_segmentationsr diarizationrrrr centroidsnum_different_speakersinactive_speakersrclustered_separationsframe_durationrsourcesrspeaker_activationremaining_gapsremaining_zerosr num_sourcesr")rrrrr#applys(-                        zSpeechSeparation.applycCstdi|jS)Nr")r r1r_r"r"r# get_metricszSpeechSeparation.get_metric) Nr&r'Fr(r)r)NNr])FN)NNNFN)"__name__ __module__ __qualname____doc__rrboolstrrrrrrrGpropertyr0setterrarerfrr rorrr}ndarrayrrrrr r  __classcell__r"r"rZr#r%:sC  F   "  5 r%)rN)/r rirrrrtypingrrrrrnumpyr}reinopsr pyannote.corerr r pyannote.metrics.diarizationr pyannote.pipeline.parameterr r rpyannote.audiorrrrpyannote.audio.core.ior#pyannote.audio.pipelines.clusteringr-pyannote.audio.pipelines.speaker_verificationrpyannote.audio.pipelines.utilsrrrpyannote.audio.utils.signalrrr$r%r"r"r"r#s*