o pis@s>dZddlZddlZddlZddlZddlZddlmZddlm Z m Z m Z m Z m Z ddlZddlZddlmZddlmZmZmZmZddlmZddlmZdd lmZdd lmZm Z m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'm(Z(m)Z)ddl*m+Z+ddl,m-Z-m.Z.m/Z/ddl0m1Z1dde2fddZ3Gddde eZ4dS)zSpeech separation pipelinesN)Path)CallableOptionalTextTupleUnion) rearrange)Audio InferenceModelPipeline) AudioFile) Clustering)PretrainedSpeakerEmbedding) PipelineModelSpeakerDiarizationMixin get_model)set_num_speakers)binarize) Annotation SlidingWindowSlidingWindowFeature)GreedyDiarizationErrorRate) Categorical ParamDictUniform)binary_dilation batch_sizecCst|g|}tj|d|iS)zBatchify iterable fillvalue)iter itertools zip_longest)iterablerrargsr%^/home/ubuntu/.local/lib/python3.10/site-packages/pyannote/audio/pipelines/speech_separation.pybatchify6sr'cs^eZdZdZ          d5d ed ed ed ed ededede e de e dfde e e dfffdd ZedefddZejdefddZddZddZeddZ d6deeeffd d!Z  d7d"ed#ed$e efd%d&Zd'ed(ejd)edefd*d+Z     d8d,ed-e ed.e ed/e ed0ed$e edefd1d2Zde fd3d4Z!Z"S)9SpeechSeparationu> Speech separation pipeline Parameters ---------- segmentation : Model, str, or dict, optional Pretrained segmentation model and separation model. See pyannote.audio.pipelines.utils.get_model for supported format. segmentation_step: float, optional The segmentation model is applied on a window sliding over the whole audio file. `segmentation_step` controls the step of this window, provided as a ratio of its duration. Defaults to 0.1 (i.e. 90% overlap between two consecuive windows). embedding : Model, str, or dict, optional Pretrained embedding model. Defaults to "speechbrain/spkrec-ecapa-voxceleb@5c0be38". See pyannote.audio.pipelines.utils.get_model for supported format. embedding_exclude_overlap : bool, optional Exclude overlapping speech regions when extracting embeddings. Defaults (False) to use the whole speech. clustering : str, optional Clustering algorithm. See pyannote.audio.pipelines.clustering.Clustering for available options. Defaults to "AgglomerativeClustering". segmentation_batch_size : int, optional Batch size used for speaker segmentation. Defaults to 1. embedding_batch_size : int, optional Batch size used for speaker embedding. Defaults to 1. der_variant : dict, optional Optimize for a variant of diarization error rate. Defaults to {"collar": 0.0, "skip_overlap": False}. This is used in `get_metric` when instantiating the metric: GreedyDiarizationErrorRate(**der_variant). token : str or bool, optional Huggingface token to be used for downloading from Huggingface hub. cache_dir: Path or str, optional Path to the folder where files downloaded from Huggingface hub are stored. Usage ----- >>> pipeline = SpeechSeparation() >>> diarization, separation = pipeline("/path/to/audio.wav") >>> diarization, separation = pipeline("/path/to/audio.wav", num_speakers=4) >>> diarization, separation = pipeline("/path/to/audio.wav", min_speakers=2, max_speakers=10) Hyper-parameters ---------------- segmentation.min_duration_off : float Fill intra-speaker gaps shorter than that many seconds. segmentation.threshold : float Mark speaker has active when their probability is higher than this. clustering.method : {'centroid', 'average', ...} Linkage used for agglomerative clustering clustering.min_cluster_size : int Minium cluster size. clustering.threshold : float Clustering threshold used to stop merging clusters. separation.leakage_removal : bool Zero-out sources when speaker is inactive. separation.asr_collar When using leakage removal, keep that many seconds before and after each speaker turn References ---------- Joonas Kalda, Clément Pagés, Ricard Marxer, Tanel Alumäe, and Hervé Bredin. "PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordings" Odyssey 2024. https://arxiv.org/abs/2403.02288 pyannote/separation-ami-1.0皙?Jspeechbrain/spkrec-ecapa-voxceleb@5c0be3875fda05e81f3c004ed8c7c06be308de1eFAgglomerativeClusteringN segmentationsegmentation_step embeddingembedding_exclude_overlap clusteringembedding_batch_sizesegmentation_batch_size der_varianttoken cache_dirc  sRt||_t|| | d} ||_||_||_||_||_|p$ddd|_ | j dj } t | | |j| d|d|_ |j jj djrKttddd |_n ttd d tddd |_|jd kr`d} nt|j| | d|_t|jjdd|_|jj} zt|}Wntytddttjdw|j| d|_ tt!ddgtddd|_"dS)N)r6r7F)collar skip_overlaprT)durationstepskip_aggregationr?)min_duration_offr*g?) thresholdr?OracleClusteringnot_applicabledownmix) sample_ratemonozclustering must be one of [, ])metric)leakage_removal asr_collar)#super__init__segmentation_modelrr/r0r3r1 klusteringr5specificationsr;r _segmentationmodelpowersetrrr.r _embeddingr rD_audiorHrKeyError ValueErrorjoinlist __members__valuer2r separation)selfr.r/r0r1r2r3r4r5r6r7rQsegmentation_durationrH Klustering __class__r%r&rLsX        zSpeechSeparation.__init__returncCs|jjSNrPrr\r%r%r&r4sz(SpeechSeparation.segmentation_batch_sizercCs ||j_dSrbrc)r\rr%r%r&r4s cCstrb)NotImplementedErrorrdr%r%r&default_parameterssz#SpeechSeparation.default_parametersccs d} d|dV|d7}q)NrTSPEAKER_02dr-r%)r\speakerr%r%r&classess zSpeechSeparation.classescCsdS)Nztraining_cache/segmentationr%rdr%r%r&CACHED_SEGMENTATIONsz$SpeechSeparation.CACHED_SEGMENTATIONcCs~|dur t|dd}|jr2|j|vr||j\}}||fS|j||d\}}||f||j<||fS|j||d\}}||fS)aIApply segmentation model Parameter --------- file : AudioFile hook : Optional[Callable] Returns ------- segmentations : (num_chunks, num_frames, num_speakers) SlidingWindowFeature separations : (num_chunks, num_samples, num_speakers) SlidingWindowFeature Nr.hook) functoolspartialtrainingrkrP)r\filerm segmentations separationsr%r%r&get_segmentationss z"SpeechSeparation.get_segmentationsbinary_segmentationsexclude_overlaprmcsjr"dt}d|vr"jjjdjs|djjkr"|dSj j }j j \}}} |rYj j} |j j} t|| | dtjj ddddk} tj | j n d tj j fd d } t| jd d }t|| j}g}|dur|dd|ddt|dD]1\}}ttdd|\}}t|}t|}j ||d}|||dur|d|||dqt|}t|d|d}jrjjjdjrd|id<|Sjj|dd<|S)aExtract embeddings for each (chunk, speaker) pair Parameters ---------- file : AudioFile binary_segmentations : (num_chunks, num_frames, num_speakers) SlidingWindowFeature Binarized segmentation. exclude_overlap : bool, optional Exclude overlapping speech regions when extracting embeddings. In case non-overlapping speech is too short, use the whole speech. hook: Optional[Callable] Called during embeddings after every batch to report the progress Returns ------- embeddings : (num_chunks, num_speakers, dimension) array ztraining_cache/embeddings embeddingsrsegmentation.thresholdr>Taxiskeepdimsc3stD]M\\}}\}}jj|dd\}}tj|ddtj}tj|ddtj}t|j|jD]\}}t|krD|}n|}|dt |dfVq6qdS)Npad)moder8)nan) ziprTcropnp nan_to_numastypefloat32Tsumtorch from_numpy)chunkmasks_ clean_maskswaveformspeaker_activation_with_context clean_mask used_maskruclean_segmentationsrqmin_num_framesr\r%r&iter_waveform_and_mask<s(  z?SpeechSeparation.get_embeddings..iter_waveform_and_mask)NN)rrN)total completedr-cSs |dduS)Nrr%)br%r%r&js z1SpeechSeparation.get_embeddings..)rz(c s) d -> c s d)c)rxrw)rpgetdictrPrQrOrRr.r@sliding_windowr;datashaperSmin_num_samplesrDmathceilrrrr'r3 enumeraterfilterrvstackappendr)r\rqrurvrmcacher; num_chunks num_frames num_speakersr num_samples clean_framesrbatches batch_countembedding_batchesibatch waveformsrwaveform_batch mask_batchembedding_batchrwr%rr&get_embeddingssp        zSpeechSeparation.get_embeddingsrr hard_clusterscountcCs|jj\}}}t|d}tjt|||f}tt||D]+\} \} \} } t| D]} | dkr4q-tj| dd| | kfdd|| dd| f<q-q t ||j }|S)a;Build final discrete diarization out of clustered segmentation Parameters ---------- segmentations : (num_chunks, num_frames, num_speakers) SlidingWindowFeature Raw speaker segmentation. hard_clusters : (num_chunks, num_speakers) array Output of clustering step. count : (total_num_frames, 1) SlidingWindowFeature Instantaneous number of active speakers. Returns ------- discrete_diarization : SlidingWindowFeature Discrete (0s and 1s) diarization. r-Nr{) rrrmaxrzerosrruniquerr)r\rrrrrrlocal_num_speakers num_clustersclustered_segmentationsrclusterrr.kr%r%r& reconstructs& zSpeechSeparation.reconstructrqr min_speakers max_speakersreturn_embeddingsc s|j||d}t|||d\}}}|j||d\}}|d||d||jjjdjr/|} n t||jj dd} |j | |jjj dd } |d | t | jd krit|d d } |re| dt d|jjffS| dfS|jdkrs|ssd} n|j|| |j|d} |d| |j| | |||||jjj d\} }}t | d}||ks||krttd|d|d|d|d t | j|t j| _t j| jdddk}d| |<| || | }|!|| }t j|dddk}|jdd|f|_|jj"\}}|d|| || | }|j#j$|jj"d}t%|d|d}t&j'||dd dd}|jj"\}}t (|jd dtd||ff|_|jdd|f|_|j)j*rt+|jj,t-|j)j.|j/j0}|dkrt 1|j}t2|D] }|jj3|}|dk}t4|dgd|}|t j|j3|<qft5||j#}|j|6|j|_|jt jt 7|jddd!d"|_|j8|d |jj9d#} |d | _:d$|vr|d$r|j;|d$| dd%\}fd&d'| <Dnd(d't=| <|>D| j?d)} d*d'@D|jddfd+d,| <Df|_|s| |fS|dur| |dfStA| <|j"dkr:t (|dtA| <|j"dfd f}|fd-d,| <D}| ||fS).aApply speaker diarization Parameters ---------- file : AudioFile Processed file. num_speakers : int, optional Number of speakers, when known. min_speakers : int, optional Minimum number of speakers. Has no effect when `num_speakers` is provided. max_speakers : int, optional Maximum number of speakers. Has no effect when `num_speakers` is provided. return_embeddings : bool, optional Return representative speaker embeddings. hook : callable, optional Callback called after each major steps of the pipeline as follows: hook(step_name, # human-readable name of current step step_artefact, # artifact generated by current step file=file) # file being processed Time-consuming steps call `hook` multiple times with the same `step_name` and additional `completed` and `total` keyword arguments usable to track progress of current step. Returns ------- diarization : Annotation Speaker diarization sources : SlidingWindowFeature Separated sources embeddings : np.array, optional Representative speaker embeddings such that `embeddings[i]` is the speaker embedding for i-th speaker in diarization.labels(). Only returned when `return_embeddings` is True. rl)rrrr.rsrF)onset initial_state)r8r8)warm_upspeaker_countingr8uri)rNrA)rvrmrw)rwrrr min_clusters max_clustersrqframesr-z2 The detected number of speakers (z/) is outside the given bounds [rFzS]. This can happen if the given audio file is too short to contain zh or more speakers. Try to lower the desired minimal number of speakers. rrdiscrete_diarizationry)r<r;T)rhammingmissing skip_average)rrrzg:0yE>)min_duration_onr? annotation)return_mappingcsi|] }|||qSr%)r).0keymappingr%r& sz*SpeechSeparation.apply..cSsi|]\}}||qSr%r%)rlabelexpected_labelr%r%r&rsrcSsi|]\}}||qSr%r%)rindexrr%r%r&rscg|]}|qSr%r%rrinverse_mappingr%r& z*SpeechSeparation.apply..crr%r%rrr%r&rr)B setup_hookrrtrPrQrOrRrr.r@ speaker_countreceptive_fieldrnanmaxrrrrS dimensionrNrr1r2rwarningswarntextwrapdedentminimumrint8rrto_diarizationrrr;rr aggregater~r[rIintrroundrJrTrD zeros_likerangerrralignabs to_annotationr?roptimal_mappinglabelsrrj rename_labelsitemslen) r\rqrrrrrmrrrsbinarized_segmentationsr diarizationrwrr centroidsnum_different_speakersinactive_speakersractive_speakersrclustered_separationsframe_durationrsources num_sourcesasr_collar_framesdilated_speaker_activationsrspeaker_activation non_silentdilated_non_silentr%)rrr&applys"-                       zSpeechSeparation.applycCstdi|jS)Nr%)rr5rdr%r%r& get_metricszSpeechSeparation.get_metric) r)r*r+Fr,r-r-NNNrb)FN)NNNFN)#__name__ __module__ __qualname____doc__rfloatboolstrrrrrrrrLpropertyr4setterrfrjrkrrrtrrrndarrayrr rr rr __classcell__r%r%r_r&r(=sC   G   "  4 r()rN)5rrnr!rrrpathlibrtypingrrrrrnumpyrreinopsrpyannote.audior r r r pyannote.audio.core.ior #pyannote.audio.pipelines.clusteringr-pyannote.audio.pipelines.speaker_verificationrpyannote.audio.pipelines.utilsrrr*pyannote.audio.pipelines.utils.diarizationrpyannote.audio.utils.signalr pyannote.corerrrpyannote.metrics.diarizationrpyannote.pipeline.parameterrrr scipy.ndimagerrr'r(r%r%r%r&s0