o pi)b@s~ddlZddlZddlmZddlmZmZmZmZm Z m Z ddl Z ddl Z ddlmZmZddlmZddlmZmZmZddlmZddlmZmZmZmZmZm Z m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'dd l(m)Z)m*Z*dd l+m,Z,ddl-m.Z.m/Z/ddl0m1Z1ddl2m3Z3ddl4m5Z5e6e/j7Z8e6e.j7Z9GdddeZ:dde;de;de;fddZZ>e>?e<dSdS)N)Counter)DictLiteralOptionalSequenceTextUnion) MLFlowLoggerTensorBoardLogger)pyplot)Problem ResolutionSpecifications)SegmentationTask)DetectionErrorRateDiarizationErrorRateDiarizationPrecisionDiarizationRecallFalseAlarmRateMissedDetectionRateSpeakerConfusionRate)nll_loss) permutate)Powerset)SegmentSlidingWindowFeature)SpeakerDiarizationProtocol)ScopeSubset)track)BaseWaveformTransform)Metriccs4eZdZdZ             d*dedeeedfd ed ee d e d ee e d ee de dee de dee deee eeeeffdee dedffdd Zd+fdd ZddZde ded efddZd ejfd!d"Zd#e fd$d%Zd eee eeeefffd&d'Zd#e fd(d)ZZS),SpeakerDiarizationu Speaker diarization Parameters ---------- protocol : SpeakerDiarizationProtocol pyannote.database protocol cache : str, optional As (meta-)data preparation might take a very long time for large datasets, it can be cached to disk for later (and faster!) re-use. When `cache` does not exist, `Task.prepare_data()` generates training and validation metadata from `protocol` and save them to disk. When `cache` exists, `Task.prepare_data()` is skipped and (meta)-data are loaded from disk. Defaults to a temporary path. duration : float, optional Chunks duration. Defaults to 2s. max_speakers_per_chunk : int, optional Maximum number of speakers per chunk (must be at least 2). Defaults to estimating it from the training set. max_speakers_per_frame : int, optional Maximum number of (overlapping) speakers per frame. Defaults to 2. balance: Sequence[Text], optional When provided, training samples are sampled uniformly with respect to these keys. For instance, setting `balance` to ["database","subset"] will make sure that each database & subset combination will be equally represented in the training samples. weight: str, optional When provided, use this key as frame-wise weight in loss function. batch_size : int, optional Number of training samples per batch. Defaults to 32. num_workers : int, optional Number of workers used for generating training samples. Defaults to multiprocessing.cpu_count() // 2. pin_memory : bool, optional If True, data loaders will copy tensors into CUDA pinned memory before returning them. See pytorch documentation for more details. Defaults to False. augmentation : BaseWaveformTransform, optional torch_audiomentations waveform transform, used by dataloader during training. metric : optional Validation metric(s). Can be anything supported by torchmetrics.MetricCollection. Defaults to DiarizationErrorRate and its components. References ---------- Alexis Plaquet and Hervé Bredin "Powerset multi-class cross entropy loss for neural speaker diarization" Proc. Interspeech 2023 Hervé Bredin and Antoine Laurent "End-To-End Speaker Segmentation for Overlap-Aware Resegmentation." Proc. Interspeech 2021 N$@ Fprotocolcachedurationmax_speakers_per_chunkmax_speakers_per_framebalanceweight batch_size num_workers pin_memory augmentationmetricmax_num_speakersloss)bcemsec stj|||| | | | |dt|tstd|dur&| dur&| }td|dur/td|dkr;td|d||_||_||_ ||_ dS)N)r(r-r.r/r0r1r'z>SpeakerDiarization task requires a SpeakerDiarizationProtocol.zL`max_num_speakers` has been deprecated in favor of `max_speakers_per_chunk`.z-`loss` has been deprecated and has no effect.z5`max_speakers_per_frame` must be 1 or more (you used z).) super__init__ isinstancer ValueErrorwarningswarnr)r*r+r,)selfr&r'r(r)r*r+r,r-r.r/r0r1r2r3 __class__i/home/ubuntu/.local/lib/python3.10/site-packages/pyannote/audio/tasks/segmentation/speaker_diarization.pyr8ps:    zSpeakerDiarization.__init__c st||jdur-|jddtdk}g}d|jdd}tt |d|dD]}|jd t |jd d |kd}|jd t |jd d |kd}|D]U}|d } |d |d } |t |d | k|d| kd} t | | |jd|jD]&} | |j} | t | d | k| d| kd}| t t |dqqWq-ttt|\}}t|t|}}t|}||}||}t|t|}t||D]-\}}|dkrtd|ddq|dkrtd|ddqtd|dd|dqtd|t |dkdd|_td|jd|jd|j|jkrAtd|jd|jdttjtj|j|jd d!t |jD|jd"d#|_!dS)$Naudio-metadatasubsettrainz*Estimating maximum number of speakers per gzs chunk in the training setr descriptionannotations-segmentsfile_idzannotations-regionsstartr(endg?file_label_idxz - z7.2%z( of all chunks contain no speech at all.r6z contain 1 speaker or lessz contain z speakers or lessr$g ףp= ?z$Setting `max_speakers_per_chunk` to za. You can override this value (or avoid this estimation step) by passing `max_speakers_per_chunk=z` to the task constructor.z`max_speakers_per_frame` (z1) must be smaller than `max_speakers_per_chunk` ()cSsg|] }d|dqS)zspeaker#r6r@).0ir@r@rA sz,SpeakerDiarization.setup..T)problem resolutionr( min_durationclassespowerset_max_classespermutation_invariant)"r7setupr) prepared_dataSubsetsindexr(rnpwherearangeappendlenuniqueziplistritemsarrayargsortcumsumsumprintmaxr*r:rr MONO_LABEL_CLASSIFICATIONr FRAMErSrangespecifications)r=stagetrainingnum_unique_speakersprogress_descriptionrI annotationsannotated_regionsregion region_start region_endregion_annotations window_start window_endwindow_annotations num_speakerscountssorting_indicesratioskratior>r@rArWs             zSpeakerDiarization.setupcCstt|jj|jj|j_dSN)rr_rmrTrUmodelpowersetr=r@r@rAsetup_loss_funcs  z"SpeakerDiarization.setup_loss_funcrI start_timecs||}t|jd|d}|d}t|||}t}|jj||\|d<} |jd|\} } |jd| | } | | d|jk| d|j k@} |jj j }d |jj j }t | d|j |j |}t d t ||t}t | d|j|j |}t ||t}tt | |}t|}||jkr |jt||jjj}t j||ft jd }d d t|D}t||| |D]\}}}||}d|||d|f<qt||jj |d|d<|jd|fdd jj D|d<||dd<|S)aPrepare chunk Parameters ---------- file_id : int File index start_time : float Chunk start time duration : float Chunk duration. Returns ------- sample : dict Dictionary containing the chunk data with the following keys: - `X`: waveform - `y`: target as a SlidingWindowFeature instance where y.labels is in meta.scope space. - `meta`: - `scope`: target scope (0: file, 1: database, 2: global) - `database`: database index - `file`: file index rBscope _label_idxXzaudio-segments-idsrHrJrK?r)dtypecSsi|]\}}||qSr@r@)rNidxlabelr@r@rA Tz4SpeakerDiarization.prepare_chunk..r6)labelsycsi|]}||qSr@r@)rNkeymetadatar@rAr_rmetafile)!get_fileScopesrXrdictraudiocroprKrJreceptive_fieldstepr(r[maximumroundastypeintminimumrbr`r_r) num_frameshparams sample_ratezerosuint8 enumeraterarrnames)r=rIrr(r label_scopelabel_scope_keychunksample_start_idend_idrrchunk_annotationsrhalfrJ start_idxrKend_idxr num_labelsrrmappingr mapped_labelr@rrA prepare_chunksH      z SpeakerDiarization.prepare_chunkreturncCsg}|D]I}|dj}t|dj}||jkr2tjtj|dd dd}|dd|d|jf}n||jkrGtj|dd|j|ffdd}n ||qt t |S)a Parameters ---------- batch : list List of samples to collate. "y" field is expected to be a SlidingWindowFeature. Returns ------- y : torch.Tensor Collated target tensor of shape (num_frames, self.max_speakers_per_chunk) If one chunk has more than `self.max_speakers_per_chunk` speakers, we keep the max_speakers_per_chunk most talkative ones. If it has less, we pad with zeros (artificial inactive speakers). rr)axisN)rrconstant)mode) datar_rr)r[rergpadr^torch from_numpystack)r=batch collated_ybrr{indicesr@r@rA collate_yds     zSpeakerDiarization.collate_y batch_idxc CsZ|d}|d}tjtj|dddd}||jk}||}||}|s'dS||}|j\}} } t|dd} || tj|| d|jj d} |jj |} t | |\}} |jj |}t|tj|dd| d }|jjd |d d d d d |js|j}t|tr|n|g}|D]}|q|j||D]}|jj|ddd|qd|iS)a8Compute permutation-invariant segmentation loss Parameters ---------- batch : (usually) dict of torch.Tensor Current batch. batch_idx: int Batch index. Returns ------- loss : {str: torch.tensor} {"loss": loss} rrr6dimNr,devicer,z loss/trainFTon_stepon_epochprog_barloggerg@norm)gradient_clip_valgradient_clip_algorithmr3)rrganyr)rshapegetattrgetonesrr to_multilabelr to_powersetfloatrargmaxlogautomatic_optimization optimizersr9rb zero_gradmanual_backwardclip_gradientsr)r=rrtargetwaveformr{keep predictionr-rr weight_keyr, multilabelpermutated_targetpermutated_target_powersetr3r optimizerr@r@rA training_steps^         z SpeakerDiarization.training_stepcCs0tdtdtdtdtdtdtddS)z1Returns diarization error rate and its componentsr)rzDiarizationErrorRate/ConfusionzDiarizationErrorRate/MisszDiarizationErrorRate/FalseAlarmzDiarizationErrorRate/PrecisionzDiarizationErrorRate/Recallz'DiarizationErrorRate/DetectionErrorRate)rrrrrrrrr@r@rAdefault_metricsz!SpeakerDiarization.default_metricc Cs|d}|d}||}|j\}}}t|dd} || tj||d|jjd} |jj|} t | |\} }|jj | } t |tj | dd| d }|jjd |d d d d d |jt| ddt|dd|jj|jjd d d d d |jjdkst|jjddks|dkrdS| }| }t|jd|jd}tt|}t||}tjd||dd d\}}tj||dk<t|jdkr|ddddtj f}|t!|jd9}t"|D]d}||}||}||dd|f}||}|#||$dt||%d|jd|&'d |('d ||dd|f}||}|#||%dd|$dt||&'d qt)|jj*D]+}t+|t,ri|j-.d||jjqUt+|t/r|j-j0|j1|d|jjddqUt2|dS)zCompute validation loss and metric Parameters ---------- batch : dict of torch.Tensor Current batch. batch_idx: int Batch index. rrr,Nr6rrrrzloss/valFTrr$r ))nrowsncolsfigsizesqueezegg?samples samples_epochz.png)run_idfigure artifact_file)3rrrrrrrrrrrrrrrvalidation_metric transposelog_dict current_epochmathlog2cpunumpyminr-ceilsqrtpltsubplotsr[nanr_newaxisr]rlplotset_xlimset_ylim get_xaxis set_visible get_yaxis tight_layoutloggersr9r experiment add_figurer log_figurerclose)r=rrrrrr-rrrr,rrrr3ry_pred num_samplesrrfigaxes sample_idxrow_idxcol_idxax_refsample_yax_hyp sample_y_predrr@r@rAvalidation_steps                  z"SpeakerDiarization.validation_step) Nr#Nr$NNr%NFNNNNr)__name__ __module__ __qualname____doc__rrrstrrrrrboolr r!rrr8rWrrrTensorrrrr" __classcell__r@r@r>rAr":sn8      7aV.U r"testpyannote/segmentationr&rCrcsZddlm}ddlm}ddlm}ddlm}ddlm }m }ddl m } |dd \} |} ||d |id }t t||} | Zj|jt| d } jd dd ddttdttffdd }||| d}| D]'}j|dd|d}||||d}|d}| |||d}| qnWdn1swY| jdd}dS)zEvaluate a segmentation modelr) Inference) get_devices)DiscreteDiarizationErrorRate)binarize) FileFinder get_protocol)Progressr6)needsr) preprocessors)total Processingg?N completedr6csj||ddS)N)r8)update)r8r6 file_taskprogressr@rA progress_hookszevaluate..progress_hookrurirF annotation)hook annotated)uemT)display)NN)pyannote.audior-pyannote.audio.pipelines.utilsr.pyannote.audio.utils.metricr/pyannote.audio.utils.signalr0pyannote.databaser1r2 rich.progressr3rbradd_tasknamer_rrr9advancereport)r&rCrr-r.r/r0r1r2r3rr1files main_taskr= inferencer reference hypothesisrBrr@r:rAevaluates2      "  rS__main__)r+r,)@rr; collectionsrtypingrrrrrrrr[rlightning.pytorch.loggersr r matplotlibr rpyannote.audio.core.taskr r r(pyannote.audio.tasks.segmentation.mixinsrpyannote.audio.torchmetricsrrrrrrrpyannote.audio.utils.lossr pyannote.audio.utils.permutationrpyannote.audio.utils.powersetr pyannote.corerrpyannote.database.protocolr#pyannote.database.protocol.protocolrrrIr/torch_audiomentations.core.transforms_interfacer torchmetricsr!rb__args__rYrr"r'rSr#typerrunr@r@r@rAs@    $        I#