o pi@sddlZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZddlZddlZddlmZmZddlmZddlmZmZmZddlmZmZdd lm Z m!Z!m"Z"m#Z#m$Z$dd l%m&Z&dd l'm(Z(dd l)m*Z*dd l+m,Z,m-Z-ddl.m/Z/ddl0m1Z1m2Z2ddl3m4Z4ddl5m6Z6m7Z7ddl8m9Z9ddl:m;Z;z ddlZ>dZ?Wn e@ydZ?YnweAe2jBZCeAe1jBZDGddde7ZEGdddeZFdS)N)Counter)partial)DictLiteralOptionalSequenceTextUnion) MLFlowLoggerTensorBoardLogger)pyplot)Problem ResolutionSpecifications)SegmentationTaskTask)OptimalDiarizationErrorRate$OptimalDiarizationErrorRateThresholdOptimalFalseAlarmRateOptimalMissedDetectionRateOptimalSpeakerConfusionRate)binary_cross_entropy) permutate)create_rng_for_worker)SegmentSlidingWindowFeature)SpeakerDiarizationProtocol)ScopeSubset)track) DataLoaderIterableDataset)BaseWaveformTransform)Metric)MixITLossWrappermultisrc_neg_sisdrTFcs6eZdZdZdeffdd ZddZddZZS) ValDatasetzValidation dataset class Val dataset needs to be iterable so that mixture of mixture generation can be performed in the same way for both training and development. Parameters ---------- task : PixIT Task instance. taskcst||_dSN)super__init__r')selfr' __class__Y/home/ubuntu/.local/lib/python3.10/site-packages/pyannote/audio/tasks/separation/PixIT.pyr*Rs  zValDataset.__init__cC |jSr()r' val__iter__r+r.r.r/__iter__V zValDataset.__iter__cCr0r()r' val__len__r2r.r.r/__len__Yr4zValDataset.__len__) __name__ __module__ __qualname____doc__rr*r3r6 __classcell__r.r.r,r/r&Fs  r&c!seZdZdZ               d@dedeeedfd ed ee d ee d e d ee e dee de dee de dee deee eeeeffdee deddef fdd ZdAfdd Zde ded efddZd efd!d"Zd#d$Zd%ejfd&d'Zd%ejfd(d)Zd%ejfd*d+ZdBd-d.Zd ejfd/d0Z dAd1ejd2ejdeejd ejfd3d4Z d5d6Z!d7d8Z"d9e fd:d;Z#d eee eeeefffdd?Z%Z&S)CPixITuW Joint speaker diarization and speaker separation task based on PixIT Parameters ---------- protocol : SpeakerDiarizationProtocol pyannote.database protocol cache : str, optional As (meta-)data preparation might take a very long time for large datasets, it can be cached to disk for later (and faster!) re-use. When `cache` does not exist, `Task.prepare_data()` generates training and validation metadata from `protocol` and save them to disk. When `cache` exists, `Task.prepare_data()` is skipped and (meta)-data are loaded from disk. Defaults to a temporary path. duration : float, optional Chunks duration. Defaults to 5s. max_speakers_per_chunk : int, optional Maximum number of speakers per chunk (must be at least 2). Defaults to estimating it from the training set. max_speakers_per_frame : int, optional Maximum number of (overlapping) speakers per frame. Setting this value to 1 or more enables `powerset multi-class` training. Default behavior is to use `multi-label` training. weigh_by_cardinality: bool, optional Weigh each powerset classes by the size of the corresponding speaker set. In other words, {0, 1} powerset class weight is 2x bigger than that of {0} or {1} powerset classes. Note that empty (non-speech) powerset class is assigned the same weight as mono-speaker classes. Defaults to False (i.e. use same weight for every class). Has no effect with `multi-label` training. balance: Sequence[Text], optional When provided, training samples are sampled uniformly with respect to these keys. For instance, setting `balance` to ["database","subset"] will make sure that each database & subset combination will be equally represented in the training samples. weight: str, optional When provided, use this key as frame-wise weight in loss function. batch_size : int, optional Number of training samples per batch. Defaults to 32. num_workers : int, optional Number of workers used for generating training samples. Defaults to multiprocessing.cpu_count() // 2. pin_memory : bool, optional If True, data loaders will copy tensors into CUDA pinned memory before returning them. See pytorch documentation for more details. Defaults to False. augmentation : BaseWaveformTransform, optional torch_audiomentations waveform transform, used by dataloader during training. metric : optional Validation metric(s). Can be anything supported by torchmetrics.MetricCollection. Defaults to AUROC (area under the ROC curve). separation_loss_weight : float, optional Scaling factor between diarization and separation losses. Defaults to 0.5. References ---------- Joonas Kalda, Clément Pagés, Ricard Marxer, Tanel Alumäe, and Hervé Bredin. "PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordings" Odyssey 2024. https://arxiv.org/abs/2403.02288 N@F ?protocolcachedurationmax_speakers_per_chunkmax_speakers_per_frameweigh_by_cardinalitybalanceweight batch_size num_workers pin_memory augmentationmetricmax_num_speakersloss)bcemseseparation_loss_weightc ststdtj||| | | | | |dt|tstd|dur,|dur,|}td|dur5td|dur=t d| ddkrGtd ||_ ||_ ||_ ||_ ||_||_ttd d |_dS) Nz'asteroid' must be installed to train separation models with PixIT . `pip install pyannote-audio[separation]` should do the trick.)rBrHrIrJrKrLrAz>SpeakerDiarization task requires a SpeakerDiarizationProtocol.zL`max_num_speakers` has been deprecated in favor of `max_speakers_per_chunk`.z-`loss` has been deprecated and has no effect.zODiarization is done on masks separately which is incompatible powerset trainingrz-`batch_size` must be divisible by 2 for PixITT) generalized)ASTEROID_IS_AVAILABLE ImportErrorr)r* isinstancer ValueErrorwarningswarnNotImplementedErrorrCrDrErFrGrQr$r% mixit_loss)r+r@rArBrCrDrErFrGrHrIrJrKrLrMrNrQr,r.r/r*sL   zPixIT.__init__c st||jdur-|jddtdk}g}d|jdd}tt |d|dD]}|jd t |jd d |kd}|jd t |jd d |kd}|D]U}|d } |d |d } |t |d | k|d| kd} t | | |jd|jD]&} | |j} | t | d | k| d| kd}| t t |dqqWq-ttt|\}}t|t|}}t|}||}||}t|t|}t||D]-\}}|dkrtd|ddq|dkrtd|ddqtd|dd|dqtd|t |dkdd|_td|jd|jd|jdurG|j|jkrGtd|jd|jdt|jtj|jdurUtjntjd d!d"t |jD|jd#}t|jtjtjd$d"t |jDd%}||f|_!dS)&Naudio-metadatasubsettrainz*Estimating maximum number of speakers per gzs chunk in the training setr) descriptionannotations-segmentsfile_idannotations-regionsstartrBendg?file_label_idxz - z7.2%z( of all chunks contain no speech at all.z contain 1 speaker or lessz contain z speakers or lessrRg ףp= ?z$Setting `max_speakers_per_chunk` to za. You can override this value (or avoid this estimation step) by passing `max_speakers_per_chunk=z` to the task constructor.z`max_speakers_per_frame` (z1) must be smaller than `max_speakers_per_chunk` ()TcSg|] }d|dqSzspeaker#rgr..0ir.r.r/ BzPixIT.setup..)rB resolutionproblempermutation_invariantclassespowerset_max_classescSrirjr.rkr.r.r/rnJro)rBrprqrs)"r)setuprC prepared_dataSubsetsindexrBrnpwherearangeappendlenuniqueziplistritemsarrayargsortcumsumsumprintmaxrDrWrrFRAMEr MULTI_LABEL_CLASSIFICATIONMONO_LABEL_CLASSIFICATIONrangespecifications)r+stagetrainingnum_unique_speakersprogress_descriptionrb annotationsannotated_regionsregion region_start region_endregion_annotations window_start window_endwindow_annotations num_speakerscountssorting_indicesratioskratiospeaker_diarizationspeaker_separationr,r.r/rus               z PixIT.setuprb start_timecs||}t|jd|d}|d}t|||}t}|jj||\|d<} |jd|jdd|k} | | d|jk| d|j k@} |jj j } d |jj j } t | d|j |j | }t d t || t}t | d|j|j | }t || t}tt | |}t|}||jkr |jt||jjj}t j||ft jd }d d t|D}t||| |D]\}}}||}d|||d|f<qt||jj |d|d<|jd|fdd jj D|d<||dd<|S)aPrepare chunk Parameters ---------- file_id : int File index start_time : float Chunk start time duration : float Chunk duration. Returns ------- sample : dict Dictionary containing the chunk data with the following keys: - `X`: waveform - `y`: target as a SlidingWindowFeature instance where y.labels is in meta.scope space. - `meta`: - `scope`: target scope (0: file, 1: database, 2: global) - `database`: database index - `file`: file index r\scope _label_idxXrarbrdrer?rdtypecSsi|]\}}||qSr.r.)rlidxlabelr.r.r/ z'PixIT.prepare_chunk..rg)labelsycsi|]}||qSr.r.rlkeymetadatar.r/rrmetafile)!get_fileScopesrvrdictmodelaudiocroprerdreceptive_fieldsteprBrymaximumroundastypeintminimumrr~r}rC num_frameshparams sample_ratezerosuint8 enumeraterrrnames)r+rbrrBr label_scopelabel_scope_keychunksample_rchunk_annotationsrhalfrd start_idxreend_idxr num_labelsrrmappingr mapped_labelr.rr/ prepare_chunkOsJ      zPixIT.prepare_chunkreturnc Cs(tt||j|j|jdt|jdddS)zwValidation data loader Returns ------- DataLoader Validation data loader. Tr^)r)rHrIrJ drop_last collate_fn)r r&rHrIrJrrr2r.r.r/val_dataloaders zPixIT.val_dataloaderc#stj}tdd}|dur|}n't}tfdd|DD]}ddt||D}j|fi|||<q% |durI|||}t |Vq>)aXIterate over validation samples Yields ------ dict: X: (time, channel) Audio chunks. y: (frame, ) Frame-level targets. Note that frame < time. `frame` is infered automagically from the example model output. ... rFNcsg|]}j|qSr.)metadata_unique_valuesrr2r.r/rnrz%PixIT.val__iter__..cSsi|]\}}||qSr.r.)rlrvaluer.r.r/rrz%PixIT.val__iter__..) rrgetattrval__iter__helperr itertoolsproductrchoicenext)r+rngrFchunks subchunksrfiltersr.r2r/r1s     zPixIT.val__iter__rc%ks|jddt|k}|D]\}}||jd||jd||kM}qt|d}|jd|}t|t|} |j} t |dd} || | } |jd t|jd d | kd} t | D]}t|jd d | kd}t|jd d |t|jd d |}|| | }|jd |\}}}| |||| }| | d || k| d|k@}tt|d}| t| d|}|jdkr|| || }tj j||d}|jd |\}}}| |||| }|| || }|dj|dj}t||jkr|V|Vqk|d d|ddgg}|D]&\}}}}}}|d}||dkrJt|d||d<q,|||gq,g}d}|jd d |d} |D]H}!|!d|jd d |||jd d ||kr|d7}|jd d ||} |!d| | kr|| |!d|!d| f|!d} qdgd}"tj||"d}t|d dkr|| || }|d t|d }#tj j||#d}!|!\}}}| ||| }$|| |$| }|dj|dj}t||jkr|V|VqkqK)abIterate over samples with optional domain filtering Mixtures are paired so that they have no speakers in common, come from the same file, and the combined number of speakers is no greater than max_speaker_per_chunk. Parameters ---------- rng : random.Random Random number generator filters : dict, optional When provided (as {key: value} dict), filter files so that only files such as file[key] == value are used for generating chunks. Yields ------ chunk : dict Chunks. r\r]rrzaudio-annotatednum_chunks_per_filergTrarbrcrBrdrerf)pr))rdf)rer)rBrr)rvrwrxrryrzrrrBr searchsortedrandomruniformrr~isinsizerrrr}rCrr|r)%r+splitrr split_filesrrfile_idsannotated_durationcum_prob_annotated_durationrBrrbrrannotated_region_indices#cum_prob_annotated_regions_durationannotated_region_indexregion_durationrdrrprevious_speaker_labelsrepeated_speaker_annotations first_chunk second_chunkrmerged_repeated_segmentsreprevioussegments_without_repeatcurrent_region_index previous_timesegmentrprob_segments_durationnew_start_timer.r.r/common__iter__helpers                   zPixIT.common__iter__helpercK|jd|fi|S)aIterate over validation samples with optional domain filtering Parameters ---------- rng : random.Random Random number generator filters : dict, optional When provided (as {key: value} dict), filter validation files so that only files such as file[key] == value are used for generating chunks. Yields ------ chunk : dict validation chunks. developmentrr+rrr.r.r/rzPixIT.val__iter__helpercKr)aIterate over training samples with optional domain filtering Parameters ---------- rng : random.Random Random number generator filters : dict, optional When provided (as {key: value} dict), filter training files so that only files such as file[key] == value are used for generating chunks. Yields ------ chunk : dict Training chunks. r^rrr.r.r/train__iter__helperrzPixIT.train__iter__helperr^cCsb||}||}||}|jj|dkd|j||jjj|dd}|j |j d|dS)aCollate function used for most segmentation tasks This function does the following: * stack waveforms into a (batch_size, num_channels, num_samples) tensor batch["X"]) * apply augmentation when in "train" stage * convert targets into a (batch_size, num_frames, num_classes) tensor batch["y"] * collate any other keys that might be present in the batch using pytorch default_collate function Parameters ---------- batch : list of dict List of training samples. Returns ------- batch : dict Collated batch as {"X": torch.Tensor, "y": torch.Tensor} dict. r^moderg)samplesrtargets)rrr) collate_X collate_y collate_metarKr^rrr unsqueezer r squeeze)r+batchr collated_X collated_y collated_meta augmentedr.r.r/rs    zPixIT.collate_fncCsg}|D]I}|dj}t|dj}||jkr2tjtj|dd dd}|dd|d|jf}n||jkrGtj|dd|j|ffdd}n ||qt t |S)a Parameters ---------- batch : list List of samples to collate. "y" field is expected to be a SlidingWindowFeature. Returns ------- y : torch.Tensor Collated target tensor of shape (num_frames, self.max_speakers_per_chunk) If one chunk has more than `self.max_speakers_per_chunk` speakers, we keep the max_speakers_per_chunk most talkative ones. If it has less, we pad with zeros (artificial inactive speakers). rr)axisN)rrconstantr ) datar}rrCryrrpadr|torch from_numpystack)r+rrbrrindicesr.r.r/rs     zPixIT.collate_ypermutated_predictiontargetcCst|||d}|S)aPermutation-invariant segmentation loss Parameters ---------- permutated_prediction : (batch_size, num_frames, num_classes) torch.Tensor Permutated speaker activity predictions. target : (batch_size, num_frames, num_speakers) torch.Tensor Speaker activity. weight : (batch_size, num_frames, 1) torch.Tensor, optional Frames weight. Returns ------- seg_loss : torch.Tensor Permutation-invariant segmentation loss rG)rfloat)r+r!r"rGseg_lossr.r.r/segmentation_losss zPixIT.segmentation_losscCs |jd}||}|jdddkjdd}|jdddkjdd}g} t|D]S} tj|| dd|| jdddkf|| dd|| jdddkffdd} |jd|| || } tj|jd| f| jd} tj| | fdd} | | q't| } || ||fS)a Creates mixtures of mixtures and corresponding diarization targets. Keeps track of how many speakers came from each mixture in order to reconstruct the original mixtures. Parameters ---------- mix1 : torch.Tensor First mixture. mix2 : torch.Tensor Second mixture. target1 : torch.Tensor First mixture diarization targets. target2 : torch.Tensor Second mixture diarization targets. Returns ------- mom : torch.Tensor Mixtures of mixtures. targets : torch.Tensor Diarization targets for mixtures of mixtures. num_active_speakers_mix1 : torch.Tensor Number of active speakers in the first mixture. num_active_speakers_mix2 : torch.Tensor Number of active speakers in the second mixture. rrgdimNrRdevice) shaperrrcatrr*r|r)r+mix1mix2target1target2rHmomnum_active_speakers_mix1num_active_speakers_mix2r rmr" padding_dimpadding_tensorr.r.r/create_mixtures_of_mixtures3s2  ""   z!PixIT.create_mixtures_of_mixturesc Csp|d}|d}|jd}|dkrdS|ddkr|dd}|dddd}|dddd}||||ddd|ddd\}}} } tj|ddd|ddd|fdd}|tj|||fdd\} } | |d} | j\} }} t|d d}||tj| |d|jj d }t || \}} |j |||d }| | ddt||f dd}||| ||fS) a8Common step for training and validation Parameters ---------- batch : dict of torch.Tensor Current batch. Returns ------- seg_loss : torch.Tensor Segmentation loss. separation_loss : torch.Tensor Separation loss. diarization : torch.Tensor Diarization predictions. permutated_diarization : torch.Tensor Permutated diarization predictions that minizimise seg_loss. target : torch.Tensor Diarization target. rrrrRNrrgr'rGr)r#)r+rr6rr,rrgetonesr*rr&r[ transposermean)r+rr"waveformbszr-r.r1 mom_targetr diarizationsources mom_sourcesrHr weight_keyrGpermutated_diarizationr%separation_lossr.r.r/ common_stepjsJ   "(   zPixIT.common_step batch_idxc Cs||\}}}}}|jjd|ddddd|jjd|dddddd|j||j|}t|r5dS|jjd|ddddd|jsv|j} t| t rP| n| g} | D]} | qU|j || D]} |jj | |jj d d | qdd |iS) a(Compute PixIT loss for training Parameters ---------- batch : (usually) dict of torch.Tensor Current batch. batch_idx: int Batch index. Returns ------- loss : {str: torch.tensor} {"loss": loss} zloss/train/separationFTon_stepon_epochprog_barloggerzloss/train/segmentationrgNz loss/trainnorm)gradient_clip_valgradient_clip_algorithmrN)rDrlogrQrisnanautomatic_optimization optimizersrVr zero_gradmanual_backwardclip_gradientsrLr) r+rrEr%rCr>rBr"rNrQ optimizerr.r.r/ training_stepsh       zPixIT.training_stepcCstttttdS)z1Returns diarization error rate and its components)DiarizationErrorRatezDiarizationErrorRate/ThresholdzDiarizationErrorRate/ConfusionzDiarizationErrorRate/MisszDiarizationErrorRate/FalseAlarm)rrrrrr2r.r.r/default_metrics zPixIT.default_metriccCs||\}}}}}|jjd|ddddd|jjd|dddddd|j||j|}|jjd|ddddd|jt|ddt|dd|jj|jjddddd|jjd ksmt |jjdd ksm|d krod S| } | } t|jd } t t | } t | | } tjd| | d dd \}}tj| | d k<t| jdkr| d d d d tjf} | t| jd9} t| D]d}|| }|| }||dd |f}| |}|||d t||d|jd|d| d||dd|f}| |}|||dd|d t||dqt!|jj"D]+}t#|t$rH|j%&d||jjq4t#|t'r^|j%j(|j)|d|jjddq4t*|d S)zCompute validation loss and metric Parameters ---------- batch : dict of torch.Tensor Current batch. batch_idx: int Batch index. zloss/val/separationFTrFzloss/val/segmentationrgzloss/valrRrN ))nrowsncolsfigsizerrgg?r  samples_epochz.png)run_idfigure artifact_file)+rDrrNrQvalidation_metricrr9log_dict current_epochmathlog2r$cpunumpyminrHceilsqrtpltsubplotsrynanr}r+newaxisr{rplotset_xlimset_ylim get_xaxis set_visible get_yaxis tight_layoutloggersrVr experiment add_figurer log_figurer`close)r+rrEr%rCr>rBr"rNry_pred num_samplesr\r]figaxes sample_idxrow_idxcol_idxax_refsample_yax_hyp sample_y_predrJr.r.r/validation_steps                zPixIT.validation_step)Nr=NNFNNr>NFNNNNr?r()r^)'r7r8r9r:rrr strr$rboolrrr"r#rrr*rurr rr1rRandomrrr rrTensorrr&r6rDrVrXrr;r.r.r,r/r<]s?      EpW(9 +2 7OM r<)GrrfrrX collectionsr functoolsrtypingrrrrrr riryrlightning.pytorch.loggersr r matplotlibr rmpyannote.audio.core.taskr rr(pyannote.audio.tasks.segmentation.mixinsrrpyannote.audio.torchmetricsrrrrrpyannote.audio.utils.lossr pyannote.audio.utils.permutationrpyannote.audio.utils.randomr pyannote.corerrpyannote.database.protocolr#pyannote.database.protocol.protocolrr rich.progressrtorch.utils.datar r!/torch_audiomentations.core.transforms_interfacer" torchmetricsr#asteroid.lossesr$r%rTrUr__args__rwrr&r<r.r.r.r/sD