o pi~@sddlmZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z mZddlmZddlmZdd lmZdd lmZmZmZmZmZmZmZmZddlZddlZ ddl!Z"ddl#Z#dd l$m%Z%m&Z&dd l'm(Z(dd l)m*Z*ddl+m,Z,m-Z-ddl.m/Z/m0Z0m1Z1ddl2m3Z3ddl4m5Z5ddl6m7Z7m8Z8e9e-j:Z;e9e,j:ZGddde?Z@e GdddZAGddde1ZBGddde0ZCd'd#d$ZDGd%d&d&ejEZFdS)() annotationsN) defaultdict) dataclass)Enum)cached_propertypartial)Number)Path)mkstemp)DictListLiteralOptionalSequenceTextTupleUnion)binary_cross_entropynll_loss)check_protocol)Protocol)ScopeSubset) DataLoaderDatasetIterableDataset)Identity)BaseWaveformTransform)MetricMetricCollectionc@s eZdZdZdZdZdZdZdS)ProblemrN)__name__ __module__ __qualname__BINARY_CLASSIFICATIONMONO_LABEL_CLASSIFICATIONMULTI_LABEL_CLASSIFICATIONREPRESENTATION REGRESSIONr-r-L/home/ubuntu/.local/lib/python3.10/site-packages/pyannote/audio/core/task.pyr ;s r c@seZdZdZdZdS) Resolutionr!r"N)r%r&r'FRAMECHUNKr-r-r-r.r/Gsr/c@s eZdZdS)UnknownSpecificationsErrorN)r%r&r'r-r-r-r.r2Lsr2c@seZdZUded<ded<ded<dZded <d Zd ed <dZd ed<dZded<dZded<e dddZ e dddZ ddZ ddZ dS) Specificationsr problemr/ resolutionfloatdurationNOptional[float] min_duration)r:zOptional[Tuple[float, float]]warm_upzOptional[List[Text]]classes Optional[int]powerset_max_classesFboolpermutation_invariantreturncCs&|jdurdS|jtjkrtddS)NFzQ`powerset_max_classes` only makes sense with multi-class classification problems.T)r>r4r r) ValueErrorselfr-r-r.powersetms  zSpecifications.powersetintcs&ttfddtdjdDS)Nc3s$|] }tjtj|VqdSN)scipyspecialbinomlenr<).0irCr-r. s  z6Specifications.num_powerset_classes..rr!)rFsumranger>rCr-rCr.num_powerset_classesys  z#Specifications.num_powerset_classescCdS)Nr!r-rCr-r-r.__len__zSpecifications.__len__ccs |VdSrGr-rCr-r-r.__iter__s zSpecifications.__iter__rAr?)rArF)r%r&r'__annotations__r9r;r<r>r@rrErQrSrUr-r-r-r.r3Ps         r3c.eZdZd fdd ZddZddZZS) TrainDatasettaskTaskct||_dSrGsuper__init__rZrDrZ __class__r-r.r_  zTrainDataset.__init__cC |jSrG)rZ train__iter__rCr-r-r.rU zTrainDataset.__iter__cCrdrG)rZ train__len__rCr-r-r.rSrfzTrainDataset.__len__rZr[)r%r&r'r_rUrS __classcell__r-r-rar.rYrYcrX) ValDatasetrZr[cr\rGr]r`rar-r.r_rczValDataset.__init__cCs |j|SrG)rZval__getitem__)rDidxr-r-r. __getitem__s zValDataset.__getitem__cCrdrG)rZ val__len__rCr-r-r.rSrfzValDataset.__len__rh)r%r&r'r_rnrSrir-r-rar.rkrjrkvaluerFrAstrcs.gd}fdd|D}|sdS|ddS)aMReturn the most suitable type for storing the value passed in parameter in memory. Parameters ---------- value: int value whose type is best suited to storage in memory Returns ------- str: numpy formatted type (see https://numpy.org/doc/stable/reference/arrays.dtypes.html) ))b)ii2)lrMcs$g|]\}}|tkr||fqSr-)abs)rLmax_valtyperpr-r. szget_dtype..i8rr!r-)rp types_list filtered_listr-rxr. get_dtypes  r}cs,eZdZdZ         dYdZfdd ZddZd[d!d"Zd\d#d$Zed]d&d'Z e j d^d*d'Z ed_d,d-Z e j d`d/d-Z d0d1Z d2d3Z d4d5Zdad7d8Zdbd:d;Z d\dcd>d?ZdddCdDZdedEdFZdGdHZdIdJZdfdLdMZdedNdOZdgdPdQZedhdSdTZdUdVZedWdXZZS)ir[u Base task class A task is the combination of a "problem" and a "dataset". For example, here are a few tasks: - voice activity detection on the AMI corpus - speaker embedding on the VoxCeleb corpus - end-to-end speaker diarization on the VoxConverse corpus A task is expected to be solved by a "model" that takes an audio chunk as input and returns the solution. Hence, the task is in charge of generating (input, expected_output) samples used for training the model. Parameters ---------- protocol : Protocol pyannote.database protocol cache : str, optional As (meta-)data preparation might take a very long time for large datasets, it can be cached to disk for later (and faster!) re-use. When `cache` does not exist, `Task.prepare_data()` generates training and validation metadata from `protocol` and save them to disk. When `cache` exists, `Task.prepare_data()` is skipped and (meta)-data are loaded from disk. Defaults to a temporary path. duration : float, optional Chunks duration in seconds. Defaults to two seconds (2.). min_duration : float, optional Sample training chunks duration uniformely between `min_duration` and `duration`. Defaults to `duration` (i.e. fixed length chunks). warm_up : float or (float, float), optional Use that many seconds on the left- and rightmost parts of each chunk to warm up the model. This is mostly useful for segmentation tasks. While the model does process those left- and right-most parts, only the remaining central part of each chunk is used for computing the loss during training, and for aggregating scores during inference. Defaults to 0. (i.e. no warm-up). batch_size : int, optional Number of training samples per batch. Defaults to 32. num_workers : int, optional Number of workers used for generating training samples. Defaults to multiprocessing.cpu_count() // 2. pin_memory : bool, optional If True, data loaders will copy tensors into CUDA pinned memory before returning them. See pytorch documentation for more details. Defaults to False. augmentation : BaseWaveformTransform, optional torch_audiomentations waveform transform, used by dataloader during training. metric : optional Validation metric(s). Can be anything supported by torchmetrics.MetricCollection. Defaults to value returned by `default_metric` method. Attributes ---------- specifications : Specifications or tuple of Specifications Task specifications (available after `Task.setup` has been called.) N@r: FprotocolrcacheOptional[Union[str, None]]r7r6r9r8r;!Union[float, Tuple[float, float]] batch_sizerF num_workersr= pin_memoryr? augmentationOptional[BaseWaveformTransform]metric2Union[Metric, Sequence[Metric], Dict[str, Metric]]c stt|\|_} | d|_| d|_|jstd| d|_|r(t|n||_ ||_ |dur4|n||_ ||_ t |trC||f}||_|durPtd}|dkrntjdkrntjddkrntjd d krntd d}||_||_| pztd d |_| |_dS)Nhas_validation has_scopezQProtocol must provide 'scope' information (e.g. 'file', 'database', or 'global'). has_classesr"rdarwinr#r!zUnum_workers > 0 is not supported with macOS and Python 3.8+: setting num_workers = 0.dict) output_type)r^r_rrrrrBrr rr7r9r isinstancerr;multiprocessing cpu_countsysplatform version_infowarningswarnrrrr_metric) rDrrr7r9r;rrrrrchecksrar-r.r_s<        z Task.__init__c& s|jr|jr|jjdkrdS|jjjdddnttd|_tt t d<t d<t }t t }t }t }t }t }t }i}|j rft tt d|jtt d |j} n tt d|j} d} d} t| D]C\} \} }t}|d d vrd |d d |d |d <t | |d<t |d|d<t|tgd }|D]<}||}t|tr||vrӈ||||||<qt|tr|||<qtjd |d t|dtdq|t }|t|d| }d}|dD]!}|j |j krq| |j |j!f}||||j 7}| d7} q||||| f| }|dj"ddD]l\}}}t |d}||vr_||||}d}}|dkr|d }||vrzg||<|||vr||||||}|dkr||vr||||}|| |j!|j#|||f| d7} qI||| fqyfddDfddtD} dt$t%dd|Dfddg}!dt$t%dd|Dfddd t$t%d!d|Dfd"t$t%d#d|Dfd$t$t%d%d|Dfg}"i}#|jj&|#d&<t'j(|t'j)d'|#d(<|*t'j(| d'|#d)<*t'(||#d*<|*t'j(||!d'|#d+<|*t'j(|d,d-gd'|#d.<|*t'j(||"d'|#d/<|*t'j(|d,d-gd'|#d0<|*|#d1<|+D]\}}$t'j(|$t'j)d'|#d2|d3<q|*t'j(|t'j)d'|#d4<|*|j r|,|#|-|#t.|jd5}%t'j/|%fi|#WddS1swYdS)6aUse this to prepare data from task protocol Notes ----- Called only once on the main process (and only on it), for global_rank 0. After this method is called, the task should have a `prepared_data` attribute with the following dictionary structure: prepared_data = { 'protocol': name of the protocol 'audio-path': array of N paths to audio 'audio-metadata': array of N audio infos such as audio subset, scope and database 'audio-annotated': array of N annotated duration (usually equals file duration but might be shorter if file is not fully annotated) 'annotations-regions': array of M annotated regions 'audio-regions-ids': array of N start/end indices of annotated regions 'annotations-segments': array of M' annotated segments 'audio-segments-ids': array of N start/end indices of annotated segments 'metadata-values': dict of lists of values for subset, scope and database 'metadata-`database-name`-labels': array of `database-name` labels. Each database with "database" scope labels has it own array. 'metadata-labels': array of global scope labels } rNT)parentsexist_okr!subsetscopetrain developmentdatabase)urirraudiorr< annotation annotatedz Ignoring 'z ' metadata because of its type (z*). Only str and int are supported for now.)categoryrr:rr) yield_labelcs"g|] tfddDqS)c3s|] }|dVqdS)rN)getrLkey metadatumr-r.rN s/Task.prepare_data...)tuple)rL)metadata_unique_valuesrr.rysz%Task.prepare_data..cs.g|]\}|ttfddDfqS)c3s|]}|VqdSrGr-)rLmrMr-r.rN r)r}maxr)metadatarr.ry sfile_idcs|]}|dVqdSrNr-)rLarr-r-r.rNrz$Task.prepare_data..)r7f)startrcsrrr-rLar-r-r.rNr)endrfile_label_idxcsr)r#Nr-rr-r-r.rN#rdatabase_label_idxcsr)r$Nr-rr-r-r.rN$rglobal_label_idxcsr)Nr-rr-r-r.rN%rr)dtypez audio-pathzaudio-metadatazaudio-annotatedzannotations-regions)rrM)rrMzaudio-regions-idszannotations-segmentszaudio-segments-idszmetadata-valuesz metadata-z-labelszmetadata-labelswb)0rexistsstatst_sizeparentmkdirr r rlistSubsetsScopesr itertoolschainziprepeatrrr enumeraterappendindexsetrrqrFrrrw UserWarningr7r itertracksrr}rnamenparraystr_clearitemsprepare_validationpost_prepare_dataopensavez_compressed)&rDaudiosannotated_durationannotated_regionsaudio_regions_idsraudio_segments_ids unique_labelsdatabase_unique_labels files_iter regions_id segments_idrrfilerremaining_metadata_keysrrpfile_unique_labels _regions_id_annotated_durationsegmentannotated_region _segments_id_labelrrrrrmetadata_dtype region_dtype segment_dtype prepared_datalabels cache_filer-)rrr. prepare_data5s6                              $zTask.prepare_datarr cCrR)aMethod for completing `prepared_data` with task-specific data. For instance, for a classification task, this could be a list of possible classes. Parameters ---------- prepared_data: dict dictionnary containing protocol data prepared by `prepare_data()` Note ---- This method does not return anything. Thus, user have to directly modify `prepared_data`, for updates to be taken into account Nr-)rDrr-r-r.r_szTask.post_prepare_datacCs|dkr |jj|j|_z"t|jd}ttj|dd|_Wdn1s)wYWn t y<t ddw|j j |jdkrVt d |j j d |jdd dS) z>Setup data cached by prepare_data into the task on each devicefitrbT) allow_pickleNzICached data for protocol not found. Ensure that prepare_data() was calledzJ and executed correctly or/and that the path to the task cache is correct.rz!Protocol specified for the task (z)) does not correspond to the cached one ())trainerstrategy broadcastrrrrloadrFileNotFoundErrorprintrrrB)rDstagerr-r-r.setupps*  z Task.setuprAcCs|jjSrGmodelautomatic_optimizationrCr-r-r.r szTask.automatic_optimizationautomatic_optimisationNonecCs ||j_dSrGr )rDr r-r-r.r s ,Union[Specifications, Tuple[Specifications]]cCst|ds td|jS)N_specificationsaTask specifications are not available. This is most likely because they depend on the content of the training subset. Use `task.prepare_data()` and `task.setup()` to go over the training subset and fix this, or let lightning trainer do that for you in `trainer.fit(model)`.)hasattrr2rrCr-r-r.specificationss zTask.specificationsrcCs ||_dSrG)r)rDrr-r-r.rs cCsdSrGr-rCr-r-r.setup_loss_funcrTzTask.setup_loss_funccCd|jjd}t|)N Missing 'z.train__iter__' method.rbr%NotImplementedErrorrDmsgr-r-r.rezTask.train__iter__cCr)Nrz.train__len__' method.rrr-r-r.rgrzTask.train__len__rcCsd|jjd}t|)Nrz.collate_fn' method.r)rDbatchrrr-r-r. collate_fnszTask.collate_fnrc Cs(tt||j|j|jdt|jdddS)NTrrrrr drop_lastr)rrYrrrrrrCr-r-r.train_dataloaders zTask.train_dataloaderr3 torch.TensorcCsH|jtjtjfvrt|||dS|jtjfvrt|||dSd}t|)a8Guess and compute default loss according to task specification Parameters ---------- specifications : Specifications Task specifications target : torch.Tensor * (batch_size, num_frames) for binary classification * (batch_size, num_frames) for multi-class classification * (batch_size, num_frames, num_classes) for multi-label classification prediction : torch.Tensor (batch_size, num_frames, num_classes) weight : torch.Tensor, optional (batch_size, num_frames, 1) Returns ------- loss : torch.Tensor Binary cross-entropy loss in case of binary and multi-label classification, Negative log-likelihood loss in case of multi-class classification. weightz+TODO: implement for other types of problems)r4r r(r*rr)rr)rDrtarget predictionr"rr-r-r. default_lossszTask.default_loss batch_idxrLiteral['train', 'val']c Cst|jtr td||d}|j\}}}|d}|dkr%t|ddnd} || tj ||d|jj d} t |j d |j |} d | ddd| f<t |j d|j |} d | dd|| df<|j|j||| d } t| rudS|jjd || d dd ddd| iS)aDefault training or validation step according to task specification * binary cross-entropy loss for binary or multi-label classification * negative log-likelihood loss for regular classification If "weight" attribute exists, batch[self.weight] is also passed to the loss function during training (but has no effect in validation). Parameters ---------- batch : (usually) dict of torch.Tensor Current batch. batch_idx: int Batch index. stage : {"train", "val"} "train" for training step, "val" for validation step Returns ------- loss : {str: torch.tensor} {"loss": loss} zCDefault training/validation step is not implemented for multi-task.Xyrr"Nr!)devicerr:r!zloss/FT)on_stepon_epochprog_barloggerloss)rrrrr shapegetattrrtorchonesr*roundr;r7r%isnanlog)rDrr&ry_predr num_framesrr) weight_keyr" warm_up_left warm_up_rightr/r-r-r. common_steps8   zTask.common_stepcC|||dS)Nrr<rDrr&r-r-r. training_step+zTask.training_stepcCsd|jjd}t|)Nrz.val__getitem__' method.r)rDrmrr-r-r.rl.rzTask.val__getitem__cCr)Nrz.val__len__' method.rrr-r-r.ro3rzTask.val__len__Optional[DataLoader]c Cs2|jrtt||j|j|jdt|jdddSdS)NFvalrr)rrrkrrrrrrCr-r-r.val_dataloader8s  zTask.val_dataloadercCr=)NrCr>r?r-r-r.validation_stepGrAzTask.validation_stepcCr)zDefault validation metricrz.default_metric' method.rrr-r-r.default_metricJrzTask.default_metricrcCs|jdur ||_t|jSrG)rrFrrCr-r-r.rOs   z Task.metriccCs0|j}|dur||j_|jj|jjdSdSrG)rr validation_metrictor*)rDrr-r-r.setup_validation_metricVs zTask.setup_validation_metriccCs*tt|j\}}||jrdfSdfS)aQuantity (and direction) to monitor Useful for model checkpointing or early stopping. Returns ------- monitor : str Name of quantity to monitor. mode : {'min', 'max} Minimize See also -------- lightning.pytorch.callbacks.ModelCheckpoint lightning.pytorch.callbacks.EarlyStopping rmin)nextiterrrhigher_is_better)rDrrr-r-r. val_monitor\szTask.val_monitor) Nr~Nr:rNFNN)rrrrr7r6r9r8r;rrrFrr=rr?rrrr)rr rGrV)r r?rAr )rAr)rr)r)rAr)rr3rAr )r&rFrr')r&rF)rArB)rAr)rAr)r%r&r'__doc__r_rrrpropertyr setterrrrergrrr%r<r@rlrorDrErFrrrIrNrir-r-rar.r[sX>< ,        ' G    r[)rprFrArq)G __future__rrrrr collectionsr dataclassesrenumr functoolsrrnumbersrpathlibr tempfiler typingr r r rrrrr lightningnumpyr scipy.specialrHr2pyannote.audio.utils.lossrrpyannote.audio.utils.protocolrpyannote.databaser#pyannote.database.protocol.protocolrrtorch.utils.datarrrtorch_audiomentationsr/torch_audiomentations.core.transforms_interfacer torchmetricsrrr__args__rrr r/ Exceptionr2r3rYrkr}LightningDataModuler[r-r-r-r.sH       (       ;