o piq=@sddlZddlZddlmZmZmZmZmZmZm Z ddl Z ddl Z ddl mmZddlmZmZmZddlmZddlmZmZddlmZddlmZddlmZdd l m!Z!Gd d d eZ"dS) N)DictListOptionalSequenceTextTupleUnion)Problem ResolutionSpecifications)SegmentationTask)SegmentSlidingWindowFeature)Protocol)SegmentationProtocol)BaseWaveformTransform)MetriccseZdZdZ           d%dedeeedfd eeed e d ee e e e ffd ee e d ee de dee dedeedeee eeeeffffdd ZdefddZd&fdd Zde de d e fddZde fdd Zde fd!d"Zed#d$ZZS)'MultiLabelSegmentationa Generic multi-label segmentation Multi-label segmentation is the process of detecting temporal intervals when a specific audio class is active. Example use cases include speaker tracking, gender (male/female) classification, or audio event detection. Parameters ---------- protocol : Protocol cache : str, optional As (meta-)data preparation might take a very long time for large datasets, it can be cached to disk for later (and faster!) re-use. When `cache` does not exist, `Task.prepare_data()` generates training and validation metadata from `protocol` and save them to disk. When `cache` exists, `Task.prepare_data()` is skipped and (meta)-data are loaded from disk. Defaults to a temporary path. classes : List[str], optional List of classes. Defaults to the list of classes available in the training set. duration : float, optional Chunks duration. Defaults to 2s. warm_up : float or (float, float), optional Use that many seconds on the left- and rightmost parts of each chunk to warm up the model. While the model does process those left- and right-most parts, only the remaining central part of each chunk is used for computing the loss during training, and for aggregating scores during inference. Defaults to 0. (i.e. no warm-up). balance: Sequence[Text], optional When provided, training samples are sampled uniformly with respect to these keys. For instance, setting `balance` to ["database","subset"] will make sure that each database & subset combination will be equally represented in the training samples. weight: str, optional When provided, use this key to as frame-wise weight in loss function. batch_size : int, optional Number of training samples per batch. Defaults to 32. num_workers : int, optional Number of workers used for generating training samples. Defaults to multiprocessing.cpu_count() // 2. pin_memory : bool, optional If True, data loaders will copy tensors into CUDA pinned memory before returning them. See pytorch documentation for more details. Defaults to False. augmentation : BaseWaveformTransform, optional torch_audiomentations waveform transform, used by dataloader during training. metric : optional Validation metric(s). Can be anything supported by torchmetrics.MetricCollection. Defaults to AUROC (area under the ROC curve). N@ Fprotocolcacheclassesdurationwarm_upbalanceweight batch_size num_workers pin_memory augmentationmetricc sRt|tstdt|dtj||||| | | | |d ||_||_||_dS)NzHMultiLabelSegmentation task expects a SegmentationProtocol but you gave z. )rrrrr r!r"r) isinstancer ValueErrortypesuper__init__rrr) selfrrrrrrrrrr r!r" __class__`/home/ubuntu/.local/lib/python3.10/site-packages/pyannote/audio/tasks/segmentation/multilabel.pyr'\s$   zMultiLabelSegmentation.__init__ prepared_datac s jdur js td}jrtjj }nj}jdurvt t }|D]8}| dd}|sMtd|dd|dd}t ||D] }|vrZ |qO| fdd |Dq/tjtjd |d <_ngt }|D]V}| dd}|std|dd|dd}t |t|tj}|rtd|dd|dd d |d}t|| fdd t|tj@Dq{tjjtjd |d <tjt|tjftjd } t|D] \} d| | f<q| |d<|dS)NaE Could not infer list of classes. Either provide a list of classes when instantiating the task, or make sure that the training protocol provides a 'classes' entry. See https://github.com/pyannote/pyannote-database#segmentation for more details. rz File "uriz" (from databaseaW database) does not provide a 'classes' entry. Please make sure the corresponding training protocol provides a 'classes' entry for all files. See https://github.com/pyannote/pyannote-database#segmentation for more details. csg|]}|qSr+)index.0klass)rr+r, z.dtype classes-listz; database) provides extra classes (z, z,) that are ignored. csg|]}j|qSr+)rr0r1r(r+r,r4s Tclasses-annotated)r has_classestextwrapdedenthas_validation itertoolschainrtrain developmentlistgetr$appendnparraystr_setjoinprintzeroslenbool_ enumerateclear) r(r-msg files_iterannotated_classesfile file_classesr3 extra_classesannotated_classes_arrayfile_idr+)rr(r,post_prepare_datas          z(MultiLabelSegmentation.post_prepare_datacs6t|t|jdtjtj|j|j |j d|_ dS)Nr8)rproblem resolutionr min_durationr) r&setupr r-r MULTI_LABEL_CLASSIFICATIONr FRAMErr\rspecifications)r(stager)r+r,r]s  zMultiLabelSegmentation.setuprX start_timecs||}t|||}t}|jj||\|d<}|jd|jdd|k}||d|jk|d|jk@} |jj j } d|jj j } t | d|j|j| } t dt | | t} t | d|j|j| }t || t}|jt||jjj}t j|t|jdft jd  }d|d d |jd |f<t| || d D]\} }}d || |d |f<qt||jj |jd|d<|jd|fddjjD|d<||dd<|S)aPrepare chunk for multi-label segmentation Parameters ---------- file_id : int File index start_time : float Chunk start time duration : float Chunk duration. Returns ------- sample : dict Dictionary containing the chunk data with the following keys: - `X`: waveform - `y`: target (see Notes below) - `meta`: - `database`: database index - `file`: file index Notes ----- y is a trinary matrix with shape (num_frames, num_classes): - 0: class is inactive - 1: class is active - -1: we have no idea Xzannotations-segmentsrXstartendg?rr8r6Nr:global_label_idx)labelsyzaudio-metadatacsi|]}||qSr+r+)r2keymetadatar+r, Fr5z8MultiLabelSegmentation.prepare_chunk..metarT)get_filer dictmodelaudiocropr-rerdreceptive_fieldsteprrFmaximumroundastypeintminimum num_frameshparams sample_rateonesrMint8ziprrr7names)r(rXrbrrTchunksample_ annotationschunk_annotationsruhalfrd start_idxreend_idxr{rilabelr+rkr, prepare_chunksJ       z$MultiLabelSegmentation.prepare_chunk batch_idxcCs|d}||}|d}|j|jksJ|dk}||}||}t||tj}t|r2dS|jjd|dddddd|iS) Nrcriz loss/trainFTon_stepon_epochprog_barloggerloss) rqshapeFbinary_cross_entropyr%torchfloatisnanlogr(batchrrcy_predy_truemaskrr+r+r, training_stepKs&  z$MultiLabelSegmentation.training_stepcCsv|d}||}|d}|j|jksJ|dk}||}||}t||tj}|jjd|dddddd|iS) Nrcrirloss/valFTrr)rqrrrr%rrrrr+r+r,validation_stephs" z&MultiLabelSegmentation.validation_stepcCsdS)aQuantity (and direction) to monitor Useful for model checkpointing or early stopping. Returns ------- monitor : str Name of quantity to monitor. mode : {'min', 'max} Minimize See also -------- lightning.pytorch.callbacks.ModelCheckpoint lightning.pytorch.callbacks.EarlyStopping )rminr+r9r+r+r, val_monitorsz"MultiLabelSegmentation.val_monitor) NNrrNNrNFNN)N)__name__ __module__ __qualname____doc__rrrstrrrrrrryboolrrrr'rYr]rrrpropertyr __classcell__r+r+r)r,r(sZ6       (g Tr)#r?r<typingrrrrrrrnumpyrFrtorch.nn.functionalnn functionalrpyannote.audio.core.taskr r r (pyannote.audio.tasks.segmentation.mixinsr pyannote.corer rpyannote.databaserpyannote.database.protocolr/torch_audiomentations.core.transforms_interfacer torchmetricsrrr+r+r+r,s$