o pi'G@sdZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl Z ddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZd d lmZGdddZddZdS)ah Pipeline Usage: pyannote-pipeline train [options] [(--forever | --iterations=)] pyannote-pipeline best [options] pyannote-pipeline apply [options] pyannote-pipeline -h | --help pyannote-pipeline --version Common options: Experimental protocol (e.g. "Etape.SpeakerDiarization.TV") --registry= Path to, comma-separated, database configuration files. [default: ~/.pyannote/db.yml] --subset= Set subset. Defaults to 'development' in "train" mode, and to 'test' in "apply" mode. "train" mode: Set experiment root directory. This script expects a configuration file called "config.yml" to live in this directory. See "Configuration file" section below for more details. --iterations= Number of iterations. [default: 1] --forever Iterate forever. --sampler= Choose sampler between RandomSampler or TPESampler [default: TPESampler]. --pruner= Choose pruner between MedianPruner or SuccessiveHalvingPruner. Defaults to no pruning. --pretrained= Use parameters in existing training directory to bootstrap the optimization process. In practice, this will simply run a first trial with this set of parameters. --average-case Optimize for average case instead of worst case. "apply" mode: Path to the directory containing trained hyper- parameters (i.e. the output of "train" mode). --use-filter Apply pipeline only to files that pass the filter. Configuration file: The configuration of each experiment is described in a file called /config.yml that describes the pipeline. ................... /config.yml ................... pipeline: name: Yin2018 params: sad: tutorials/pipeline/sad scd: tutorials/pipeline/scd emb: tutorials/pipeline/emb metric: angular # preprocessors can be used to automatically add keys into # each (dict) file obtained from pyannote.database protocols. preprocessors: audio: ~/.pyannote/db.yml # load template from YAML file video: ~/videos/{uri}.mp4 # define template directly # filters can be used to filter out some files from the protocol # (e.g. to only keep files with a specific number of speakers) filters: pyannote.audio.utils.protocol.FilterByNumberOfSpeakers: num_speakers: 2 # one can freeze some hyper-parameters if needed (e.g. when # only part of the pipeline needs to be updated) freeze: speech_turn_segmentation: speech_activity_detection: onset: 0.5 offset: 0.5 # pyannote.audio pipelines will run on CPU by default. # use `device` key to send it to GPU. device: cuda ................................................................... "train" mode: Tune the pipeline hyper-parameters /..yml "best" mode: Display current best loss and corresponding hyper-paramters. "apply" mode Apply the pipeline (with best set of hyper-parameters) N)Optional)Path)docopt)tqdm)datetime) FileFinder)registry) get_annotated)get_class_by_name) OptimizercseZdZdZdZdZdZed!dede ddfd d Z d!d ede ffd d Z      d"de de e de edede e de e de fddZd#de de fddZ  d$de dede e de fdd ZZS)% ExperimentzPipeline experiment Parameters ---------- experiment_dir : `Path` Experiment root directory. training : `bool`, optional Switch to training mode z{experiment_dir}/config.ymlz*{experiment_dir}/train/{protocol}.{subset}z{train_dir}/apply/{date}F train_dirtrainingreturncCsB|jd}|||d}|d}ttj||_|j||S)a6Load pipeline from train directory Parameters ---------- train_dir : `Path` Path to train directory training : `bool`, optional Switch to training mode. Returns ------- xp : `Experiment` Pipeline experiment. r r params.yml) parentsr fromtimestampospathgetmtimemtime_ pipeline_ load_params)clsrrexperiment_dirxp params_ymlrP/home/ubuntu/.local/lib/python3.10/site-packages/pyannote/pipeline/experiment.pyfrom_train_dirs   zExperiment.from_train_dirrc st||_|jj|jd}t|d}tj|tjd|_ Wdn1s)wYi}|j di D]@\}}t |t rXt|ddd}|di| di||<q9z t|d ||<Wq9tyy} z |} | ||<WYd} ~ q9d} ~ ww||_g|j d i D]\}} t|}|di| qd tffd d } | |_|j dd} t| dd}|di|j d di|_d|j vr|j d} |j| d|j vrddl}||j d}|j|dSdS)N)rrLoader preprocessorsnamezpyannote.pipeline)default_module_nameparams) database_ymlfiltersrcstfddDS)Nc3s|]}|VqdS)Nr).0firr sz;Experiment.__init__..all_filters..)allr-r*r-r all_filterssz(Experiment.__init__..all_filterspipelinezpyannote.pipeline.blocksfreezedevicerr)super__init__r CONFIG_YMLformatopenyamlload SafeLoaderconfig_getitems isinstancedictr rFileNotFoundErrorpreprocessors_appendboolfilters_rr4torchr5to)selfrr config_ymlfpr%key preprocessorKlassetemplater(r2 pipeline_namerHr5 __class__r1r r7sR       zExperiment.__init__ developmentNr protocol_namesubset pretrained n_iterationssamplerpruner average_casec Cst|jj|j||d}|jdddtj||jd} d} t|j |d| |||d} |j dkr3d nd } |d } t d d dd}| d| d |rs|d }t|dd}tj|tjd}Wdn1siwY|d}nd}tt|jt| |}| j||dd}z| j}Wnty}z | tj}WYd}~nd}~ww|d krtnt|}t||D]3\}}|d}| || |kr|d}|}|j j| ||ddd|dd}|j |d| d qdS)ayTrain pipeline Parameters ---------- protocol_name : `str` Name of pyannote.database protocol to use. subset : `str`, optional Use this subset for training. Defaults to 'development'. pretrained : Path, optional Use parameters in "pretrained" training directory to bootstrap the optimization process. In practice this will simply run a first trial with this set of parameters. n_iterations : `int`, optional Number of iterations. Defaults to 1. sampler : `str`, optional Choose sampler between RandomSampler and TPESampler pruner : `str`, optional Choose between MedianPruner or SuccessiveHalvingPruner. average_case : `bool`, optional Optimise for average case. Defaults to False (i.e. worst case). rprotocolrWTrexist_okr%defaulttrials.journal)db study_namerZr[r\minimizer rtrialr)unitpositionleavezFirst trial in progressr"moder#Nr() warm_start show_progressloss)r(rpz Best trial: dg%)desc) r TRAIN_DIRr9rmkdirr get_protocolrDr r get_directionrset_descriptionupdater:r;r<r=listfilterrGgetattr tune_iter best_loss ValueErrornpinf itertoolscountrangezip dump_params)rJrVrWrXrYrZr[r\rr^re optimizer directionr progress_barpre_params_ymlrL pre_paramsrninputs iterationsrrPrr.statusrp best_paramsrtrrr trainsn       zExperiment.trainc Cst|jj|j||d}d}t|j|d|d}z|j}Wnty4}z tdWYd}~dSd}~ww|j }tdd|d d t j |d d } t| dS) a Print current best pipeline Parameters ---------- protocol_name : `str` Name of pyannote.database protocol used for training. subset : `str`, optional Subset used for training. Defaults to 'development'. r]rbrc)rdrez4Still waiting for at least one iteration to succeed.NzLoss = rqrrz&% with the following hyper-parameters:F)default_flow_style) rrur9rr rrrprintrr;dump) rJrVrWrrerrrPrcontentrrr bestXs,    zExperiment.besttest output_dir use_filterc Cstj||jd}z|j}Wnty#}zd}WYd}~nd}~ww|jddd|r<||d|d|jj}n||d|d|jj}t|ddS} t t ||} |rat |j | } d |d |d } t | | d d D]*} || } |j| | | dd}|durd}|durqqt| }||| |d}qqWdn1swY|jd}|r||||durd|d}t|dS|r||d|d}n ||d|d}t|d} | t|WddS1swYdS)zApply current best pipeline Parameters ---------- protocol_name : `str` Name of pyannote.database protocol to process. subset : `str`, optional Subset to process. Defaults to 'test' raNTr_.z _INCOMPLETE.wrlz Processing z ()file)iterablertri annotation)uemlatestzWFor some (possibly good) reason, the output of this pipeline could not be evaluated on z_INCOMPLETE.evalz.eval)rrwrDr get_metricNotImplementedErrorrv write_formatr:r{r}r|rGrwriter?r parentexistsunlink symlink_torstr)rJrVrrWrr^metricrP output_extrLfilesrt current_fileoutput referencer_rmsg output_evalrrr apply}sf      "zExperiment.apply)F)rUNr NNF)rU)rF)__name__ __module__ __qualname____doc__r8ru APPLY_DIR classmethodrrFr!r7rrintrrr __classcell__rrrSr r sR H b)r c Csttdd}|ddD]}t|q |d}|d}|drp|dur'd}|d r.d }nt|d }|d }|d }|d}|rLt|jdd}|d}t|d} | jdd} t | dd} | j |||||||d|dr|durzd}t|d} | jdd} t | dd} | j ||d|dr|durd}|d} t|d} | jdd} t j | dd} t| j j| | jdd} | j|| || ddSdS)NzTunable pipelines)versionz --registry,zz--subsetrrUz --foreverrgz --iterationsz --samplerz--prunerz --pretrainedT)strictz--average-casezr)rWrYrXrZr[r\rF)rWrrz --use-filterz z %Y%m%d-%H%M%S)rdate)rWr)rrsplitr load_databaserr expanduserresolver rrr!rr9rstrftimer) argumentsr)rVrWrrZr[rXr\r experimentrrrrrr mainsh           r)rros.pathr;numpyrtypingrpathlibrrrrrpyannote.databaserrr pyannote.core.utils.helperr rr r rrrrr s(Z           J