o 8wiZB@sdZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl Z ddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZd d lmZGdddZddZdS)a+ Pipeline Usage: pyannote-pipeline train [options] [(--forever | --iterations=)] pyannote-pipeline best [options] pyannote-pipeline apply [options] pyannote-pipeline -h | --help pyannote-pipeline --version Common options: Experimental protocol (e.g. "Etape.SpeakerDiarization.TV") --registry= Path to, comma-separated, database configuration files. [default: ~/.pyannote/db.yml] --subset= Set subset. Defaults to 'development' in "train" mode, and to 'test' in "apply" mode. "train" mode: Set experiment root directory. This script expects a configuration file called "config.yml" to live in this directory. See "Configuration file" section below for more details. --iterations= Number of iterations. [default: 1] --forever Iterate forever. --sampler= Choose sampler between RandomSampler or TPESampler [default: TPESampler]. --pruner= Choose pruner between MedianPruner or SuccessiveHalvingPruner. Defaults to no pruning. --pretrained= Use parameters in existing training directory to bootstrap the optimization process. In practice, this will simply run a first trial with this set of parameters. --average-case Optimize for average case instead of worst case. "apply" mode: Path to the directory containing trained hyper- parameters (i.e. the output of "train" mode). Configuration file: The configuration of each experiment is described in a file called /config.yml that describes the pipeline. ................... /config.yml ................... pipeline: name: Yin2018 params: sad: tutorials/pipeline/sad scd: tutorials/pipeline/scd emb: tutorials/pipeline/emb metric: angular # preprocessors can be used to automatically add keys into # each (dict) file obtained from pyannote.database protocols. preprocessors: audio: ~/.pyannote/db.yml # load template from YAML file video: ~/videos/{uri}.mp4 # define template directly # one can freeze some hyper-parameters if needed (e.g. when # only part of the pipeline needs to be updated) freeze: speech_turn_segmentation: speech_activity_detection: onset: 0.5 offset: 0.5 # pyannote.audio pipelines will run on CPU by default. # use `device` key to send it to GPU. device: cuda ................................................................... "train" mode: Tune the pipeline hyper-parameters /..yml "best" mode: Display current best loss and corresponding hyper-paramters. "apply" mode Apply the pipeline (with best set of hyper-parameters) N)Optional)Path)docopt)tqdm)datetime) FileFinder)registry) get_annotated)get_class_by_name) OptimizercseZdZdZdZdZdZed dede ddfd d Z d d ede ffd d Z      d!de de e de edede e de e de fddZd"de de fddZ d#de dede e fddZZS)$ ExperimentzPipeline experiment Parameters ---------- experiment_dir : `Path` Experiment root directory. training : `bool`, optional Switch to training mode z{experiment_dir}/config.ymlz*{experiment_dir}/train/{protocol}.{subset}z{train_dir}/apply/{date}F train_dirtrainingreturncCsB|jd}|||d}|d}ttj||_|j||S)a6Load pipeline from train directory Parameters ---------- train_dir : `Path` Path to train directory training : `bool`, optional Switch to training mode. Returns ------- xp : `Experiment` Pipeline experiment. r r params.yml) parentsr fromtimestampospathgetmtimemtime_ pipeline_ load_params)clsrrexperiment_dirxp params_ymlrY/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/pyannote/pipeline/experiment.pyfrom_train_dirs   zExperiment.from_train_dirrc st||_|jj|jd}t|d}tj|tjd|_ Wdn1s)wYi}|j di D]@\}}t |t rXt|ddd}|di| di||<q9z t|d ||<Wq9tyy} z |} | ||<WYd} ~ q9d} ~ ww||_|j d d} t| d d}|di|j d  di|_d |j vr|j d } |j| d |j vrddl} | |j d }|j|dSdS)N)rrLoader preprocessorsnamezpyannote.pipeline)default_module_nameparams) database_ymlpipelinezpyannote.pipeline.blocksfreezedevicerr)super__init__r CONFIG_YMLformatopenyamlload SafeLoaderconfig_getitems isinstancedictr rFileNotFoundErrorpreprocessors_rr+torchr,to)selfrr config_ymlfpr%key preprocessorKlassetemplate pipeline_namer(r<r, __class__rr r.sF       zExperiment.__init__ developmentNr protocol_namesubset pretrained n_iterationssamplerpruner average_casec Cst|jj|j||d}|jdddtj||jd} d} t|j |d| |||d} |j dkr3d nd } |d } t d d dd}| d| d |rs|d }t|dd}tj|tjd}Wdn1siwY|d}nd}tt| |}| j||dd}z| j}Wnty}z | tj}WYd}~nd}~ww|d krtnt|}t||D]3\}}|d}| || |kr|d}|}|j j| ||ddd|dd}|j |d| d qdS)ayTrain pipeline Parameters ---------- protocol_name : `str` Name of pyannote.database protocol to use. subset : `str`, optional Use this subset for training. Defaults to 'development'. pretrained : Path, optional Use parameters in "pretrained" training directory to bootstrap the optimization process. In practice this will simply run a first trial with this set of parameters. n_iterations : `int`, optional Number of iterations. Defaults to 1. sampler : `str`, optional Choose sampler between RandomSampler and TPESampler pruner : `str`, optional Choose between MedianPruner or SuccessiveHalvingPruner. average_case : `bool`, optional Optimise for average case. Defaults to False (i.e. worst case). rprotocolrKTrexist_okr%defaulttrials.journal)db study_namerNrOrPminimizer rtrialr)unitpositionleavezFirst trial in progressr"moder#Nr() warm_start show_progressloss)r(rdz Best trial: dg%)desc)r TRAIN_DIRr0rmkdirr get_protocolr;r r get_directionrset_descriptionupdater1r2r3r4listgetattr tune_iter best_loss ValueErrornpinf itertoolscountrangezip dump_params)r>rJrKrLrMrNrOrPrrRrY optimizer directionr progress_barpre_params_ymlr@ pre_paramsrbinputs iterationsrrrDrwistatusrd best_paramsrhrrr trainsn       zExperiment.trainc Cst|jj|j||d}d}t|j|d|d}z|j}Wnty4}z tdWYd}~dSd}~ww|j }tdd|d d t j |d d } t| dS) a Print current best pipeline Parameters ---------- protocol_name : `str` Name of pyannote.database protocol used for training. subset : `str`, optional Subset used for training. Defaults to 'development'. rQrVrW)rXrYz4Still waiting for at least one iteration to succeed.NzLoss = rerfz&% with the following hyper-parameters:F)default_flow_style) rrir0rr rrrrsprintrr2dump) r>rJrKrrYr{rrrDrcontentrrr bestDs,    zExperiment.besttest output_dirc Cstj||jd}z|j}Wnty#}zd}WYd}~nd}~ww|jddd||d|d|jj}t|ddK}t t ||} d|d |d } t | | d d D]*} || } |j || | d d} | durrd}|durwqXt| }|| | |d}qXWdn1swY|jd}|r||||durd|d}t|dS||d|d}t|d}| t|WddS1swYdS)zApply current best pipeline Parameters ---------- protocol_name : `str` Name of pyannote.database protocol to process. subset : `str`, optional Subset to process. Defaults to 'test' rUNTrS.wr`z Processing z ()file)iterablerhr] annotation)uemlatestzWFor some (possibly good) reason, the output of this pipeline could not be evaluated on z.eval)rrkr;r get_metricNotImplementedErrorrj write_formatr1rorprwriter6r parentexistsunlink symlink_torstr)r>rJrrKrRmetricrD output_extr@filesrh current_fileoutput referencer_rmsg output_evalrrr applyisT     "zExperiment.apply)F)rINr NNF)rI)r)__name__ __module__ __qualname____doc__r/ri APPLY_DIR classmethodrboolr!r.rrintrrr __classcell__rrrGr r sL = a&r c Csttdd}|ddD]}t|q |d}|d}|drp|dur'd}|d r.d }nt|d }|d }|d }|d}|rLt|jdd}|d}t|d} | jdd} t | dd} | j |||||||d|dr|durzd}t|d} | jdd} t | dd} | j ||d|dr|durd}t|d} | jdd} t j | dd} t| j j| | jdd} | j|| |ddSdS)NzTunable pipelines)versionz --registry,zz--subsetrrIz --foreverr[z --iterationsz --samplerz--prunerz --pretrainedT)strictz--average-casezr)rKrMrLrNrOrPrF)rKrrz z %Y%m%d-%H%M%S)rdate)rrsplitr load_databaserr expanduserresolver rrr!rr0rstrftimer) argumentsr)rJrKrrNrOrLrPr experimentrrrrr mainsb          r)rros.pathr2numpyrttypingrpathlibrrrvrrpyannote.databaserrr pyannote.core.utils.helperr r{r r rrrrr s(R           -