o Si&@sddlZddlmZddlmZmZmZddlmZm Z m Z m Z m Z ddl mZddlmZddd Z  dd ed eed edeeeeee e ffffddZddZddZddZddZdS)N)Path)DictOptionalUnion) Recording RecordingSetSupervisionSegmentSupervisionSet$validate_recordings_and_supervisions) fix_manifests)PathlikereturncCs tddS)NaV Unfortunately you need to download the data manually due to licensing reason. See info and instructions how to obtain BVCC dataset used for VoiceMOS challange: - https://arxiv.org/abs/2105.02373 - https://nii-yamagishilab.github.io/ecooper-demo/VoiceMOS2022/index.html - https://codalab.lisn.upsaclay.fr/competitions/695)print) target_dirrG/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/recipes/bvcc.py download_bvccsr corpus_dir output_dirnum_jobscs t|}|d}|sJd||dd}|dd|r)s1Jd|d|d}|s=J||d }|sIJ||d }|sUJ||d }|sfJd ||dd} |dd| rzsJd|d| d } | sJ| | d} | sJ| | d } | sJ| | d } | sJ| i}tjd|d}tdtt t t | |t |fdd}t|\}t||d|d<tdtt t t | |t |fdd}t|\}t||d|d<fddt | D}dtdd|Di|d<fddt |  D}dtdd|Di|d <fd!dt |  D}dtd"d|Di|d#<tjd|d}td$tt t t |  |t|fd%d}t|\}t||d|d&<td'tt t t |  |t|fd(d}t|\}t||d|d)<|durt|}|jd*d*d+|D]$\}}|d|d,|d-d.|vr |d.|d/|d-q|S)0Nz phase1-mainzMain track dir is missing DATAsetswavz!Have you run data preparation in ?DEVSETTRAINSETztest.scpz phase1-oodz#Out of domain track dir is missing zunlabeled_mos_list.txtz*.wav)patternrzPreparing main1_devc |jvSNidrec) main1_dev_suprrO zprepare_bvcc..) recordings supervisions main1_devzPreparing main1_traincrrr r")main1_train_suprrr%br& main1_traincg|]}|qSrstrip.0name) main1_wavrr m z prepare_bvcc..r'cs|]}t|VqdSrr from_filer0prrr q  zprepare_bvcc.. main1_testcr,rr-r/ood1_wavrrr3wr4csr5rr6r8rrrr:{r;ood1_unlabeledcr,rr-r/r=rrr3r4csr5rr6r8rrrr:r; ood1_testzPreparing ood1_devcrrr r") ood1_dev_suprrr%r&ood1_devzPreparing ood1_traincrrr r")ood1_train_suprrr%r& ood1_trainT)parentsexist_okbvcc_recordings_z .jsonl.gzr(bvcc_supervisions_)rresolveexistsrfrom_dirlogginginfor from_segmentsgen_supervision_per_uttsortedopen readlinesparse_main_linefilterr r from_recordingsparse_ood_linemkdiritemsto_file)rrr phase1_main main1_sets main1_devp main1_trainp main1_testp phase1_ood ood1_setsood1_unlabeledp ood1_devp ood1_trainp ood1_testp manifests main1_recsmain1_dev_recsmain1_train_recsmain1_test_wavpathsunlabeled_wavpathsood1_test_wavpaths ood1_recs ood1_dev_recsood1_train_recspartdr)r$r*r2rArCr>r prepare_bvccs                           rqc Cs|d\}}}}}|d\}}}} }}} | dvsJ| | dkr%d} n| dkr,d} n| dkr3d } ntd | | d vsBJ| | d k} ||||| | |d fS)am For context see phase1-main/README: TRAINSET and DEVSET contain the individual ratings from each rater, along with some demographic information for the rater. The format is as follows: sysID,uttID,rating,ignore,listenerinfo The listener info is as follows: {}_AGERANGE_LISTENERID_GENDER_[ignore]_[ignore]_HEARINGIMPAIRMENT ,_)MaleFemaleOthersrtMruFrvOzUnsupported value )YesNorz)r!M_F impairmentage)split ValueError) linesysiduttidrating_ignore listenerinforsagerange listenerid listener_mfhaveimpairmentrrrrSs*rSc CsJ|d\}}}}}|d\}}}}}}}|dvsJ|||||dfS)af For context see phase1-ood/README: TRAINSET and DEVSET contain the individual ratings from each rater, along with some demographic information for the rater. (TRAINSET only contains information about the labeled training data, not for the unlabeled samples.) The format is as follows: sysID,uttID,rating,ignore,listenerinfo The listener info is as follows: {}_na_LISTENERID_na_na_na_LISTENERTYPE LISTENERTYPE may take the following values: EE: speech experts EP: paid listeners, native speakers of Chinese (any dialect) ER: voluntary listeners rrrs)EEEPER)r!type)r) rrrrrrrsr listenertyperrrrVs rVccsdg}}|D]&}|}||}|d}||kr)t||EdH||g}}q||qt|dkr?t||EdHdSdS)Nr)r.segment_from_runappendlen)linesr' parse_line prev_uttid prev_supsrrMrrrrrO s   rOc csi}i}d\}}|D]=\}}}} | d} t||| <| || <|dur'|}n ||ks4J|d||dur;|}q ||ksHJ|d|q |dur|dur]t|dkr]t|dks_J|drj|dd}||j} t||d| ddd||ddVdSdS) N)NNr!z vs rz.wav)MOS listeners)r! recording_idstartdurationtextlanguagespeakercustom)popintrendswithrr) infosr'MOSd LISTENERsduttidAsysidArrr listenerdrrrrrrs<  $    r)r N)Nr)rLpathlibrtypingrrrlhotserrrr r lhotse.qar lhotse.utilsr rrstrrqrSrVrOrrrrrs.     $+"