o Si|0@sdZddlZddlmZddlmZmZmZmZddl m Z ddl m Z m Z m Z mZmZmZmZmZddlmZddlmZmZgd Z d d edeeeeefdeedefddZ d!d edededefddZ  d"dedeedeeeeefdedeeeeeeee efffff ddZddZ   d#dedeedededeeeeee effff ddZ dS)$a This recipe provides functionality for downloading and preparing the fleurs corpus. The data is hosted on huggingface and to enable more control of the download format, we use the streaming download interface and save each audio file as it is streamed. The download can take quite some time. The fleurs corpus consist of data in 102 languages spoken by multiple speakers. There is about 10 hrs of trainign data in each language with smaller accompanying dev and test sets. Full details can be found in @inproceedings{conneau2023fleurs, title={Fleurs: Few-shot learning evaluation of universal representations of speech}, author={Conneau, Alexis and Ma, Min and Khanuja, Simran and Zhang, Yu and Axelrod, Vera and Dalmia, Siddharth and Riesa, Jason and Rivera, Clara and Bapna, Ankur}, booktitle={2022 IEEE Spoken Language Technology Workshop (SLT)}, pages={798--805}, year={2023}, organization={IEEE} } N)Path)DictOptionalSequenceUnion)tqdm) Recording RecordingSetSupervisionSegmentSupervisionSetaudio fix_manifests"get_ffmpeg_torchaudio_info_enabled"set_ffmpeg_torchaudio_info_enabled) parallel_map)Pathlikeis_module_available)faf_zaam_etar_egas_inast_esaz_azbe_bybg_bgbn_inbs_baca_esceb_phckb_iq cmn_hans_cncs_czcy_gbda_dkde_deel_gren_uses_419et_eefa_irff_snfi_fifil_phfr_frga_iegl_esgu_inha_nghe_ilhi_inhr_hrhu_huhy_amid_idig_ngis_isit_itja_jpjv_idka_gekam_kekea_cvkk_kzkm_khkn_inko_krky_kglb_lulg_ugln_cdlo_lalt_ltluo_kelv_lvmi_nzmk_mkml_inmn_mnmr_inms_mymt_mtmy_mmnb_none_npnl_nlnso_zany_mwoc_from_etor_inpa_inpl_plps_afpt_brro_roru_rusd_insk_sksl_sisn_zwso_sosr_rssv_sesw_keta_inte_intg_tjth_thtr_truk_uaumb_aour_pkuz_uzvi_vnwo_snxh_zayo_ng yue_hant_hkzu_za.allF target_dir languagesforce_downloadreturncCs~t|}|d}|d}|jdddt|tr|dks"|ddkr$t}t|tr,|g}t|D] }||}t|||q0|S)a Download the specified fleurs datasets. :param target_dir: The path to which the corpus will be downloaded. :type target_dir: Pathlike :param languages: Optional list of str or str specifying which languages to download. The str specifier for a language has the ISOCODE_COUNTRYCODE format, and is all lower case. By default this is set to "all", which will download the entire set of languages. :type languages: Optional[Union[str, Sequence[str]]] :param force_download: Specifies whether to overwrite an existing archive. :type force_download: bool :return: The root path of the downloaded data :rtype: Path fleursmetadataTparentsexist_okrzr)rmkdir isinstancestrDEFAULT_LANGUAGESrdownload_single_fleurs_language)r{r|r} corpus_dir metadata_dirlanglang_dirrI/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/recipes/fleurs.pydownload_fleurss   rlanguagec Cstdstdddlm}dd}t|}|jdd|}|jddd |jddd |d |d }|rB|sBt d |St gd D]}|d||d|d}g} |dkr[dn|} || } | jddd t |d|d| dD]R} t | dt| ddj | dd| ddt| dt| ddj | d| ddd| ddt| d | d!d"krd#nd$g} | | qtt|| d%d&}| D] }td'||d(qWd)n1swYqH||S)*a Download a single fleurs language :param target_dir: The path to which one langauge will be downloaded :type target_dir: Pathlike :param language: The code for the specified language :type language: str :param force_download: Specifies whether to overwrite an existing archive. :type force_download: bool :return: The path to the downloaded data for the specified language :rtype: Path datasetsz[The huggingface datasets package is not installed. Please install ...(pip install datasets)r) load_datasetcSs|S)Nr)xrrr _identitysz2download_single_fleurs_language.._identityrTrry _completedz5Skipping dowload because {completed_detector} exists.)train validationtestz google/fleurs) cache_dir streamingsplitrdevzDownloading data from -desc/r patharray sampling_rateidraw_transcription transcription |z | num_samplesgenderFEMALEMALE.tsvw )fileN)r ImportErrorrrrrris_filelogginginforr save_audionamerjoinrappendopenprinttouch)r{rr}rrrcompleted_detectorrrrosplit split_dirdata metadata_fmdrrrrsb        rrr output_dirnum_jobsc Cspt|tr t|}|sJd|t|trt|}|jddddt}t|tr4|dkr4|g}nt|ts>t|trF|ddkrF|}i}|D]/}||}|sbt d|d|d qJ||}|jddddt ||||d ||<qJ|d ur|D]5} d D]0} || | d  || d| d| d|| | d || d| d| dqq|S)a  Prepares the manifest for all of the FLEURS languages requested. :param corpus_dir: Path to the root where the FLEURS data are stored. :type corpus_dir: Pathlike, :param output_dir: The directory where the .jsonl.gz manifests will be written. :type output_dir: Pathlike, :param langauges: str or str sequence specifying the languages to prepare. The str 'all' prepares all 102 languages. :return: The manifest :rtype: Dict[str, Dict[str, Union[RecordingSet, Supervisions]]]] zNo such directory: iT)moderrrzrz Skipping z. No directory z found.)rrNrrr supervisionszfleurs-_supervisions_z .jsonl.gz recordings _recordings_) rrris_dirrrlisttuplerrprepare_single_fleurs_languageto_file) rrr|r langs_list manifestsrcorpus_dir_langoutput_dir_langldsetrrrprepare_fleurssJ      rcCstj|t|jdS)N recording_id)r from_filerstem)rrrr_make_recordingNsrcsttr tgggd}gggd}dD]tddi}tjddjd_}|D]T}|d} | \} } } } }}}| |vrQd|| <|| d7<t| j} | t | d || d | | d t t |d d d| || d || |d | id q7Wdn1swYqdD]'t ttfdd|D|dd|ddD] }| |qqi}dD]t|}t|}t||\}}||d|<q|S)a Prepares manifests using a single FLEURS language. :param corpus_dir: Path to the root where the FLEURS data are stored. :type corpus_dir: Pathlike, :param output_dir: The directory where the .jsonl.gz manifests will be written. :type output_dir: Pathlike, :param langauge: str specifying the language to prepare. :return: The manifest :rtype: Dict[str, Dict[str, Union[RecordingSet, Supervisions]]]] rz Preparing z ...rrrrr_gi>raw_text) rrstartdurationchanneltextrspeakerrcustomNc3s&|]}d|jdVqdS)rz.wavNr).0srrrr s  z1prepare_single_fleurs_language..)rzMaking recordings from rr)rr)rrrrrrrstriprrr roundintrrrr from_segmentsr from_recordingsr )rrrrrr prompt_idsrrvals prompt_idfnamerrrnsamplesrrecorsupsrecosrrrrRsn        r)ryrzF)F)Nrzr)Nrr)!__doc__rpathlibrtypingrrrrrlhotserr r r r r rrlhotse.parallelr lhotse.utilsrrrrboolrrrrrrrrrrsx  (  k / M" ?