o SiP @sdZddlZddlZddlZddlZddlZddlmZddlm Z m Z m Z ddl m Z mZddlmZddlmZddlmZmZmZmZd Z dd edededefddZ ddede ede ee ee eeffffddZdede fddZdS)aX The Edinburgh International Accents of English Corpus Citation Sanabria, Ramon; Markl, Nina; Carmantini, Andrea; Klejch, Ondrej; Bell, Peter; Bogoychev, Nikolay. (2023). The Edinburgh International Accents of English Corpus, [dataset]. University of Edinburgh. School of Informatics. The Institute for Language, Cognition and Computation. The Centre for Speech Technology Research. https://doi.org/10.7488/ds/3832. Description English is the most widely spoken language in the world, used daily by millions of people as a first or second language in many different contexts. As a result, there are many varieties of English. Although the great many advances in English automatic speech recognition (ASR) over the past decades, results are usually reported based on test datasets which fail to represent the diversity of English as spoken today around the globe. We present the first release of The Edinburgh International Accents of English Corpus (EdAcc). This dataset attempts to better represent the wide diversity of English, encompassing almost 40 hours of dyadic video call conversations between friends. Unlike other datasets, EdAcc includes a wide range of first and second-language varieties of English and a linguistic background profile of each speaker. Results on latest public, and commercial models show that EdAcc highlights shortcomings of current English ASR models. The best performing model, trained on 680 thousand hours of transcribed data, obtains an average of 19.7% WER -- in contrast to the the 2.7% WER obtained when evaluated on US English clean read speech. Across all models, we observe a drop in performance on Jamaican, Indonesian, Nigerian, and Kenyan English speakers. Recordings, linguistic backgrounds, data statement, and evaluation scripts are released on our website under CC-BY-SA. Source: https://datashare.ed.ac.uk/handle/10283/4836 N)Path)DictOptionalUnion) fix_manifests$validate_recordings_and_supervisions) RecordingSet)SupervisionSet)Pathlikeis_module_availableresumable_download safe_extracti}.F$https://datashare.ed.ac.uk/download/ target_dirforce_downloadbase_urlreturnc Csd}t|}|d}|jddd|d}|r$td|d|S||}t|d|||d tj|dd t |}|j |d Wd n1sQwYd }t || } t | |Wd n1sowY||S)ad Download and extract the EDACC dataset. :param target_dir: Pathlike, the path of the dir to store the dataset. :param force_download: Bool, if True, download the data even if it exists. :param base_url: str, the url of the website used to fetch the archive from. :return: the path to downloaded and extracted directory with data. zDS_10283_8983.zipedaccTparentsexist_okz .completedz Skipping EDACC download because z exists./)filenamer) ignore_errors)pathNzedacc_v1.0.tar.gz)rmkdiris_filelogginginfor shutilrmtreezipfileZipFile extractalltarfileopenr touch) rrr archive_name corpus_dircompleted_detector archive_pathziptar_nametarr/H/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/recipes/edacc.pydownload_edacc's0    r1r) output_dirc Cs>ddlm}tdstdt|d}|d}|dur't|}|jddd i}t|d }t|d }|D]}|j d ksEJd ||j t ksLJq7dd|D}dD]} || } t | d} t dd| D} Wdn1svwYt | dd} t|D]\} }| | vrt|| dqWdn1swY|| t d\}}}t | d} t ttj| }Wdn1swY|D]}d|_|j|v|_||jD] \}}t|||qqt||\}}t||||d|| <|dur||d| d||d| dqV|S)aP Returns the manifests which consist of the Recordings and Supervisions. :param corpus_dir: a path to the unzipped EDACC directory (has ``edacc_v1.0`` inside). :param output_dir: an optional path where to write the manifests. :return: a dict with structure ``{"dev|test": {"recordings|supervisions": }}`` r)load_kaldi_data_dirpandasz/Please install pandas via 'pip install pandas'.z edacc_v1.0dataNTrzlinguistic_background.csvz*.wavz$Unexpected multi-channel recording: cSs(i|]}|j|jd|jdjqS) r)idsourcessource).0rr/r/r0 rs(z!prepare_edacc..)devtestsegmentscss|] }|dVqdS)r6N)split)r;lr/r/r0 ysz prepare_edacc..zwav.scpw)file) sampling_ratez conv.listEnglish) recordings supervisionsedacc_recordings_z .jsonl.gzedacc_supervisions_) lhotse.kaldir3r ValueErrorrrparse_linguistic_backgroundrfrom_dir num_channelsrF_EDACC_SAMPLING_RATEr&setsorteditemsprintmapstrstriplanguage recording_idis_conversationalspeakersetattrrrto_file)r)r2r3 audio_dir manifestsspk2metarHr<wav_scprAdata_dirf split_rec_idsridrstrrI_ conv_rec_idsskeyvalr/r/r0 prepare_edaccSs^         rmrc srddl}||}|jdddddddd d }d |j|d <d |j|d <dtfddfdd|D}|S)Nrgender ethnicity educationaccentother_languages birth_yearstart_english_year)zWhat is your gender?u!What’s your ethnic background? z'What is your higher level of education?zIHow would you describe your accent in English? (e.g. Italian, Glaswegian)z^Do you speak any second languages? separate them with commas (e.g., Mandarin,Catalan,French )u)What’s your year of birth? (e.g., 1992)z6What year did you start learning English? (e.g., 1999))columnsiageyears_speaking_englishrcSsh|dkrt|r dSt|S|dkr)t|trt|rgSdd|dDSt|tr2|S|S)NrwrrcSsg|]}|qSr/)rX)r;vr/r/r0 sz>parse_linguistic_background..parse..,)mathisnanint isinstancefloatrArWrX)rkrlr/r/r0parses  z*parse_linguistic_background..parsecs*i|]\}dfdddDqS)PARTICIPANT_IDcsi|] }|||qSr/r/)r;m)rrowr/r0r=sz:parse_linguistic_background...)rnrorprqrrrvrwr/)r;rhr)rr0r=s  z/parse_linguistic_background..)r4read_csvrenamersrtriterrows)rpddfrar/rr0rNs&    rN)rFr)N)__doc__rr{r r%r"pathlibrtypingrrrlhotserr lhotse.audiorlhotse.supervisionr lhotse.utilsr r r r rQboolrWr1rmrNr/r/r/r0sD    . C