o 2wiY @s dZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z ddl mZmZddlmZmZddlmZmZddlmZmZd Zd ed efd d Zddefdede ede ed efddZdefddZ  ddede eded e eeffddZdS)a About the Earnings 21 dataset: The Earnings 21 dataset ( also referred to as earnings21 ) is a 39-hour corpus of earnings calls containing entity dense speech from nine different financial sectors. This corpus is intended to benchmark automatic speech recognition (ASR) systems in the wild with special attention towards named entity recognition (NER). This dataset has been submitted to Interspeech 2021. The paper describing methods and results can be found on arXiv at https://arxiv.org/pdf/2104.11348.pdf @misc{delrio2021earnings21, title={Earnings-21: A Practical Benchmark for ASR in the Wild}, author={Miguel Del Rio and Natalie Delworth and Ryan Westerman and Michelle Huang and Nishchal Bhandari and Joseph Palakapilly and Quinten McNamara and Joshua Dong and Piotr Zelasko and Miguel Jette}, year={2021}, eprint={2104.11348}, archivePrefix={arXiv}, primaryClass={cs.CL} } N)Path)DictListOptionalTupleUnion) fix_manifests$validate_recordings_and_supervisions) Recording RecordingSet)SupervisionSegmentSupervisionSet)Pathlikeresumable_downloadzIhttps://codeload.github.com/revdotcom/speech-datasets/zip/refs/heads/maintextreturncCs"|tddtj}|}|S)N) translatestr maketransstring punctuationlower)rrV/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/lhotse/recipes/earnings21.py normalize)sr.F target_dirforce_downloadurlcCstdt|}|jddd|d}|d}|d}|r+td|d|St|||d tj|dd t |}| D] }d|vrP|j ||d qCWd n1s[wYt t |d dt |t|d ||S)aGDownload and untar the dataset. :param target_dir: Pathlike, the path of the dir to store the dataset. The extracted files are saved to target_dir/earnings21/ Please note that the github repository contains other additional datasets and using this call, you will be downloading all of them and then throwing them out. :param force_download: Bool, if True, download the tar file no matter whether it exists or not. :param url: str, the url to download the dataset. :return: the path to downloaded and extracted directory with data. zDownloading Earnings21 from github repository is not very efficient way how to obtain the corpus. You will be downloading other data as well.Tparentsexist_ok earnings21zspeech-datasets-main.zipz.lhotse-download.completedz Skipping - z exists.)filenamer) ignore_errors)pathNzspeech-datasets-main)logginginformkdiris_filershutilrmtreezipfileZipFilenamelistextractmovertouch)rrr extracted_dirzip_pathcompleted_detectorzipfrrrdownload_earnings211s4  r8r$cCs`t|"}t}||D]}|d}||dq|WdS1s)wYdS)N|r)openlistreadlinesplitappend)r$r7 transcriptlinerrrparse_nlp_file`s  $rA corpus_dir output_dirnormalize_textc CsLt|}|sJd||durt|}|jddd|d}t|d}t|dks1J|tdd |D}|d d }t|d }t|dksTJt}|D]%} | j } d t | } |rkt | } t | | d|| jdd| d} || qYt|} t|| \}} t|| |dur| |d||d|| fS)ay Returns the manifests which consist of the Recordings and Supervisions. When all the manifests are available in the ``output_dir``, it will simply read and return them. :param corpus_dir: Pathlike, the path of the data dir. The structure is expected to mimic the structure in the github repository, notably the mp3 files will be searched for in [corpus_dir]/media and transcriptions in the directory [corpus_dir]/transcripts/nlp_references :param output_dir: Pathlike, the path where to write the manifests. :param normalize_text: Bool, if True, normalize the text. :return: (recordings, supervisions) pair .. caution:: The `normalize_text` option removes all punctuation and converts all upper case to lower case. This includes removing possibly important punctuations such as dashes and apostrophes. zNo such directory: NTr mediaz*.mp3,css|]}t|VqdS)N)r from_file).0prrr s  z%prepare_earnings21.. transcriptsnlp_referencesz*.nlp grEnglish)id recording_idstartdurationchannellanguagerz$earnings21_supervisions_all.jsonl.gzz"earnings21_recordings_all.jsonl.gz)ris_dirr)r;globlensortr from_recordingsstemjoinrArr rRr>r from_segmentsrr to_file)rBrCrD media_dir audio_files recording_setnlp_dir nlp_filessupervision_segmentsnlp_filerOrssupervision_setrrrprepare_earnings21jsL    rg)NF) __doc__r'r+rr-pathlibrtypingrrrrrlhotserr lhotse.audior r lhotse.supervisionr r lhotse.utilsrr _DEFAULT_URLrrboolr8rArgrrrrsL   /