o Si@sdZddlZddlZddlmZddlmZmZddlm Z ddl m Z m Z ddl mZddlmZmZdd lmZmZdd lmZmZ dd ed dfddZded efddZ  ddedededed eeeeee effff ddZdS)uU Description taken from official website: https://datasets.kensho.com/datasets/spgispeech SPGISpeech consists of 5,000 hours of recorded company earnings calls and their respective transcriptions. The original calls were split into slices ranging from 5 to 15 seconds in length to allow easy training for speech recognition systems. Calls represent a broad cross-section of international business English; SPGISpeech contains approximately 50,000 speakers, one of the largest numbers of any speech corpus, and offers a variety of L1 and L2 English accents. The format of each WAV file is single channel, 16kHz, 16 bit audio. Transcription text represents the output of several stages of manual post-processing. As such, the text contains polished English orthography following a detailed style guide, including proper casing, punctuation, and denormalized non-standard words such as numbers and acronyms, making SPGISpeech suited for training fully formatted end-to-end models. Official reference: O’Neill, P.K., Lavrukhin, V., Majumdar, S., Noroozi, V., Zhang, Y., Kuchaiev, O., Balam, J., Dovzhenko, Y., Freyberg, K., Shulman, M.D., Ginsburg, B., Watanabe, S., & Kucsko, G. (2021). SPGISpeech: 5, 000 hours of transcribed financial audio for fully formatted end-to-end speech recognition. ArXiv, abs/2104.02014. ArXiv link: https://arxiv.org/abs/2104.02014 N)Path)DictUnion)tqdm) Recording RecordingSet) parallel_map)manifests_existread_manifests_if_cached)SupervisionSegmentSupervisionSet)PathlikeSeconds. target_dirreturncCstddS)z Download and untar the dataset. NOTE: This function just returns with a message since SPGISpeech is not available for direct download. :param target_dir: Pathlike, the path of the dir to storage the dataset. zSPGISpeech is not available for direct download. Please fill out the form at https://datasets.kensho.com/datasets/spgispeech to download the corpus.N)logginginfo)rrM/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/recipes/spgispeech.pydownload_spgispeech&s rtextcCs"|tddtj}|}|S)N) translatestr maketransstring punctuationlower)rrrr normalize7srT corpus_dir output_dirnormalize_textnum_jobsc CsNt|}|sJd||dr|n|d}ddg}i}t|}|jdddt||dddd}|D]}td |t||ddd rStd |d q7i}d tdtfddat |d|d&} t t t|| d|dddD] } | j|| j<| | qzWdn1swYt |d|de} t||ddK} t| t | ddD]8} | d}|dddd d!}|d"}|rt|}|dd}t||||d||d#d$}| |qWdn1swYWdn 1s wYt | jt| jd%||<q7td&|S)'a Returns the manifests which consist of the Recordings and Supervisions. When all the manifests are available in the ``output_dir``, it will simply read and return them. :param corpus_dir: Pathlike, the path of the data dir. :param output_dir: Pathlike, the path where to write the manifests. :param normalize_text: Bool, if True, normalize the text (similar to ESPNet recipe). :param num_jobs: int, the number of jobs to use for parallel processing. :return: a Dict whose key is the dataset part, and the value is Dicts with the keys 'audio' and 'supervisions'. .. note:: Unlike other recipes, output_dir is not Optional here because we write the manifests to the output directory while processing to avoid OOM issues, since it is a large dataset. .. caution:: The `normalize_text` option removes all punctuation and converts all upper case to lower case. This includes removing possibly important punctuations such as dashes and apostrophes. zNo such directory: train spgispeechvalT)parentsexist_okzjsonl.gz) dataset_partsr"prefixsuffixlazyzProcessing SPGISpeech subset: )partr"r+r,zSPGISpeech subset: z already prepared - skipping.prcSs"tj||jjd|jd}|S)N_) recording_id)r from_fileparentstem)r/rrrraudio_read_workerzsr6spgispeech_recordings_z .jsonl.gzz*.wav)r$z Processing SPGISpeech recordings)descNspgispeech_supervisions_z.csvr5zProcessing utterances|r/r0z.wavrEnglish)idr1rspeakerstartdurationlanguage) recordings supervisionszManifests are lazily materialized. You may want to call `lhotse.qa.fix_manifests()` to ensure that all supervisions fall within the corresponding recordings.)ris_dirmkdirr rrr rr6r open_writerrrrglobrAr>writer opennextstripsplitreplacerr from_jsonl_lazypathwarning)r!r"r#r$ audio_dirr* manifestsr. durations rec_writer recording sup_writerflinepartsr1rspkidsegmentrrrprepare_spgispeech?s       r])r)Tr )__doc__rrpathlibrtypingrr tqdm.autor lhotse.audiorrlhotse.parallelrlhotse.recipes.utilsr r lhotse.supervisionr r lhotse.utilsr rrrrboolintr]rrrrs>