o Nig+@sdZddlmZddlmZddlmZddlZddlZddlZddl m m Z ddl mZGdddejjZdd ZGd d d ejjZd d ZddZddZddZdS)$TED-LIUM speech recognition dataset.)absolute_import)division)print_functionNcs(eZdZdZejjfddZZS)TedliumReleaseConfigz4BuilderConfig for a release of the TED-LIUM dataset.c s>tt|jddtjdi|||_||_||_||_ dS)Nversionz1.0.1) superr__init__tfdscoreVersionurl download_url split_pathscitation)selfrrrrkwargs __class__rU/home/ubuntu/.local/lib/python3.10/site-packages/tensorflow_datasets/audio/tedlium.pyr #s  zTedliumReleaseConfig.__init__) __name__ __module__ __qualname____doc__r r disallow_positional_argsr __classcell__rrrrr src Cstdddddtjjtjddftjjtjddftjjtjdd fgd }td d d ddtjjtjddftjjtjddftjjtjdd fgd }tddddtj j dtj j j dtjjtjdddftjjtjddd ftjjtjddfgd }|||gS)zCCreates builder configs for all supported Tedlium dataset releases.release1aB The TED-LIUM corpus is English-language TED talks, with transcriptions, sampled at 16kHz. It contains about 118 hours of speech. This is the TED-LIUM corpus release 1, licensed under Creative Commons BY-NC-ND 3.0 (http://creativecommons.org/licenses/by-nc-nd/3.0/deed.en). ab @inproceedings{rousseau2012tedlium, title={TED-LIUM: an Automatic Speech Recognition dedicated corpus}, author={Rousseau, Anthony and Del{\'e}glise, Paul and Est{\`e}ve, Yannick}, booktitle={Conference on Language Resources and Evaluation (LREC)}, pages={125--129}, year={2012} } zhttps://www.openslr.org/7/z:http://www.openslr.org/resources/7/TEDLIUM_release1.tar.gzTEDLIUM_release1traindevtest)name descriptionrrrrrelease2a This is the TED-LIUM corpus release 2, licensed under Creative Commons BY-NC-ND 3.0 (http://creativecommons.org/licenses/by-nc-nd/3.0/deed.en). All talks and text are property of TED Conferences LLC. The TED-LIUM corpus was made from audio talks and their transcriptions available on the TED website. We have prepared and filtered these data in order to train acoustic models to participate to the International Workshop on Spoken Language Translation 2011 (the LIUM English/French SLT system reached the first rank in the SLT task). Contains 1495 talks and transcripts. aj @inproceedings{rousseau2014tedlium2, title={Enhancing the {TED-LIUM} Corpus with Selected Data for Language Modeling and More {TED} Talks}, author={Rousseau, Anthony and Del{\'e}glise, Paul and Est{\`e}ve, Yannick}, booktitle={Conference on Language Resources and Evaluation (LREC)}, year={2014} } zhttps://www.openslr.org/19/z;http://www.openslr.org/resources/19/TEDLIUM_release2.tar.gzTEDLIUM_release2release3a This is the TED-LIUM corpus release 3, licensed under Creative Commons BY-NC-ND 3.0. All talks and text are property of TED Conferences LLC. This new TED-LIUM release was made through a collaboration between the Ubiqus company and the LIUM (University of Le Mans, France) Contents: - 2351 audio talks in NIST sphere format (SPH), including talks from TED-LIUM 2: be careful, same talks but not same audio files (only these audio file must be used with the TED-LIUM 3 STM files) - 452 hours of audio - 2351 aligned automatic transcripts in STM format - TEDLIUM 2 dev and test data: 19 TED talks in SPH format with corresponding manual transcriptions (cf. 'legacy' distribution below). - Dictionary with pronunciations (159848 entries), same file as the one included in TED-LIUM 2 - Selected monolingual data for language modeling from WMT12 publicly available corpora: these files come from the TED-LIUM 2 release, but have been modified to get a tokenization more relevant for English language Two corpus distributions: - the legacy one, on which the dev and test datasets are the same as in TED-LIUM 2 (and TED-LIUM 1). - the 'speaker adaptation' one, especially designed for experiments on speaker adaptation. a @inproceedings{hernandez2018tedlium3, title={TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation}, author={Hernandez, Fran{\c{c}}ois and Nguyen, Vincent and Ghannay, Sahar and Tomashenko, Natalia and Est{\`e}ve, Yannick}, booktitle={International Conference on Speech and Computer}, pages={198--208}, year={2018}, organization={Springer} } zhttps://www.openslr.org/51/z9http://www.openslr.org/resources/51/TEDLIUM_release-3.tgz)rextract_methodzTEDLIUM_release-3legacydata) rr SplitTRAINospathjoin VALIDATIONTESTdownloadResource ExtractMethod TAR_STREAM)rr$r&rrr_make_builder_configs/sb   !  =r5c@s.eZdZdZeZddZddZddZdS) Tedliumrc Csftjj||jjtjtjjddtjt j tjj gddt j dd|jj |jj tjjdddS)Ni>) sample_rate)unknownfemalemale)namesspeechtext speaker_idgenderid)r=r>)builderr#featuressupervised_keyshomepagermetadata)r r DatasetInfobuilder_configr#rC FeaturesDictAudioTexttfstring ClassLabelrr MetadataDict)rrrr_infos   z Tedlium._infocCsP||jj}g}|jjD]\}}dtj||i}|tj j ||dq |S)N directory)r" gen_kwargs) download_and_extractrHrrr,r-r.appendr r SplitGenerator)r dl_manager extracted_dirsplitssplitr-rrrr_split_generatorsszTedlium._split_generatorscCs<tjjj}tjjtj |dd}|| |B| t BS)Nstmz*stm)r r lazy_imports apache_beamrLiogfileglobr,r-r.CreateFlatMap _generate_examples_from_stm_file)rpipelinerQbeam stm_filesrrr_build_pcollections zTedlium._build_pcollectionN) rrrrr5BUILDER_CONFIGSrPrZrgrrrrr6s  r6c cstj|}tjtj|d}tjj|Q}|D]E}|}| dd\}}}}} } } t | } d|} t tj|| t |t |t | } d||| | g}| | |t| |d}||fVqWddS1smwYdS)z+Generate examples from a TED-LIUM stm file.sph z%s.sph-r<N)r,r-dirnamer.rLr^r_GFilestriprY_maybe_trim_suffix_extract_audio_segmentintfloat _parse_gender)stm_pathstm_dirsph_dirflinefnchannelspeakerstartendlabel transcript audio_filesampleskeyexamplerrrrcs.  "rccCsB|dd}|d}t|dkr|d}|ds|d|7}|S)Nrjr()rsplitlen startswith)rrXsuffixrrrrps    rpcCsTtd|ddd}|sd}|S|dkrd}|S|dkr"d}|S|dkr(d}|S) z-Parse gender string from STM "