o NiJ@sdZddlmZddlmZddlmZddlZddlZddlZddlZddl Z ddl m m ZddlmZdZedddZd eZd Zd Zgd Zd dZddZGdddejjZdS)zVoxForge dataset.)absolute_import)division)print_functionNz @article{maclean2018voxforge, title={Voxforge}, author={MacLean, Ken}, journal={Ken MacLean.[Online]. Available: http://www.voxforge.org/home.[Acedido em 2012]}, year={2018} } ia VoxForge is a language classification dataset. It consists of user submitted audio clips submitted to the website. In this release, data from 6 languages is collected - English, Spanish, French, German, Russian, and Italian. Since the website is constantly updated, and for the sake of reproducibility, this release contains only recordings submitted prior to {}. The samples are splitted between train, validation and testing so that samples from each speaker belongs to exactly one split. zhttp://www.voxforge.org/zMhttps://storage.googleapis.com/tfds-data/downloads/voxforge/voxforge_urls.txt)deenesfritrucCst||krtdjt||dtdd|D}td|dkr)td|g}d}|D]\}}|}||7}||t||t||fq/|d d |d d|f|d <|S) aComputes boundary indices for each of the splits in split_probs. Args: split_probs: List of (split_name, prob), e.g. [('train', 0.6), ('dev', 0.2), ('test', 0.2)] n_items: Number of items we want to split. Returns: The item indices of boundaries between different splits. For the above example and n_items=100, these will be [('train', 0, 60), ('dev', 60, 80), ('test', 80, 100)]. z]Not enough items for the splits. There are {splits} splits while there are only {items} items)splitsitemscs|]\}}|VqdSN).0nameprrV/home/ubuntu/.local/lib/python3.10/site-packages/tensorflow_datasets/audio/voxforge.py Mz,_compute_split_boundaries..rg:0yE>z"Probs should sum up to 1. probs={}gr)len ValueErrorformatsumabsappendint) split_probsn_items total_probssplit_boundariessum_prrprevrrr_compute_split_boundaries<s$  "r%cCsttdd|D}tj|}||t|t|}i}|D]\}}} t|| D]} |||| <q+q!t t} |D]\} } || }| | | q<| S)aDSplit items to train/dev/test, so all items in group go into same split. Each group contains all the samples from the same speaker ID. The samples are splitted between train, validation and testing so that samples from each speaker belongs to exactly one split. Args: items_and_groups: Sequence of (item_id, group_id) pairs. split_probs: List of (split_name, prob), e.g. [('train', 0.6), ('dev', 0.2), ('test', 0.2)] split_number: Generated splits should change with split_number. Returns: Dictionary that looks like {split name -> set(ids)}. csrrr)ritem_idgroup_idrrrrnrz-_get_inter_splits_by_group..) sortedsetnprandom RandomStateshuffler%rrange collections defaultdictadd)items_and_groupsr split_numbergroupsrngr"group_id_to_split split_namei_starti_endi split_to_idsr&r'splitrrr_get_inter_splits_by_group^s    r=c@sDeZdZdZejdZe d e Z ddZ ddZdd Zd S) Voxforgez@A Language classification dataset based on the VoxForge website.z1.0.0aD VoxForge requires manual download of the audio archives. The complete list of archives can be found in {}. It can be downloaded using the following command: wget -i voxforge_urls.txt -x Note that downloading and building the dataset locally requires ~100GB disk space (but only ~60GB will be used permanently). c Cs@tjj|ttjtjjdddtjjtdt j ddt t dS)Nwavi>) file_format sample_rate)namesaudiolabel speaker_id)rDrE)builder descriptionfeaturessupervised_keyshomepagecitation) tfdscore DatasetInfo _DESCRIPTIONrI FeaturesDictAudio ClassLabelLABELStfstring _HOMEPAGE _CITATION)selfrrr_infos zVoxforge._infocCsP|dti}g}tjj|d4}|D])}|dddd}tj |j |}tjj |s9t d|||qWdn1sIwY|}g} t||D]\} }tj |\} } | dd} | | | fqWgd }t| |d}tjjtjj|d |d d tjjtjj|d |d d tjjtjj|d|d d gS)zReturns SplitGenerators. urls_list"'z5VoxForge requires manual download. Path {} is missingN-r))traingffffff?) validationg?)testg?r`) file_names dl_manager)r gen_kwargsrarb)download_URLS_LIST_FILErUiogfileGFilestripreplaceospathjoin manual_direxistsAssertionErrorrrzipr<r=rMrNSplitGeneratorSplitTRAIN VALIDATIONTEST)rYrdr[ archive_urlsfline archive_url archive_patharchivesarchives_and_speaker_idsarchive_ archive_namerFrr rrr_split_generatorssX  zVoxforge._split_generatorsc cs|D]Q}tj|\}}|dd}|dd}|||d}||} | D])\} } | ds4q*tj| \} } d||| dtd }|| ||dfVq*qdS) zYields examples.r_rz/Trunkz.wavz{}_{}_{}NrC)rmrnr<index iter_archiveendswithrr)rYrcrdfnamefolderrrF label_idxrErwav_pathwav_objrwav_namekeyrrr_generate_exampless   zVoxforge._generate_examplesN)__name__ __module__ __qualname____doc__rMrNVersionVERSIONtextwrapdedentrrgMANUAL_DOWNLOAD_INSTRUCTIONSrZrrrrrrr>s  ,r>) r __future__rrrr/datetimermrnumpyr*tensorflow.compat.v2compatv2rUtensorflow_datasets.public_api public_apirMrXdate _LAST_DATEr isoformatrPrWrgrTr%r=rNGeneratorBasedBuilderr>rrrrs,       ""