o Ni}2@s dZddlmZddlmZddlmZddlZddlZddlm Z ddl m m Z ddlmZdZdZd Zd Zd Zd Zeed fZdZdZdZdZdZdZdZGdddejj Z!Gddde!Z"Gddde!Z#Gdddejj$Z%Gddde!Z&dd Z'd!d"Z(dS)#z(MNIST, Fashion MNIST, KMNIST and EMNIST.)absolute_import)division)print_functionN)urllibz3https://storage.googleapis.com/cvdf-datasets/mnist/ztrain-images-idx3-ubyte.gzztrain-labels-idx1-ubyte.gzzt10k-images-idx3-ubyte.gzzt10k-labels-idx1-ubyte.gz `'z@article{lecun2010mnist, title={MNIST handwritten digit database}, author={LeCun, Yann and Cortes, Corinna and Burges, CJ}, journal={ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist}, volume={2}, year={2010} } an@article{DBLP:journals/corr/abs-1708-07747, author = {Han Xiao and Kashif Rasul and Roland Vollgraf}, title = {Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms}, journal = {CoRR}, volume = {abs/1708.07747}, year = {2017}, url = {http://arxiv.org/abs/1708.07747}, archivePrefix = {arXiv}, eprint = {1708.07747}, timestamp = {Mon, 13 Aug 2018 16:47:27 +0200}, biburl = {https://dblp.org/rec/bib/journals/corr/abs-1708-07747}, bibsource = {dblp computer science bibliography, https://dblp.org} } ar @online{clanuwat2018deep, author = {Tarin Clanuwat and Mikel Bober-Irizar and Asanobu Kitamoto and Alex Lamb and Kazuaki Yamamoto and David Ha}, title = {Deep Learning for Classical Japanese Literature}, date = {2018-12-03}, year = {2018}, eprintclass = {cs.CV}, eprinttype = {arXiv}, eprint = {cs.CV/1812.01718}, } aF@article{cohen_afshar_tapson_schaik_2017, title={EMNIST: Extending MNIST to handwritten letters}, DOI={10.1109/ijcnn.2017.7966217}, journal={2017 International Joint Conference on Neural Networks (IJCNN)}, author={Cohen, Gregory and Afshar, Saeed and Tapson, Jonathan and Schaik, Andre Van}, year={2017} } c@s8eZdZdZeZejdZ ddZ ddZ ddZ d S) MNISTzMNIST.z3.0.1c Cs:tjj|dtjtjjtdtjjtddddt dS)Nz)The MNIST database of handwritten digits.shape num_classesimagelabelz!http://yann.lecun.com/exdb/mnist/builder descriptionfeaturessupervised_keyshomepagecitation) tfdscore DatasetInfor FeaturesDictImageMNIST_IMAGE_SHAPE ClassLabelMNIST_NUM_CLASSES_MNIST_CITATIONselfr%b/home/ubuntu/.local/lib/python3.10/site-packages/tensorflow_datasets/image_classification/mnist.py_infoes  z MNIST._infocsvttttd}|fdd|D}tjjtj j t t |d|dddtjjtj j t t|d|d ddgS) zReturns SplitGenerators. train_data train_labels test_data test_labelscs"i|] \}}|tjj|qSr%)rparseurljoinURL).0kvr#r%r& |s"z+MNIST._split_generators..r)r* num_examples data_path label_pathname gen_kwargsr+r,)_MNIST_TRAIN_DATA_FILENAME_MNIST_TRAIN_LABELS_FILENAME_MNIST_TEST_DATA_FILENAME_MNIST_TEST_LABELS_FILENAMEdownload_and_extractitemsrrSplitGeneratorSplitTRAINdict_TRAIN_EXAMPLESTEST_TEST_EXAMPLES)r$ dl_manager filenames mnist_filesr%r#r&_split_generatorsrs2zMNIST._split_generatorsc csRt||}t||}tt||}t|D]\}\}} || d} || fVqdS)zGenerate MNIST examples as dicts. Args: num_examples (int): The number of example. data_path (str): Path to the data files label_path (str): Path to the labels Yields: Generator yielding the next examples rN)_extract_mnist_images_extract_mnist_labelslistzip enumerate) r$r5r6r7imageslabelsdataindexrrrecordr%r%r&_generate_exampless   zMNIST._generate_examplesN) __name__ __module__ __qualname____doc__ _MNIST_URLr/rrVersionVERSIONr'rKrVr%r%r%r&r _s  r c@eZdZdZddZdS) FashionMNISTz;http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/c C>tjj|dtjtjjtdtjjgdddddtdS)NzFashion-MNIST is a dataset of Zalando's article images consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes.r ) z T-shirt/topTrouserPulloverDressCoatSandalShirtSneakerBagz Ankle bootnamesrz0https://github.com/zalandoresearch/fashion-mnistr) rrrrrrrr _FASHION_MNIST_CITATIONr#r%r%r&r's  zFashionMNIST._infoNrWrXrYr/r'r%r%r%r&r_s r_c@r^)KMNISTz-http://codh.rois.ac.jp/kmnist/dataset/kmnist/c Cr`)Na,Kuzushiji-MNIST is a drop-in replacement for the MNIST dataset (28x28 grayscale, 70,000 images), provided in the original MNIST format as well as a NumPy format. Since MNIST restricts us to 10 classes, we chose one character to represent each of the 10 rows of Hiragana when creating Kuzushiji-MNIST.r ) okisutsunahamayareworirz+http://codh.rois.ac.jp/kmnist/index.html.enr) rrrrrrrr _K_MNIST_CITATIONr#r%r%r&r's z KMNIST._infoNrlr%r%r%r&rms rmcs(eZdZdZejjfddZZS) EMNISTConfigz BuilderConfig for EMNIST CONFIG.c s:tt|jddtjddi|||_||_||_dS)a?BuilderConfig for EMNIST class number. Args: class_number: There are six different splits provided in this dataset. And have different class numbers. train_examples: number of train examples test_examples: number of test examples **kwargs: keyword arguments forwarded to super. versionz3.0.0z6New split API (https://tensorflow.org/datasets/splits)Nr%) superry__init__rrr\ class_numbertrain_examples test_examples)r$r}r~rkwargs __class__r%r&r|s  zEMNISTConfig.__init__) rWrXrYrZrrdisallow_positional_argsr| __classcell__r%r%rr&rysryc @seZdZdZdZdZedddddd ed d ddd d ed d dddd edddddd edddddd edddddd gZdd Zd!d"Z dS)#EMNISTzEmnist dataset.z:https://www.itl.nist.gov/iaui/vip/cs_links/EMNIST/gzip.zipNbyclass>iL iczEMNIST ByClass)r9r}r~rrbymerge/zEMNIST ByMergebalancediipIzEMNIST Balancedletters%iZi9zEMNIST Lettersdigitsrii@z EMNIST Digitsmnistr r z EMNIST MNISTc Cs>tjj|dtjtjjtdtjj|jj ddddt dS)NaThe EMNIST dataset is a set of handwritten character digits derived from the NIST Special Database 19 and converted to a 28x28 pixel image format and dataset structure that directly matches the MNIST dataset. Note: Like the original EMNIST data, images provided here are inverted horizontally and rotated 90 anti-clockwise. You can use `tf.transpose` within `ds.map` to convert the images to a human-friendlier format.r rrz=https://www.nist.gov/itl/products-and-services/emnist-datasetr) rrrrrrrr builder_configr}_EMNIST_CITATIONr#r%r%r&r'&s z EMNIST._infocsd|jjd|jjd|jjd|jjd}tj||jd|fdd| D}t j j t j jt|jj|d |d d d t j j t j jt|jj|d |dd d gS)Nz$emnist-{}-train-images-idx3-ubyte.gzz$emnist-{}-train-labels-idx1-ubyte.gzz#emnist-{}-test-images-idx3-ubyte.gzz#emnist-{}-test-labels-idx1-ubyte.gzr(gzipcs i|] \}}|tj|qSr%)ospathjoin)r0r1fnamedir_namer%r&r3Psz,EMNIST._split_generators..r)r*r4r8r+r,)formatrr9rrrr?r/extractr@rrrArBrCrDr~rFr)r$rHrI extractedr%rr&rK?sDzEMNIST._split_generators) rWrXrYrZr/r]ryBUILDER_CONFIGSr'rKr%r%r%r&rsb. rcCsptjj|d&}|d|tt|}tj|tjd |ttd}|WdS1s1wYdS)Nrbdtyper) tfiogfileGFileread_MNIST_IMAGE_SIZEnp frombufferuint8reshape)image_filepath num_imagesfbufrSr%r%r&rLfs  $rLcCsdtjj|d }|d||}tj|tjdtj }|WdS1s+wYdS)Nrr) rrrrrrrrastypeint64)labels_filepath num_labelsrrrRr%r%r&rMqs   $rM))rZ __future__rrrrnumpyr six.movesrtensorflow.compat.v2compatv2rtensorflow_datasets.public_api public_apirr[r;r<r=r>rrr!rErGr"rkrxrrGeneratorBasedBuilderr r_rm BuilderConfigryrrLrMr%r%r%r&s<         Gs