o 8wi7$@sddlZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z ddl m Z ddlmZddlZddlmZddlmZmZmZddlmZdd lmZdd lmZd ZGd d d eZdededefddZGdddeZdS)N)Sized)AnyCallableOptionalUnion) HTTPError)warn)Tensor) DataLoaderDataset random_split) _IS_WINDOWS)LightningDataModule)_TORCHVISION_AVAILABLEz./datac seZdZdZdZdZdZdZ d,ded e d e d e d e d df fdd Z de d e ee ffddZd e fddZed efddZded e fddZd-d e d dfddZded dfddZed.d!ed"e d#ed e eeffd$d%Zed/d'ed(ee efd)ee efd efd*d+ZZS)0_MNISTaDCarbon copy of ``tests_pytorch.helpers.datasets.MNIST``. We cannot import the tests as they are not distributed with the package. See https://github.com/Lightning-AI/pytorch-lightning/pull/7614#discussion_r671183652 for more context. .. warning:: This is meant for testing/debugging and is experimental. )zChttps://pl-public-data.s3.amazonaws.com/MNIST/processed/training.ptz?https://pl-public-data.s3.amazonaws.com/MNIST/processed/test.ptz training.ptztest.ptcompleteTg_)Ǻ?gGr?roottrain normalizedownloadkwargsreturnNc sZt||_||_||_|||jr|jn|j}|t j |j |\|_ |_dSN)super__init__rrr prepare_dataTRAIN_FILE_NAMETEST_FILE_NAME _try_loadospathjoincached_folder_pathdatatargets)selfrrrrr data_file __class__e/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/pytorch_lightning/demos/mnist_datamodule.pyr6s  "z_MNIST.__init__idxcCsV|j|d}t|j|}|jdur't|jdkr'|j|g|jR}||fS)Nr)r$float unsqueezeintr%rlennormalize_tensor)r&r,imgtargetr*r*r+ __getitem__Cs z_MNIST.__getitem__cCs t|jSr)r1r$r&r*r*r+__len__Ls z_MNIST.__len__cCstj|jd|jS)NMNIST)r r!r"rcache_folder_namer6r*r*r+r#Osz_MNIST.cached_folder_path data_foldercCs4d}|j|jfD]}|otjtj||}q|S)NT)rrr r!isfiler")r&r:existingfnamer*r*r+ _check_existsSsz_MNIST._check_existscCs4|r||js||j||jstddS)NzDataset not found.)r>r# _download RuntimeError)r&rr*r*r+rYs   z_MNIST.prepare_datacCsRtj|dd|jD]}td|tj|tj|}tj ||q dS)NT)exist_okz Downloading ) r makedirs RESOURCESlogginginfor!r"basenameurllibrequest urlretrieve)r&r:urlfpathr*r*r+r?_s  z_MNIST._download? path_datatrialsdeltac Csd\}}|s Jdtj|sJd|t|D]'}zt|}WntyA}z|}t|t WYd}~qd}~ww|dusIJ|durO||S)zHResolving loading from the same time from multiple concurrent processes.)NNz!at least some trial has to be setzmissing file: N) r r!r;rangetorchload Exceptiontimesleeprandom)rNrOrPres exception_exr*r*r+rfs    z_MNIST._try_loadtensormeanstdcCs8tj||j|jd}tj||j|jd}|||S)N)dtypedevice)rR as_tensorr`rasubdiv)r]r^r_r*r*r+r2{sz_MNIST.normalize_tensor)TrT)T)rLrM)r\rM)__name__ __module__ __qualname____doc__rCrrr9strbooltuplerrr0r r5r7propertyr#r>rr? staticmethodr.rrr2 __classcell__r*r*r(r+r#s>   &6rargsrrc Osttdd }|r6zddlm}|tddWnty5}ztd|dd}WYd}~nd}~ww|s>td t}||i|S) NPL_USE_MOCKED_MNISTFr)r8T)rzError z) downloading `torchvision.datasets.MNIST`zD`torchvision.datasets.MNIST` not available. Using our hosted version) rjr getenvtorchvision.datasetsr8_DATASETS_PATHrprintr)rortorchvision_mnist_availabler8r[r*r*r+r8s r8cseZdZdZdZedddddfded ed ed ed ed eddffdd Z e defddZ d!ddZ deddfddZ defddZdefddZdefddZe deefdd ZZS)"MNISTDataModulezStandard MNIST, train, val, test splits and transforms. >>> MNISTDataModule() # doctest: +ELLIPSIS <...mnist_datamodule.MNISTDataModule object at ...> mnistiF* data_dir val_split num_workersrseed batch_sizerNcsNt|rtrtd|dd}||_||_||_||_||_||_ dS)an Args: data_dir: where to save/load the data val_split: how many of the training images to use for the validation split num_workers: how many workers to use for loading data normalize: If true applies image normalize seed: starting seed for RNG. batch_size: desired batch size. zYou have requested num_workers=zA on Windows, but currently recommended is 0, so we set it for yourN) rrr rr{r|r}rr~r)r&r{r|r}rr~rr(r*r+rs   zMNISTDataModule.__init__cCsdS)N r*r6r*r*r+ num_classesszMNISTDataModule.num_classescCs$t|jdddt|jddddS)zSaves MNIST files to `data_dir`TrrFN)r8r{r6r*r*r+rszMNISTDataModule.prepare_datastagecCsr|jrd|jini}t|jfddd|}t|tsJt|}t|||j|jgt dd\|_ |_ dS)z"Split the train and valid dataset. transformTFrry) generatorN) default_transformsr8r{ isinstancerr1r r|rR Generator manual_seed dataset_train dataset_val)r&rextradataset train_lengthr*r*r+setupszMNISTDataModule.setupcCst|j|jd|jdddS)z7MNIST train set removes a subset to use for validation.Trshuffler} drop_last pin_memory)r rrr}r6r*r*r+train_dataloaderz MNISTDataModule.train_dataloadercCst|j|jd|jdddS)z?MNIST val set uses a subset of the training set for validation.FTr)r rrr}r6r*r*r+val_dataloaderrzMNISTDataModule.val_dataloadercCsD|jrd|jini}t|jfddd|}t||jd|jdddS)z#MNIST test set uses the test split.rFrTr)rr8r{r rr})r&rrr*r*r+test_dataloaderszMNISTDataModule.test_dataloadercCsFtsdSddlm}|jr|||jdddg}|S|}|S)Nr) transforms)g?)r^r_)r torchvisionrrComposeToTensor Normalize)r&rmnist_transformsr*r*r+rs  z"MNISTDataModule.default_transforms)rN)rerfrgrhnamersrir0rjrrlrrrr rrrrrrrnr*r*r(r+rvsB"     rv) rDr rWrUrGcollections.abcrtypingrrrr urllib.errorrwarningsrrRr torch.utils.datar r r "lightning_fabric.utilities.importsr pytorch_lightningr#pytorch_lightning.utilities.importsrrsrr8rvr*r*r*r+s&        _