o wi@spdZddlmZddlmZddlmZGdddeZGdddeeZGd d d eZGd d d eeZ d S)zdTrain PyTorch models directly from POSIX tar archive. Code works locally or over HTTP connections. )utils)IterableDataset) PipelineStagec@s eZdZdZddZddZdS) MockDatasetzCreate a mock dataset for performance testing and unit testing. Args: sample: The sample to be returned repeatedly. length (int): The length of the mock dataset. cCs||_||_dSN)samplelength)selfrrr U/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/webdataset/extradatasets.py__init__s zMockDataset.__init__ccst|jD]}|jVqdS)zYield samples from the mock dataset. Returns: Iterator: An iterator that yields the same sample repeatedly. N)rangerr)r _r r r __iter__s zMockDataset.__iter__N)__name__ __module__ __qualname____doc__r rr r r r rs rc@s"eZdZdZdddZddZdS) repeatedlya2Repeatedly yield samples from a dataset. Args: source: The source dataset to repeat. nepochs (int, optional): Maximum number of epochs to repeat. nbatches (int, optional): Maximum number of batches to repeat. length (int, optional): Length of the repeated dataset. NcCs||_||_||_dSr)sourcernbatches)r rnepochsrrr r r r 3s zrepeatedly.__init__cCstj||j|jdS)zReturn an iterator that iterates repeatedly over a source. Args: source: The source dataset to repeat. Returns: Iterator: An iterator that repeatedly yields samples from the source. )rr)rrrr)r rr r r invoke8s  zrepeatedly.invoke)NNN)rrrrr rr r r r r)s rc0eZdZdZfddZddZddZZS) with_epochaChange the actual and nominal length of an IterableDataset. This will continuously iterate through the original dataset, but impose new epoch boundaries at the given length/nominal. This exists mainly as a workaround for the odd logic in DataLoader. It is also useful for choosing smaller nominal epoch sizes with very large datasets. Args: dataset: The source IterableDataset. length (int): Declared length of the dataset. cst||_d|_dSr)superr rrr datasetr __class__r r r V  zwith_epoch.__init__cCst|j}d|d<|S)zReturn the pickled state of the dataset. This resets the dataset iterator, since that can't be pickled. Returns: dict: A dictionary representing the pickled state of the dataset. Nr)dict__dict__)r resultr r r __getstate__[s zwith_epoch.__getstate__c cs|jdur t||_t|jD]/}zt|j}Wn"ty;t||_zt|j}Wn ty8YYdSwYnw|Vqd|_dS)aReturn an iterator over the dataset. This iterator returns as many samples as given by the `length` parameter. Args: dataset: The source dataset to iterate over. Yields: Sample: The next sample from the dataset. N)riterr rnext StopIteration)r rrrr r r rgs"      zwith_epoch.invoke)rrrrr r$r __classcell__r r rr rHs   rcr) with_lengthzRepeatedly yield samples from a dataset with a specified length. Args: dataset: The source dataset. length (int): The stated length of the dataset. cst||_||_dSr)rr rrrrr r r r zwith_length.__init__cCst|S)zReturn an iterator that iterates over the source dataset. Args: dataset: The source dataset to iterate over. Returns: Iterator: An iterator over the source dataset. )r%)r rr r r rs zwith_length.invokecCs|jS)zrReturn the user specified length. Returns: int: The specified length of the dataset. )r)r r r r __len__szwith_length.__len__)rrrrr rr*r(r r rr r)s   r)N) rrpytorchrrrrrr)r r r r s   9