o wi@s^ddlZddlZddlZddlmZddlmZmZddlm Z ddZ Gdd d ee Z dS) N)islice) DataLoaderIterableDataset) PipelineStagecCs0dd}t|jjd|jtfd|i}||_|S)zAdd a length method to the given object. Args: obj: The object to which the length method will be added. Returns: The modified object with a new length method. cSs|jS)N)sizeselfr P/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/webdataset/pipeline.pylengthsz!add_length_method..length_Length__len__)type __class____name__r)objr Combinedr r r add_length_method s  rcs~eZdZdZfddZddZddZdd Zd d Zd d Z ddZ ddZ ddZ dddZ dddZdddZZS) DataPipelinezA pipeline starting with an IterableDataset and a series of filters. Args: *args: Variable length argument list of pipeline stages. **kwargs: Arbitrary keyword arguments. cs^tg|_d|_d|_d|_|D]}|durqt|tr&|j|q|j |qdS)Nr) super__init__pipeliner repetitionsnsamples isinstancelistextendappend)r argskwargsargrr r r(s  zDataPipeline.__init__cCs&|jD] }t|dr|q|`dS)z)Close the pipeline and release resources.closeN)rhasattrr$)r stepr r r r$6s  zDataPipeline.closecOsvt|ttfrt|dkrt|St|tr|j|i|St|tr't|St|r4||i|}|St |d)aApply a pipeline stage, possibly to the output of a previous stage. Args: f: The pipeline stage to invoke. *args: Variable length argument list. **kwargs: Arbitrary keyword arguments. Returns: The result of invoking the pipeline stage. Raises: ValueError: If the pipeline stage is not valid. rz: not a valid pipeline stage) rrrleniterrrunrcallable ValueError)r fr r!resultr r r invoke=s  zDataPipeline.invokecCs4||jd}|jddD]}|||}q|S)zCreate an iterator through one epoch in the pipeline. Returns: An iterator for one epoch of the pipeline. rrN)r.r)r sourcer&r r r iterator1VszDataPipeline.iterator1ccsDt|jD]}d}|D] }|V|d7}q|dkrdSqdS)zCreate an iterator through the entire dataset, using the given number of repetitions. Yields: Samples from the dataset. rrN)rangerr0)r _countsampler r r iteratoras  zDataPipeline.iteratorcCs4|jdkr|jdkrt||jS|S|S)zCreate an iterator through the pipeline, repeating and slicing as requested. Returns: An iterator through the pipeline. rr)rrrr5rr r r __iter__ps  zDataPipeline.__iter__cCs |j|S)zReturn pipeline stage i. Args: i: The index of the pipeline stage to return. Returns: The pipeline stage at index i. )r)r ir r r stage~s zDataPipeline.stagecCs|j|dS)zsAppend a pipeline stage (modifies the object). Args: f: The pipeline stage to append. N)rr)r r,r r r rszDataPipeline.appendcGs0t|}t|j|_|D]}||q|S)zAppend pipeline stages to a copy of the pipeline and return the copy. Args: *args: Variable length argument list of pipeline stages to append. Returns: A new DataPipeline object with the appended stages. )copyrr)r r r-r"r r r composes  zDataPipeline.composeFcCs|std||_t|S)aAdd a __len__ method returning the desired value. This does not change the actual number of samples in an epoch. PyTorch IterableDataset should not have a __len__ method. This is provided only as a workaround for some broken training environments that require a __len__ method. Args: n: The length value to set. silent: If True, suppress the warning message. Returns: The modified DataPipeline object with a __len__ method. z.with_length() only sets the value of __len__ for compatibility with some training environments. It does not change the number of samples in an epoch.)warningswarnrr)r nsilentr r r with_lengths zDataPipeline.with_lengthrcCstj|_t|||_|S)aChange the epoch to return the given number of samples/batches. Args: nsamples: The number of samples per epoch. nbatches: The number of batches per epoch. Returns: The modified DataPipeline object. )sysmaxsizermaxr)r rnbatchesr r r with_epochs zDataPipeline.with_epochcCs*|dkr ||_||_|Stj|_||_|S)a8Repeat iterating through the dataset for the given number of epochs up to the given number of samples. Args: nepochs: The number of epochs to repeat. nbatches: The number of batches to limit per repetition. Returns: The modified DataPipeline object. r)rrr@rA)r nepochsrCr r r repeats zDataPipeline.repeat)F)rr)r __module__ __qualname____doc__rr$r.r0r5r6r8rr:r?rDrF __classcell__r r r#r r s     r) r9r@r; itertoolsrpytorchrrutilsrrrr r r r s