o Si@sRddlZddlZddlmZddlmZmZddlmZGdddej j j Z dS)N)CutSet)get_rankget_world_size) CutSamplerc s|eZdZdZ  ddejjjdede de ddf fd d Z d e ddfd d Z ddZ defddZdeddfddZZS)IterableDatasetWrappera A wrapper that creates an iterable-style dataset out of a map-style dataset and a :class:`~lhotse.dataset.sampling.base.CutSampler`. It's intended use is for training with cuts (:class:`~lhotse.cut.Cut`) that contain binary data instead of paths/urls/keys pointing to an external storage. Currently, we support this kind of dataloading using WebDataset library (to export a :class:`~lhotse.CutSet` into WebDataset tarball format, see :func:`lhotse.dataset.webdataset.export_to_webdataset`). .. caution: With an iterable-style dataset, the sampler replicas exist in the dataloading worker subprocesses. That means unless you take extra steps to avoid data duplication, it may happen that each worker returns exactly the same data. This problem is avoided with WebDataset by using sharding -- we let WebDataset subset the shards visible in each subprocess (and each node in multi-GPU DDP training). .. note: If you are going to use this class with ``persistent_workers=True`` option of ``torch.utils.data.DataLoader``, set ``auto_increment_epoch=True`` argument. It will ensure that each epoch is shuffled differently than the previous one. Example usage:: >>> from lhotse import CutSet >>> from lhotse.dataset import K2SpeechRecognitionDataset, DynamicCutSampler >>> # Preparing data -- WebDataset takes care of sharding and de-duplicating data >>> cuts = CutSet.from_webdataset( ... "data/shard-{000000..000321}.tar", ... shuffle_shards=True, ... split_on_workers=True, ... split_on_nodes=True, ... ) >>> dataset = K2SpeechRecognitionDataset() >>> sampler = DynamicCutSampler(cuts, max_duration=200, shuffle=True) >>> # Creating terable dataset wrapper >>> iter_dset = IterableDatasetWrapper(dataset, sampler) >>> dloader = torch.utils.data.DataLoader(iter_dset, batch_size=None, num_workers=2) >>> # Training loop >>> for epoch in range(10): ... dloader.dataset.set_epoch(epoch) ... for batch in dloader: ... pass # training step Fdatasetsamplerauto_increment_epoch reset_on_iterreturnNcsnt||_||_||_||_d|_d|_|jj}|jj }|dks'|dkr5t d|d|ddSdS)Nrz8We detected you're trying to use a CutSampler with rank z and world_size a inside an IterableDatasetWrapper. Setting rank != 0 and world_size != 1 in Lhotse's CutSampler is inteded for map-style datasets, when the sampler exists in the main training loop. Make sure these settings are intentional or pass rank=0 and world_size=1 to the sampler's constructor. ) super__init__rrr r epoch _sampler_iterrank world_sizewarningswarn)selfrrr r rws __class__S/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/dataset/iterable_dataset.pyr6s zIterableDatasetWrapper.__init__rcCs^||_|j|t|jdr)t|jjtr+|jjD]}t|jdr(|j|qdSdSdS)Ncuts set_epoch)rrrhasattr isinstancertupledata)rrcsrrrrPs    z IterableDatasetWrapper.set_epochcCs |jdus|jrt|j|_|S)N)rr iterr)rrrr__iter__[s zIterableDatasetWrapper.__iter__cCsNzt|j}|||j|WSty&|jr"||jdd|_w)Nr )nextr_update_dataloading_infor StopIterationr rr)rsampledrrr__next__`s    zIterableDatasetWrapper.__next__rcCs.t}t}|D] }||jd<||jd<qdS)Nrr)rrdataloading_info)rrrrcrrrr%ks   z/IterableDatasetWrapper._update_dataloading_info)FF)__name__ __module__ __qualname____doc__torchutilsr Datasetrboolrintrr#dictr(rr% __classcell__rrrrr s&/  r) rr/lhotserlhotse.dataset.dataloadingrrlhotse.dataset.sampling.baserr0r IterableDatasetrrrrrs