o wiw@s^dZddlZddlZddlmZddZddZGd d d eZdd d Z GdddeZ dS)z1Classes for mixing samples from multiple sources.N)IterableDatasetcgsFd} zt||t|}|VWn tyYdSw|d7}q)zYield samples from multiple sources in a round-robin fashion until the shortest source is exhausted. Args: *sources: Iterable sources to draw samples from. Yields: Sample from one of the sources. rTrN)nextlen StopIterationsourcesisampler K/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/webdataset/mix.pyround_robin_shortests   r cgsrt|}d}t|dkr7|t|;}zt||}|d7}|VWn ty.||=Ynwt|dks dSdS)zYield samples from multiple sources in a round-robin fashion until all sources are exhausted. Args: *sources: Iterable sources to draw samples from. Yields: Sample from one of the sources. rrN)listrrrrr r r round_robin_longest#s       rc@s"eZdZdZdddZddZdS) RoundRobinz8Iterate over multiple datasets in a round-robin fashion.FcCs||_||_dS)zInitialize the RoundRobin iterator. Args: datasets (list): List of datasets to iterate over. longest (bool): If True, continue until the longest dataset is exhausted. N)datasetslongest)selfrrr r r __init__;s zRoundRobin.__init__cCs&dd|jD}|jrt|St|S)zReturn an iterator over the sources. Returns: iterator: An iterator that yields samples from the datasets in a round-robin fashion. cSg|]}t|qSr iter.0dr r r Kz'RoundRobin.__iter__..)rrrr rrr r r __iter__EszRoundRobin.__iter__N)F__name__ __module__ __qualname____doc__rrr r r r r8s  rFccs|dur dgt|}nt|}t|dkrTt|t|}t}t||}z t||VWnt yK|rF||=||=nYdSYnwt|dksdSdS)aYield samples randomly from multiple sources based on given probabilities. Args: sources (list): List of iterable sources to draw samples from. probs (list, optional): List of probabilities for each source. Defaults to None. longest (bool): If True, continue until all sources are exhausted. Defaults to False. Yields: Sample randomly selected from one of the sources. Nrr) rrnparraysumcumsumrandom searchsortedrr)rprobsrcumrr r r r random_samplesRs$    r-c@s"eZdZdZdddZddZdS) RandomMixzZIterate over multiple datasets by randomly selecting samples based on given probabilities.NFcCs||_||_||_dS)a:Initialize the RandomMix iterator. Args: datasets (list): List of datasets to iterate over. probs (list, optional): List of probabilities for each dataset. Defaults to None. longest (bool): If True, continue until all datasets are exhausted. Defaults to False. N)rr*r)rrr*rr r r rrs zRandomMix.__init__cCs"dd|jD}t||j|jdS)zReturn an iterator over the sources. Returns: iterator: An iterator that yields samples randomly from the datasets. cSrr rrr r r rrz&RandomMix.__iter__..)r)rr-r*rrr r r r~szRandomMix.__iter__NFrr r r r r.os  r.r/) r#r(numpyr$pytorchrr rrr-r.r r r r s