o Si@sfddlZddlmZddlmZmZddlZddlm Z ddl m Z GdddZ Gdd d e Z dS) N)deque)ListOptional)CutSet)Cutc@seZdZdZdefddZedefddZede e fdd Z d e ddfd d Z dddZdeddfddZd ddZde ddfddZd!ddZdefddZde fddZdS)" DataSourcez An iterator wrapper over CutSet that helps with the sampling process: it allows for deterministic re-shuffling of elements and "returning" sampled elements to be yielded again. itemscCsf||_|j|_d|_t|_|js#tdd|jD|_t|j|_ nd|_d|_ |j|_ |j |_ dS)Ncss|]}|jVqdSN)duration).0cr W/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/dataset/sampling/data_source.py sz&DataSource.__init__..) _orig_items_shuffled_items_iterr _reusableis_lazysum_total_durationlen _total_cuts_remaining_durationremaining_cuts)selfrr r r__init__s zDataSource.__init__returncCs|jjSr )rrrr r rr!szDataSource.is_lazycCs|jdurdStd|jS)Nr)rmaxrr r rremaining_duration%s  zDataSource.remaining_durationseedcCs&|t|}|jj|d|_|S)z Shuffles the elements using the provided random seed value. When the input CutSet is lazy, we use a streaming variant of shuffle, that may be less random. )rng)resetrandomRandomrshuffler)rr!rr r rr&-s zDataSource.shuffleothercCs||j|j|_|S)z{ Sorts the underlying CutSet to provide Cuts in the same order of cut_ids as the other DataSource. )r#r sort_liker)rr(r r rr)8szDataSource.sort_likecutNcCs8|j||js|j|j7_|jd7_dSdS)z8Push the cut in front of other cuts to be sampled again.N)rappendrrr r)rr*r r r take_backAs zDataSource.take_backcCs$d|_|j|j|_|j|_dSz'Reset the iterable state of DataSource.N)rrclearrrrrrr r rr#Hs  zDataSource.resetstepscCs.|dksJt|t|D]}t|qdSz5Advance the data source by ``steps`` amount of steps.rN)iterrangenextrr0ir r r fast_forwardOs   zDataSource.fast_forwardcCs|t|j|_|Sr )r#r2rrrr r r__iter__Vs zDataSource.__iter__cCsD|jr |j}nt|j}|js |j|j8_|jd8_|SNr+)rpopleftr4rrrr rrnext_cutr r r__next__[s  zDataSource.__next__cCs t|jSr )rrrr r r__len__es zDataSource.__len__)r(rrrrN)rr)__name__ __module__ __qualname____doc__rrpropertyboolrrfloatr intr&r)rr-r#r7r8r=r>r r r rr s    rcsbeZdZdZdededeffdd Zdd d Zd eddfd d Z dddZ de fddZ Z S)WeightedDataSourcea An iterator wrapper over CutSet that helps with the sampling process: it allows for deterministic re-shuffling of elements and "returning" sampled elements to be yielded again. Every cut has a sampling weight. At the beginning of each epoch, we pre-compute the indexes by sampling from multi-nomial distribution without replacement. The data source will be exhausted if the number of drawn cuts exceed num_samples rweights num_samplescsftj|dt|t|ksJd|t|ksJdt|}||}||_||_d|_dS)aLThe constructor of the weighted data source Args: items (CutSet): The cutset itself weights (List): A list of values representing the weight of each cut. All values must be positive num_samples (int): The number of samples to be drawn. Must smaller than the total number of cuts )rzThe length should matchzDThe number of samples to be drawn should not exceed the dataset sizeN) superrrnparrayrrIrJsampled_indexes)rrrIrJ __class__r rrus   zWeightedDataSource.__init__rNcCs*d|_d|_|j|j|_|j|_dSr.)rrNrr/rrrrrr r rr#s   zWeightedDataSource.resetr0cCs0|dksJt|t|D]}t|jqdSr1)r2r3r4rNr5r r rr7s   zWeightedDataSource.fast_forwardcCsD|t|j|_tjjt|j|j |jdd|_ t|j |_ |S)NF)preplace) r#r2rrrLr$choicerrIrJrNrr r rr8s  zWeightedDataSource.__iter__cCsJ|jr |j}n|jt|j}|js#|j|j8_|jd8_|Sr9) rr:rr4rNrrr rr;r r rr=s zWeightedDataSource.__next__r?)rrH)r@rArBrCrrrGrr#r7r8rr= __classcell__r r rOrrHis   rH)r$ collectionsrtypingrrnumpyrLlhotser lhotse.cutrrrHr r r rs   ^