o Si*@sddlmZddlmZddlmZmZmZmZm Z m Z m Z ddl Z ddlZddlmZddlmZddlmZmZGdd d eZdS) )reduce)add)AnyCallableDictListOptionalTupleUnionN)CutSet)Cut) CutSamplerSamplingDiagnosticsc sreZdZdZdddddededeeeefde d d f fd d Z e d e efd dZ e d e e fddZe d e e fddZfddZd eeefffdd Zdeeefd d ffdd ZddZd eeeeffddZd.ddZd e d d ffd!d" Zd#eegefd d fd$d%Zd&eeeed'ffd d fd(d)Ze d e fd*d+Z!d efd,d-Z"Z#S)/RoundRobinSamplera :class:`.RoundRobinSampler` takes several samplers as input, and yields a mini-batch of cuts from each of those samplers in turn. E.g., with two samplers, the first mini-batch is from ``sampler0``, the seconds from ``sampler1``, the third from ``sampler0``, and so on. It is helpful for alternating mini-batches from multiple datasets or manually creating batches of different sizes. The input samplers do not have to provide the same number of batches -- when any of the samplers becomes depleted, we continue to iterate the non-depleted samplers, until all of them are exhausted. Example:: >>> sampler = RoundRobinSampler( ... SimpleCutSampler(cuts_corpusA, max_cuts=32, shuffle=True), ... SimpleCutSampler(cuts_corpusB, max_cuts=64, shuffle=True), ... ) >>> for cut in sampler: ... pass # profit Fr) stop_early randomizeseedsamplersrrrreturnNcstjdd|d||_||_d|_ttt|j|_d|_ d|_ t |tr3t|t|jks2Jn|dkrDdt|jgt|j}||_ dS)a RoundRobinSampler's constructor. :param samplers: The list of samplers from which we sample batches in turns. :param stop_early: Should we finish the epoch once any of the samplers becomes depleted. By default, we will keep iterating until all the samplers are exhausted. This setting can be used to balance datasets of different sizes. :param randomize: Select the next sampler according to a distribution, instead of in order. If a list of floats is provided, it must contain the same number of elements as the number of samplers, and the values will be used as probabilities. If ``True`` is provided, the probabilities will be uniform. If ``False`` is provided, the samplers will be selected in order. :param seed: Random seed used to select the next sampler (only used if ``randomize`` is True) r)rank world_sizerNTg?) super__init__rrrnglistrangelen_nondepleted_samplers_indices_cur_sampler_idx_num_dl_workers isinstancer)selfrrrr __class__W/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/dataset/sampling/round_robin.pyr$s  zRoundRobinSampler.__init__cC,z tdd|jDWStyYdSw)zj Remaining duration of data left in the sampler (may be inexact due to float arithmetic). cs|]}|jVqdSN)remaining_duration.0sr%r%r& Oz7RoundRobinSampler.remaining_duration..Nsumr TypeErrorr"r%r%r&r*Is  z$RoundRobinSampler.remaining_durationcCr')z Remaining number of cuts in the sampler. Not available when the CutSet is read in lazy mode (returns None). csr(r))remaining_cutsr+r%r%r&r.Zr/z3RoundRobinSampler.remaining_cuts..Nr0r3r%r%r&r4S  z RoundRobinSampler.remaining_cutscCr')z Total number of cuts in the sampler. Not available when the CutSet is read in lazy mode (returns None). csr(r))num_cutsr+r%r%r&r.er/z-RoundRobinSampler.num_cuts..Nr0r3r%r%r&r6^r5zRoundRobinSampler.num_cutscs"t|jD]}|qdS)a8 Enables re-setting to the start of an epoch when iter() is called. This is only needed in one specific scenario: when we restored previous sampler state via ``sampler.load_state_dict()`` but want to discard the progress in the current epoch and start from the beginning. N)rallow_iter_to_reset_stater)r"r-r#r%r&r7is   z+RoundRobinSampler.allow_iter_to_reset_statec s@t}|dd|jD|j|j|j|jt|j d|S)z Return the current state of the sampler in a state_dict. Together with ``load_state_dict()``, this can be used to restore the training loop's state to the one stored in the state_dict. cSsg|]}|qSr%) state_dictr+r%r%r& }sz0RoundRobinSampler.state_dict..)rrrrr r) rr8updaterrrrr rr)r"r8r#r%r&r8ts  zRoundRobinSampler.state_dictr8cs|d|_|d|_|d|_|d|_|d|_t|jt|dks;Jdt|jdt|dd t|j|dD] \}}| |qDt |d S) aX Restore the state of the sampler that is described in a state_dict. This will result in the sampler yielding batches from where the previous training left it off. .. caution:: The samplers are expected to be initialized with the same CutSets, but this is not explicitly checked anywhere. .. caution:: The input ``state_dict`` is being mutated: we remove each consumed key, and expect it to be empty at the end of loading. If you don't want this behavior, pass a copy inside of this function (e.g., using ``import deepcopy``). .. note:: For implementers of sub-classes of CutSampler: the flag ``self._just_restored_state`` has to be handled in ``__iter__`` to make it avoid resetting the just-restored state (only once). rrrr rrzmError in RoundRobinSampler.load_state_dict(): Inconsistent number of samplers: current RoundRobinSampler has z, the state_dict has .N) poprrrr rrrzipload_state_dictr)r"r8sampler sampler_sdr#r%r&r>s"      z!RoundRobinSampler.load_state_dictcCstjj|j|jd|_|jD]}t|q|jr|St t t |j|_ d|_ d|_tjj}|durB|jt |j |_ |j|_|S)N)rrr)nprandom default_rngrepochrriter_just_restored_staterrrrrr torchutilsdataget_worker_infoid num_workers)r"r? worker_infor%r%r&__iter__s   zRoundRobinSampler.__iter__cCst|jdkr t|j|j}|j|}zt|}Wn#ty>|j|j|js3t|jdkr4|| YSw||S)Nr) rr StopIterationrrnextr<r _set_next_idx _next_batch)r" sampler_idxr?batchr%r%r&rRs     zRoundRobinSampler._next_batchcsjdur5tjdkr5ttj}fddjDfddDjj|dddd_dSjjtj_dS)NFrcsg|]}j|qSr%)r)r,ir3r%r&r9sz3RoundRobinSampler._set_next_idx..csg|]}|tqSr%)r1)r,x)pr%r&r9s)sizereplacerWr)rrrrrchoicerr )r"Nr%)rWr"r&rQs  zRoundRobinSampler._set_next_idxrDcs&|jD]}||qt|dS)a Sets the epoch for this sampler. When :attr:`shuffle=True`, this ensures all replicas use a different random ordering for each epoch. Otherwise, the next iteration of this sampler will yield the same ordering. :param epoch: Epoch number. N)r set_epochr)r"rDr-r#r%r&r\s  zRoundRobinSampler.set_epoch predicatecCs|jD]}||qdS)a  Add a constraint on individual cuts that has to be satisfied to consider them. Can be useful when handling large, lazy manifests where it is not feasible to pre-filter them before instantiating the sampler. Example: >>> cuts = CutSet(...) ... sampler = SimpleCutSampler(cuts, max_duration=100.0) ... # Retain only the cuts that have at least 1s and at most 20s duration. ... sampler.filter(lambda cut: 1.0 <= cut.duration <= 20.0) N)rfilter)r"r]r?r%r%r&r^s zRoundRobinSampler.filterrT.cCsdSr)r%)r"rTr%r%r&_log_diagnosticssz"RoundRobinSampler._log_diagnosticscCsttdd|jDS)Ncsr(r)) diagnosticsr+r%r%r&r.r/z0RoundRobinSampler.diagnostics..)rrrr3r%r%r&r`szRoundRobinSampler.diagnosticscCs |jS)zJReturns a string describing the statistics of the sampling process so far.)r` get_reportr3r%r%r&ras zRoundRobinSampler.get_report)rN)$__name__ __module__ __qualname____doc__r boolr rfloatintrpropertyrr*r4r6r7rstrrr8r>rNr r rRrQr\rr r^r_rr`ra __classcell__r%r%r#r&r sD%    !   "r) functoolsroperatorrtypingrrrrrr r numpyrArGlhotser lhotse.cutr lhotse.dataset.sampling.baser rrr%r%r%r&s $