o Si6"@sxddlmZddlmZddlmZmZmZmZm Z m Z m Z ddl m Z ddlmZddlmZmZGdddeZd S) )reduce)add)AnyCallableDictListOptionalTupleUnion)CutSet)Cut) CutSamplerSamplingDiagnosticscsPeZdZdZdddededdffdd Zedee fd d Z edee fd d Z edee fddZ fddZdeeefffdd Zdeeefddffdd ZddZdeeeeffddZde ddffdd Zdeegefddfdd Zd!eeeed"ffddfd#d$Zedefd%d&Zdefd'd(Z Z!S)) ZipSamplera- :class:`.ZipSampler` takes several samplers as input and concatenates their sampled mini-batch cuts together into a single :class:`~lhotse.cut.CutSet`, or returns a tuple of the mini-batch CutSets. It is helpful for ensuring that each batch consists of some proportion of cuts coming from different sources. The input samplers do not have to provide the same number of batches -- when any of the samplers becomes depleted, the iteration will stop (like with Python's ``zip()`` function). Example:: >>> sampler = ZipSampler( ... SimpleCutSampler(cuts_corpusA, max_duration=250, shuffle=True), ... SimpleCutSampler(cuts_corpusB, max_duration=100, shuffle=True), ... ) >>> for cut in sampler: ... pass # profit T) merge_batchessamplersrreturnNcs tjddd||_||_dS)ap ZipSampler's constructor. :param samplers: The list of samplers from which we sample batches together. :param merge_batches: Should we merge the batches from each sampler into a single CutSet, or return a tuple of CutSets. Setting this to ``False`` makes ZipSampler behave more like Python's ``zip`` function. r)rank world_sizeN)super__init__rr)selfrr __class__O/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/dataset/sampling/zip.pyr s  zZipSampler.__init__cC,z tdd|jDWStyYdSw)z Remaining duration of data left in the sampler (may be inexact due to float arithmetic). .. note: For ZipSampler, it's the minimum of remaining durations in its sub-samplers. cs|]}|jVqdSN)remaining_duration.0srrr 5z0ZipSampler.remaining_duration..Nminr TypeErrorrrrrr -s  zZipSampler.remaining_durationcCr)z Remaining number of cuts in the sampler. Not available when the CutSet is read in lazy mode (returns None). .. note: For ZipSampler, it's the minimum of remaining cuts in its sub-samplers. csrr)remaining_cutsr!rrrr$Br%z,ZipSampler.remaining_cuts..Nr&r)rrrr*9  zZipSampler.remaining_cutscCr)z Total number of cuts in the sampler. Not available when the CutSet is read in lazy mode (returns None). .. note: For ZipSampler, it's the minimum of num cuts in its sub-samplers. csrr)num_cutsr!rrrr$Or%z&ZipSampler.num_cuts..Nr&r)rrrr,Fr+zZipSampler.num_cutscs"t|jD]}|qdS)a8 Enables re-setting to the start of an epoch when iter() is called. This is only needed in one specific scenario: when we restored previous sampler state via ``sampler.load_state_dict()`` but want to discard the progress in the current epoch and start from the beginning. N)rallow_iter_to_reset_stater)rr#rrrr-Ss   z$ZipSampler.allow_iter_to_reset_statecs,t}||jdd|jDd|S)z Return the current state of the sampler in a state_dict. Together with ``load_state_dict()``, this can be used to restore the training loop's state to the one stored in the state_dict. cSsg|]}|qSr) state_dictr!rrr hsz)ZipSampler.state_dict..)rr)rr.updaterr)rr.rrrr.^s zZipSampler.state_dictr.cs||d|_t|jt|dks#Jdt|jdt|ddt|j|dD] \}}||q,t|dS)aX Restore the state of the sampler that is described in a state_dict. This will result in the sampler yielding batches from where the previous training left it off. .. caution:: The samplers are expected to be initialized with the same CutSets, but this is not explicitly checked anywhere. .. caution:: The input ``state_dict`` is being mutated: we remove each consumed key, and expect it to be empty at the end of loading. If you don't want this behavior, pass a copy inside of this function (e.g., using ``import deepcopy``). .. note:: For implementers of sub-classes of CutSampler: the flag ``self._just_restored_state`` has to be handled in ``__iter__`` to make it avoid resetting the just-restored state (only once). rrz_Error in ZipSampler.load_state_dict(): Inconsistent number of samplers: current ZipSampler has z, the state_dict has .N)poprlenrzipload_state_dictr)rr.sampler sampler_sdrrrr5ms   zZipSampler.load_state_dictcCs|jD]}t|q|Sr)riter)rr6rrr__iter__s  zZipSampler.__iter__cs||jrLg}|jD] }t|}||q |stSt|tr,tdd|DSt|}g}t |D]|tfdd|Dq6t |Sg}|jD] }|t|qQt |S)Ncss|] }|D]}|VqqdSrrr"batchcrrrr$sz)ZipSampler._next_batch..c3s"|] }|D]}|VqqdSrrr:irrr$s ) r-rrnextappendr isinstance from_cutsr3rangetuple)rcutsr6r; tuple_lencut_setsrr=r _next_batchs*     zZipSampler._next_batchepochcs&|jD]}||qt|dS)a Sets the epoch for this sampler. When :attr:`shuffle=True`, this ensures all replicas use a different random ordering for each epoch. Otherwise, the next iteration of this sampler will yield the same ordering. :param epoch: Epoch number. N)r set_epochr)rrIr#rrrrJs  zZipSampler.set_epoch predicatecCs|jD]}||qdS)a  Add a constraint on individual cuts that has to be satisfied to consider them. Can be useful when handling large, lazy manifests where it is not feasible to pre-filter them before instantiating the sampler. Example: >>> cuts = CutSet(...) ... sampler = SimpleCutSampler(cuts, max_duration=100.0) ... # Retain only the cuts that have at least 1s and at most 20s duration. ... sampler.filter(lambda cut: 1.0 <= cut.duration <= 20.0) N)rfilter)rrKr6rrrrLs zZipSampler.filterr;.cCsdSrr)rr;rrr_log_diagnosticsszZipSampler._log_diagnosticscCsttdd|jDS)Ncsrr) diagnosticsr!rrrr$r%z)ZipSampler.diagnostics..)rrrr)rrrrNszZipSampler.diagnosticscCs |jS)zJReturns a string describing the statistics of the sampling process so far.)rN get_reportr)rrrrOs zZipSampler.get_report)"__name__ __module__ __qualname____doc__r boolrpropertyrfloatr intr*r,r-rstrrr.r5r9r r r rHrJrr rLrMrrNrO __classcell__rrrrr s(      $ "rN) functoolsroperatorrtypingrrrrrr r lhotser lhotse.cutr lhotse.dataset.sampling.baser rrrrrrs $