o SiS%@s\ddlZddlmZmZmZddlmZmZddlm Z m Z ddl m Z Gddde Z dS)N)AnyDictOptional)CutSetSeconds) CutSamplerTimeConstraint) DataSourcecseZdZdZ       ddededeeded ed eed eed effd d Z e dee fddZ e deefddZ e deefddZdeeefffdd Zdeeefddffdd Zd ddZdefddZZS)!SimpleCutSamplera Samples cuts from a CutSet to satisfy the input constraints. It behaves like an iterable that yields lists of strings (cut IDs). When one of :attr:`max_duration`, or :attr:`max_cuts` is specified, the batch size is dynamic. Exactly zero or one of those constraints can be specified. Padding required to collate the batch does not contribute to max duration. Example usage:: >>> dataset = K2SpeechRecognitionDataset(cuts) >>> sampler = SimpleCutSampler(cuts, shuffle=True) >>> loader = DataLoader(dataset, sampler=sampler, batch_size=None) >>> for epoch in range(start_epoch, n_epochs): ... sampler.set_epoch(epoch) ... train(loader) NFrcuts max_durationmax_cutsshuffle drop_last world_sizerankseedc sPtj|||||dtdd||fDsJdt||_t||d|_dS)a SimpleCutSampler's constructor. :param cuts: the ``CutSet`` to sample data from. :param max_duration: The maximum total recording duration from ``cuts``. :param max_cuts: The maximum number of cuts sampled to form a mini-batch. By default, this constraint is off. :param shuffle: When ``True``, the cuts will be shuffled at the start of iteration. Convenient when mini-batch loop is inside an outer epoch-level loop, e.g.: `for epoch in range(10): for batch in dataset: ...` as every epoch will see a different cuts order. :param drop_last: When ``True``, the last batch is dropped if it's incomplete. :param world_size: Total number of distributed nodes. We will try to infer it by default. :param rank: Index of distributed node. We will try to infer it by default. :param seed: Random seed used to consistently shuffle the dataset across different processes. )rrrrrcss|]}|duVqdS)N).0vrrR/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/dataset/sampling/simple.py @s z,SimpleCutSampler.__init__..z7At least one of max_duration or max_cuts has to be set.)r r N)super__init__anyr data_sourcertime_constraint) selfr r r rrrrr __class__rrrs"   zSimpleCutSampler.__init__returncC|jjS)z Remaining duration of data left in the sampler (may be inexact due to float arithmetic). Not available when the CutSet is read in lazy mode (returns None). )rremaining_durationrrrrr"Iz#SimpleCutSampler.remaining_durationcCr!)z Remaining number of cuts in the sampler. Not available when the CutSet is read in lazy mode (returns None). )rremaining_cutsr#rrrr%Qr$zSimpleCutSampler.remaining_cutscCs|jjrdSt|jS)z Total number of cuts in the sampler. Not available when the CutSet is read in lazy mode (returns None). N)ris_lazylenr#rrrnum_cutsYs zSimpleCutSampler.num_cutscs"t}|d|ji|S)z Return the current state of the sampler in a state_dict. Together with ``load_state_dict()``, this can be used to restore the training loop's state to the one stored in the state_dict. r)r state_dictupdater)rr)rrrr)cs  zSimpleCutSampler.state_dictr)csztdi|d}|j|krtd|jd|d||_t||jr2|j|j |j |j |j j jdS)aX Restore the state of the sampler that is described in a state_dict. This will result in the sampler yielding batches from where the previous training left it off. .. caution:: The samplers are expected to be initialized with the same CutSets, but this is not explicitly checked anywhere. .. caution:: The input ``state_dict`` is being mutated: we remove each consumed key, and expect it to be empty at the end of loading. If you don't want this behavior, pass a copy inside of this function (e.g., using ``import deepcopy``). .. note:: For implementers of sub-classes of CutSampler: the flag ``self._just_restored_state`` has to be handled in ``__iter__`` to make it avoid resetting the just-restored state (only once). rzKSimpleCutSampler.load_state_dict(): Inconsistent time_constraint: expected z received z= We will overwrite the settings with the received state_dict.Nr)rpoprwarningswarnrload_state_dictrrrepoch fast_forward diagnosticscurrent_epoch_stats total_cuts)rr)rrrrr.qs  z SimpleCutSampler.load_state_dictcCs<|jr|S|j|jr|j|j|jt|j|S)zi Prepare the dataset for iterating over a new epoch. Will shuffle the data if requested. )_just_restored_stater1reset_current_epochrrrr/iterr#rrr__iter__s  zSimpleCutSampler.__iter__cCs|jg} zt|j}Wn!ty0|r'|jr |jr't|YS|j |tw| |s=|j |q|j ||jsN||n|rW|j|n td||qt|S)NTzThe first cut drawn in batch collection violates the max_duration, or max_cuts constraints - we'll return it anyway. Consider increasing max_duration/max_cuts.)rresetnextr StopIterationrclose_to_exceedingr from_cutsr1discard _filter_fndiscard_singleaddexceededappend take_backr,r-)rr next_cutrrr _next_batchs<           3zSimpleCutSampler._next_batch)NNFFNNr)r r )__name__ __module__ __qualname____doc__rrrintboolrpropertyfloatr"r%r(rstrrr)r.r7rE __classcell__rrrrr sH +  #r )r,typingrrrlhotserrlhotse.dataset.sampling.baserr#lhotse.dataset.sampling.data_sourcer r rrrrs