o Si.@s`ddlZddlmZmZmZmZddlmZmZddl m Z m Z ddl m Z Gddde ZdS)N)AnyDictOptionalTuple)CutSetSeconds) CutSamplerTimeConstraint) DataSourcecseZdZdZ        d!dedededed eed ed ed eed eedeffdd Z e dee fddZ e deefddZ e deefddZdeeefffdd Zdeeefddffdd Zd"ddZdeeeffdd ZZS)#CutPairsSamplera Samples pairs of cuts from a "source" and "target" CutSet. It expects that both CutSet's strictly consist of Cuts with corresponding IDs. It behaves like an iterable that yields lists of strings (cut IDs). When one of :attr:`max_source_duration`, :attr:`max_target_duration`, or :attr:`max_cuts` is specified, the batch size is dynamic. Exactly zero or one of those constraints can be specified. Padding required to collate the batch does not contribute to max source_duration/target_duration. NFr source_cuts target_cutsmax_source_durationmax_target_durationmax_cutsshuffle drop_last world_sizerankseedc sJtj|||| | dt||_t||_t||d|_t||d|_dS)a CutPairsSampler's constructor. :param source_cuts: the first ``CutSet`` to sample data from. :param target_cuts: the second ``CutSet`` to sample data from. :param max_source_duration: The maximum total recording duration from ``source_cuts``. :param max_target_duration: The maximum total recording duration from ``target_cuts``. :param max_cuts: The maximum number of cuts sampled to form a mini-batch. By default, this constraint is off. :param shuffle: When ``True``, the cuts will be shuffled at the start of iteration. Convenient when mini-batch loop is inside an outer epoch-level loop, e.g.: `for epoch in range(10): for batch in dataset: ...` as every epoch will see a different cuts order. :param drop_last: When ``True``, the last batch is dropped if it's incomplete. :param world_size: Total number of distributed nodes. We will try to infer it by default. :param rank: Index of distributed node. We will try to infer it by default. :param seed: Random seed used to consistently shuffle the dataset across different processes. )rrrrr) max_durationrN)super__init__r r r r source_constraintstarget_constraints) selfr r rrrrrrrr __class__U/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/dataset/sampling/cut_pairs.pyrs"   zCutPairsSampler.__init__returncC|jjS)a  Remaining duration of data left in the sampler (may be inexact due to float arithmetic). Not available when the CutSet is read in lazy mode (returns None). .. note: For :class:`.CutPairsSampler` we return the source cuts duration. )r remaining_durationrrrrr"Gsz"CutPairsSampler.remaining_durationcCr!)z Remaining number of cuts in the sampler. Not available when the CutSet is read in lazy mode (returns None). )r remaining_cutsr#rrrr$QszCutPairsSampler.remaining_cutscCs|jjrdSt|jS)z Total number of cuts in the sampler. Not available when the CutSet is read in lazy mode (returns None). N)r is_lazylenr#rrrnum_cutsYs zCutPairsSampler.num_cutscs*t}||j|jd|S)z Return the current state of the sampler in a state_dict. Together with ``load_state_dict()``, this can be used to restore the training loop's state to the one stored in the state_dict. )rr)r state_dictupdaterr)rr(rrrr(cs zCutPairsSampler.state_dictr(cstdi|d}|j|krtd|jd|d||_tdi|d}|j|kr;td|jd|d||_t||jr[|j |j |j |j |j |j |j |jjj|j |jjjdS) aX Restore the state of the sampler that is described in a state_dict. This will result in the sampler yielding batches from where the previous training left it off. .. caution:: The samplers are expected to be initialized with the same CutSets, but this is not explicitly checked anywhere. .. caution:: The input ``state_dict`` is being mutated: we remove each consumed key, and expect it to be empty at the end of loading. If you don't want this behavior, pass a copy inside of this function (e.g., using ``import deepcopy``). .. note:: For implementers of sub-classes of CutSampler: the flag ``self._just_restored_state`` has to be handled in ``__iter__`` to make it avoid resetting the just-restored state (only once). rzLCutPairsSampler.load_state_dict(): Inconsistent source_constraint: expected z received z= We will overwrite the settings with the received state_dict.rzLCutPairsSampler.load_state_dict(): Inconsistent target_constraint: expected Nr)r poprwarningswarnrrload_state_dictrr repochr fast_forward diagnosticscurrent_epoch_stats total_cuts)rr(rrrrrr-rs4   zCutPairsSampler.load_state_dictcCsZ|jr|S|j|jr!|j|j|j|j|j|jt|jt|j|S)zi Prepare the dataset for iterating over a new epoch. Will shuffle the data if requested. ) _just_restored_stater0reset_current_epochrr rr.r iterr#rrr__iter__s   zCutPairsSampler.__iter__cCs|j|jg}g} zt|j}t|j}|j|jks$JdWn7ty\|rS|jr;|j s;|j rSt |t |ksGJdt |t |fYS|j |tw||rg||sn|j |q|j||j||js|js||||n|r|j||j|ntd||||qt |t |ksJdt |t |fS)NTzSampled source and target cuts with differing IDs. Ensure that your source and target cuts have the same length, the same IDs, and the same order.zPUnexpected state: some cuts in source / target are missing their counterparts...zThe first cut drawn in batch collection violates one of the max_... constraintswe'll return it anyway. Consider increasing max_source_duration/max_cuts/etc.)rresetrnextr r id StopIterationrclose_to_exceedingr&r from_cutsr0discard _filter_fndiscard_singleaddexceededappend take_backr+r,)rr r next_source_cutnext_target_cutrrr _next_batchsr               CzCutPairsSampler._next_batch)NNNFFNNr)r r )__name__ __module__ __qualname____doc__rrrintboolrpropertyfloatr"r$r'rstrrr(r-r6rrF __classcell__rrrrr sR   2   /r )r+typingrrrrlhotserrlhotse.dataset.sampling.baserr #lhotse.dataset.sampling.data_sourcer r rrrrs