o Si @sddlZddlmZddlmZmZddlZddlm Z ddl m Z  dde de deee e fee efffd d Zdde d e de fd dZdS)N)mean)DictTuple)CutSet) CutSamplersamplerbatch_tuple_indexreturnc sddddddddtddd}zt|}t|ttr%|Wnty8tdiifYSwfd d |D}fd d |D}|D]&}t|trZ||}|D]\}}||}|||krt|||<|||<q^qO||fS) a Function for finding 'pessimistic' batches, i.e. batches that have the highest potential to blow up the GPU memory during training. We will fully iterate the sampler and record the most risky batches under several criteria: - single longest cut - single longest supervision - largest batch cuts duration - largest batch supervisions duration - max num cuts - max num supervisions .. note: It is up to the users to convert the sampled CutSets into actual batches and test them by running forward and backward passes with their model. Example of how this function can be used with a PyTorch model and a :class:`~lhotse.dataset.K2SpeechRecognitionDataset`:: sampler = SimpleCutSampler(cuts, max_duration=300) dataset = K2SpeechRecognitionDataset() batches, scores = find_pessimistic_batches(sampler) for reason, cuts in batches.items(): try: batch = dset[cuts] outputs = model(batch) loss = loss_fn(outputs) loss.backward() except: print(f"Exception caught when evaluating pessimistic batch for: {reason}={scores[reason]}") raise :param sampler: An instance of a Lhotse :class:`.CutSampler`. :param batch_tuple_index: Applicable to samplers that return tuples of :class:`~lhotse.cut.CutSet`. Indicates which position in the tuple we should look up for the CutSet. :return: A tuple of dicts: the first with batches (as CutSets) and the other with criteria values, i.e.: ``({"": , ...}, {"": , ...})`` cStdd|DS)Ncs|]}|jVqdSNduration.0crQ/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/dataset/sampling/utils.py 4=find_pessimistic_batches....maxcutsrrr4z*find_pessimistic_batches..cSr )Ncss$|] }tdd|jDVqdS)csr r r rsrrrr6rzGfind_pessimistic_batches.....N)sum supervisionsrrrrr5s rrrrrrr5cSr )Ncsr r r rrrrr8rrrrrrrr8rcSr )Ncss"|] }|jD]}|jVqqdSr )r r)rrrrrrr9s  rr"rrrrr9r!cSr )Ncss |] }|jD]}dVqqdS)N)r )rr_rrrr=s  rr"rrrrr=r!)single_longest_cutsingle_longest_supervisionlargest_batch_cuts_duration#largest_batch_supervisions_duration max_num_cutsmax_num_supervisionsz7Empty sampler encountered in find_pessimistic_batches()csi|]}|qSrr)rk first_batchrr Jrz,find_pessimistic_batches..csi|] \}}||qSrr)rr+fnr,rrr.Ks) leniternext isinstancetuple StopIterationwarningswarnitems) rrcriteria top_batches top_valuesbatchcritr/valrr,rfind_pessimistic_batches s<)      r? n_samplesc sg}g}g}g}g}g}g}g} g} t|}t|D]} zt|Wn ty,Ynwtts;tddjddt dkr| dj dj dj | dj t dj dj | t fddtdt Ddj  d} d} d}D]2}| |j | tdd |jD| |d |d | |d 7} | |d 7} ||d 7}q| | | | | |qt |}t |}t |}t |}t |}t |}d gd |d dt|d d|d dt|d d|d dt|d d||dd|d dt|d d|d dt|d d|d dt|d d||ddt | ddt |ddt | ddS)z Returns a human-readable string message about amount of padding diagnostics. Assumes that padding corresponds to segments without any supervision within cuts. zThe sampler returned a mini-batch with multiple CutSets: we will only report the padding estimate for the first CutSet in each mini-batch.rF) ascendingr#cs g|] }dj|jqSrr )rir<rr sz1report_padding_ratio_estimate..csr r r rrrrrrz0report_padding_ratio_estimate..zAn average CUT has z.1fzs (std=zs) of supervisions vs. z)s) of total duration. Average padding is z s), i.e. z.1%z. An average BATCH has z's) of combined supervised duration vs. z2s) of combined total duration. Average padding is zD. Expected variability of cut durations within a single batch is +/-z (two closest cuts: z, two most distant cuts: z). )r1ranger2r5r3rr6r7sort_by_durationr0appendrrpadrr npjoinstd)rrA supervisedtotalgapsbatch_supervised batch_total batch_gaps min_dur_diffsmean_dur_diffs max_dur_diffsrD batch_sup batch_tot batch_gapcut m_supervisedm_totalm_gapsm_batch_supervised m_batch_total m_batch_gapsrrErreport_padding_ratio_estimateYs      $               t         rcrC)r@)r6 statisticsrtypingrrnumpyrMlhotserlhotse.dataset.sampling.baserintstrfloatr?rcrrrrs    N