o wiE @s`ddlZddlmZddlmZddlmZmZddlZ ddl m Z ddl m Z mZddlmZddlmZdd lmZmZeGd d d e ZeGd d d eZ  d'de jdeeeBdedeedBdedBf ddZeGdddeZGdddZGdddZGdddZ GdddZ!Gd d!d!Z"d"e defd#d$Z#d"e defd%d&Z$dS)(N) bisect_left) dataclass)AnySequence)Cut)SamplingConstraintTokenConstraint)FixedBucketBatchSizeConstraint)ifnone) FormattableNeMoMultimodalConversationc@seZdZUdZdZedBed<dZedBed<dZ edBed<dZ edBed<dZ e ed<dZ d d Zd ed dfd dZd e fddZd e fddZdddZd ed efddZdS)MultimodalSamplingConstraintz Sampling strategy that customizes Lhotse samplers to measure sequence lengths as token counts. It provides a unified interface for audio and text examples - audio duration is converted to an equivalent token count. Ntoken_equivalent_duration batch_size batch_tokensquadratic_factorFmeasure_total_lengthcCst|j|j|jd|_dS)N) max_tokens max_examplesquadratic_length)rrrr _internalselfri/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/nemo/collections/common/data/lhotse/sampling.py __post_init__>s  z*MultimodalSamplingConstraint.__post_init__examplereturncCs ||}||_|j|dSN)measure_length num_tokensradd)rrr rrrr!Es z MultimodalSamplingConstraint.addcC |jSr)rexceededrrrrr#J z%MultimodalSamplingConstraint.exceededcCr"r)rclose_to_exceedingrrrrr%Mr$z/MultimodalSamplingConstraint.close_to_exceedingcCs|jdSr)rresetrrrrr&Psz"MultimodalSamplingConstraint.resetc Cst|tr+t|j|j}|jr)d}|jD]}|dr$|t |j 7}q||S|St|t rOz |jr8|j WS|j WSttfyN}ztd|d}~wwtdt|)NrtokenszCouldn't determine the length of a text example; have you provided both prompt_format and tokenizer when instantiating the dataloader?Unsupported example type: ) isinstancermathceildurationrr supervisions has_customlenr'r total_length input_lengthAttributeErrorAssertionError RuntimeErrortype)rraudio_len_in_tokens text_tokensserrrrSs,    z+MultimodalSamplingConstraint.measure_length)rN)__name__ __module__ __qualname____doc__rfloat__annotations__rintrrrboolrrrr!r#r%r&rrrrrr s   r c @seZdZUdZdZeed<dZee dBed<ddZ e defd d Z d e dee e fe Bfd d Zdded ededefddZdS) FixedBucketBatchSizeConstraint2Da Sampling strategy that customizes Lhotse samplers to support 2D bucket selection (it also supports 1D). It is intended only for audio examples (i.e., Lhotse Cut objects). When ``strict_2d`` is set, we only consider sub-buckets for a single bucket that is the best match. When set to ``False``, we'll promote an example to buckets with larger 1st dim if they can accommodate the 2nd dim. When ``max_ratio`` is set, it discards the examples that exceed a specific output-to-input length ratio. ``max_ratio`` must be a list with the same length as the number of buckets. ``max_ratio`` is only applied when ``strict_2d`` is set to ``True``. T strict_2dN max_ratiocCst|jdtrt|j|_|jdurJdt|jdt|jdSdS)Nrz+self.max_ratio must be a list, but we got: zlen(self.max_ratio)=z" != len(self.max_seq_len_buckets)=)r)max_seq_len_bucketsrnpasarrayrDr/rrrrr}s  z.FixedBucketBatchSizeConstraint2D.__post_init__rcCst|jtjSr)r)rErFndarrayrrrrbucketing_2d_enabledsz5FixedBucketBatchSizeConstraint2D.bucketing_2d_enabledrcCs|jr |jt|fS|jSr)rIr,_measure_tokensrrrrrrsz/FixedBucketBatchSizeConstraint2D.measure_lengthbuckets example_lencCs(|dur ||}t|j||j|jdS)N)strictrD)rfind_smallest_bucketrErCrD)rrLrrMrrr select_buckets  z.FixedBucketBatchSizeConstraint2D.select_bucket)NN)r:r;r<r=rCrAr?rDlistr>rpropertyrIrtuplerrr@rPrrrrrBls    rBTrL example_lensrNrDrc CsHt|ttfrt||}|t|krdS|S|rt|dddf|d}||jdkr.dS|}||jdkrZ||df||dfkrZ|d7}||jdkrZ||df||dfksCt|||df|d}|||krodS||}|dur|d|d||krdS|Stjt||kdd} t | } | s| | r| SdS)a[ Find the smallest bucket that fits a given example. Each bucket and ``example_lens`` are floats (1-D bucketing) or tuples of (dim0, dim1, dim2, ...) (N-D bucketing, typically 2-D). Assumes the buckets have been sorted ascendingly. Returns a tuple of (smallest_bin, bin_idx), or (None, None) if no bucket fits the example. Nr)axis) r)r>r@rr/shaperFallrGargmaxitem) rLrTrNrDidx dim0_begindim0_end dim1_beginfit_idxdoes_fit min_fit_idxrrrrOs2  &&    rOc@sLeZdZUdZdZedBed<dZeed<de dee eefBfdd Z dS) *MultimodalFixedBucketBatchSizeConstraint2Dz Sampling strategy that customizes Lhotse samplers to support both multimodal sampling and 2D bucket selection. It combines the capabilities of :class:`FixedBucketBatchSizeConstraint2D` and :class:`MultimodalSamplingConstraint` NrFrrrcCst|tr"t|j|j}t|}|jr||fS|jr ||S|St|t r9|jr0|j |j fS|jr6|j S|j St dt|)Nr()r)rr*r+r,rrJrIrr r1 output_lengthr0r4r5)rrr6r7rrrrs   z9MultimodalFixedBucketBatchSizeConstraint2D.measure_length) r:r;r<r=rr>r?rrArrSrrrrrrbs  "rbc@<eZdZdZdedBdedBddfddZdefdd ZdS) DurationFilterz Callable, returns ``True`` if a cut's duration is in range [d_min, d_max] and ``False`` otherwise. Acts as a pass-through for objects of other type than Cut. d_minNd_maxrcCs t|d|_t|td|_dS)Ninf)r rfr>rg)rrfrgrrr__init__s zDurationFilter.__init__cCsrt|tr|j|jko|jkSSt|tr7|jrdStdd|D}|j|ko4|jkSSdS)NTcss|]}|jVqdSr)r,).0crrr sz*DurationFilter.__call__..) r)rrfr,rgr is_text_onlysum list_cuts)rrtot_durrrr__call__s  zDurationFilter.__call__r:r;r<r=r>rjrArrrrrrresrec@s@eZdZdZdedBdedBdeddfddZdefd d ZdS) TokenCountFiltera Callable, returns ``True`` if an example's number of tokens is in range [t_min, t_max] and ``False`` otherwise. It is only applicable to data types that derive from class ``Formattable`` and lhotse ``Cut`` objects. Acts as a passthrough for Cuts. Raises exception if a non-Formattable and non-Cut data are provided. The ``measure_total_length`` option allows to select whether we should filter on context_ids length (=False) or input_ids length (=True). The difference is that for decoder-only models, we collapse input and output into a single sequence, so we should measure the example length using input_ids (measure_total_length=True). However, for models which have separate inputs and outputs such as encoder-decoder models, we want to measure the input lengths only here (measure_total_length=False), and enable ``TokenPerTokenFilter`` for additional filtering on the output sequence length. t_minNt_maxrrcCs@t|d|_t|td|_||_|jdkp|jtdk|_dS)Nrhrir)r rur>rvrenabled)rrurvrrrrrj s zTokenCountFilter.__init__c Cs|jrt|tr dSt|tsJd|z |jr|jn|j}Wnttfy8}z t d|d|d}~ww|j |koD|j kSS)NTzTokenCountFilter can only be applied to data examples that derive Formattable class. Formattable objects define properties input_length, output_length, and total_length that allow us to select the right sequence length for filtering. We got: z(Cannot measure token count for example: z -- did you forget to apply prompt formatting? If instantiating Lhotse dataloader, make sure you provided 'prompt_format' option and passed the tokenizer.) rwr)rr rr0r1r2r3r4rurv)rrlengthr9rrrrr&s$  zTokenCountFilter.__call__)r:r;r<r=r>rArjrrrrrrrts"rtc@rd) TokenPerSecondFilterz Callable, returns ``True`` if a cut's num_tokens (sum of len(tokens) for each supervision) is in range [tps_min, tps_max] and ``False`` otherwise. Acts as a pass-through for objects of other type than Cut. tps_minNtps_maxrcCdt|d|_t|trtd}t|td|_||ks%Jd|d||dkp.|tdk|_dS)Nrhriztps_min=z tps_max=r)r rzr)rr>r{rw)rrzr{rrrrj@  zTokenPerSecondFilter.__init__cCs8t|tr|js dSt|}|j|ko|jkSSNT)r)rrw _measure_tpsrzr{)rrtpsrrrrrHszTokenPerSecondFilter.__call__rsrrrrry9ryc@rd) TokenPerTokenFilterz Callable, returns ``True`` if a cut's num_tokens (sum of len(tokens) for each supervision) is in range [tps_min, tps_max] and ``False`` otherwise. Acts as a pass-through for audio examples (Cuts). tpt_minNtpt_maxrcCr|)Nrhriztpt_min=z tpt_max=r)r rr)rr>rrw)rrrrrrrjVr}zTokenPerTokenFilter.__init__cCsHt|ts|js dS|jjd|jjd}|j|ko!|jkSS)NTr)r)rrw answer_idsrW context_idsrr)rrtptrrrrr^szTokenPerTokenFilter.__call__rsrrrrrOrrc@s0eZdZdZdeddfddZdefddZdS) BucketingFilterz Filters out examples that did not fit into any of the buckets. Intended mainly for 2D bucketing. This filter is only active when the constraint passed to it is of type ``FixedBucketBatchSizeConstraint2D``, and is otherwise disabled. sampling_constraintrNcCs||_t|jt|_dSr) constraintr)rBrw)rrrrrrjmszBucketingFilter.__init__cCs |jsdS|j|jj|duSr~)rwrrPrErKrrrrrqszBucketingFilter.__call__)r:r;r<r=rrjrArrrrrrresrcutcCsJt|dr t|jSdd|jD}t|dksJdtdd|DS)N input_idscSsg|] }t|dr|qS)r')hasattrrkr8rrr zsz#_measure_tokens..rzCannot measure the number of tokens with untokenized supervisions. Did you forget to provide the tokenizer argument to get_lhotse_dataloader_from_config() method?css|]}t|jVqdSr)r/r'rrrrrmsz"_measure_tokens..)rr/rr-ro)rsupervisions_with_tokensrrrrJws  rJcCst|}||jSr)rJr,)rr rrrrs r)TN)%r*bisectr dataclassesrtypingrrnumpyrF lhotse.cutrlhotse.datasetrr)lhotse.dataset.sampling.dynamic_bucketingr lhotse.utilsr 1nemo.collections.common.data.lhotse.text_adaptersr r r rBrHr>rAr@rOrbrertryrrrJrrrrrsH     N.   7)*