o 2wia@sddlZddlmZddlmZddlmZddlmZm Z m Z m Z m Z ddl ZddlZddlmZddlmZmZddlmZdd lmZdd lmZmZdd lmZmZmZGd d d Z   d9dede!de ede ej"de ej#ej#ff ddZ$    d:dede!de ede%de e!de e ej#ej#fe ej#ej#efff ddZ&e&Z'   d;dede!de ede%fddZ(  ddeede ed*e%de e!d+e ede e e ej#efe e ej#ee fff d,d-Z1  d?deede ed*e%de e ej#e ej#effd.d/Z2 d@deede ede ej#fd0d1Z3 dAd2ed*e%de e!de ej#fd3d4Z4d2edej#fd5d6Z5 dBd2ed*e%de e ej#e ej#ffd7d8Z6dS)CN)Executor)partial)repeat)IterableListOptionalTupleUnion)CrossEntropyLoss)CutSet Recording)suppress_audio_loading_errors)suppress_video_loading_errors)CutMixedCut)DEFAULT_PADDING_VALUESecondscompute_num_samplesc@seZdZdZ      ddeded ed ed ed ed efddZdedee j e j ffddZ de j de j deefddZdS) TokenCollateraCollate list of tokens Map sentences to integers. Sentences are padded to equal length. Beginning and end-of-sequence symbols can be added. Call .inverse(tokens_batch, tokens_lens) to reconstruct batch as string sentences. Example: >>> token_collater = TokenCollater(cuts) >>> tokens_batch, tokens_lens = token_collater(cuts.subset(first=32)) >>> original_sentences = token_collater.inverse(tokens_batch, tokens_lens) Returns: tokens_batch: IntTensor of shape (B, L) B: batch dimension, number of input sentences L: length of the longest sentence tokens_lens: IntTensor of shape (B,) Length of each sentence after adding and but before padding. Tcutsadd_eosadd_bos pad_symbol bos_symbol eos_symbol unk_symbolc Cs||_||_||_||_||_||_dd|D}||g|r!|gng|r(|gngt|} ddt| D|_dd| D|_ dS)NcSs"h|] }|jdjD]}|q qSr) supervisionstext).0cutcharr&U/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/lhotse/dataset/collation.py 9s"z)TokenCollater.__init__..cSsi|]\}}||qSr&r&)r#idxtokenr&r&r' Az*TokenCollater.__init__..cSsg|]}|qSr&r&r#r*r&r&r' Bsz*TokenCollater.__init__..) rrrrrrsorted enumerate token2idx idx2token) selfrrrrrrrtokens tokens_uniquer&r&r'__init__'s"   zTokenCollater.__init__returncsvdd|D}tt|tdfdd|D}ttjfdd|Dtjd}tfdd|D}||fS)NcSs"g|] }ddd|jDqS) cs|]}|jVqdSN)r")r# supervisionr&r&r' Fz4TokenCollater.__call__...)joinr!r#r$r&r&r'r.Esz*TokenCollater.__call__..keycsLg|]"}jr jgngt|jrjgngjgt|qSr&)rrlistrrrlenr#seqmax_lenr3r&r'r.Kscsg|] }fdd|DqS)cg|]}j|qSr&)r1r-r3r&r'r.Ur,z5TokenCollater.__call__...r&rDrIr&r'r.Usdtypecs(g|]}t|tjtjqSr&)rCintrrrDrIr&r'r.[s)rCmaxtorch from_numpynparrayint64 IntTensor)r3rtoken_sequencesseqs tokens_batch tokens_lensr&rFr'__call__Ds&  zTokenCollater.__call__rVrWcs,jrdndfddt||D}|S)Nrc s:g|]\}}dfdd||tjDqS)crHr&)r2)r#r)rIr&r'r.isz4TokenCollater.inverse...)r>rLr)r# tokens_listendr3startr&r'r.gs z)TokenCollater.inverse..)rzip)r3rVrW sentencesr&r]r'inversecs   zTokenCollater.inverseN)TTrrrr)__name__ __module__ __qualname____doc__r boolstrr6rrNTensorrX LongTensorrSrrar&r&r&r'rs> rrightr pad_directionexecutorfeatures_dtyper7c Cstdd|Ds Jtjdd|Dtjd}|jt||d}tt|}tj t ||j |j |d}|durMt |D] \}}t|||<q>||fSt |t|D]\}} | ||<qU||fS)a1 Load features for all the cuts and return them as a batch in a torch tensor. The output shape is ``(batch, time, features)``. The cuts will be padded with silence if necessary. :param cuts: a :class:`CutSet` used to load the features. :param pad_direction: where to apply the padding (``right``, ``left``, or ``both``). :param executor: an instance of ThreadPoolExecutor or ProcessPoolExecutor; when provided, we will use it to read the features concurrently. :return: a tuple of tensors ``(features, features_lens)``. csr9r: has_featuresr?r&r&r'r<r=z#collate_features..cSsg|]}|jqSr&) num_framesr?r&r&r'r.sz$collate_features..rJ)rp directionN)allrNtensorrLpadrMitemnextiteremptyrCrp num_featuresr0_read_featuresmap) rrkrlrm features_lens first_cutfeaturesr)r$example_featuresr&r&r'collate_featuresss  rFfault_tolerantrecording_fieldc Cs"|D]"}|dur|jsJd|jq||s$Jd|d|jqg}|D]}|dur3|j}n t|jt||jd}||q)|j t dd|D|dd }t |||||d \}}}t |d j d krot|d d}ntdd|Dd dd d}tj|tjd} |r|| |fS|| fS)a Load audio samples for all the cuts and return them as a batch in a torch tensor. The output shape is ``(batch, time)``. The cuts will be padded with silence if necessary. :param cuts: a :class:`CutSet` used to load the audio samples. :param pad_direction: where to apply the padding (``right``, ``left``, or ``both``). :param executor: an instance of ThreadPoolExecutor or ProcessPoolExecutor; when provided, we will use it to read audio concurrently. :param fault_tolerant: when ``True``, the cuts for which audio loading failed will be skipped. Setting this parameter will cause the function to return a 3-tuple, where the third element is a CutSet for which the audio data were sucessfully read. :param recording_field: when specified, we will try to load recordings from a custom field with this name (i.e., ``cut.load_()`` instead of default ``cut.load_audio()``). :return: a tuple of tensors ``(audio, audio_lens)``, or ``(audio, audio_lens, cuts)``. NzMissing recording in cut zMissing custom recording field z in cut ) sampling_ratecsr9r:durationr?r&r&r'r<r=z collate_audio..Trrq preserve_id)suppress_errorsrfilter_aux_iterrrYg) padding_valuecSsg|]}|ddqS)rrY) transposer#ar&r&r'r.sz!collate_audio..rJ) has_recordingid has_custom num_samplesrrgetattrrappendrtrMread_audio_from_cutsrCshapecollate_vectorscollate_matricesrrNrsint32) rrkrlrrr$ sample_countsraudios audio_lensr&r&r' collate_audiosN   rc stdd|Ds Ji|D] }|j|jjf|j<q|jtdd|D|dd}t|||d\}}}t |}t |}tj fdd|Dtj d }tj fd d|Dtj d }|rf|||||fS||||fS) aw Load video and audio for all cuts and return them as a batch in torch tensors. The output video shape is ``(batch, time, channel, height, width)``. The output audio shape is ``(batch, channel, time)``. The cuts will be padded with silence if necessary. .. note:: We expect each video to contain audio and the same number of audio channels. We may support padding missing channels at a later time. :param cuts: a :class:`CutSet` used to load the audio samples. :param pad_direction: where to apply the padding (``right``, ``left``, or ``both``). :param executor: an instance of ThreadPoolExecutor or ProcessPoolExecutor; when provided, we will use it to read video concurrently. :param fault_tolerant: when ``True``, the cuts for which video/audio loading failed will be skipped. Setting this parameter will cause the function to return a 5-tuple, where the fifth element is a CutSet for which the audio data were sucessfully read. :return: a tuple of tensors ``(video, video_lens, audio, audio_lens)``, or ``(video, video_lens, audio, audio_lens, cuts)``. csr9r:) has_videor?r&r&r'r<r=z collate_video..csr9r:rr#cr&r&r'r<r=Trrcg|] }|jdqSr rr?id2lensr&r'r. z!collate_video..rJcr)rYrr?rr&r'r.r) rrrvideorprrtrMread_video_from_cutsrNstackrsr) rrkrlrr$videosrr video_lensr&rr' collate_videos&   rfield pad_valuecsddlm}m}t|dt|r0tfdd|Ds$Jdtfdd|DSt|r|durGt d d t d t }j fd d|D}tj fd d|Dtj d}t|tjd}|j} t|g|jR} |jtfddtjtjtjtj fDrtj|tj| d} t|D]Q\} } | j}|dkrtd|n&|dkrt| || n|dkr| |d}t|| |ntd|d| ftfddtt| jD}| | |<q| |fSt fdd|DS)a Load custom arrays for all the cuts and return them as a batch in a torch tensor. The output shapes are: - ``(batch, d0, d1, d2, ...)`` for :class:`lhotse.array.Array` of shape ``(d0, d1, d2, ...)``. Note: all arrays have to be of the same shape, as we expect these represent fixed-size embeddings. - ``(batch, d0, pad_dt, d1, ...)`` for :class:`lhotse.array.TemporalArray` of shape ``(d0, dt, d1, ...)`` where ``dt`` indicates temporal dimension (variable-sized), and ``pad_dt`` indicates temporal dimension after padding (equal-sized for all cuts). We expect these represent temporal data, such as alignments, posteriors, features, etc. - ``(batch, )`` for anything else, such as int or float: we will simply stack them into a list and tensorize it. .. note:: This function disregards the ``frame_shift`` attribute of :class:`lhotse.array.TemporalArray` when padding; it simply pads all the arrays to the longest one found in the mini-batch. Because of that, the function will work correctly even if the user supplied inconsistent meta-data. .. note:: Temporal arrays of integer type that are smaller than torch.int64, will be automatically promoted to torch.int64. :param cuts: a :class:`CutSet` used to load the features. :param field: name of the custom field to be retrieved. :param pad_value: value to be used for padding the temporal arrays. Ignored for non-temporal array and non-array attributes. :param pad_direction: where to apply the padding (``right``, ``left``, or ``both``). :return: a collated data tensor, or a tuple of tensors ``(collated_data, sequence_lens)``. r)Array TemporalArrayc3s"|] }t|jjkVqdSr:)rrr)rfirst_manifestr&r'r<As z'collate_custom_field..zCannot collate manifests of type Array with different shapes, because we don't know which dimension must be padded. Use TemporalArray manifests and try again.cg|] }t|qSr&rNrO load_customrrr&r'r.Fz(collate_custom_field..Nz6Argument 'pad_value' not passed -- we will pad field 'z' with .crr&rrrr&r'r.Yrcsg|]}|jqSr&rr) temporal_dimr&r'r.[r,rJr@c3s|]}|kVqdSr:r&)r#drJr&r'r<asrjleftbothrz$Unexpected pad_direction argument: ''c3s(|]}|kr ntdddVqdSr:)slice)r#i)rtemporal_slicer&r'r<ss  csg|]}t|qSr&)rrrr&r'r.}r,) lhotse.arrayrrr isinstancerrrNrwarningswarnrrrsrrMnumelrrCrKanyuint8int8int16rRonesr0r ValueErrortuplerange)rrrrkrrarrsarr_lens largest_arrmaxlencollated_shapetensorsaidxralenhalfindicesr&)rKrrrrr'collate_custom_fieldsZ%    &      rcCstdd|Ds Jtdd|DsJ||}tt|}tt|t|j|j|j }t |D]\}}t |j dd||<q4|S)a5 Load features for all the cuts and return them as a batch in a torch tensor. The cuts have to be of type ``MixedCut`` and their tracks will be interpreted as individual channels. The output shape is ``(batch, channel, time, features)``. The cuts will be padded with silence if necessary. csr9r:rnr?r&r&r'r<r=z1collate_multi_channel_features..css|]}t|tVqdSr:)rrr?r&r&r'r<sF)mixed) rrrtrvrwrNrxrCtracksrpryr0rO load_features)rr}r~r)r$r&r&r'collate_multi_channel_featuress  rrrmatching_shapescsdd|D}tdd|DsJdt|ddd|r-tfd d|Ds-Jd t|jd |}t|D]\}}|||d |jd f<q>|S) a Convert an iterable of 1-D tensors (of possibly various lengths) into a single stacked tensor. :param tensors: an iterable of 1-D tensors. :param padding_value: the padding value inserted to make all tensors have the same length. :param matching_shapes: when ``True``, will fail when input tensors have different shapes. :return: a tensor with shape ``(B, L)`` where ``B`` is the number of input tensors and ``L`` is the number of items in the longest tensor. cS&g|]}t|tjr |nt|qSr&rrNrhrOr#tr&r&r'r.z#collate_vectors..cs|] }t|jdkVqdS)rYNrCrrr&r&r'r<z"collate_vectors..z Expected only 1-D input tensors.cS |jdSNrrrr&r&r' z!collate_vectors..r@c3|] }|jjkVqdSr:rrlongestr&r'r<  IAll tensors must have the same shape when matching_shapes is set to True.rNrrrMnew_onesrCrr0rrrresultrrr&rr'rs  rcsdd|D}tdd|DsJdt|ddd|r-tfd d|Ds-Jd jt|gjR|}t|D]\}}|||d |jd f<q?|S) a, Convert an iterable of 2-D tensors (of possibly various first dimension, but consistent second dimension) into a single stacked tensor. :param tensors: an iterable of 2-D tensors. :param padding_value: the padding value inserted to make all tensors have the same length. :param matching_shapes: when ``True``, will fail when input tensors have different shapes. :return: a tensor with shape ``(B, L, F)`` where ``B`` is the number of input tensors, ``L`` is the largest found shape[0], and ``F`` is equal to shape[1]. cSrr&rrr&r&r'r.rz$collate_matrices..csr)rNrrr&r&r'r<rz#collate_matrices..z Expected only 2-D input tensors.cSrrrrr&r&r'rrz"collate_matrices..r@c3rr:rrrr&r'r<rrNrrrr&rr'rs  rrrc Csd}|dur tdg}d}|durtn|j}g}g}g} tt||tt||d||D]\} \} } } | dur8q,|| || | | q,|t|f}|rV|| f}|S)a Loads audio data from an iterable of cuts. :param cuts: a CutSet or iterable of cuts. :param executor: optional Executor (e.g., ThreadPoolExecutor or ProcessPoolExecutor) to perform the audio reads in parallel. :param suppress_errors: when set to ``True``, will enable fault-tolerant data reads; we will skip the cuts and audio data for the instances that failed (and emit a warning). When ``False`` (default), the errors will not be suppressed. :param recording_field: when specified, we will try to load recordings from a custom field with this name (i.e., ``cut.load_()`` instead of default ``cut.load_audio()``). :param filter_aux_iter: when specified, we will iterate over this iterator and discard the elements for which a corresponding cut failed to load audio, if ``suppress_errors`` is set to ``True``. This iterator is expected to be of the same length as ``cuts``. :return: a tuple of two items: a list of audio tensors (with different shapes), and a list of cuts for which we read the data successfully. If ``filter_aux_iter`` is specified, it returns a 3-tuple where the third element is the filtered auxiliary iterator. TNF)rr) rr{r0r_r _read_audiorr from_cuts)rrlrrr aux_requestedmap_fnrok_cuts aux_iter_outr)r$ maybe_audioaux_itemansr&r&r'rs>     rc Cs|durtn|j}g}g}g}tt||tt|d|D]\}\}} | dur(q| \} } || || ||q||t|fS)a Loads audio data from an iterable of cuts. :param cuts: a CutSet or iterable of cuts. :param executor: optional Executor (e.g., ThreadPoolExecutor or ProcessPoolExecutor) to perform the audio reads in parallel. :param suppress_errors: when set to ``True``, will enable fault-tolerant data reads; we will skip the cuts and audio data for the instances that failed (and emit a warning). When ``False`` (default), the errors will not be suppressed. :return: a tuple of two items: a list of audio tensors (with different shapes), and a list of cuts for which we read the data successfully. Nr)r{r0r_r _read_videorr r) rrlrrrrrr)r$ maybe_ansraudior&r&r'rs.    rcCs |durtn|j}t|t|Sr:)r{rBrz)rrlrr&r&r'read_features_from_cuts>srr$cCst|d=|dur|}nt||}t|ts%Jd|dt|||}|jddkr6|d}t |WdS1sEwYdS)z{ Loads audio data from cut, or returns None if there was an error and ``suppress_errors`` was set to ``True``. enabledNzExpected 'getattr(cut, z)' to yield Recording, got rrY) r load_audiorrr typerrsqueezerNrO)r$rrrattrr&r&r'rEs     $rcCst|Sr:)rNrOr)r$r&r&r'rzZsrzcCs6t|d |WdS1swYdS)z Loads video + audio data from cut, or returns None if there was an error and ``suppress_errors`` was set to ``True``. rN)r load_video)r$rr&r&r'r^s $r)rjNN)rjNFN)rjNF)Nrj)rF)NFNN)NFr:)FN)F)7rconcurrent.futuresr functoolsr itertoolsrtypingrrrrr numpyrPrNtorch.nnr lhotser r lhotse.audior lhotse.audio.utilsr lhotse.cutrr lhotse.utilsrrrrrgrKrhrrfrcollate_multi_channel_audiorrLfloatrr ignore_indexndarrayrrrrrrrzrr&r&r&r's0      c #  I 9  j     ( > ,