o SiHn@sddlZddlmZddlmZddlmZddlmZm Z m Z m Z m Z ddl ZddlZddlmZddlmZmZddlmZdd lmZdd lmZmZdd lmZmZGd d d Z   d?dede de ede ej!de ej"ej"ff ddZ#    d@dede de ede$de e de e ej"ej"fe ej"ej"efff ddZ%e%Z&     dAdede$de de ede$de e f ddZ'  dBdede d e de(e)fde de ej"e ej"ej"fff d!d"Z*dedej"fd#d$Z+ej,ddfd%ee ej"ej-fd&e e(e)fde d'e$dej"f d(d)Z. dCd%ee ej"ej-fd&e e(e)fd'e$dej"fd*d+Z/    dDdeede ed,e$de e d-e ede e e ej"efe e ej"ee fff d.d/Z0    dEdeede$de ed,e$de e de e ej"e ej"eff d0d1Z1 dFdeede ede ej"fd2d3Z2 dGd4ed,e$de e de ej"fd5d6Z3d4edej"fd7d8Z4   dHd4ede$d,e$de e de e ej"e ej"ff d9d:Z5 ;dIdedZ6dS)JN)Executor)partial)repeat)IterableListOptionalTupleUnion)CrossEntropyLoss)CutSet Recording)suppress_audio_loading_errors)suppress_video_loading_errors)CutMixedCut)DEFAULT_PADDING_VALUEcompute_num_samplesc@seZdZdZ      ddeded ed ed ed ed efddZdedee j e j ffddZ de j de j deefddZdS) TokenCollateraCollate list of tokens Map sentences to integers. Sentences are padded to equal length. Beginning and end-of-sequence symbols can be added. Call .inverse(tokens_batch, tokens_lens) to reconstruct batch as string sentences. Example: >>> token_collater = TokenCollater(cuts) >>> tokens_batch, tokens_lens = token_collater(cuts.subset(first=32)) >>> original_sentences = token_collater.inverse(tokens_batch, tokens_lens) Returns: tokens_batch: IntTensor of shape (B, L) B: batch dimension, number of input sentences L: length of the longest sentence tokens_lens: IntTensor of shape (B,) Length of each sentence after adding and but before padding. Tcutsadd_eosadd_bos pad_symbol bos_symbol eos_symbol unk_symbolc Cs||_||_||_||_||_||_dd|D}||g|r!|gng|r(|gngt|} ddt| D|_dd| D|_ dS)NcSs"h|] }|jdjD]}|q qSr) supervisionstext).0cutcharr%L/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/dataset/collation.py 9s"z)TokenCollater.__init__..cSsi|]\}}||qSr%r%)r"idxtokenr%r%r& Az*TokenCollater.__init__..cSsg|]}|qSr%r%r"r)r%r%r& Bsz*TokenCollater.__init__..) rrrrrrsorted enumerate token2idx idx2token) selfrrrrrrrtokens tokens_uniquer%r%r&__init__'s"   zTokenCollater.__init__returncsvdd|D}tt|tdfdd|D}ttjfdd|Dtjd}tfdd|D}||fS)NcSs"g|] }ddd|jDqS) cs|]}|jVqdSN)r!)r" supervisionr%r%r& Fz4TokenCollater.__call__...)joinr r"r#r%r%r&r-Esz*TokenCollater.__call__..keycsLg|]"}jr jgngt|jrjgngjgt|qSr%)rrlistrrrlenr"seqmax_lenr2r%r&r-Kscsg|] }fdd|DqS)cg|]}j|qSr%)r0r,r2r%r&r-Ur+z5TokenCollater.__call__...r%rCrHr%r&r-Usdtypecs(g|]}t|tjtjqSr%)rBintrrrCrHr%r&r-[s)rBmaxtorch from_numpynparrayint64 IntTensor)r2rtoken_sequencesseqs tokens_batch tokens_lensr%rEr&__call__Ds&  zTokenCollater.__call__rUrVcs,jrdndfddt||D}|S)Nrc s:g|]\}}dfdd||tjDqS)crGr%)r1)r"r(rHr%r&r-isz4TokenCollater.inverse...)r=rKr)r" tokens_listendr2startr%r&r-gs z)TokenCollater.inverse..)rzip)r2rUrV sentencesr%r\r&inversecs   zTokenCollater.inverseN)TTrrrr)__name__ __module__ __qualname____doc__r boolstrr5rrMTensorrW LongTensorrRrr`r%r%r%r&rs> rrightr pad_directionexecutorfeatures_dtyper6c Cstdd|Ds Jtjdd|Dtjd}|jt||d}tt|}tj t ||j |j |d}|durMt |D] \}}t|||<q>||fSt |t|D]\}} | ||<qU||fS)a1 Load features for all the cuts and return them as a batch in a torch tensor. The output shape is ``(batch, time, features)``. The cuts will be padded with silence if necessary. :param cuts: a :class:`CutSet` used to load the features. :param pad_direction: where to apply the padding (``right``, ``left``, or ``both``). :param executor: an instance of ThreadPoolExecutor or ProcessPoolExecutor; when provided, we will use it to read the features concurrently. :return: a tuple of tensors ``(features, features_lens)``. csr8r9 has_featuresr>r%r%r&r;r<z#collate_features..cSsg|]}|jqSr%) num_framesr>r%r%r&r-sz$collate_features..rI)ro directionN)allrMtensorrKpadrLitemnextiteremptyrBro num_featuresr/_read_featuresmap) rrjrkrl features_lens first_cutfeaturesr(r#example_featuresr%r%r&collate_featuresss  rFfault_tolerantrecording_fieldc Cs"|D]"}|dur|jsJd|jq||s$Jd|d|jqg}|D]}|dur3|j}n t|jt||jd}||q)|j t dd|D|dd }t |||||d \}}}t |d j d krot|d d}ntdd|Dd dd d}tj|tjd} |r|| |fS|| fS)a Load audio samples for all the cuts and return them as a batch in a torch tensor. The output shape is ``(batch, time)``. The cuts will be padded with silence if necessary. :param cuts: a :class:`CutSet` used to load the audio samples. :param pad_direction: where to apply the padding (``right``, ``left``, or ``both``). :param executor: an instance of ThreadPoolExecutor or ProcessPoolExecutor; when provided, we will use it to read audio concurrently. :param fault_tolerant: when ``True``, the cuts for which audio loading failed will be skipped. Setting this parameter will cause the function to return a 3-tuple, where the third element is a CutSet for which the audio data were sucessfully read. :param recording_field: when specified, we will try to load recordings from a custom field with this name (i.e., ``cut.load_()`` instead of default ``cut.load_audio()``). :return: a tuple of tensors ``(audio, audio_lens)``, or ``(audio, audio_lens, cuts)``. NzMissing recording in cut Missing custom recording field  in cut ) sampling_ratecsr8r9durationr>r%r%r&r;r<z collate_audio..Trrp preserve_id)suppress_errorsrfilter_aux_iterrrXg) padding_valuecSsg|]}|ddqS)rrX) transposer"ar%r%r&r-sz!collate_audio..rI) has_recordingid has_custom num_samplesrrgetattrrappendrsrLread_audio_from_cutsrBshapecollate_vectorscollate_matricesrrMrrint32) rrjrkrrr# sample_countsraudios audio_lensr%r%r& collate_audiosN   rT with_audioc sp|D]3}|dur|jsJd|jq||s$Jd|d|jt||js5Jd|d|jqi|D]%}|durG|j}|j}nt||j}t|jt||j}||j f|j<q:|j t dd|D|d d }t ||||d \} } }t | } t jfd d |Dt jd} |rt | } t jfdd |Dt jd} nd\} } |r| | | | |fS| | | | fS)aw Load video and audio for all cuts and return them as a batch in torch tensors. The output video shape is ``(batch, time, channel, height, width)``. The output audio shape is ``(batch, channel, time)``. The cuts will be padded with silence if necessary. .. note:: We expect each video to contain audio and the same number of audio channels. We may support padding missing channels at a later time. :param cuts: a :class:`CutSet` used to load the audio samples. :param with_audio: should the audio data be loaded. :param pad_direction: where to apply the padding (``right``, ``left``, or ``both``). :param executor: an instance of ThreadPoolExecutor or ProcessPoolExecutor; when provided, we will use it to read video concurrently. :param fault_tolerant: when ``True``, the cuts for which video/audio loading failed will be skipped. Setting this parameter will cause the function to return a 5-tuple, where the fifth element is a CutSet for which the audio data were sucessfully read. :param recording_field: when specified, we will try to load recordings from a custom field with this name (i.e., ``cut.load_()`` instead of default ``cut.load_video()``). :return: a tuple of tensors ``(video, video_lens, audio, audio_lens)``, or ``(video, video_lens, audio, audio_lens, cuts)``. Nz&Missing video in the recording of cut rrz(Missing video in custom recording field z of cut csr8r9rr"cr%r%r&r;r<z collate_video..Tr)rrkrcg|] }|jdqS)rXrr>id2lensr%r&r-"z!collate_video..rIcrrrr>rr%r&r-&r)NN) has_videorrrvideorrrrrorsrLread_video_from_cutsrMstackrrr) rrrjrkrrr#rrvideosr video_lensrr%rr& collate_videosX    rfield pad_valuecs:ddlm}m}ddlm}t|dt|r6tfdd|Ds*Jdt fdd|DSt|r|d urMt d d t d t }j fd d|D}tjfdd|Dtjd}t|tjd} | j} t|g| jR} | jtfddtjtjtjtjfDrtj|tj| d} t|D]Q\} }|j}|dkrtd|n&|dkrt| || n|dkr| |d}t|| |ntd|d| ftfddtt|jD}|| |<q| |fSt|rt |Stt!rt"||dStfdd|DS)a Load custom arrays for all the cuts and return them as a batch in a torch tensor. The output shapes are: - ``(batch, d0, d1, d2, ...)`` for :class:`lhotse.array.Array` of shape ``(d0, d1, d2, ...)``. Note: all arrays have to be of the same shape, as we expect these represent fixed-size embeddings. - ``(batch, d0, pad_dt, d1, ...)`` for :class:`lhotse.array.TemporalArray` of shape ``(d0, dt, d1, ...)`` where ``dt`` indicates temporal dimension (variable-sized), and ``pad_dt`` indicates temporal dimension after padding (equal-sized for all cuts). We expect these represent temporal data, such as alignments, posteriors, features, etc. - ``(batch, )`` for anything else, such as int or float: we will simply stack them into a list and tensorize it. .. note:: This function disregards the ``frame_shift`` attribute of :class:`lhotse.array.TemporalArray` when padding; it simply pads all the arrays to the longest one found in the mini-batch. Because of that, the function will work correctly even if the user supplied inconsistent meta-data. .. note:: Temporal arrays of integer type that are smaller than torch.int64, will be automatically promoted to torch.int64. :param cuts: a :class:`CutSet` used to load the features. :param field: name of the custom field to be retrieved. :param pad_value: value to be used for padding the temporal arrays. Ignored for non-temporal array and non-array attributes. :param pad_direction: where to apply the padding (``right``, ``left``, or ``both``). :return: a collated data tensor, or a tuple of tensors ``(collated_data, sequence_lens)``. r)Array TemporalArray)Imagec3s"|] }t|jjkVqdSr9)rrr)rfirst_manifestr%r&r;]s z'collate_custom_field..zCannot collate manifests of type Array with different shapes, because we don't know which dimension must be padded. Use TemporalArray manifests and try again.cg|] }t|qSr%rMrN load_customrrr%r&r-bz(collate_custom_field..Nz6Argument 'pad_value' not passed -- we will pad field 'z' with .crr%rrrr%r&r-urcsg|]}|jqSr%rr) temporal_dimr%r&r-wr+rIr?c3s|]}|kVqdSr9r%)r"drIr%r&r;}srileftbothrz$Unexpected pad_direction argument: ''c3s(|]}|kr ntdddVqdSr9)slice)r"i)rtemporal_slicer%r&r;s  )rrjcsg|]}t|qSr%)rrrr%r&r-r+)# lhotse.arrayrr lhotse.imagerr isinstancerqrMrwarningswarnrrrrrrLnumelrrBrJanyuint8int8int16rQonesr/r ValueErrortuplerangecollate_imagesr r)rrrrjrrrarrsarr_lens largest_arrmaxlencollated_shapetensorsaidxralenhalfindicesr%)rJrrrrr&collate_custom_field1sd%     &         rcCstdd|Ds Jtdd|DsJ||}tt|}tt|t|j|j|j }t |D]\}}t |j dd||<q4|S)a5 Load features for all the cuts and return them as a batch in a torch tensor. The cuts have to be of type ``MixedCut`` and their tracks will be interpreted as individual channels. The output shape is ``(batch, channel, time, features)``. The cuts will be padded with silence if necessary. csr8r9rmr>r%r%r&r;r<z1collate_multi_channel_features..css|]}t|tVqdSr9)rrr>r%r%r&r;sF)mixed) rqrsrurvrMrwrBtracksrorxr/rN load_features)rr|r}r(r#r%r%r&collate_multi_channel_featuress  rrrmatching_shapescsdd|D}tdd|DsJd|dvrtd|t|dd d |r8tfd d|Ds8Jd t|jd |}t|D] \}}|dkr]|||d|jd f<qI||||jd  df<qI|S)a@ Convert an iterable of 1-D tensors (of possibly various lengths) into a single stacked tensor. :param tensors: an iterable of 1-D tensors. :param padding_value: the padding value inserted to make all tensors have the same length. :param pad_direction: where to apply the padding (``right`` or ``left``). :param matching_shapes: when ``True``, will fail when input tensors have different shapes. :return: a tensor with shape ``(B, L)`` where ``B`` is the number of input tensors and ``L`` is the number of items in the longest tensor. cS&g|]}t|tjr |nt|qSr%rrMrgrNr"tr%r%r&r-z#collate_vectors..cs|] }t|jdkVqdS)rXNrBrrr%r%r&r;z"collate_vectors..z Expected only 1-D input tensors.)rriz-pad_direction must be 'left' or 'right', got cS |jdSNrrrr%r%r& z!collate_vectors..r?c3|] }|jjkVqdSr9rrlongestr%r&r;  IAll tensors must have the same shape when matching_shapes is set to True.rriN)rqrrLnew_onesrBrr/)rrrjrresultrrr%rr&rs*  rcsdd|D}tdd|DsJdt|ddd|r-tfd d|Ds-Jd jt|gjR|}t|D]\}}|||d |jd f<q?|S) a, Convert an iterable of 2-D tensors (of possibly various first dimension, but consistent second dimension) into a single stacked tensor. :param tensors: an iterable of 2-D tensors. :param padding_value: the padding value inserted to make all tensors have the same length. :param matching_shapes: when ``True``, will fail when input tensors have different shapes. :return: a tensor with shape ``(B, L, F)`` where ``B`` is the number of input tensors, ``L`` is the largest found shape[0], and ``F`` is equal to shape[1]. cSrr%rrr%r%r&r-rz$collate_matrices..csr)rNrrr%r%r&r;rz#collate_matrices..z Expected only 2-D input tensors.cSrrrrr%r%r&rrz"collate_matrices..r?c3rr9rrrr%r&r;rrNr)rqrLrrBrr/)rrrrrrr%rr&rs  rrrc Csd}|dur tdg}d}|durtn|j}g}g}g} tt||tt||d||D]\} \} } } | dur8q,|| || | | q,|t|f}|rV|| f}|S)a Loads audio data from an iterable of cuts. :param cuts: a CutSet or iterable of cuts. :param executor: optional Executor (e.g., ThreadPoolExecutor or ProcessPoolExecutor) to perform the audio reads in parallel. :param suppress_errors: when set to ``True``, will enable fault-tolerant data reads; we will skip the cuts and audio data for the instances that failed (and emit a warning). When ``False`` (default), the errors will not be suppressed. :param recording_field: when specified, we will try to load recordings from a custom field with this name (i.e., ``cut.load_()`` instead of default ``cut.load_audio()``). :param filter_aux_iter: when specified, we will iterate over this iterator and discard the elements for which a corresponding cut failed to load audio, if ``suppress_errors`` is set to ``True``. This iterator is expected to be of the same length as ``cuts``. :return: a tuple of two items: a list of audio tensors (with different shapes), and a list of cuts for which we read the data successfully. If ``filter_aux_iter`` is specified, it returns a 3-tuple where the third element is the filtered auxiliary iterator. TNF)rr) rrzr/r^r _read_audiorr from_cuts)rrkrrr aux_requestedmap_fnrok_cuts aux_iter_outr(r# maybe_audioaux_itemansr%r%r&rs>     rc Cs|durtn|j}g}g}g}tt||tt|||d|D]\} \} } | dur*q| \} } || || || q||t|fS)a Loads audio data from an iterable of cuts. :param cuts: a CutSet or iterable of cuts. :param with_audio: should the audio data be loaded. :param executor: optional Executor (e.g., ThreadPoolExecutor or ProcessPoolExecutor) to perform the audio reads in parallel. :param suppress_errors: when set to ``True``, will enable fault-tolerant data reads; we will skip the cuts and audio data for the instances that failed (and emit a warning). When ``False`` (default), the errors will not be suppressed. :param recording_field: when specified, we will try to load recordings from a custom field with this name (i.e., ``cut.load_()`` instead of default ``cut.load_video()``). :return: a tuple of two items: a list of audio tensors (with different shapes), and a list of cuts for which we read the data successfully. N)rrr)rzr/r^r _read_videorr r)rrrkrrrrrrr(r# maybe_ansraudior%r%r&r<s2   rcCs |durtn|j}t|t|Sr9)rzrAry)rrkrr%r%r&read_features_from_cutsnsrr#cCst|d=|dur|}nt||}t|ts%Jd|dt|||}|jddkr6|d}t |WdS1sEwYdS)z{ Loads audio data from cut, or returns None if there was an error and ``suppress_errors`` was set to ``True``. enabledNExpected 'getattr(cut, )' to yield Recording, got rrX) r load_audiorrr typerrsqueezerMrN)r#rrrattrr%r%r&rus     $rcCst|Sr9)rMrNr)r#r%r%r&rysrycCst|d6|dur|j|dWdSt||}t|ts-Jd|dt||j||dWdS1s>wYdS)z Loads video + audio data from cut, or returns None if there was an error and ``suppress_errors`` was set to ``True``. rN)rrr)r load_videorrr rr)r#rrrr r%r%r&rs    $rimage image_fieldcs fdd|D}t|}|S)a8 Load images for all cuts and return them as a batch in a torch tensor. The output image shape is ``(batch, height, width, channel)``. :param cuts: a :class:`CutSet` used to load the images. :param image_field: the field in the cut to load the images from. :return: tensor of collated imagescrr%)rM as_tensorrr>r r%r&r-rz"collate_images..)rMr)rr imagesr%rr&rs r)riNN)riNFN)TriNFN)Nri)rF)NFNN)TNFNr9)FN)TFN)r )7rconcurrent.futuresr functoolsr itertoolsrtypingrrrrr numpyrOrMtorch.nnr lhotser r lhotse.audior lhotse.audio.utilsr lhotse.cutrr lhotse.utilsrrrrfrJrgrrercollate_multi_channel_audiorrKfloatrr ignore_indexndarrayrrrrrrryrrr%r%r%r&sj      c #  I T  o  )  ( > 3