o æS™i8Zã@sðddlZddlZddlmZddlmZmZddlmZddl m Z ddl m Z m Z mZmZmZmZmZmZddlZddlZddlmZddlmZdd lmZdd lmZdd lm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(eGd d „d eƒƒZ)dS)éN)Ú dataclass)ÚpartialÚreduce)Úgroupby)Úadd)ÚAnyÚCallableÚIterableÚListÚOptionalÚSequenceÚTupleÚUnion)Ú Recording)ÚDataCut)ÚFeatures)ÚSupervisionSegment) Ú add_durationsÚfastcopyÚhash_str_to_intÚis_equal_or_containsÚmerge_items_with_delimiterÚoverlapsÚrich_exception_infoÚto_listÚuuid4c@sÆeZdZUdZeeed<edefdd„ƒZe d-de e eeefde e j fdd„ƒZe d-de e eeefde e j fd d „ƒZe  d.de e eeefd ede eeje ejffd d„ƒZ     d/de e eefdedededeede ede eddfdd„Z  d0dedede eeeegefddfdd„Zd e eeefdefd!d"„Zed#eddfd$d%„ƒZ d1d&ede d'ed'ffd(d)„Z!ed*e"ddfd+d,„ƒZ#dS)2ÚMultiCutuE :class:`~lhotse.cut.MultiCut` is a :class:`~lhotse.cut.Cut` that is analogous to the MonoCut. While MonoCut represents a single channel of a recording, MultiCut represents multi-channel recordings where supervisions may or may not be shared across channels. It is intended to be used to store, for example, segments of a microphone array recording. The following diagrams illustrate some examples for MultiCut usage: >>> 2-channel telephone recording with 2 supervisions, one for each channel (e.g., Switchboard): â•”â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â• MultiCut â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•— â•‘ ┌──────────────────────────┠║ Channel 1 ──╬─│ Hello this is John. │──────────────────────────────╬──────── â•‘ └──────────────────────────┘ â•‘ â•‘ ┌──────────────────────────â”â•‘ Channel 2 ──╬───────────────────────────────│ Hey, John. How are you? │╠──────── â•‘ └──────────────────────────┘║ â•šâ•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â• >>> Multi-array multi-microphone recording with shared supervisions (e.g., CHiME-6), along with close-talk microphones (A and B are distant arrays, C is close-talk): â•”â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•— â•‘ ┌───────────────────┠┌───────────────────┠║ A-1 ──╬─┤ ├─────────────────────────┤ ├───────╬─ â•‘ │ What did you do? │ │I cleaned my room. │ â•‘ A-2 ──╬─┤ ├─────────────────────────┤ ├───────╬─ â•‘ └───────────────────┘ ┌───────────────────┠└───────────────────┘ â•‘ B-1 ──╬────────────────────────┤Yeah, we were going├──────────────────────────────╬─ â•‘ │ to the mall. │ â•‘ B-2 ──╬────────────────────────┤ ├──────────────────────────────╬─ â•‘ └───────────────────┘ ┌───────────────────┠║ C ──╬─────────────────────────────────────────────────────┤ Right. ├─╬─ â•‘ └───────────────────┘ â•‘ â•šâ•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â• MultiCut â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â•â• By definition, a MultiCut has the same attributes as a MonoCut. The key difference is that the Recording object has multiple channels, and the Supervision objects may correspond to any of these channels. The channels that the MultiCut can be a subset of the Recording channels, but must be a superset of the Supervision channels. See also: - :class:`lhotse.cut.Cut` - :class:`lhotse.cut.MonoCut` - :class:`lhotse.cut.CutSet` - :class:`lhotse.cut.MixedCut` ÚchannelÚreturncCstt|jƒƒS©N)Úlenrr)Úself©r"úD/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/cut/multi.pyÚ num_channelsRszMultiCut.num_channelsNcCsŽ|jrE|jj|j|j|dur|jn|d}|jd|jdkr*|d|j…df}|S|jd|jdkrCtj ||dd…dffdd}|SdS)a˜ Load the features from the underlying storage and cut them to the relevant [begin, duration] region of the current MultiCut. :param channel: The channel to load the features for. If None, all channels will be loaded. This is useful for the case when we have features extracted for each channel of the multi-cut, and we want to selectively load them. N)ÚstartÚdurationÚ channel_idré.éÿÿÿÿ)Úaxis) Ú has_featuresÚfeaturesÚloadr%r&rÚshapeÚ num_framesÚnpÚ concatenate)r!rÚfeatsr"r"r#Ú load_featuresVs ý þzMultiCut.load_featurescCs.|jr|jj|dur |jn||j|jdSdS)au Load the audio by locating the appropriate recording in the supplied Recording. The audio is trimmed to the [begin, end] range specified by the MultiCut. :param channel: optional int or list of int, the subset of channels to load (all by default). :return: a numpy ndarray with audio samples, with shape (C , N ) N)ÚchannelsÚoffsetr&)Ú has_recordingÚ recordingÚ load_audiorr%r&)r!rr"r"r#r8ss ýzMultiCut.load_audioTÚ with_audiocCs0|jr|jj|dur |jn||j|j|dSdS)að Load the subset of video (and audio) from attached recording. The data is trimmed to the [begin, end] range specified by the MonoCut. :param channel: optional int or list of int, the subset of channels to load (all by default). :param with_audio: bool, whether to load and return audio alongside video. True by default. :return: a tuple of video tensor and optionally audio tensor (or ``None``), or ``None`` if this cut has no video. N)r4r5r&r9)Ú has_videor7Ú load_videorr%r&)r!rr9r"r"r#r;†süzMultiCut.load_videoF©rÚ rir_recordingÚnormalize_outputÚ early_onlyÚaffix_idÚ rir_channelsÚ room_rng_seedÚsource_rng_seedc sÖ|jsJdƒ‚|jrt d¡d|_ˆdur4|jdksJdƒ‚|dur-tttƒƒ|j ƒ}|dur3|}nt ‡fdd„|DƒƒsCJdƒ‚|j j ˆ||ˆ|||d }‡fd d „|j Dƒ} t|ˆrd|j ›d n|j || d S)aj Return a new ``MultiCut`` that will convolve the audio with the provided impulse response. If the `rir_recording` is multi-channel, the `rir_channels` argument determines which channels will be used. This list must be of the same length as the number of channels in the `MultiCut`. If no ``rir_recording`` is provided, we will generate an impulse response using a fast random generator (https://arxiv.org/abs/2208.04101), only if the MultiCut has exactly one channel. At the moment we do not support simulation of multi-channel impulse responses. :param rir_recording: The impulse response to use for convolving. :param normalize_output: When true, output will be normalized to have energy as input. :param early_only: When true, only the early reflections (first 50 ms) will be used. :param affix_id: When true, we will modify the ``MonoCut.id`` field by affixing it with "_rvb". :param rir_channels: The channels of the impulse response to use. First channel is used by default. If multiple channels are specified, this will produce a MixedCut instead of a MonoCut. :param room_rng_seed: The seed for the room configuration. :param source_rng_seed: The seed for the source positions. :return: a modified copy of the current ``MonoCut``. z;Cannot apply reverberation on a MultiCut without Recording.z¥Attempting to reverberate a MultiCut that references pre-computed features. The feature manifest will be detached, as we do not support feature-domain reverberation.Nr(zlWe do not support reverberation simulation for multi-channel recordings. Please provide an impulse response.c3s|]}|ˆjkVqdSr)r$©Ú.0Úc)r=r"r#Ú Òs€  ÿz&MultiCut.reverb_rir..z(Invalid channel index in `rir_channels`.)r=r>r?r@rArBrCcsg|]}|jˆd‘qS)©r@)Ú reverb_rir©rEÚsrHr"r#Ú às ýÿÿz'MultiCut.reverb_rir..Ú_rvb)Úidr7Ú supervisions)r6r+ÚloggingÚwarningr,r$rÚstrrrNÚallr7rIrOr) r!r=r>r?r@rArBrCÚ recording_rvbÚsupervisions_rvbr")r@r=r#rIžsR ÿþÿ ÿ€  ÿþù üüzMultiCut.reverb_rirÚ delimiterÚ merge_policyÚmerge_channelsÚcustom_merge_fncsttd|dkd‰|dur|‰n‡fdd„‰t|jdd„d}t|ƒd kr'|S|rHtƒ}|D]}tt|jƒƒ}| |¡q.t|ƒ}t |ƒ|i}nd d „t t|d d„dd d„dDƒ}g} d } |  ¡D]š\} ‰ˆdj } ˆdj } t| | |jd}tdd„ˆDƒƒ}tdd„ˆDƒƒ}tdd„tˆˆd d…ƒDƒƒr³tdd„ˆDƒƒr³| s³t d|j›d¡d} |  tˆdd„ˆDƒƒˆdj| |t| ƒd dd„ˆDƒ¡ˆdd„ˆDƒƒˆdd„ˆDƒƒˆdd„ˆDƒƒ‡‡fdd „|Dƒ‡fd d „|Dƒd! ¡qct|| d"S)#a  Return a copy of the cut that has all of its supervisions merged into a single segment. The ``channel`` attribute of all the segments in this case will be set to the union of all channels. If ``merge_channels`` is set to ``False``, the supervisions will be merged into a single segment per channel group. The ``channel`` attribute will not change in this case. The new start is the start of the earliest superivion, and the new duration is a minimum spanning duration for all the supervisions. The text fields of all segments are concatenated with a whitespace. :param merge_policy: one of "keep_first" or "delimiter". If "keep_first", we keep only the first segment's field value, otherwise all string fields (including IDs) are prefixed with "cat#" and concatenated with a hash symbol "#". This is also applied to ``custom`` fields. Fields with a ``None`` value are omitted. :param merge_channels: If true, we will merge all supervisions into a single segment. If false, we will merge supervisions per channel group. Default: True. :param custom_merge_fn: a function that will be called to merge custom fields values. We expect ``custom_merge_fn`` to handle all possible custom keys. When not provided, we will treat all custom values as strings. It will be called roughly like: ``custom_merge_fn(custom_key, [s.custom[custom_key] for s in sups])`` ú#Ú keep_first)rVÚ return_firstNcsˆtt|ƒƒSr)ÚmaprR)ÚkÚvs)Ú merge_func_r"r#Úsz-MultiCut.merge_supervisions..cSó|jSr)r%©rKr"r"r#raó©Úkeyr(cSsi|] \}}t|ƒt|ƒ“qSr")ÚtupleÚlist)rErFÚcsupsr"r"r#Ú (s ÿÿz/MultiCut.merge_supervisions..cSrbr©rrcr"r"r#ra+rdFrr))Ú sampling_ratecsó.|]}|jdur|j ¡D]}|VqqdSr)ÚcustomÚkeys©rErKr^r"r"r#rG6ó€$ÿz.MultiCut.merge_supervisions..csrmr)Ú alignmentrorpr"r"r#rG9rqcss|] \}}t||ƒVqdSr)r)rEÚs1Ús2r"r"r#rG>s€css|]}|jduVqdSr©ÚtextrJr"r"r#rG?ó€z¦You are merging overlapping supervisions that have text transcripts. The result is likely to be unusable if you are going to train speech recognition models (cut id: z).Tcsó|]}|jVqdSr)rNrJr"r"r#rGKó€ú csó|] }|jr|jVqdSrrurJr"r"r#rGPó€csr{r)ÚspeakerrJr"r"r#rGQr|csr{r)ÚlanguagerJr"r"r#rGRr|csr{r)ÚgenderrJr"r"r#rGSr|cs&i|]‰ˆˆˆ‡fdd„ˆDƒƒ“qS)c3ó.|]}|jdurˆ|jvr|jˆVqdSr)rnrJ©r^r"r#rGWó€þýú9MultiCut.merge_supervisions...r"©rE)riÚ merge_customrr#rjTs ø þþÿcs&i|]‰ˆtt‡fdd„ˆDƒƒ“qS)c3r€r)rrrJrr"r#rGcr‚rƒ)rrr„)rirr#rj_s ø þþþ) rNÚ recording_idr%r&rrvr}r~rrnrr©rO)rrÚsortedrOr ÚsetrrÚupdatergrÚitemsr%ÚendrrlÚanyÚzipÚwarningsÚwarnrNÚappendrr†rhÚjoinr)r!rWrXrYÚsupsÚ all_channelsrKrFÚsups_by_channelÚmsupsÚtext_overlap_warningrÚmstartÚmendÚ mdurationÚ custom_keysÚalignment_keysr")rir…r`r#Úmerge_supervisionsîs†ý   ÿþ  ÿÿÿþýþÿ  ÷  öëÿ %zMultiCut.merge_supervisionsr4c s²tˆtƒ}t|r ˆgnˆƒ t|jjƒ¡s!Jdˆ›d|jj›ƒ‚|p(tˆƒdk}|rSddlm}|s6ˆ\‰||j ›dˆ›|j|j |j ˆ‡fdd„|j Dƒ|j dSt|ˆd S) a- Select specified channels from this cut. Supports extending to other channels available in the underlying :class:`Recording`. If a single channel is provided, we'll return a :class:`~lhotse.cut.MonoCut`, otherwise we'll return a :class:`~lhotse.cut.MultiCut`. zCannot select channels=z= because they are not a subset of self.recording.channel_ids=r(©ÚMonoCutú-có$g|]}t|jˆƒrt|ˆd‘qS©rk©rrrrJ©r4r"r#rLˆó  ý ÿz*MultiCut.with_channels..©rNr7r%r&rrOrnrk)Ú isinstanceÚintr‰Úissubsetr7Ú channel_idsr ÚmonorŸrNr%r&rOrnr)r!r4Úchannel_is_intr«rŸr"r¤r#Ú with_channelsps.  ÿþ  þõ zMultiCut.with_channelsÚcutscsÄddlm‰t‡fdd„|DƒƒsJdƒ‚tdd„t|dd„d Dƒƒdks*Jd ƒ‚ttd d„|Dƒƒƒt|ƒks=Jd ƒ‚|d  ¡}| d¡t dii|¥t dd„|Dƒƒdd„|Dƒdœ¥¤ŽS)a Convert one or more MonoCut to a MultiCut. If multiple mono cuts are provided, they must match in all fields except the channel. Each cut must have a distinct channel. :param cuts: the input cut(s). :return: a MultiCut with a single track. r(ržc3s|]}t|ˆƒVqdSr)r§rDržr"r#rGrwz%MultiCut.from_mono..zAll cuts must be MonoCutscss|]}dVqdS)r(Nr")rEÚ_r"r"r#rGŸs€ ÿcSs|j|j|jfSr)r†r%rŒ)rFr"r"r#ra sz$MultiCut.from_mono..rez,Cuts must match in all fields except channelcsrxrrkrDr"r"r#rG¤ryz%All cuts must have a distinct channelrÚtypecSsg|]}|j‘qSr"rkrDr"r"r#rL®sz&MultiCut.from_mono..cSsg|] }|jD]}|‘qqSr"r‡)rErFrKr"r"r#rL¯s)rrONr") r«rŸrSÚsumrr r‰Úto_dictÚpoprrˆ)r®Údatar"ržr#Ú from_mono’s0 ÿýüÿþ  ÿ ýÿzMultiCut.from_monoÚ mono_downmixrcsbddlm}m‰ddlm‰‡‡fdd„tˆjƒDƒ}|s|S|ˆj‡fdd„|Dƒd}| ¡S)ar Convert a MultiCut to either a list of MonoCuts (one per channel) or a single MonoCut obtained by downmixing all channels. :param mono_downmix: If true, we will downmix all channels into a single MonoCut. If false, we will return a list of MonoCuts, one per channel. :return: a list of MonoCuts or a single MonoCut. r()ÚMixedCutÚMixTrackržc sFg|]‰ˆˆj›dˆ›ˆjˆjˆjˆ‡fdd„ˆjDƒˆjd‘qS)r cr¡r¢r£rJrkr"r#rLÆr¥z/MultiCut.to_mono...r¦)rNr7r%r&rOrnr„)rŸr!rkr#rL¿só þõÿz$MultiCut.to_mono..csg|] }ˆ|ddd‘qS)gN)Úcutr5Úsnrr")rEÚmono_cut)r¸r"r#rLÕsÿ)rNÚtracks) Úmixedr·r¸r«rŸrrrNÚto_mono)r!r¶r·Ú mono_cutsÚ mixed_cutr")r¸rŸr!r#r¾³s  ò ÿþzMultiCut.to_monor´cCs°ddlm}| dd¡d|vrt | d¡¡nd}d|vr&t | d¡¡nd}d|vr1| d¡ng}d|vr=||dƒd|vrF| d¡td i|¤||dd „|Dƒd œ¤ŽS) Nr)Údeserialize_custom_fieldr°r,r7rOrncSsg|]}t |¡‘qSr")rÚ from_dictrJr"r"r#rLôsz&MultiCut.from_dict..)r,r7rOr")Úlhotse.serializationrÁr³rrÂrr)r´rÁr,r7Úsupervision_infosr"r"r#rÂÛs$  ÿÿ  ÿ  üzMultiCut.from_dictr)NT)NTFTr<NN)rVTN)F)$Ú__name__Ú __module__Ú __qualname__Ú__doc__r r¨Ú__annotations__Úpropertyr$rr rr0Úndarrayr3r8Úboolr ÚtorchÚTensorr;rrr rIrRrr rrr­Ú staticmethodrµr¾ÚdictrÂr"r"r"r#rs  1ÿÿ þÿÿ þýþý üøþýüûúùø ÷Rüþýü û" (r)*rPrÚ dataclassesrÚ functoolsrrÚ itertoolsrÚoperatorrÚtypingrrr r r r r rÚnumpyr0rÍÚ lhotse.audiorÚlhotse.cut.datarÚlhotse.featuresrÚlhotse.supervisionrÚ lhotse.utilsrrrrrrrrrrr"r"r"r#Ús    (    ,