o SiB@sddlmZddlmZmZmZmZddlZddl Z ddl m m Z ddlmZddlmZddlmZmZddlmZmZmZddlmZGd d d e jjjZdd dZdeddfddZ dS)) defaultdict)CallableDictListUnionN)validate)CutSet)BatchIOPrecomputedFeatures) LOG_EPSILONcompute_num_framesifnone)Hdf5MemoryIssueFixcseZdZdZdddddddeedf dededed ed ed e e e ge fd e e e j ge j fd ededeffdd Zde deeee j e efffddZZS) K2SurtDataseta The PyTorch Dataset for the multi-talker ASR task using k2 library. We support the modeling framework known as Streaming Unmixing and Recognition Transducer (SURT), as described in [1] and [2], but this dataset can also be used for other multi-talker ASR approaches such as MT-RNNT [3] and SOT [4]. See icefall recipe for usage: https://github.com/k2-fsa/icefall/pull/1126. We take a cut containing possibly overlapping speech and split the supervision segments into one of N channels based on their start times (N is provided), known as ``heuristic error assignment training`` (HEAT) [1]. The supervision segments in each channel are then concatenated and used as the supervision for that channel. If we have features for the source cuts, we can also return them for use in masking losses, for instance. [1] Lu, L., Kanda, N., Li, J., & Gong, Y. (2021). Streaming end-to-end multi-talker speech recognition. IEEE Signal Processing Letters, 28, 803-807. [2] Raj, D., Lu, L., Chen, Z., Gaur, Y., & Li, J. (2022, May). Continuous streaming multi-talker asr with dual-path transducers. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 7317-7321). IEEE. [3] Sklyar, I., Piunova, A., Zheng, X., & Liu, Y. (2022, May). Multi-turn RNN-T for streaming recognition of multi-party speech. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 8402-8406). IEEE. [4] Kanda, N., Gaur, Y., Wang, X., Meng, Z., & Yoshioka, T. (2020). Serialized output training for end-to-end overlapped speech recognition. arXiv preprint arXiv:2003.12687. .. hint:: Training mixtures can be simulated from single-speaker utterances using :class:`~lhotse.workflows.MeetingSimulation` workflow. This dataset expects to be queried with lists of cut IDs, for which it loads features and automatically collates/batches them. To use it with a PyTorch DataLoader, set ``batch_size=None`` and provide a :class:`SimpleCutSampler` sampler. Each item in this dataset is a dict of: .. code-block:: { 'inputs': float tensor with shape determined by :attr:`input_strategy`: - single-channel: - features: (B, T, F) - audio: (B, T) - multi-channel: currently not supported 'input_lens': int tensor of shape (B,) 'supervisions': list of lists of supervision segments, where the outer list is batch, and the inner list is indexed by channel. So ``len(supervisions) == B``, and ``len(supervisions[i]) == num_channels``. Note that some channels may have no supervision segments. 'text': list of lists of strings, where the outer list is batch, and the inner list is indexed by channel. So ``len(text) == B``, and ``len(text[i]) == num_channels``. Each element contains the text of the supervision segments in that channel, joined by the :attr:`text_delimiter`. Note that some channels may have no supervision segments, so the corresponding text will be an empty string. } If ``return_cuts`` is ``True``, each item will also contain a ``cuts`` field with the :class:`~lhotse.cut.Cut` objects used to create the batch. Additionally, if ``return_sources`` is ``True``, each item will contain: .. code-block:: { 'source_feats': list of list of float tensors. The outer list is batch, and the inner list is number of segments in the mixture. Each element denotes the features of the source cut in the mixture. 'source_boundaries': list of list of tuples, where the outer list is batch, and the inner list is number of segments in the mixture. Each element denotes the start and end frame of the source cut in the mixture. } In order to return the source features and boundaries, we expect the cuts to contain some additional fields: - ``source_feats``: a float tensor representing the features of all the source segments, concatenated together along T dimension, in order of their start times. - ``source_feat_offsets``: a list of ints, where each element denotes the offset of the source segments in the ``source_feats`` tensor. See https://github.com/lhotse-speech/lhotse/discussions/1008#discussioncomment-5511746 for example code to create these fields. Dimension symbols legend: * ``B`` - batch size (number of Cuts) * ``S`` - number of supervision segments (greater or equal to B, as each Cut may have multiple supervisions) * ``T`` - number of frames of the longest Cut * ``F`` - number of features The 'sequence_idx' field is the index of the Cut used to create the example in the Dataset. F N return_cutsreturn_sourcesreturn_alignments num_channelstext_delimitercut_transformsinput_transformsinput_strategy pad_valuestrictc sbt||_||_||_||_||_t|g|_t|g|_ ||_ | |_ | |_ t dd|_dS)a k2 ASR IterableDataset constructor. :param return_cuts: When ``True``, will additionally return a "cut" field in each batch with the Cut objects used to create that batch. :param return_sources: When ``True``, will additionally return a "source_feats" field and a "source_boundaries" field in each batch. The "source_feats" field contains the features of the source cuts from which the mixture was created, and "source_boundaries" contains the boundaries of the source cuts in the mixture (in number of frames). This requires that the cuts contain additional fields ``source_feats`` (which is a TemporalArray) and ``source_feat_offsets`` (which is a list of ints). :param return_alignments: When ``True``, will keep the supervision alignments if they are present in the cuts. :param num_channels: Number of output branches. The supervision utterances will be split into the channels based on their start times. :param text_delimiter: The delimiter used to join the text of the supervision segments in each channel. :param cut_transforms: A list of transforms to be applied on each sampled batch, before converting cuts to an input representation (audio/features). Examples: cut concatenation, noise cuts mixing, etc. :param input_transforms: A list of transforms to be applied on each sampled batch, after the cuts are converted to audio/features. Examples: normalization, SpecAugment, etc. :param input_strategy: The strategy used to convert the cuts to audio/features. :param pad_value: The value used to pad the source features to resolve one-off errors. :param strict: If ``True``, we will remove cuts that have more simultaneous supervisions than the number of channels. If ``False``, we will keep them. d)reset_intervalN)super__init__rrrrrr rrrrrrhdf5_fix) selfrrrrrrrrrr __class__G/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/dataset/surt.pyrps (  zK2SurtDataset.__init__cutsreturncst|jjs|}|jdd}jD]}||}qtt}gg}g}|D]݉ddt j D}ddt j D}g}g} d} t j dddD]L} d} t j D]&} t || d ksk|| | jkr|| | t|| | j|| <d } nqZ| sd } |t|}||| t||| j||<qQjrj}t |t j ksJd t |d t j d ddt|ddD}fddt j dddD} fddt|| D}| rjrjq,||j<jr |||| q,t d kr)tdt dt |d|fdd}|}t |dkr;|\}}}n|\}}||t|fdd|Dd}j r[||d<jrg||d<||d<|S)z Return a new batch, with the batch size automatically determined using the constraints of max_duration and max_cuts. F) ascendingcSsg|]}gqSr$r$.0_r$r$r% z-K2SurtDataset.__getitem__..cSsg|]}dqS)r$r)r$r$r%r,r-cSs|jSN)startsr$r$r%sz+K2SurtDataset.__getitem__..)keyrTzWThe number of source feature offsets should be equal to the number of supervisions.Got z offsets for z supervisions.cSsg|]}t|qSr$)torch from_numpy)r*xr$r$r%r,sNcs0g|]}t|jjjt|jjjfqSr$)r r0 frame_shift sampling_rateendr*supcutr$r%r,s cSs |j|jfSr/)r0speakerr1r$r$r%r3s cs(g|]\}\}}t|||jdqS)) padding_value)adjust_source_featsr)r*r7r0r;r!r$r%r,s z WARNING: z cuts were removed out of z0 due to more overlapping speakers than channels.cs |jvSr/)idr>) invalid_cutsr$r%r3s csg|] }fdd|DqS)cs"g|] }jdd|DqS)cSsg|]}|jqSr$)textstripr<r$r$r%r,'szCK2SurtDataset.__getitem__....)rjoin)r*sups_chrCr$r%r,&sz8K2SurtDataset.__getitem__...r$)r*cut_supsrCr$r%r,%s  )inputs input_lens supervisionsrGr& source_featssource_boundaries)!validate_for_asrr updaterdrop_alignmentssort_by_durationrrlistrangersortedrNlenr0appendmaxr;indexminrsource_feat_offsetsnpsplitload_source_featsziprrDprintfilterrvaluesr)r!r&tnfmrNrOrPrK last_sup_end cut_sourcescut_source_boundaries invalid_cutr=assignedimin_end_channelr] input_tplrLrMbatchr$)r?rEr!r% __getitem__s                  zK2SurtDataset.__getitem__)__name__ __module__ __qualname____doc__r r boolintstrrrrr5Tensorr floatrrrro __classcell__r$r$r"r%rsDb   0:rr.rcCs|jd|kr |St|jd||kr"td|jdd|d|jd|kr:tj|ddd||jdf|dS|d|S)aU Adjust the number of frames in the source features to match the supervision. If the source features have fewer frames than the supervision, we pad them to match the supervision. If the source features have more frames than the supervision, we trim them to match the supervision. Args: feats: Source features. num_frames: Number of frames in the supervision. padding_value: Value to use for padding. tol: Tolerance for checking if the number of frames in the source features is close to the number of frames in the supervision. rz)Number of frames in the source features (z;) is not close to the number of frames in the supervision (z).)valueN)shapeabs ValueErrorFpad)feats num_framesrAtolr$r$r%rB5s" rBr&r'cCsvt|d}|D]0}|jD]*}|j| ks"Jd|jd|jd|j|j|ks7Jd|jd|jdq qdS)NgMb`?zHSupervisions starting before the cut are not supported for ASR (sup id: z , cut id: )zESupervisions ending after the cut are not supported for ASR (sup id: )rrNr0rDduration)r&rr? supervisionr$r$r%rQTs( rQ)r.r)! collectionsrtypingrrrrnumpyr^r5torch.nn.functionalnn functionalr~lhotser lhotse.cutrlhotse.dataset.input_strategiesr r lhotse.utilsr r r lhotse.workaroundsrutilsdataDatasetrrBrQr$r$r$r%s     (