o Sin@sdZddlZddlZddlZddlmmZddlZddl m Z ddl m Z ddl mZmZmZmZmZmZddlmZddlmZddlmZmZmZdd lmZdd lmZdd l m!Z!dd l"m#Z#m$Z$m%Z%dd l&m'Z'm(Z(m)Z)m*Z*gdddggddZ+gddggdgdZ,   d?de dee-dee.dee.ddf dd Z/ !    " d@de'd#ee'd$ee'dee-dee.dee.de fd%d&Z0Gd'd(d(eZ1 )dAd$e'd*e.deeee.e.e.fee$fee.ee.e2ffffd+d,Z3   dBd-ee'd.ee.ee.e2ffd/e-d0e'def d1d2Z4  dCd-ee'd/e-d0e'defd3d4Z5d5ed6ee.ee1fd.ee.ee.e2ffde%fd7d8Z6d5ed6ee.ee1fde%fd9d:Z7    ; dDd#e'd$ee'd0ee'dee.dZ8dS)Eav The data preparation recipe for the ICSI Meeting Corpus. It follows the Kaldi recipe by Pawel Swietojanski: https://git.informatik.fh-nuernberg.de/poppto72658/kaldi/-/commit/d5815d3255bb62eacf2fba6314f194fe09966453 ICSI data comprises around 72 hours of natural, meeting-style overlapped English speech recorded at International Computer Science Institute (ICSI), Berkley. Speech is captured using the set of parallel microphones, including close-talk headsets, and several distant independent microhones (i.e. mics that do not form any explicitly known geometry, see below for an example layout). Recordings are sampled at 16kHz. The correponding paper describing the ICSI corpora is [1] [1] A Janin, D Baron, J Edwards, D Ellis, D Gelbart, N Morgan, B Peskin, T Pfau, E Shriberg, A Stolcke, and C Wooters, The ICSI meeting corpus. in Proc IEEE ICASSP, 2003, pp. 364-367 ICSI data did not come with any pre-defined splits for train/valid/eval sets as it was mostly used as a training material for NIST RT evaluations. Some portions of the unrelased ICSI data (as a part of this corpora) can be found in, for example, NIST RT04 amd RT05 evaluation sets. This recipe, however, to be self-contained factors out training (67.5 hours), development (2.2 hours and evaluation (2.8 hours) sets in a way to minimise the speaker-overlap between different partitions, and to avoid known issues with available recordings during evaluation. This recipe follows [2] where dev and eval sets are making use of {Bmr021, Bns001} and {Bmr013, Bmr018, Bro021} meetings, respectively. [2] S Renals and P Swietojanski, Neural networks for distant speech recognition. in Proc IEEE HSCMA 2014 pp. 172-176. DOI:10.1109/HSCMA.2014.6843274 Below description is (mostly) copied from ICSI documentation for convenience. ================================================================================= Simple diagram of the seating arrangement in the ICSI meeting room. The ordering of seat numbers is as specified below, but their alignment with microphones may not always be as precise as indicated here. Also, the seat number only indicates where the participant started the meeting. Since most of the microphones are wireless, they were able to move around. Door 1 2 3 4 ----------------------------------------------------------------------- | | | | S | | | | c | | | | r 9| D1 D2 | D3 PDA D4 | | e | | | | e | | | | n | | | | ----------------------------------------------------------------------- 8 7 6 5 D1, D2, D3, D4 - Desktop PZM microphones PDA - The mockup PDA with two cheap microphones The following are the TYPICAL channel assignments, although a handful of meetings (including Bmr003, Btr001, Btr002) differed in assignment. The mapping from the above, to the actual waveform channels in the corpora, and (this recipe for a signle distant mic case) is: D1 - chanE - (this recipe: sdm3) D2 - chanF - (this recipe: sdm4) D3 - chan6 - (this recipe: sdm1) D4 - chan7 - (this recipe: sdm2) PDA left - chanC PDA right - chanD ----------- Note (Pawel): The mapping for headsets is being extracted from mrt files. In cases where IHM channels are missing for some speakers in some meetings, in this recipe we either back off to distant channel (typically D2, default) or (optionally) skip this speaker's segments entirely from processing. This is not the case for eval set, where all the channels come with the expected recordings, and split is the same for all conditions (thus allowing for direct comparisons between IHM, SDM and MDM settings). NOTE on data: The ICSI data is freely available from the website (see `download` below) and also as LDC corpora. The annotations that we download below are same as LDC2004T04, but there are some differences in the audio data, specifically in the session names. Some sessions (Bns...) are named (bns...) in the LDC corpus, and the Mix-Headset wav files are not available from the LDC corpus. So we recommend downloading the public version even if you have an LDC subscription. The public data also includes annotations of roles, dialog, summary etc. but we have not included them in this recipe. N) defaultdict)Path)DictList NamedTupleOptionalTupleUnion)tqdm)$validate_recordings_and_supervisions) AudioSource Recording RecordingSet)read_sph) fix_manifests)normalize_text_ami) AlignmentItemSupervisionSegmentSupervisionSet)PathlikeSeconds add_durationsresumable_download)FBdb001Bed002Bed003Bed004Bed005Bed006Bed008Bed009Bed010Bed011Bed012Bed013Bed014Bed015Bed016Bed017Bmr001Bmr002Bmr003Bmr005Bmr006Bmr007Bmr008Bmr009Bmr010Bmr011Bmr012Bmr014Bmr015Bmr016Bmr019Bmr020Bmr022Bmr023Bmr024Bmr025Bmr026Bmr027Bmr028Bmr029Bmr030Bmr031Bns002Bns003Bro003Bro004Bro005Bro007Bro008Bro010Bro011Bro012Bro013Bro014Bro015Bro016Bro017Bro018Bro019Bro022Bro023Bro024Bro025Bro026Bro027Bro028Bsr001Btr001Btr002Buw001Bmr021Bns001)Bmr013Bmr018Bro021traindevtest) 0123456789ABrn)EFrnro)ihmsdmmdmihm-mixF&http://https://groups.inf.ed.ac.uk/amirv target_dirforce_downloadurlmicreturnc CsttjtddD]o}|dvr]t|D]E}|d|d|d}||}|jddd|d |d}z t|||d Wqt j j y[} zt d |WYd} ~ qd} ~ wwq |d |d }||}|jddd|d}t|||d q dS)NzDownloading ICSI meetingsdesc)rvrwrxz/ICSIsignals/SPH/z/chanz.sphTparentsexist_okchanfilenamer|zSkipping failed download from z/ICSIsignals/NXT/z.interaction.wavzMix-Headset.wav)r itertoolschain from_iterable PARTITIONSvaluesMIC_TO_CHANNELSmkdirrurlliberror HTTPErrorloggingwarning) r{r|r}r~itemchannelwav_urlwav_dirwav_patherG/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/recipes/icsi.pydownload_audios6    r.http://groups.inf.ed.ac.uk/ami audio_dirtranscripts_dirc Cs.t|}|r t|n|d}|rt|n|d}t||||td|r4|s4td||S|d}|d}t||d|dt||d |dt|d }|||rht|d |Wd n1srwYt|d}| d |Wd |S1swY|S) a Download ICSI audio and annotations for provided microphone setting. :param target_dir: Pathlike, the path in which audio and transcripts dir are created by default. :param audio_dir: Pathlike (default = '/audio'), the path to store the audio data. :param transcripts_dir: Pathlike (default = '/transcripts'), path to store the transcripts data :param force_download: bool (default = False), if True, download even if file is present. :param url: str (default = 'http://groups.inf.ed.ac.uk/ami'), download URL. :param mic: str {'ihm','ihm-mix','sdm','mdm'}, type of mic setting. :return: the path to downloaded and extracted directory with data. speech transcriptszDownloading ICSI annotationsz/Skip downloading transcripts as they exist in: z4/ICSICorpusAnnotations/ICSI_original_transcripts.zipz(/ICSICorpusAnnotations/ICSI_core_NXT.zipzICSI_original_transcripts.ziprzICSI_core_NXT.zipNztranscripts/preambles.mrt) rrrinfoexistsrzipfileZipFile extractallrenameextract) r{rrr|r}r~annotations_url_mrtannotations_url_nxtzrrr download_icsisF       rc@sBeZdZUeed<eed<eed<eed<eed<eeed<dS)IcsiSegmentAnnotationtextspeakergender start_timeend_timewordsN)__name__ __module__ __qualname__str__annotations__rrrrrrrrs rupper normalizec)stt}tt}tt}t|d[}t|}|D]I}|jdkre|jd}|D]:} | jdkrd| D]0} | jdkrFddt | D||<q3| jdkrc| D]} d | jvrY| jd nd ||| jd <qMq3q*qWdn1spwYi} |d  d D]t} | j d\}}}g}d}t| =}t|}|D]+}|jdkrq|durd|jvr|jd}t |jd}t |jd}|||fqWdn1swY|dust|dkrq~||f}|||}|||f| |<q~i}|d d D]} | j d\}}}||f}|| vrq| |\}}}g}d}t| T}t|}t |D]@\}}|jdksRd|jvsR|jddksRd|jvsR|jddkrTq,t |jd}t |jd}||||jfq,Wdn 1sxwY|||f||<qtt}| D]\}\}}}||\}}}|d||f}|D]\ttfdd|}t|dkrq|dd} |dd}!g}"|D]Z}#t| t|#ddd}$t|!t|#ddd}%t|%|$ dd}&t|#d |d!}'t|'dkrq|&dkrtd"|dd|d|d#| d$|!d% q|"t|$|&|'d&qd'd(d)|"D}(||t|(||d| |!|"d*qq||fS)+Nz preambles.mrtMeetingSessionPreambleChannelscSsi|] \}}|jd|qS)Name)attrib).0idxrrrr s z*parse_icsi_annotations.. ParticipantsChannelchan6rSegmentsz*.xmlrsegment participant starttimeendtimerWordsFwcs|dko |dkS)Nrr)rseg_end seg_startrrXsz(parse_icsi_annotations..r)ndigitsi>) sampling_raterSegment z at time -z5 has a word with zero or negative duration. Skipping.)startdurationsymbol css|]}|jVqdSN)r)rrrrr osz)parse_icsi_annotations..)rrrrrr)rlistdictopenETparsegetroottagr enumerateglobstemsplitfloatappendlenritemsfiltermaxroundminrrrrrjoinr))rr annotationschannel_to_idx_mapspk_to_channel_mapfrootchild meeting_id grandchildgreatgrandchildrsegmentsfilemeet_idlocal_id_ spk_segmentsspk_idtreesegrrkeyrr seg_wordscombine_with_nextiword spk_wordsnew_keyrendword_alignmentsrw_startw_endw_durw_symbolrrrrparse_icsi_annotationss                        & $r audio_pathsr save_to_wav output_dirc s0ddl}ddlm}|dd|}durttg}t|ddD]o\}vr8ddtgd D<t|d\} } |rtt |d } | j d d d t|D]\} } t| \}}| | j d }| ||j | ||| <qU|tfddt|D| | jd| jd| dq#t|S)Nr)groupbycSs |jdS)N)parts)prrrrs z'prepare_audio_grouped..Preparing audiorcSsi|]\}}||qSrr)rrcrrrrs z)prepare_audio_grouped..)chanEchanFrchan7wavsTr.wavcs8g|]}|jvrtd|jgt|dqS)rtypechannelssource)rr r)r audio_pathr session_namerr sz)prepare_audio_grouped..ridsourcesr num_samplesr) soundfilecytoolzrrrr rrrrrrwriteTrr sortedshaperfrom_recordings)rrrrsfr channel_wavs recordings channel_pathsaudio_sf samplerate session_dirr r)audiorrrr*rprepare_audio_groupedsD         r@c Csddl}g}t|ddD]c}|jd}|jdkr'||}|j}|j} |j} n.t|\}} |j \} }|rUt |d|} | j ddd| |j d} | | |j| | }|t|td tt| t|d g| ||| d q t|S) Nrrrrr$r#Trrr%r-)r1r rsuffix SoundFileframesr'r=rr6rrrr3r4rr r rrangerrr7) rrrr8r:r)r+r< num_frames num_channelsr=r>rrrrprepare_audio_singles@       rGr?rc sfddD}g}t|ddD]e}|jD]_}|j\}||j|f}|dur*qt|D]G\} } | j| j} | j|jkrOt d|jd|d| dq.| dkru| t |jd|d| |j| j| |d | j | j| jd | jid q.qqt|S) Ncs.i|]}|d|d|df|qS)rrr)rr rrrrrs z+prepare_supervision_ihm..Preparing supervisionrrrz8 exceeds recording duration. Not adding to supervisions.rEnglishr  r. recording_idrrrlanguagerrr alignment)r r/r'getr.rrrrrrrrrrrrr from_segments) r?rrannotation_by_id_and_channelr recordingr(r annotationseg_idxseg_inforrrHrprepare_supervision_ihmsH     "rVc Cstt}|D] \}}||d|qg}t|ddD]`}||j}|jd}|dur9t d|jqt |j dkrKt d|jdqt |D].\} } | j | j} | dkr}|t|jd| |j| j| |jd | j| j| jd | jid qOqt|S) NrrIrz"No annotation found for recording rz"More than 1 channels in recording z. Skipping this recording.rrJr rK)rrrextendr rOr.r/rrrr'rrrrr channel_idsrrrrrrP) r?rannotation_by_idr valuerrRrSr(rTrUrrrrprepare_supervision_other sF     r[kaldinormalize_textc s t|}|dur t|n|d}|sJd||s&Jd||tvs4Jd|d|r>|dus>Jd|durMt|}|jdddtd t||d \}}td d t|}|d ksn|dkr| d|d} t t | |d kr|nd||} n"|dks|dkrt |r| d|dn| d} t t | ||} td|d krt| ||nt| |} tt} dD]H| fdd} | fdd}t| |\} }t| ||dur| |d|dd||d|dd| |d| <qt| S)a Returns the manifests which consist of the Recordings and Supervisions :param audio_dir: Pathlike, the path which holds the audio data :param transcripts_dir: Pathlike, the path which holds the transcripts data :param output_dir: Pathlike, the path where to write the manifests - `None` means manifests aren't stored on disk. :param mic: str {'ihm','ihm-mix','sdm','mdm'}, type of mic to use. :param normalize_text: str {'none', 'upper', 'kaldi'} normalization of text :param save_to_wav: bool, whether to save the sph audio to wav format :return: a Dict whose key is ('train', 'dev', 'test'), and the values are dicts of manifests under keys 'recordings' and 'supervisions'. NrzNo such directory: zMic z not supportedz/output_dir must be specified when saving to wavTrzParsing ICSI transcriptsrzPreparing recording manifestsrrvrxzchan[z].sphrwryz*.wavzPreparing supervision manifestsrdc|jtvSr)r.rxpartrrrzprepare_icsi..cr^r)rLrr_rarrrrczicsi- _recordings_z .jsonl.gz_supervisions_)r: supervisions)ris_dirrkeysrrrrrrglobr@rrrGrVr[rrrrr to_file)rrrr~r]rrrr'rr? supervision manifests audio_partsupervision_partrrar prepare_icsiKsj       ro)Frzrv)rNNFrrv)r)NFN)FN)NNrvr\F)9__doc__rrrxml.etree.ElementTreeetree ElementTreerr collectionsrpathlibrtypingrrrrrr tqdm.autor lhotser lhotse.audior r rlhotse.audio.backendr lhotse.qarlhotse.recipes.utilsrlhotse.supervisionrrr lhotse.utilsrrrrrrboolrrrrintrr@rGrVr[rorrrrs \           " B ,  < 1 4 -