o 2wi*q!@sdZddlZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z m Z m Z ddlmZddlmZddlmZmZmZmZmZmZddlZddlmZdd lmZmZdd lm Z m!Z!m"Z"dd l#m$Z$m%Z%dd l&m'Z'm(Z(dd l)m*Z*m+Z+m,Z,gdddgddgdZ-gdddggddZ.dZ/dZ0 d7de*defddZ1       ! " d8d#e*d$ee*d%eee2ee2fd&e2d'e3d(e3d)e3d*e4d+e4d,e*d-e2d.e3dee2ee2ee"e(ffffd/d0Z5 d9d#e*d1e4d2eee2de3fd3d4Z6Gd5d6d6Z7dS):a The CHiME-6 dataset is a collection of over 50 hours of conversational speech recordings collected from twenty real dinner parties that have taken place in real homes. The recordings have been made using multiple 4-channel microphone arrays and have been fully transcribed. The dataset features: - simultaneous recordings from multiple microphone arrays; - real conversation, i.e. talkers speaking in a relaxed and unscripted fashion; - a range of room acoustics from 20 different homes each with two or three separate recording areas; - real domestic noise backgrounds, e.g., kitchen appliances, air conditioning, movement, etc. Fully-transcribed utterances are provided in continuous audio with ground truth speaker labels and start/end time annotations for segmentation. The dataset was used for the 5th and 6th CHiME Speech Separation and Recognition Challenge. Further information and an open source baseline speech recognition system are available online (http://spandh.dcs.shef.ac.uk/chime_challenge/chime2018). NOTE: The CHiME-5 and CHiME-6 datasets are the same, with the only difference that additional software was provided in CHiME-6 to perform array synchronization. We expect that users have already downloaded the CHiME-5 dataset here: https://licensing.sheffield.ac.uk/product/chime5 NOTE: Users can also additionally perform array synchronization as described here: https://github.com/kaldi-asr/kaldi/blob/master/egs/chime6/s5_track1/local/generate_chime6_data.sh We also provide this option in the `prepare_chime6` function. N) defaultdict)ProcessPoolExecutorThreadPoolExecutorwait)datetime)Path)AnyDictListOptionalSequenceUnion)tqdm) fix_manifests$validate_recordings_and_supervisions) AudioSource Recording RecordingSet)TimeFormatConverternormalize_text_chime6)SupervisionSegmentSupervisionSet)Pathlike add_durationsresumable_download)S03S04S05S06S07S08S12S13S16S17S18S19S20S22S23S24S02S09S01S21)traindeveval)rrrrrr r!r"r#r$r%r(r)r*)r-r&r'r.zfhttps://raw.githubusercontent.com/chimechallenge/chime6-synchronisation/master/chime6_audio_edits.jsonz`https://raw.githubusercontent.com/chimechallenge/chime6-synchronisation/master/audio_md5sums.txt. target_dirreturncCs2tdtdtdtdt|t|S)a{ Download the original dataset. This cannot be done automatically because of the license agreement. Please visit the following URL and download the dataset manually: https://licensing.sheffield.ac.uk/product/chime5 :param target_dir: Pathlike, the path of the dir to storage the dataset. :return: the path to downloaded and extracted directory with data. z5We cannot download the CHiME-6 dataset automatically.zAPlease visit the following URL and download the dataset manually:z0https://licensing.sheffield.ac.uk/product/chime5z>Then, please extract the tar files to the following directory:)printr)r3r6R/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/lhotse/recipes/chime6.pydownload_chime6Os   r8allmdmF /usr/bin/soxkaldi corpus_dir output_dir dataset_partsmicuse_reference_arrayperform_array_syncverify_md5_checksumsnum_jobsnum_threads_per_jobsox_pathnormalize_textuse_chime7_splitc 'sddl} dvs Jd|rt|}|jdddd|vr$tt}nt|tr,|g}t| ttsEJdttd |tt j d d |D} |r|sZt d |d }|jddd|D]}|d|jddd|d|jdddqgt| dgddd}|dksJdt||| |dt|t| }td|dt|dfdd | D}t|}Wdn1swYnt|}|rt||| drtdntd|odk}tt}|D]}gg}| rt|t|<dkrit|D]}| r|d vrd!n|}d"d |d| |d#D}t|dkrCt!d$|d%|d&g}t"|D]/\}}|#t$d'd(|d(|d)gt|d*|j%d+d)}d(|d(|d)g||f<qI| &t|d}#t'||t(|j)|j*|j*|j)d,qn]t|D]X}| r|d vrd!n|}d-d |d| |d.D}g}t"t+|D]\}}|#t$d'|gt|d*q| &t|d}#t'||t(|j)|j*|j*|j)d,qt,-dEfd/d0 }t|D]}| r|d vrd!n|}t.|d||d1t}t/0|}t"|D]b\}} | d2}||||rD|d!krD| d3ndd4}!t12| d5}"t12| d6}#|"|#kr]q-|#t3|d7|||"t4|#|" d8d9|!t5| d:| d;d<||d!krd=| vrd=| d=indd> q-Wdn 1swYqt67|}t8|d?\}$}%t9|$|%|dur|rd@n}&|%:|dA|&dB|dC|$:|dAdD|dC|$|%d?||<q|S)Fa Returns the manifests which consist of the Recordings and Supervisions :param corpus_dir: Pathlike, the path of the data dir, either the original CHiME-5 data or the synchronized CHiME-6 data. If former, the `perform_array_sync` must be True. :param output_dir: Pathlike, the path where to write the manifests. :param mic: str, the microphone type to use, choose from "ihm" (close-talk) or "mdm" (multi-microphone array) settings. For MDM, there are 6 array devices with 4 channels each, so the resulting recordings will have 24 channels. :param use_reference_array: bool, if True, use the reference array for MDM setting. Only the supervision segments have the reference array information in the `channel` field. The recordings will still have all the channels in the array. Note that the train set does not have the reference array information. :param perform_array_sync: Bool, if True, perform array synchronization based on: https://github.com/chimechallenge/chime6-synchronisation :param num_jobs: int, the number of jobs to run in parallel for array synchronization. :param num_threads_per_job: int, number of threads to use per job for clock drift correction. Large values may require more memory, so we recommend using a job scheduler. :param sox_path: Pathlike, the path to the sox v14.4.2 binary. Note that different versions of sox may produce different results. :param normalize_text: str, the text normalization method, choose from "none", "upper", "kaldi". The "kaldi" method is the same as Kaldi's text normalization method for CHiME-6. :param verify_md5_checksums: bool, if True, verify the md5 checksums of the audio files. Note that this step is slow so we recommend only doing it once. It can be sped up by using the `num_jobs` argument. :param use_chime7_split: bool, if True, use the new split for CHiME-7 challenge. :return: a Dict whose key is the dataset part ("train", "dev" and "eval"), and the value is Dicts with the keys 'recordings' and 'supervisions'. NOTE: If `perform_array_sync` is True, the synchronized data will be written to `output_dir`/CHiME6. This may take a long time and the output will occupy approximately 160G of storage. We will also create a temporary directory for processing, so the required storage in total will be approximately 300G. rN)ihmr:z"mic must be either 'ihm' or 'mdm'.T)parentsexist_okr9zdataset_parts must be one of z. Found cSsg|]}t|qSr6) DATASET_PARTS).0partr6r6r7 sz"prepare_chime6..z@If `perform_array_sync` is True, `output_dir` must be specified.CHiME6audiotranscriptionsz --versionzutf-8 zv14.4.2a5The sox version must be 14.4.2. Please download the sox v14.4.2 binary from https://sourceforge.net/projects/sox/files/sox/14.4.2/ and specify the path to the binary with the `sox_path` argument.You can also install it in a Conda environment with the following command: conda install -c conda-forge sox=14.4.2)r>r?rG num_workersz&Performing array synchronization with z! jobs. This may take a long time. max_workerscsg|] }jj|dqS))session)submitsynchronize_session)rNrY)chime6_array_synchronizerexr6r7rPs)rVsessionszMD5 checksums verified. All OK.zXMD5 checksums do not match. Please prepare the array-synchronized CHiME-6 dataset again.r:rJ)r&r'r/cSg|]}|qSr6r6rNpr6r6r7rP_P*.wavzNo audio found for session z in z set.filer;)typechannelssource_)idsources sampling_rate num_samplesdurationcSr_r6r6r`r6r6r7rP%rb_U*.wavcsFdkr ||fS|}stt|jSfddt|jDS)NrJcsg|] \}}|jvr|qSr6)rh)rNisrefr6r7rPEsz8prepare_chime6.._get_channel..)listrange num_channels enumeraterk)spk_idrYrs recording)global_spk_channel_maprA recordingsrrr7 _get_channel=s z$prepare_chime6.._get_channel.jsonspeakerrsrr start_timeend_time->)rlwords) normalizeEnglishlocation) rj recording_idstartrnchanneltextlanguager~custom)r{ supervisionsz-refzchime6-_supervisions_z .jsonl.gz _recordings_N); soundfilermkdirrtrMkeys isinstancestrsetissubset itertoolschain from_iterable ValueError subprocess check_outputdecodestripsplitChime6ArraySynchronizerminlenlogginginforr_verify_md5_checksumsr5 RuntimeErrorrdictDATASET_PARTS_CHIME7rglobFileNotFoundErrorrwappendrstem SoundFilerint samplerateframessortedrfrom_recordingsopenjsonloadrhms_to_secondsrrrr from_segmentsrrto_file)'r>r?r@rArBrCrDrErFrGrHrIsfr^ chime6_dirrO sox_versionfuturesri manifestsrrYpart_ audio_pathsrkidx audio_pathrxaudio_sfr|f transcriptsegmentrrend recording_setsupervision_set mic_affixr6)r\r]rzrAr{r7prepare_chime6asj2                   (        #   rrVr^c s.ddltt}|d}ttt|ddit|d}|D]}|j ddd\}}|t|j <q!Wdn1s@wYd t d t ffd d } t |d } durefdd| D} td|dt|d} t t| | | t| d} Wdt| S1swYt| S)Nrz md5sums.txtzDownloading checksum file)descrrTr;)maxsplitrdr4csBtt|d}t|j}|vr||krdSdS)NrbFT)md5rrread hexdigestr)rdchecksumfilename) checksumshashlibr6r7_verify_checksums  z/_verify_md5_checksums.._verify_checksumz*.wavcs$g|]}|jddvr|qS)rir)rr)rNr)r^r6r7rPs$z)_verify_md5_checksums..zVerifying checksum with z workers...rW)total)rrtempfilemkdtemprCHIME6_MD5SUM_FILErrrrrrboolrtrr5rrmaprr9) r>rVr^temp_dir checksum_filerlinerrr all_filesr]resultsr6)rrr^r7rs0     rc @seZdZdZ  d!dededededd f d d Zd edd fd dZ d"ded ede e ee fdd fddZ dedede e edd fddZ d#ded ede e ee fdedd f ddZdedede eeeee ffdd fddZ d"d ede e ee fdd fdd Zd S)$rz: Class for synchronizing CHiME6 array recordings. soxr;r>r?rGrVr4NcCst||_t||_t||_||_|jjddd|jd}ttt|t |}t t ||_ Wdn1s=wYt|jdsR|jd|_dSdS)NTrLrKzaudio_edits.jsonr)rr>r?rGrVrrCHIME6_AUDIO_EDITS_JSONrrrrr audio_editsendswith)selfr>r?rGrVaudio_edits_jsonrr6r6r7__init__s     z Chime6ArraySynchronizer.__init__rYcCsttjd|d|jd}||jvrtd|dS|j|}td|d|j|||dtd|d |j ||||j d td |d |j ||d t |dS)z6 Synchronize a single CHiME6 session. chime6_ri)prefixdirz!No audio edits found for session Nz Correcting z for frame drops...) frame_dropsz for clock drift...) linear_fit num_threadszAdjusting timestamps in z JSON files...)r)rrrr?rrwarningr5correct_frame_dropscorrect_clock_driftrVadjust_json_timestampsshutilrmtree)rrYrsession_audio_editsr6r6r7r[s(   z+Chime6ArraySynchronizer.synchronize_sessionrcCs2t|jd|d}|D]}||j}||}|jjdddt||qt|jd|d}tt } |D]}|j dd dd } | |  |q=| D]>\} } | |vrmtd | d |d qXgg} } | D]}||j}||}| || |qt|| | || d qXdS)NrRrcTrror2rrirUzArray in session z has no frame drops information.edits)rr>r relative_toparentrossymlinkrrtrrritemsrr_apply_edits_to_wav)rr?rYrsession_binaural_wavswavwav_relative_pathwav_output_pathsession_array_wavs array_wavsarray_idwavsin_wavsout_wavsr6r6r7rs:     z+Chime6ArraySynchronizer.correct_frame_dropsrr rcCs"ddl}tjdd|Ddd}|dd|dd|dd}tj|jd|f|jd}|jd}|D]3} | d} t| d|} | d} | | | } | |krSn|dd| d| f|dd| d| f<q6|ddd| f}t|D]\}}|jt |tj ||ddd d d qxdS) NrcSsg|] }t|qSr6)r from_file load_audio)rNrr6r6r7rP'sz?Chime6ArraySynchronizer._apply_edits_to_wav..)axisrUrer;)shapedtyperWAV)rddatarformat) rnp concatenatezerosr rrrwwriter expand_dims)rrr rrx max_spacex_newlength_xeditin_fromin_toout_fromout_torprr6r6r7r!s0$  .z+Chime6ArraySynchronizer._apply_edits_to_wavrrc Cst|d|d}t|dQ}g}|D]8}||} |j| } |jdddd} | |vr@td| d |d q| | |j || || qt |D]} | qSWddS1sewYdS) NrRz_*.wavrWr2rrirUzChannel rz has no clock drift information.)rrrrr?namerrrrrZ_apply_clock_drift_correctionrresult) rr>rYrr session_wavsr]rrrrmic_idfuturer6r6r7rDs6     z+Chime6ArraySynchronizer.correct_clock_driftin_wavout_wavc Cs&|d}|d}|j}t|j}t|}t|}t|tr|}|dddg} |g} d} tj|jd} tt || |D]\} \}}}| d|dt| d }| d|d t| d }|d d ||g}| dkr|dkry| }d |dg}nd|ddd dg}n || 7}d t |dg}||7}|dkr||}||dg7}|dkr| |} qrrr?rrrrrseconds_to_hmsrdump)rrYrin_json relative_pathout_jsoncorrected_uttsfinfoutrrpidr(r)delta_trrr6r6r7rsJ       ( z.Chime6ArraySynchronizer.adjust_json_timestamps)rr;r)Nr;)__name__ __module__ __qualname____doc__rrrrr[r r rrr rrr floatr!rr6r6r6r7rs % /  ' $ Qr)r2) Nr9r:FFFr;r;r<r=F)r;N)8rOrrrrrrr collectionsrconcurrent.futuresrrrrdtpathlibrtypingrr r r r r numpyrrlhotserr lhotse.audiorrrlhotse.recipes.utilsrrlhotse.supervisionrr lhotse.utilsrrrrMrrrr8rrrrrrr6r6r6r7s              *  +