o Si|#@sNdZddlZddlmZddlmZddlmZddlm Z m Z m Z ddl m Z ddl mZmZdd lmZmZmZdd lmZdd lmZmZdd lmZmZmZmZd ZgdgddZ  d$dede e defddZ!d%de"de de"fddZ#    d&dede ed e e"d!e e"de e de e"e e"e eeffff d"d#Z$dS)'u DiPCo is a speech data corpus that simulates a “dinner party” scenario taking place in an everyday home environment. The corpus was created by recording multiple groups of four Amazon employee volunteers having a natural conversation in English around a dining table. The participants were recorded by a single-channel close-talk microphone and by five far-field 7-microphone array devices positioned at different locations in the recording room. The dataset contains the audio recordings and human labeled transcripts of a total of 10 sessions with a duration between 15 and 45 minutes. The corpus was created to advance in the field of noise robust and distant speech processing and is intended to serve as a public research and benchmarking data set. The corpus is made availabe under the CDLA-Permissive license. More details and download link: https://www.amazon.science/publications/dipco-dinner-party-corpus N) defaultdict)datetime)Path)DictOptionalUnion)tqdm) fix_manifests$validate_recordings_and_supervisions) AudioSource Recording RecordingSet)normalize_text_chime6)SupervisionSegmentSupervisionSet)Pathlike add_durationsresumable_download safe_extractz(https://s3.amazonaws.com/dipco/DiPCo.tgz)S02S04S05S09S10)S01S03S06S07S08deveval.F target_dirforce_downloadreturncCsht|}|jddd|d}tt||dt|}t||dWd|S1s-wY|S)a Download and untar the dataset :param target_dir: Pathlike, the path of the dir to storage the dataset. :param force_download: Bool, if True, download the tars no matter if the tars exist. :return: the path to downloaded and extracted directory with data. Tparentsexist_okz DiPCo.tgz)filenamer$)pathN)rmkdirr CORPUS_URLtarfileopenr)r#r$tar_pathtarr1H/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/recipes/dipco.pydownload_dipco's   r3sessionuse_chime7_offsetcCs(|rt|dd}dd|dS|S)z Returns the session ID :param session: str, the session ID. :param use_chime7_offset: bool, if True, offset session IDs (from CHiME-7 challenge). :return: str, the session ID. NS02d)int)r4r5session_numberr1r1r2get_session_id;s r<mdmkaldi corpus_dir output_dirmicnormalize_textcCsxddl}ddl}|dvsJdt|}|sJd|tt}|dur2t|}|jddddD]}g} g} |d kri} t|D]Y} d d |d || d D} g}t | D]\}}| t d|gt |d|j dd}|| | |f<q^|t | d}| tt| ||t|j|j|j|jdqEnRt|D]M} dd |d || dD} g}t t| D]\}}| t d|gt |dq|t | d}| tt| ||t|j|j|j|jdqdd}t|D]} t|d|| dl}||}t |D]Z\}}|d}|d kr&| | |fnttd}||dd}||dd}t| |}| t|d|||t|| dd |t|d!|d"d#||d$|d%|d&d'd( qWdn 1sywYqtt| t !| \}}t"|||dur|#|d)|d*|d+|#|d)|d,|d+||d-||<q4|S).a Returns the manifests which consist of the Recordings and Supervisions :param corpus_dir: Pathlike, the path of the data dir. :param output_dir: Pathlike, the path where to write the manifests. :param mic: str, the microphone type to use, choose from "ihm" (close-talk) or "mdm" (multi-microphone array) settings. For MDM, there are 5 array devices with 7 channels each, so the resulting recordings will have 35 channels. :param normalize_text: str, the text normalization to apply. Choose from "none", "upper", or "kaldi". "kaldi" is the default and is the same normalization used in Kaldi's CHiME-6 recipe. :param use_chime7_offset: bool, if True, offset session IDs (from CHiME-7 challenge). :return: a Dict whose key is the dataset part ("dev" and "eval"), and the value is Dicts with the keys 'recordings' and 'supervisions'. rN)ihmr=z!mic must be one of 'ihm' or 'mdm'zNo such directory: Tr&rrCcSg|]}|qSr1r1.0pr1r1r2 vz!prepare_dipco..audioz_P*.wavfile)typechannelssource_r6)idsources sampling_rate num_samplesdurationcSrDr1r1rEr1r1r2rHrIz_U*.wavcSst|dtdddS)Nz %H:%M:%S.%filr6)dtstrptime total_seconds)xr1r1r2szprepare_dipco..transcriptionsz.json speaker_id# start_timez close-talkend_time-i>)rRwords) normalizeEnglishgender nativeness mother_tongue)rdre) rP recording_idstartrTchanneltextlanguagespeakerrccustomzdipco-_supervisions_z .jsonl.gz _recordings_) recordings supervisions)$json soundfileris_dirrdictr+ DATASET_PARTSrglob enumerateappendr strstemsplit SoundFiler r<r: samplerateframessortedr.loadlistrangerrrr r from_recordingsr from_segmentsr to_file)r?r@rArBr5rqsf manifestspartrorpglobal_spk_channel_mapr4 audio_pathsrQidx audio_pathspk_idaudio_sf _get_timef transcriptsegmentrhrgend session_id recording_setsupervision_setr1r1r2 prepare_dipcoKs               r)r"F)F)Nr=r>F)%__doc__r- collectionsrrrUpathlibrtypingrrrrlhotser r lhotse.audior r r lhotse.recipes.utilsrlhotse.supervisionrr lhotse.utilsrrrrr,ruboolr3ryr<rr1r1r1r2sV