o Si'@sdZddlZddlmZddlmZmZddlmZddl m Z m Z m Z ddl mZddlmZmZmZmZdd lmZmZdd lmZmZmZmZ    d"d ed e ede ede edede ee eefff ddZ   d#d ed e ede edede ee eefff ddZ  d$d edede edede ee eefff ddZ  d%dedededefddZ!dedefd d!Z"dS)&a About the Callhome American English CALLHOME American English Speech was developed by the Linguistic Data Consortium (LDC) and consists of 120 unscripted 30-minute telephone conversations between native speakers of English. All calls originated in North America; 90 of the 120 calls were placed to various locations outisde of North America, while the remaining 30 calls were made within North America. Most participants called family members or close friends. This script support setup of two different tasks -- either ASR or SRE For ASR, the following LDC corpora are relevant Speech : LDC97S42 Transcripts : LDC97T14 Lexicon : LDC97L20 (not actually used) For SRE, this script prepares data for speaker diarization on a portion of CALLHOME used in the 2000 NIST speaker recognition evaluation. The 2000 NIST SRE data is required. LDC catalog number LDC2001S97. N)Counter)DecimalInvalidOperation)Path)DictOptionalUnion)tqdm) Recording RecordingSetSupervisionSegmentSupervisionSet) fix_manifests$validate_recordings_and_supervisions)Pathlikecheck_and_rglobresumable_download safe_extractF audio_dirrttm_dirtranscript_dir output_dirabsolute_pathsreturncCs$|dur t||||St||||S)a< Prepare manifests for the CallHome American English corpus. We create two manifests: one with recordings, and the other one with text supervisions. Depending on the value of transcript_dir, will prepare either * data for ASR task (expected LDC corpora ``LDC97S42`` and ``LDC97T14``) * or the SRE task (expected corpus ``LDC2001S97``) :param audio_dir: Path to ``LDC97S42``or ``LDC2001S97`` content :param transcript_dir: Path to the ``LDC97T14`` content :param rttm_dir: Path to the transcripts directory. If not provided, the transcripts will be downloaded. :param absolute_paths: Whether to return absolute or relative (to the corpus dir) paths for recordings. :return: A dict with manifests. The keys are: ``{'recordings', 'supervisions'}``. N)prepare_callhome_english_asrprepare_callhome_english_sre)rrrrrrS/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/recipes/callhome_english.pyprepare_callhome_english%srcs|durt}|d}t|}t|d}tfddt|D}t||\}}t|||durKt|}|j ddd| |d| |d ||d S) a Prepare manifests for the Callhome American English portion prepartion. We create two manifests: one with recordings, and the other one with text supervisions. :param audio_dir: Path to ``LDC2001S97`` package. :param rttm_dir: Path to the transcripts directory. If not provided, the transcripts will be downloaded. :param output_dir: Directory where the manifests should be written. Can be omitted to avoid writing. :param absolute_paths: Whether to return absolute or relative (to the corpus dir) paths for recordings. :return: A dict with manifests. The keys are: ``{'recordings', 'supervisions'}``. Nz fullref.rttm*.sphc3&|]}tj|r dnddVqdSN)relative_path_depthr from_file.0prrr f  z/prepare_callhome_english_sre..Tparentsexist_okzrecordings.jsonzsupervisions.json recordings supervisions) download_callhome_metadata read_rttmrr from_recordingsr rrrmkdirto_json)rrrr rttm_pathr1 audio_pathsr0rr)rrKs    rcsbt|}t|}i}dD]!}t|d|ddd}tfddt|D}t|d|d }g} |D]} d } t} | D]U} | } | sMqD| d rSqDz!| j d d \}}}}t t |t |}|d krnWqD| | WqDty| dd| | d<YqDty| dd| | d<YqDw| D]R} | j}| j d d \}}}}|dd}t t |t |}|d krqt |}| t|||t|d td|d|d|d|dd| d|d| d7} qq7t| } t|| \}} t|| |dur't|}|jddd||d|d| |d|d|| d||<q |S)aX Prepare manifests for the CallHome American English corpus. We create two manifests: one with recordings, and the other one with text supervisions. :param audio_dir: Path to ``LDC97S42`` content :param transcript_dir: Path to the ``LDC97T14`` content :param output_dir: Directory where the manifests should be written. Can be omitted to avoid writing. :param absolute_paths: Whether to return absolute or relative (to the corpus dir) paths for recordings. :return: A dict with manifests. The keys are: ``{'recordings', 'supervisions'}``. )evaltesttraindevtestdatar9evltestrc3r r!r$r&r)rrr*r+z/prepare_callhome_english_asr.. transcrptz*.txtr#)maxsplit :A_z0>2sz0>5d) recording_idstartdurationchannelspeakeridtextNTr,zcallhome-english_recordings_z .jsonl.gzzcallhome-english_supervisions_r/)rrreplacer r4r list read_text splitlinesstrip startswithsplitfloatrappendr ValueErrorstemr ordr from_segmentsrrr5to_file)rrrr manifestsrVr8r0transcript_pathsr1r(idxpostprocessed_lineslinerIendspkrNrJrHrr)rrvs          r.6http://www.openslr.org/resources/10/sre2000-key.tar.gz target_dirforce_downloadurlcCst|}|d}|r|S|jdddd}||}t|||dt|}t||dWd|S1s9wY|S)Nz sre2000-keyTr,zsre2000-key.tar.gz)filenamerh)path)ris_dirr5rtarfileopenr)rgrhrisre_dirtar_nametar_pathtarrrrr2s  r2rkc Cst|}g}t}|D]E}|\ }}}}} }}} }}t|t| t|}} }| dkr2q||d7<|t|d||||| ||d| ddqt |S)NgrOrGEnglish)rMrHrIrJrKrLlanguage) rrRrSrrVrWintrXr r r\) rklinessupsrec_cntrrbrGrHrKrIrJrLrrrr3s*  r3)NNNF)NNF)NF)reFrf)#__doc__rm collectionsrdecimalrrpathlibrtypingrrr tqdm.autor lhotser r r r lhotse.qarr lhotse.utilsrrrrboolstrrrrr2r3rrrrs    ( . k