o Si[#@sdZddlZddlmZddlmZddlmZddlm Z m Z m Z m Z ddl mZddlmZmZdd lmZdd lmZmZdd lmZmZmZgd Zd dZ  dde ede ede edede ee eefff ddZde ede efddZ dS)a The GALE Arabic Broadcast corpus consists of the following LDC corpora: GALE Arabic phase 2 Conversation Speech LDC2013S02: http://catalog.ldc.upenn.edu/LDC2013S02 LDC2013S07: http://catalog.ldc.upenn.edu/LDC2013S07 LDC2013T17: http://catalog.ldc.upenn.edu/LDC2013T17 LDC2013T04: http://catalog.ldc.upenn.edu/LDC2013T04 # GALE Arabic phase 2 News Speech LDC2014S07: http://catalog.ldc.upenn.edu/LDC2014S07 LDC2015S01: http://catalog.ldc.upenn.edu/LDC2015S01 LDC2014T17: http://catalog.ldc.upenn.edu/LDC2014T17 LDC2015T01: http://catalog.ldc.upenn.edu/LDC2015T01 # GALE Arabic phase 3 Conversation Speech LDC2015S11: http://catalog.ldc.upenn.edu/LDC2015S11 LDC2016S01: http://catalog.ldc.upenn.edu/LDC2016S01 LDC2015T16: http://catalog.ldc.upenn.edu/LDC2015T16 LDC2016T06: http://catalog.ldc.upenn.edu/LDC2016T06 # GALE Arabic phase 3 News Speech LDC2016S07: http://catalog.ldc.upenn.edu/LDC2016S07 LDC2017S02: http://catalog.ldc.upenn.edu/LDC2017S02 LDC2016T17: http://catalog.ldc.upenn.edu/LDC2016T17 LDC2017T04: http://catalog.ldc.upenn.edu/LDC2017T04 # GALE Arabic phase 4 Conversation Speech LDC2017S15: http://catalog.ldc.upenn.edu/LDC2017S15 LDC2017T12: http://catalog.ldc.upenn.edu/LDC2017T12 # GALE Arabic phase 4 News Speech LDC2018S05: http://catalog.ldc.upenn.edu/LDC2018S05 LDC2018T14: http://catalog.ldc.upenn.edu/LDC2018T14 # Training: 941h Testing: 10.4h The data has two types of speech: conversational and report. There is no separate dev set provided with the corpus. The `S` corpora contain speech data and the `T` corpora contain the corresponding transcriptions. This recipe prepares any subset of these corpora provided as arguments, but pairs of speech and transcript corpora must be present. E.g. to only prepare phase 3 news speech, the arguments `audio_dirs = ["/export/data/LDC2016S07","/export/data/LDC2017S02"]` and `transcript_dirs = ["/export/data/LDC2016T17","/export/data/LDC2017T04"]` must be provided to the `prepare_gale_arabic` method. This data is not available for free - your institution needs to have an LDC subscription. N) defaultdict)chain)Path)DictListOptionalUnion)$validate_recordings_and_supervisions) Recording RecordingSet) fix_manifests)SupervisionSegmentSupervisionSet)Pathlikecheck_and_rglobis_module_available) "ALAM_WITHEVENT_ARB_20070116_205800"ALAM_WITHEVENT_ARB_20070206_205801"ALAM_WITHEVENT_ARB_20070213_205800"ALAM_WITHEVENT_ARB_20070227_205800"ALAM_WITHEVENT_ARB_20070306_205800"ALAM_WITHEVENT_ARB_20070313_205800$ARABIYA_FROMIRAQ_ARB_20070216_175800$ARABIYA_FROMIRAQ_ARB_20070223_175801$ARABIYA_FROMIRAQ_ARB_20070302_175801$ARABIYA_FROMIRAQ_ARB_20070309_175800cCstdstddS)NpandaszGale Arabic data preparation requires the 'pandas' package to be installed. Please install it with 'pip install pandas' and try again)r ImportErrorrrN/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/recipes/gale_arabic.pycheck_dependenciesPs r T audio_dirstranscript_dirs output_dirabsolute_pathsreturnc s|t|t|ks Jdtdttddtdd|DD}tdd|D}dd|D}td tfd d | D}td t t |}t ||\}}t||tt}|d d|ddd|d<|dd|ddd|d<|durtdt|}|jddddD] } || d|d| d|| d|d| dq|S)a} Prepare manifests for GALE Arabic Broadcast speech corpus. :param audio_dirs: List of paths to audio corpora. :param transcripts_dirs: List of paths to transcript corpora. :param output_dir: Directory where the manifests should be written. Can be omitted to avoid writing. :return: A dict with manifests. The keys are: ``{'recordings', 'supervisions'}``. z@Paths to the same speech and transcript corpora must be providedz5Reading audio and transcript paths from provided dirscSsi|]}|j|qSr)stem.0prrr psz'prepare_gale_arabic..cSs$g|]}dD] }t||ddqqS))z*.wavz*.flacF)strictr)r(dirextrrr ss z'prepare_gale_arabic..cSsg|]}t|dqS)z*.tdfr,)r(r-rrrr/|cSsg|]}|qSrrr'rrrr/~zPreparing recordings manifestc3s&|]}tj|r dnddVqdS)N)relative_path_depth)r from_filer'r$rr s  z&prepare_gale_arabic..zPreparing supervisions manifestcS |jtvSNidTESTrrrr z%prepare_gale_arabic..cSr7r8 recording_idr;srrrr>r?) recordings supervisionstestcS |jtvSr8r9r<rrrr>r?cSrGr8r@rBrrrr>r?trainNz Writing manifests to JSONL filesT)parentsexist_ok)rHrFrDzgale-arabic_recordings_z .jsonl.gzrEzgale-arabic_supervisions_)lenlogginginforrr from_iterabler from_recordingsvaluesr from_segmentsparse_transcriptsr r dictfiltermkdirto_file) r!r"r#r$ audio_pathstranscript_pathsrDrE manifestspartrr5rprepare_gale_arabicXs^             r[rXc srtddlg}t}|D]}j|ddtdgdttttttddddd }||jd k}|d  d d |d <|d fdd |d<|d fdd |d<| D]\\}}|d d|dd|}t |d|ddd}||vs|dkrqY| || t||d |d||d|dd|d|d|d|d|d|d|d|dd d! qYq|S)"Nr r2 ) reco_idchannelstartendspeakergenderdialecttextsectionturnsegment section_typesu_type)r^r_r`rarbreTF) delimiterskiprowsusecolsnamesdtypeskipinitialspaceerror_bad_lineswarn_bad_linesz no speakerr^cSs|ddS)Nz.sph)stripreplacexrrrr>r1z#parse_transcripts..rbcs|s |ddS|S)N*rs)isnullrurtrvpdrrr>srecs|s |S|Sr8)ryrtrvrzrrr>r0-rar`)ndigitsrcArabicr_rdrfrgrhrirj)rdrfrgrhrirj) r:rAr`durationrbrclanguagerer_custom)r rsetread_csvrangestrintfloatrbapplyiterrowsroundaddappendr ) rXrEsupervision_idsfiledfidxrowsupervision_idrrrzrrRsp!  rR)NT)!__doc__rL collectionsr itertoolsrpathlibrtypingrrrrlhotser lhotse.audior r lhotse.qar lhotse.supervisionr r lhotse.utilsrrrr;r boolrr[rRrrrrs63       O