o 2wi$@sdZddlZddlmZddlmZddlmZmZm Z m Z ddl m Z ddl mZddlmZmZdd lmZdd lmZmZdd lmZmZmZgd Z  ddedede ededeee eefff ddZdededeeeeffddZdefddZ ddZ!ddZ"dS) aV This is a part of English HUB4 corpus. It contains Broadcast News data, i.e. audio and transcripts of TV news. We currently support the following LDC packages: 1997 English Broadcast News Train (HUB4) Speech LDC98S71 Transcripts LDC98T28 This data is not available for free - your institution needs to have an LDC subscription. N)chain)Path)DictListOptionalUnion)sliding_window)$validate_recordings_and_supervisions) Recording RecordingSet) fix_manifests)SupervisionSegmentSupervisionSet)Pathlikecheck_and_rglobrecursion_limit)z3s  z)prepare_broadcast_news..icSsg|] \}}t||qSr)make_supervisions)rrrrrr ;sz*prepare_broadcast_news..Ncs|]}|dVqdS)sectionsNrrsupsrrrr?csr#)segmentsNrr%rrrrBr'T)parentsexist_okz&broadcast-news_recordings_all.jsonl.gzz$broadcast-news_sections_all.jsonl.gzz$broadcast-news_segments_all.jsonl.gz) recordingsr$r()rr from_recordingsrzipr from_segmentsr from_iterabler r rmkdirto_file) rrrr audio_paths sgml_pathsr+supervisions_listsection_supervisionssegment_supervisionsrrrprepare_broadcast_news s@    r7 sgml_path recordingcCst|}|d}g}g}d}t|ddD]\}}t|jd} |t|jd|d|j| t t|jd| dd d|jd |jd |jd d d|dD]} | j D]y} ddt |  dD} | slq[g} g}t | D]\}}td|}| t|d||qt| t| jdttd| |D]/\\}}}|t|jd|d|j|t ||dd d|jd || jd| jdd q|d7}q[qVq||dS)zICreate supervisions for sections and segments for a given HUB4 recording.episodersection starttime_section03dendtimer)ndigitslanguagetypeprogram)r;rC)id recording_idstartdurationchannelrAcustomturncs.g|]trtfddtDsqS)c3s|]}|VqdSN startswith)rblrrrsz/make_supervisions...)lenanyEXCLUDE_BEGINNINGS)rrrOrr"sz%make_supervisions.. zsec="?(\d+\.?\d*)"?_segment04dspeakerspkrtype) rDrErFrGrHrAtextrZgender)r$r() try_parsefind enumeratefind_allfloatattrsappendr rDroundchildrenstrsplitgroup_lines_in_time_markerresearchgroupr-rstrip)r8r9docr:r5text_supervisionstext_idxsec_idxr; sec_startrJchildlinestimestexts time_markerr\matchrFendrrrr Zsh       ,r c Cszddlm}Wntdz||dWStyqddl}ddlm}|jd|ddd  d d  d d  }|$}|j d|d|j d|dddd||dWdYS1siwYYdSw)z Return a BeautifulSoup object created from an SGML file. If it runs into Unicode decoding errors, it will try to determine the file's encoding and use iconv to automatically convert it to UTF-8. r) BeautifulSoupzVBefore running BroadcastNews data preparation, you should "pip install beautifulsoup4"z html.parserN)NamedTemporaryFilez file -bi T)shellr\;zcharset=z iconv -f z -t utf-8 -o  )r|checkr\)bs4rz ImportError read_textUnicodeDecodeError subprocesstempfiler{ check_outputrhreplacermrunnameread)r8rzrr{encodingfrrrr^s6   *r^cCs,ddlm}tdd||dddDdS) zXThis is a helper for the situation when a