o
Si" @ s d Z ddlZddlZddlZddlZddlZddlZddlZddlm Z ddl
m Z m Z m
Z
ddlmZ ddlmZ ddlmZmZ ddlmZ dd lmZmZ dd
lmZm Z m Z m Z m Z d.d
ede e de fddZ!dddddZ"e#dZ$e#dZ%e#dZ&e#dZ'e#dZ(d e)d e)d e)d e)d e)d!e)f d"d#Z*d$e d%e+de fd&d'Z, ( ( ) ) )d/d*ed+e e d e e) d e e) d e e) d e e) d!e e) de e)e
eef f fd,d-Z-dS )0a
The ATCOSIM Air Traffic Control Simulation Speech corpus is a speech database of air traffic control (ATC) operator speech, provided by Graz University of Technology (TUG) and Eurocontrol Experimental Centre (EEC). It consists of ten hours of speech data, which were recorded during ATC real-time simulations using a close-talk headset microphone. The utterances are in English language and pronounced by ten non-native speakers. The database includes orthographic transcriptions and additional information on speakers and recording sessions. It was recorded an annotated by Konrad Hofbauer.
See https://www.spsc.tugraz.at/databases-and-tools/atcosim-air-traffic-control-simulation-speech-corpus.html for more details.
N)Path)DictOptionalUnion)tqdm)$validate_recordings_and_supervisions) Recording RecordingSet)
fix_manifests)SupervisionSegmentSupervisionSet)PathlikeSecondscompute_num_samplesis_module_availableresumable_download.F
target_dirforce_downloadreturnc C s t dstddd l}t| } | jddd d}| | d }| | }|d }| r:td | d
| d |S td | d|d
|d t
t |d
dkrXtdtj|dd | }| | | rndn
| rtdn| rzdndd }t|jd i |dig} | r| }
|j|
|dkd} | d} |
rt || |j d i || iD ]}
|
r|
s|
s| !|
qn|
" rt#d n
|j$|| fi || i | s|% |& |S ) Npycdlibz#Please 'pip install pycdlib' first.r Tparentsexist_okatcosimz.isoz
.completedz Skipping z because z exists.z3https://www2.spsc.tugraz.at/databases/ATCOSIM/.ISO/l 5 )filenamecompleted_file_sizer rb cd5f0c82be46242a75d3382e340f6dcazMD5 checksum does not match)
ignore_errorsudfrrjolietiso_path/rr_path) rockridgezsymlink not implemented )'r ImportErrorr r mkdiris_filelogginginfor hashlibmd5openread hexdigest RuntimeErrorshutilrmtreePyCdlibhas_udfhas_rock_ridge
has_joliet collectionsdeque
get_recordpopleftfull_path_from_dirrecordlstripis_dirosmakedirs
list_childrenis_dot is_dotdotappend
is_symlinkwarningget_file_from_isoclosetouch)r r r dataset_nameiso_path
corpus_dircompleted_detectorr# path_argrecordsrabs_pathrel_pathchildr( r( J/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/recipes/atcosim.pydownload_atcosim! sb
rW hmmohhellovicinity)hmohhhallo viscinityz \s*z(.*?)z=(\w+)z(\w+)=z +text silence_sym
breath_sym foreign_sym partial_sym unknown_symc C s t d| } g }| D ])}|d dks|d dkr$||dd q |tv r0|t| q || q d| } | d|} | d|} t|| } |d krZ| d d
} n t || } t
|| } d D ]}| ||} qh| d d
} t d| } | } | S )Nz\1r @~ z[EMPTY]z[HNOISE]= )z
[FRAGMENT]z
[NONSENSE]z [UNKNOWN]z AIR SPACEAIRSPACE)
OFF_TALK_PATTERNsubsplitrF FIX_TYPOSjoinupperreplaceFOREIGN_PATTERNINTERRUPTED_PATTERN1INTERRUPTED_PATTERN2WHITESPACE_PATTERNstrip) r` ra rb rc rd re resultwunkr( r( rV text_normalizeg s, r| duration
sampling_ratec C s t | || S )aL
A handful of supervision durations do not compute to a round number of
samples at the original recording sampling rate.
This causes problem later using compute_num_frames(). Full description:
https://github.com/lhotse-speech/lhotse/issues/1064
Return: duration that computes to a round number of samples.
)r )r} r~ r( r( rV fix_duration s
r rk rN
output_dirc C s t dstdddl}t| } | sJ d| |dur*t|}|jddd | d }|j|tjd } t j
|d
dd }
t j
|d dd } t |
d
t| dD ]k\} }
|
jr]qUt|
j|||||d}|dkrmqUt| d |
j |
j |
j d }tj||
jd}t|
jd }td|
j dddd|d|
jdt |
j|j dd||
j |
j d d|
jid
}|
| | | qUW d n1 sw Y W d n1 sw Y t !|
j"}t !| j"}t#$d ||fS ) a7
Returns the manifests which consist of the Recordings and Supervisions
:param corpus_dir: Pathlike, the path of the data dir.
:param output_dir: Pathlike, the path where to write the manifests.
:param silence_sym: str, silence symbol
:param breath_sym: str, breath symbol
:param foreign_sym: str, foreign symbol.
:param partial_sym: str, partial symbol. When set to None, will output partial words
:param unknown_sym: str, unknown symbol
:return: The RecordingSet and SupervisionSet with the keys 'audio' and 'supervisions'.
pandasz"Please 'pip install pandas' first.r NzNo such directory: Tr zTXTdata/fulldata.csv)quotingz atcosim_recordings_all.jsonl.gz) overwritez!atcosim_supervisions_all.jsonl.gz Preparing)desctotal)ra rb rc rd re rk WAVdataz.wav) recording_idd atcosim__06dg Englishrh orig_text)
idr startr} channellanguager` speakergendercustomzManifests are lazily materialized. You may want to call `lhotse.qa.fix_manifests()` to ensure that all supervisions fall within the corresponding recordings.)%r r) r r r@ r* read_csvcsv
QUOTE_NONEr open_writerr r iterrowslenrecording_corruptr|
transcriptionstr directory subdirectoryr r from_filer int
length_secr r r~
speaker_idrr writefrom_jsonl_lazypathr, rH )rN r ra rb rc rd re pdcsv_pathdf recs_writer sups_writeridxrowr` wav_path recording length100segment
recordings supervisionsr( r( rV prepare_atcosim s
7 r )r F)Nrk rk r r r ).__doc__r: r r. r, rA rer4 pathlibr typingr r r tqdm.autor lhotser lhotse.audior r lhotse.qar
lhotse.supervisionr r lhotse.utilsr
r r r r boolrW rp compilert rm ru rv rw r r| r r r r( r( r( rV s
9
*