o Si@spdZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z m Z mZmZmZmZddlmZddlmZmZdd lmZmZdd lmZdd lmZmZdd lm Z m!Z!d Z"dZ#dZ$  d#de de%de fddZ&de de'deeeeffddZ( d$de'de de)deeeffddZ*  d%de d ee de)de e'e e'eeeffffd!d"Z+dS)&a About the medical corpus A dataset of simulated patient-physician medical interviews with a focus on respiratory cases. The simulated medical conversation dataset is available on figshare.com. The dataset is divided into two sets of files: audio files of the simulated conversations in mp3 format, and the transcripts of the audio files as text files. There are 272 mp3 audio files and 272 corresponding transcript text files. Each file is titled with three characters and four digits. RES stands for respiratory, GAS represents gastrointestinal, CAR is cardiovascular, MSK is musculoskeletal, DER is dermatological, and the four following digits represent the case number of the respective disease category. It is covered in more detail at https://www.nature.com/articles/s41597-022-01423-1.pdf N) defaultdict)ProcessPoolExecutor)Path)DictListOptionalSequenceTupleUnion)tqdm) Recording RecordingSet) fix_manifests$validate_recordings_and_supervisions)manifests_exist)SupervisionSegmentSupervisionSet)Pathlikeresumable_download)testdevtrain)z audio.tar.gzzcleantext.tar.gzzmedical_test.infozmedical_dev.infozmedical_train.infoz=https://huggingface.co/datasets/yfyeung/medical/resolve/main/.F target_dirforce_downloadreturnc Cst|}|jdddt}t|ddD]D}td|||}t|dd}tt |||dd |vrWt j |dd t | }||Wd n1sRwYq|S) a Download and unzip Medical dataset. :param target_dir: Pathlike, the path of the dir to store the dataset. :param force_download: bool, if True, download the archive even if it already exists. :return: the path to downloaded and extracted directory with data. Tparentsexist_okzDownloading MedicaldesczDownloading part: z.tar.gz)filenamerztar.gz) ignore_errorsN)rmkdirMEDICAL_SPLITSr logginginfostrreplacerMEDICAL_BASE_URLshutilrmtreetarfileopen extractall)rr dataset_partspart part_pathpart_dirtarr5J/home/ubuntu/.local/lib/python3.10/site-packages/lhotse/recipes/medical.pydownload_medical*s&   r7 corpus_dir audio_infoc Cs|ddddddd\}}}}t|dddd}||}|s5td|dStj||d }t |d tt ||t |t |t |d d |d }||fS)N, []r!z.mp3zaudio/zNo such file: )path recording_id_rEnglish)idr?startdurationchannellanguagetext) r)splitr(resolveis_filer&warningr from_filerhashfloat) r8r9 audio_pathrCendrG file_name recordingsegmentr5r5r6_parse_utteranceNs$    rTsubsetnum_jobsc Cs(t|}|d|d}t|}|}Wdn1s!wYt|^}g}g}g} t|ddD] } ||t|| q7t|ddD]} | } | durUqJ| \} }| |vrb|| | |qJt |}t | }t||\}}t||Wd||fS1swY||fS)z Returns the RecodingSet and SupervisionSet given a dataset part. :param subset: str, the name of the subset. :param corpus_dir: Pathlike, the path of the data dir. :return: the RecodingSet and SupervisionSet for train and valid. medical_z.infoNzDistributing tasksr Processing)rr.read splitlinesrr appendsubmitrTresultr from_recordingsr from_segmentsrr)rVr8rW text_pathf audio_infosexfutures recordings supervisionsr9futurer^rRrS recording_setsupervision_setr5r5r6_prepare_subsetjs6         rk output_dircCst|}|sJd|tdt}|dur%t|}|jdddtt}t|ddD]E}td|t ||d d d rKtd |d q/t |||\}}|durm| |d|d| |d|d||d||<q/|S)aF Returns the manifests which consist of the Recordings and Supervisions. :param corpus_dir: Path to the Medical dataset. :param output_dir: Pathlike, the path where to write the manifests. :return: a Dict whose key is the dataset part, and the value is Dicts with the keys 'recordings' and 'supervisions'. zNo such directory: zPreparing Medical...NTrz Dataset partsrzProcessing Medical subset: medicalzjsonl.gz)r1rlprefixsuffixzMedical subset: z already prepared - skipping.medical_supervisions_z .jsonl.gzmedical_recordings_)rfrg) ris_dirr&r'MEDICALr$rdictr rrkto_file)r8rlrWsubsets manifestsr1rirjr5r5r6prepare_medicals4  rx)rF)rU)NrU),__doc__r&osr+r- collectionsrconcurrent.futures.processrpathlibrtypingrrrrr r tqdm.autor lhotse.audior r lhotse.qarrlhotse.recipes.utilsrlhotse.supervisionrr lhotse.utilsrrrsr%r*boolr7r(rTintrkrxr5r5r5r6sn        $   -