o 5tiD@sbdZddlZddlZddlmZmZddlmZmZm Z ddl m Z m Z m Z Gddded ZdS) z+ The base class for all types of datasets. N)ABCMetaabstractmethod)DictListOptional) SACREBLEU_DIR download_file smart_openc @seZdZddddeeeeffdedeeedeedeedeeef ddZd d Ze d d Z d dZ ddZ ddZ ed$d%ddZdeefddZddZddZddZddZd d!Zd"d#ZdS)&DatasetNnamedata descriptioncitationmd5cKsR||_||_||_||_||_||_||_tj t |j|_ tj |j d|_ dS)av Params come from the values in DATASETS. :param name: Name of the dataset. :param data: URL of the raw data of the dataset. :param description: Description of the dataset. :param citation: Citation for the dataset. :param md5: MD5 checksum of the dataset. :param langpairs: List of available language pairs. rawN) r r rrr langpairskwargsospathjoinr_outdir_rawdir)selfr r rrrrrrJ/home/ubuntu/.local/lib/python3.10/site-packages/sacrebleu/dataset/base.py__init__ szDataset.__init__cCsntj|jdd|jr|jndgt|j}t|j|D]\}}tj|j| |}t |||j|dqdS)z If the dataset isn't downloaded, use utils/download_file() This can be implemented here in the base class. It should write to ~/.sacreleu/DATASET/raw exactly as it does now. T)exist_okN) extract_to expected_md5) rmakedirsrrlenr ziprr_get_tarball_filenamer )rexpected_checksumsurlrtarballrrrmaybe_download0s zDataset.maybe_downloadcCstdd|S)z Removes trailing and leading spaces and collapses multiple consecutive internal spaces to a single one. :param s: The string. :return: A cleaned-up string. z\s+ )resubstrip)srrr_cleanAszDataset._cleancCs|jdddtj|S)z Produces a local filename for tarball. :param url: The url to download. :return: A name produced from the dataset identifier and the URL basename. /_.)r replacerrbasename)rr%rrrr#KszDataset._get_tarball_filenamecCs:|jdd}|dd}tj|j|d|d|S)a# Given the language pair and fieldname, return the path to the text file. The format is: ~/.sacrebleu/DATASET/DATASET.LANGPAIR.FIELDNAME :param langpair: The language pair. :param fieldname: The fieldname. :return: The path to the text file. r.r/:-r0)r r1rrrr)rlangpair fieldnamer rrr_get_txt_file_pathSs  zDataset._get_txt_file_pathcCsD|dur |j}|S||jvrtd|jd|||j|i}|S)a  Given a language pair, return the metadata for that language pair. Deal with errors if the language pair is not available. :param langpair: The language pair. e.g. "en-de" :return: Dict format which is same as self.langpairs. NzNo such language pair r.)r Exceptionr )rr5rrrr_get_langpair_metadatabs zDataset._get_langpair_metadatareturncCsdS)zProcesses raw files to plain text files. :param langpair: The language pair to process. e.g. "en-de". If None, all files will be processed. Nrrr5rrrprocess_to_textsszDataset.process_to_textcCsddgS)a Return a list of all the field names. For most source, this is just the source and the reference. For others, it might include the document ID for each line, or the original language (origLang). get_files() should return the same number of items as this. :param langpair: The language pair (e.g., "de-en") :return: a list of field names srcrefrr;rrr fieldnames{s zDataset.fieldnamescc2||}dd|D}t|D]}|VqdS)zs Iterates over all fields (source, references, and other metadata) defined by the dataset. cSg|]}t|qSrr .0frrr z$Dataset.__iter__..N) get_filesr")rr5 all_filesall_finsitemrrr__iter__s   zDataset.__iter__ccsN||}t|}|D]}|Vq WddS1s wYdS)z; Return an iterable over the source lines. N)get_source_filer r+)rr5 source_filefinlinerrrsources   "zDataset.sourceccr@)z9 Return an iterable over the references. cSrArrBrCrrrrFrGz&Dataset.references..N)get_reference_filesr")rr5 ref_filesref_finsrKrrr referencess   zDataset.referencescCs&||}||}|d}||S)Nr=)rHr?index)rr5rI all_fieldsrVrrrrMs   zDataset.get_source_filecCs,||}||}ddt||D}|S)NcSsg|] \}}|dr|qS)r>) startswith)rDrEfieldrrrrFs z/Dataset.get_reference_files..)rHr?r")rr5rIrWrSrrrrRs  zDataset.get_reference_filescsB}fdd|D}|D] }tj|sq|S)aE Returns the path of the source file and all reference files for the provided test set / language pair. Downloads the references first if they are not already local. :param langpair: The language pair (e.g., "de-en") :return: a list of the source file and all reference files csg|]}|qSr)r7)rDrYr5rrrrFsz%Dataset.get_files..)r?rrexistsr<)rr5fieldsfilesfilerrZrrHs   zDataset.get_files)N)r:N)__name__ __module__ __qualname__rstrrrrr' staticmethodr-r#r7r9rr<r?rLrQrUrMrRrHrrrrr s@   #       r ) metaclass)__doc__rr)abcrrtypingrrrutilsrr r r rrrrs