o 5ti@sTddlZddlmZddlmZddlmZddlm Z ddZ Gd d d eZ dS) N) smart_open)Dataset) defaultdictcCs|sdSd|S)Nrefzref: translatorrrM/home/ubuntu/.local/lib/python3.10/site-packages/sacrebleu/dataset/wmt_xml.py_get_field_by_translator s r c@sFeZdZdZeddZddZdddZd d Zd d Z d dZ dS) WMTXMLDatasetzy The 2021+ WMT dataset format. Everything is contained in a single file. Can be parsed with the lxml parser. c sht|}ttt}}}|dD] }||dq|dD]}||d|d}||q)t|dksKJd|t|dks]Jdt|d|g}g} g} g} d d |D} tt } d \}}}|d D]}d |j vrq||d7}dd |dD}dd|d}fdd |D}|d}fdd |D}t | D]dt fdd|Dsq|D]}| t||||iidq||| D] }| |||q| |j d| |j d| |d|ddu}|d7}qq|d|i| | | d| }|r2| |d<|S)a Unwraps the XML file from wmt21 or later. This script is adapted from https://github.com/wmt-conference/wmt-format-tools :param raw_file: The raw xml file to unwrap. :return: Dictionary which contains the following fields (each a list with values for each sentence): - `src`: The source sentences. - `docid`: ID indicating which document the sentences belong to. - `origlang`: The original language of the document. - `domain`: Domain of the document. - `ref:{translator}`: The references produced by each translator. - `ref`: An alias for the references from the first translator. z.//srclangz.//refr rz-Multiple source languages found in the file: zFound z( reference languages found in the file: cSsi|]}t|gqSrr ).0r rrr =sz8WMTXMLDataset._unwrap_wmt21_or_later..)rrFz.//doc testsuitecSsi|] }t|d|jqS)idintgettextrsegrrr rHsz .//src//segcSsdd|dDS)NcSs(i|]}t|d|jr|jndqS)rrrrrr rMszKWMTXMLDataset._unwrap_wmt21_or_later..get_sents..z.//seg)findall)docrrr get_sentsLsz7WMTXMLDataset._unwrap_wmt21_or_later..get_sentsci|] }|d|qSr r)rref_docrrr rTz.//hypcr)systemr)rhyp_docr!rr rYr"csg|]}|dqS)rr)rvalue)seg_idrr _sz8WMTXMLDataset._unwrap_wmt21_or_later..rroriglangdomainNsrc)docidr()ETparsesetgetrootraddrlenrlistattribsortedkeysanyvaluesr append)raw_filetree src_langs ref_langs translatorssrc_docr r r*docids orig_langsdomainsrefssystemssrc_sent_count doc_count seen_domainr src_sentsref_docs trans_to_refhyp_docshyps system_namefieldsr)rr&r _unwrap_wmt21_or_latersr            z$WMTXMLDataset._unwrap_wmt21_or_latercCs8|||}t|tr|dn|d}tj|j|S)z Returns the path for this language pair. This is useful because in WMT22, the language-pair data structure can be a dict, in order to allow for overriding which test set to use. pathr)_get_langpair_metadata isinstancedictosrOjoin_rawdir)selflangpair langpair_datarel_pathrrr _get_langpair_pathtsz WMTXMLDataset._get_langpair_pathNc Cs|t||D]_}||}t| }||}Wdn1s(wY|D]<}|||}tj |rFtj |dkrFq/t|d}||D] }t | ||dqPWdn1sfwYq/q dS)zProcesses raw files to plain text files. :param langpair: The language pair to process. e.g. "en-de". If None, all files will be processed. Nrw)file)maybe_downloadr4rPr5rZrrN_get_txt_file_pathrSrOexistsgetsizeprint_clean) rVrWrawfilefinrM fieldnametextfilefoutlinerrr process_to_text~s$      zWMTXMLDataset.process_to_textcCsJ|jdg}|||}t|tr|d|}n|}dd|D}|S)a Returns the preferred references for this language pair. This can be set in the language pair block (as in WMT22), and backs off to the test-set-level default, or nothing. There is one exception. In the metadata, sometimes there is no translator field listed (e.g., wmt22:liv-en). In this case, the reference is set to "", and the field "ref" is returned. rBcSsg|]}t|qSrr)rrrrr r'sz.)kwargsrrPrQrR)rVrWdefaultsrX allowed_refsrrr _get_langpair_allowed_refss  z(WMTXMLDataset._get_langpair_allowed_refscs:||||}||}fddt||D}|S)z Returns the requested reference files. This is defined as a default at the test-set level, and can be overridden per language. csg|] \}}|vr|qSrr)rffieldrlrr r'sz5WMTXMLDataset.get_reference_files..)rm get_files fieldnameszip)rVrW all_files all_fields ref_filesrrpr get_reference_filess    z!WMTXMLDataset.get_reference_filescCsP|||}t| }||}Wdn1swYt|S)a Return a list of all the field names. For most source, this is just the source and the reference. For others, it might include the document ID for each line, or the original language (origLang). get_files() should return the same number of items as this. :param langpair: The language pair (e.g., "de-en") :return: a list of field names N)r]rZrrNr2r5)rVrWrcrdrMrrr rrs     zWMTXMLDataset.fieldnames)N) __name__ __module__ __qualname____doc__ staticmethodrNrZrirmrwrrrrrr r s ]  r ) rS lxml.etreeetreer,utilsrbaser collectionsrr r rrrr s