o
5ti @ sL d dl Z d dlZddlmZ ddlmZ G dd deZG dd d eZdS )
N )
smart_open )Datasetc @ s2 e Zd ZdZdd Zdd Zd ddZd d
ZdS ) FakeSGMLDatasetz
The fake SGML format used by WMT prior to 2021. Can't be properly parsed.
Source and reference(s) in separate files.
c
C s t |@}t |d"}|D ]}|dr$| tdd|}t||d q
W d n1 s/w Y W d dS W d dS 1 sGw Y dS )zK
Extract data from raw file and convert to raw txt format.
wt(.*).*?z\1fileN)r
startswith_cleanresubprintselfinput_file_pathoutput_filep_pathfinfoutline r O/home/ubuntu/.local/lib/python3.10/site-packages/sacrebleu/dataset/fake_sgml.py_convert_format s
Pz FakeSGMLDataset._convert_formatc C s t |P}t |d2}d}|D ]%}|dr)t| d|}|dur(|d}q|dr4t||d qW d n1 s?w Y W d dS W d dS 1 sWw Y dS ) zP
Extract metadata from document tags, projects across segments.
r z7 s z3FakeSGMLDataset.process_to_text..c s$ g | ]}t jj d qS )r r! )r& _langpair langpairsr r r r' < s srcrefN)
maybe_download_get_langpair_metadata
fieldnamesrangezipr" r# r$ r% _get_txt_file_pathr r r )r r* r1 origin_filesr origin_file output_filer r) r process_to_text, s$
z FakeSGMLDataset.process_to_textc C s | |}t|| }|dksJ d| j ddg}|dkr%|d nt|| dd D ] \}}|d| q/| jd sH|g d
7 }|S ) a)
Return a list of all the field names. For most source, this is just
the source and the reference. For others, it might include the document
ID for each line, or the original language (origLang).
get_files() should return the same number of items as this.
r zEach language pair in z must have at least 2 fields.r- r. r Nzref:wmt08)docidgenreoriglang)r0 lennameappend enumerater )r r* metalengthfieldsir( r r r r1 K s
zFakeSGMLDataset.fieldnames)N)__name__
__module__ __qualname____doc__r r r8 r1 r r r r r s
r c s e Zd ZdZ fddZ ZS )WMTAdditionDatasetz=
Handle special case of WMT Google addition dataset.
c s | dr t ||S t|3}t|d}|D ]
}t| |d qW d n1 s.w Y W d d S W d d S 1 sFw Y d S )Nz.sgmr r )endswithsuperr r r rstripr __class__r r r m s
"z"WMTAdditionDataset._convert_format)rE rF rG rH r
__classcell__r r rM r rI h s rI )r" r
utilsr baser r rI r r r r s `