o 5ti`@sddlmZmZddlmZddlmZddlmZddl m Z dddZ d d e DZ eed d e d DZeedd e d DZide ddgddgdgdgdgdgdgdgdgdgdgdgdgd dgd d!e d!d"gd#d$gd%gd&gd'gd(gd)d*gd+d,gd-gd.gd/gd0d*gd+d1gd2gd3gd4gd5dgd d6e d6d7gd8d9gid:d;dgd?d@gdAdBgdCdDdgd?d@dAgdSdBdCgdDdEdFgdGdHdIgdJdKdLgdMdNgdOdPgdQdRdSedSdTgdUdVgdWdXgdYdZgd[d\gd]d^gd_d`gdadbgdcdddeddddegdfddggdhdEd-digidRdjedjddkgdlddmgdhdEd-dngidRdoedoddpgdqddrgdhdEd-dsgidRdtedtddugdvddwgdhdEd-dxgidRdyedyddzgd{dd|gdhdEd-d}gidRd~ed~ddgdddgdhdEd-dgidRdeddgdgddddgddgddgddgddgddgddgddgddgddgddgddgddgddgdddeddgdgdddgddgddgddgddgddgddgddgddgddgddgddgddgdÐdgdddedŐdgdgdȐdɐdgdːdgd͜ddedΐdgdgddѐdҐdgdԐdgd֐dgdؐdgdڐdgdܐdgdސdgddgddgddgddgddgddgddgdddeddgdgdd/ddgiddeddgdgdd/gdiddeddgdgdddgddgdddeddgdgdddgddgddgd d gd dd ed d dgddgdddugdidideddgdgddddgddgddgddgddgd d!gd"d#gd$d%gd&d'gd(d)gd*d+gd,d-gd. dd/ed/dgdgd0d/dd1gidd2ed2dgdgd3d/gd4idd5ed5d6gd7gd8d9d:gd;dgd?d@gdAddBedBdCgdDgddEdFdGgdHdIgdJdKgdLdMgdNdOgdPdQgdRdSgdTdUgdVdWgdXdYgdZ dd[ed[d\gd]gdd^d_d`gdadbgdcddgdedfgdgdhgdidjgdkdlgdmdngdodpgdqdrgds ddtedtdugdvgdwdxdygdzd{gd|d}gd~dgddgddgddgddgddgddgds ddeddgdgddddgddgddgddgddgddgddgddgddgddgd ddeddgdgddddgddgddgddgddgddgddgddgdddeddgdgddddgddgddgddgddgddgddgddgdddeddgdgddddgddgddgddgddgddgddgddgdddeddgdgddddgddgddgddgddgddgddgddgddgddgddgddgd ddeddgdgddÐdĐdgdŐdgdƐdgdŐdgdǐdgdŐdgdȐdgdŐdgdɐdgdŐdgdʜ ddedːdgdgd̐d͐dgdΐdgdϜddedАdgdgdѐdҐdgdӐdgdԐdgdӐdgdՐdgdӐdgd֜ddedgdآgd٢dڐdېdܐdgdݐdgdސdgdߐdgddgddgddgddgddgddgddgddgd ddedgdgddddgddgddgddgddgddgddidedgdgddddgddgddgddgddgddgdddedgdgddddgddgddgddgddgddgdddedgdgddddgddgddgddgd d gd d gddd ed gdgddd d gd d gddgddgddgddgdddedgdgddddgddgddgddgddgddgdddedgdgddddgddgddgddgddgddgddd ed gdgddd!d"gd"d!gd#d$gd$d#gd%d&gd&d%gddd'ed'd(gd)d*d+gd,d-gd.d/gd0d1gd2d3gd4d5gd6d7gd8d9gd:d;gd<d=gd>d?gd@dAgdBdCgdDdEgdF dGdHedHdIgdJd*dKgdLdMgdNdOgdPdQgdRdSgdTdUgdVdWgdXdYgdZd[gd\d]gd^d_gd`dagdbdcgdddegdF dGdfedfdggdhgdidjdkdlgdkdmgdkdngdoddpedpdggdhgdqdrdsdtgdsdugdvddwedwgdxgdydzd{d|d}gd|d~gd|dgdoddeddgddgddgddgddgddgdddeddgdddgddgddgddgddgddGdeddgdddgddgddgddgddgddGdeddgdddgddgddgddgddgddGZdS()FakeSGMLDatasetWMTAdditionDataset)IWSLTXMLDataset)PlainTextDataset) TSVDataset) WMTXMLDataseta rt.com.68098=US-crime guardian.181611=US-politics bbc.310963=GB-sport washpost.116881=US-politics scotsman.104228=GB-sport timemagazine.75207=OTHER-world-ID euronews-en.117981=OTHER-crime-AE smh.com.au.242810=US-crime msnbc.53726=US-politics euronews-en.117983=US-politics msnbc.53894=US-crime theglobeandmail.com.62700=US-business bbc.310870=OTHER-world-AF reuters.196698=US-politics latimes.231739=US-sport thelocal.51929=OTHER-world-SE cbsnews.198694=US-politics reuters.196718=OTHER-sport-RU abcnews.255599=EU-sport nytimes.127256=US-entertainment scotsman.104225=GB-politics dailymail.co.uk.233026=GB-scitech independent.181088=GB-entertainment brisbanetimes.com.au.181614=OTHER-business-AU washpost.116837=US-politics dailymail.co.uk.232928=GB-world thelocal.51916=OTHER-politics-IT bbc.310871=US-crime nytimes.127392=EU-business-DE euronews-en.118001=EU-scitech-FR washpost.116866=OTHER-crime-MX dailymail.co.uk.233025=OTHER-scitech-CA latimes.231829=US-crime guardian.181662=US-entertainment msnbc.53731=US-crime rt.com.68127=OTHER-sport-RU latimes.231782=US-business latimes.231840=US-sport reuters.196711=OTHER-scitech guardian.181666=GB-entertainment novinite.com.24019=US-politics smh.com.au.242750=OTHER-scitech guardian.181610=US-politics telegraph.364393=OTHER-crime-ZA novinite.com.23995=EU-world dailymail.co.uk.233028=GB-scitech independent.181071=GB-sport telegraph.364538=GB-scitech timemagazine.75193=US-politics independent.181096=US-entertainment upi.140602=OTHER-world-AF bbc.310946=GB-business independent.181052=EU-sport a bbc.381790=GB-politics rt.com.91337=OTHER-politics-MK nytimes.184853=US-world upi.176266=US-crime guardian.221754=GB-business dailymail.co.uk.298595=GB-business cnbc.com.6790=US-politics nytimes.184837=OTHER-world-ID upi.176249=GB-sport euronews-en.153835=OTHER-world-ID dailymail.co.uk.298732=GB-crime telegraph.405401=GB-politics newsweek.51331=OTHER-crime-CN abcnews.306815=US-world cbsnews.248384=US-politics reuters.218882=GB-politics cbsnews.248387=US-crime abcnews.306764=OTHER-world-MX reuters.218888=EU-politics bbc.381780=GB-crime bbc.381746=GB-sport euronews-en.153800=EU-politics bbc.381679=GB-crime bbc.381735=GB-crime newsweek.51338=US-world bbc.381765=GB-crime cnn.304489=US-politics reuters.218863=OTHER-world-ID nytimes.184860=OTHER-world-ID cnn.304404=US-crime bbc.381647=US-entertainment abcnews.306758=OTHER-politics-MX cnbc.com.6772=US-business reuters.218932=OTHER-politics-MK upi.176251=GB-sport reuters.218921=US-sport cnn.304447=US-politics guardian.221679=GB-politics scotsman.133765=GB-sport scotsman.133804=GB-entertainment guardian.221762=OTHER-politics-BO cnbc.com.6769=US-politics dailymail.co.uk.298692=EU-entertainment scotsman.133744=GB-world reuters.218911=US-sport newsweek.51310=US-politics independent.226301=US-sport reuters.218923=EU-sport reuters.218861=US-politics dailymail.co.uk.298759=US-world scotsman.133791=GB-sport cbsnews.248484=EU-scitech dailymail.co.uk.298630=US-scitech newsweek.51329=US-entertainment bbc.381701=GB-crime dailymail.co.uk.298738=GB-entertainment bbc.381669=OTHER-world-CN foxnews.94512=US-politics guardian.221718=GB-entertainment dailymail.co.uk.298686=GB-politics cbsnews.248471=US-politics newsweek.51318=US-entertainment rt.com.91335=US-politics newsweek.51300=US-politics cnn.304478=US-politics upi.176275=US-politics telegraph.405422=OTHER-world-ID reuters.218933=US-politics newsweek.51328=US-politics newsweek.51307=US-business bbc.381692=GB-world independent.226346=GB-entertainment bbc.381646=GB-sport reuters.218914=US-sport scotsman.133758=EU-sport rt.com.91350=EU-world scotsman.133773=GB-scitech rt.com.91334=EU-crime bbc.381680=GB-politics guardian.221756=US-politics scotsman.133783=GB-politics cnn.304521=US-sport dailymail.co.uk.298622=GB-politics bbc.381789=GB-sport dailymail.co.uk.298644=GB-business dailymail.co.uk.298602=GB-world scotsman.133753=GB-sport independent.226317=GB-entertainment nytimes.184862=US-politics thelocal.65969=OTHER-world-SY nytimes.184825=US-politics cnbc.com.6784=US-politics nytimes.184804=US-politics nytimes.184830=US-politics scotsman.133801=GB-sport cnbc.com.6770=US-business bbc.381760=GB-crime reuters.218865=OTHER-world-ID newsweek.51339=US-crime euronews-en.153797=OTHER-world-ID abcnews.306774=US-crime dailymail.co.uk.298696=GB-politics abcnews.306755=US-politics reuters.218909=US-crime independent.226349=OTHER-sport-RU newsweek.51330=US-politics bbc.381705=GB-sport newsweek.51340=OTHER-world-ID cbsnews.248411=OTHER-world-FM abcnews.306776=US-crime bbc.381694=GB-entertainment rt.com.91356=US-world telegraph.405430=GB-entertainment telegraph.405404=EU-world bbc.381749=GB-world telegraph.405413=US-politics bbc.381736=OTHER-politics-KP cbsnews.248394=US-politics nytimes.184822=US-world telegraph.405408=US-politics euronews-en.153799=OTHER-politics-SY euronews-en.153826=EU-sport cnn.304400=US-world)wmt18wmt19cCs$i|]\}}|dd|DqS)cSs&i|]}|dd|ddqS)=rsplit).0drN/home/ubuntu/.local/lib/python3.10/site-packages/sacrebleu/dataset/__init__.py Es&z.r )rkvrrrrDsrcCh|] }|ddqS)-r r rrrrr Hrr cCr)rrr rrrrrIrwmt24z_https://github.com/wmt-conference/wmt24-news-systems/releases/download/v1.1/data_onlyxml.tar.gzz4WMT24 official test set release, v1.1 (excluding TS) 299963fcb7b4e86d6d212bf69beb9580zxml/wmttest2024.cs-uk.all.xmlzxml/wmttest2024.en-cs.all.xmlzxml/wmttest2024.en-de.all.xmlzxml/wmttest2024.en-es.all.xmlzxml/wmttest2024.en-hi.all.xmlzxml/wmttest2024.en-is.all.xmlzxml/wmttest2024.en-ja.all.xmlzxml/wmttest2024.en-ru.all.xmlzxml/wmttest2024.en-uk.all.xmlzxml/wmttest2024.en-zh.all.xmlzxml/wmttest2024.ja-zh.all.xml) cs-uken-csen-deen-esen-hien-isen-jaen-ruen-uken-zhzja-zhrefA)data descriptionmd5 langpairsrefswmt23zShttps://github.com/wmt-conference/wmt23-news-systems/archive/refs/tags/v.0.1.tar.gzz.Official evaluation and system data for WMT23. 63576405e4ce07130a19ad76ba7eb75bz6wmt23-news-systems-v.0.1/xml/wmttest2023.cs-uk.all.xmlz6wmt23-news-systems-v.0.1/xml/wmttest2023.de-en.all.xmlz6wmt23-news-systems-v.0.1/xml/wmttest2023.en-cs.all.xmlz6wmt23-news-systems-v.0.1/xml/wmttest2023.en-de.all.xmlz6wmt23-news-systems-v.0.1/xml/wmttest2023.en-he.all.xmlrefB)pathr+z6wmt23-news-systems-v.0.1/xml/wmttest2023.en-ja.all.xmlz6wmt23-news-systems-v.0.1/xml/wmttest2023.en-ru.all.xmlz6wmt23-news-systems-v.0.1/xml/wmttest2023.en-uk.all.xmlz6wmt23-news-systems-v.0.1/xml/wmttest2023.en-zh.all.xmlz6wmt23-news-systems-v.0.1/xml/wmttest2023.he-en.all.xmlz6wmt23-news-systems-v.0.1/xml/wmttest2023.ja-en.all.xmlz6wmt23-news-systems-v.0.1/xml/wmttest2023.ru-en.all.xmlz6wmt23-news-systems-v.0.1/xml/wmttest2023.uk-en.all.xmlz6wmt23-news-systems-v.0.1/xml/wmttest2023.zh-en.all.xml)rde-enrrzen-her"r#r$r%zhe-enja-enru-enuk-enzh-enwmt22zRhttps://github.com/wmt-conference/wmt22-news-systems/archive/refs/tags/v1.1.tar.gzz.Official evaluation and system data for WMT22. 0840978b9b50b9ac3b2b081e37d620b9cs-enz4wmt22-news-systems-1.1/xml/wmttest2022.cs-en.all.xmlBrz4wmt22-news-systems-1.1/xml/wmttest2022.cs-uk.all.xmlr0z4wmt22-news-systems-1.1/xml/wmttest2022.de-en.all.xmlzde-frz4wmt22-news-systems-1.1/xml/wmttest2022.de-fr.all.xmlrz4wmt22-news-systems-1.1/xml/wmttest2022.en-cs.all.xmlrz4wmt22-news-systems-1.1/xml/wmttest2022.en-de.all.xmlzen-hrz4wmt22-news-systems-1.1/xml/wmttest2022.en-hr.all.xmlr"z4wmt22-news-systems-1.1/xml/wmttest2022.en-ja.all.xmlzen-livz5wmt22-news-systems-1.1/xml/wmttest2022.en-liv.all.xmlr#z4wmt22-news-systems-1.1/xml/wmttest2022.en-ru.all.xmlr$z4wmt22-news-systems-1.1/xml/wmttest2022.en-uk.all.xmlr%z4wmt22-news-systems-1.1/xml/wmttest2022.en-zh.all.xmlzfr-dez4wmt22-news-systems-1.1/xml/wmttest2022.fr-de.all.xmlr1z4wmt22-news-systems-1.1/xml/wmttest2022.ja-en.all.xmlzliv-enz5wmt22-news-systems-1.1/xml/wmttest2022.liv-en.all.xmlr2z4wmt22-news-systems-1.1/xml/wmttest2022.ru-en.all.xmlzru-sahz5wmt22-news-systems-1.1/xml/wmttest2022.ru-sah.all.xmlz5wmt22-news-systems-1.1/xml/wmttest2022.sah-ru.all.xmlz4wmt22-news-systems-1.1/xml/wmttest2022.uk-cs.all.xmlz4wmt22-news-systems-1.1/xml/wmttest2022.uk-en.all.xmlz4wmt22-news-systems-1.1/xml/wmttest2022.zh-en.all.xml)zsah-ruzuk-csr3r4Az wmt21/systemszRhttps://github.com/wmt-conference/wmt21-news-systems/archive/refs/tags/v1.3.tar.gzzWMT21 system output. a6aee4099da58f98f71eb3fac1694237z5wmt21-news-systems-1.3/xml/newstest2021.de-fr.all.xmlz5wmt21-news-systems-1.3/xml/newstest2021.en-de.all.xmlen-haz5wmt21-news-systems-1.3/xml/newstest2021.en-ha.all.xmlr!z5wmt21-news-systems-1.3/xml/newstest2021.en-is.all.xmlz5wmt21-news-systems-1.3/xml/newstest2021.en-ja.all.xmlz5wmt21-news-systems-1.3/xml/newstest2021.fr-de.all.xmlha-enz5wmt21-news-systems-1.3/xml/newstest2021.ha-en.all.xmlis-enz5wmt21-news-systems-1.3/xml/newstest2021.is-en.all.xmlz5wmt21-news-systems-1.3/xml/newstest2021.ja-en.all.xmlr4z5wmt21-news-systems-1.3/xml/newstest2021.zh-en.all.xmlz5wmt21-news-systems-1.3/xml/newstest2021.en-zh.all.xmlz5wmt21-news-systems-1.3/xml/newstest2021.cs-en.all.xmlz5wmt21-news-systems-1.3/xml/newstest2021.de-en.all.xmlz5wmt21-news-systems-1.3/xml/newstest2021.en-cs.all.xmlz5wmt21-news-systems-1.3/xml/newstest2021.en-ru.all.xmlz5wmt21-news-systems-1.3/xml/newstest2021.ru-en.all.xmlzbn-hiz7wmt21-news-systems-1.3/xml/florestest2021.bn-hi.all.xmlz7wmt21-news-systems-1.3/xml/florestest2021.hi-bn.all.xmlz7wmt21-news-systems-1.3/xml/florestest2021.xh-zu.all.xmlz7wmt21-news-systems-1.3/xml/florestest2021.zu-xh.all.xml)zhi-bnzxh-zuzzu-xhwmt21z7https://data.statmt.org/wmt21/translation-task/test.tgzz#Official evaluation data for WMT21. 32e7ab995bc318414375d60f0269af92ztest/newstest2021.de-fr.xmlztest/newstest2021.en-de.xmlztest/newstest2021.en-ha.xmlztest/newstest2021.en-is.xmlztest/newstest2021.en-ja.xmlztest/newstest2021.fr-de.xmlztest/newstest2021.ha-en.xmlztest/newstest2021.is-en.xmlztest/newstest2021.ja-en.xmlztest/newstest2021.zh-en.xmlztest/newstest2021.en-zh.xmlztest/newstest2021.cs-en.xmlztest/newstest2021.de-en.xmlztest/newstest2021.en-cs.xmlztest/newstest2021.en-ru.xmlztest/newstest2021.ru-en.xmlztest/florestest2021.bn-hi.xmlztest/florestest2021.hi-bn.xmlztest/florestest2021.xh-zu.xmlztest/florestest2021.zu-xh.xmlzwmt21/Bz4Official evaluation data for WMT21 with reference B.)r7r0rr#r%r2zwmt21/ABz;Official evaluation data for WMT21 with references A and B.)r7r0rrr#r%r2zwmt21/Cz3Official evaluation data for WMT21 with reference CCzwmt21/ACz:Official evaluation data for WMT21 with references A and Czwmt21/Dz3Official evaluation data for WMT21 with reference DDz wmt21/devz6https://data.statmt.org/wmt21/translation-task/dev.tgzuYDevelopment data for WMT21，if multiple references are available, the first one is used. 165da59ac8dfb5b7cafd7e90b1cac672zdev/xml/newsdev2021.en-ha.xmlzdev/xml/newsdev2021.ha-en.xmlzdev/xml/newsdev2021.en-is.xmlzdev/xml/newsdev2021.is-en.xml)r<r=r!r>z wmt20/tworefsz7https://data.statmt.org/wmt20/translation-task/test.tgzz(WMT20 news test sets with two references 3b1f777cfd2fb15ccf66e9bfdb2b1699) sgm/newstest2020-deen-src.de.sgm sgm/newstest2020-deen-ref.en.sgmz!sgm/newstestB2020-deen-ref.en.sgm) sgm/newstest2020-ende-src.en.sgm sgm/newstest2020-ende-ref.de.sgmz!sgm/newstestB2020-ende-ref.de.sgm) sgm/newstest2020-enzh-src.en.sgm sgm/newstest2020-enzh-ref.zh.sgmz!sgm/newstestB2020-enzh-ref.zh.sgm) sgm/newstest2020-ruen-src.ru.sgm sgm/newstest2020-ruen-ref.en.sgmz!sgm/newstestB2020-ruen-ref.en.sgm) sgm/newstest2020-zhen-src.zh.sgm sgm/newstest2020-zhen-ref.en.sgmz!sgm/newstestB2020-zhen-ref.en.sgm)r0rr%r2r4)r'r(r)r*wmt20z"Official evaluation data for WMT20z sgm/newstest2020-csen-src.cs.sgmz sgm/newstest2020-csen-ref.en.sgmrErFz sgm/newstest2020-defr-src.de.sgmz sgm/newstest2020-defr-ref.fr.sgmz sgm/newstest2020-encs-src.en.sgmz sgm/newstest2020-encs-ref.cs.sgmrGrHen-iuz sgm/newstest2020-eniu-src.en.sgmz sgm/newstest2020-eniu-ref.iu.sgmz sgm/newstest2020-enja-src.en.sgmz sgm/newstest2020-enja-ref.ja.sgmzen-kmz sgm/newstest2020-enkm-src.en.sgmz sgm/newstest2020-enkm-ref.km.sgmen-plz sgm/newstest2020-enpl-src.en.sgmz sgm/newstest2020-enpl-ref.pl.sgmzen-psz sgm/newstest2020-enps-src.en.sgmz sgm/newstest2020-enps-ref.ps.sgmz sgm/newstest2020-enru-src.en.sgmz sgm/newstest2020-enru-ref.ru.sgmen-taz sgm/newstest2020-enta-src.en.sgmz sgm/newstest2020-enta-ref.ta.sgmrIrJz sgm/newstest2020-frde-src.fr.sgmz sgm/newstest2020-frde-ref.de.sgmiu-enz sgm/newstest2020-iuen-src.iu.sgmz sgm/newstest2020-iuen-ref.en.sgmz sgm/newstest2020-jaen-src.ja.sgmz sgm/newstest2020-jaen-ref.en.sgmzkm-enz sgm/newstest2020-kmen-src.km.sgmz sgm/newstest2020-kmen-ref.en.sgmz sgm/newstest2020-plen-src.pl.sgmz sgm/newstest2020-plen-ref.en.sgmz sgm/newstest2020-psen-src.ps.sgmz sgm/newstest2020-psen-ref.en.sgmrKrLz sgm/newstest2020-taen-src.ta.sgmz sgm/newstest2020-taen-ref.en.sgmrMrN)pl-enzps-enr2ta-enr4z wmt20/devz6https://data.statmt.org/wmt20/translation-task/dev.tgzz'Development data for tasks new to 2020. 037f2b37aab74febbb1b2307dc2afb54zdev/newsdev2020-iuen-src.iu.sgmzdev/newsdev2020-iuen-ref.en.sgmzdev/newsdev2020-eniu-src.en.sgmzdev/newsdev2020-eniu-ref.iu.sgmzdev/newsdev2020-jaen-src.ja.sgmzdev/newsdev2020-jaen-ref.en.sgmzdev/newsdev2020-enja-src.en.sgmzdev/newsdev2020-enja-ref.ja.sgmzdev/newsdev2020-plen-src.pl.sgmzdev/newsdev2020-plen-ref.en.sgmzdev/newsdev2020-enpl-src.en.sgmzdev/newsdev2020-enpl-ref.pl.sgmzdev/newsdev2020-taen-src.ta.sgmzdev/newsdev2020-taen-ref.en.sgmzdev/newsdev2020-enta-src.en.sgmzdev/newsdev2020-enta-ref.ta.sgm)rSrPr1r"rTrQrUrRzwmt20/robust/set1zEhttps://data.statmt.org/wmt20/robustness-task/robustness20-3-sets.zip a12ac9ebe89b72195041518dffc4a9d5zWMT20 robustness task, set 1z-robustness20-3-sets/robustness20-set1-enja.enz-robustness20-3-sets/robustness20-set1-enja.jaz-robustness20-3-sets/robustness20-set1-ende.enz-robustness20-3-sets/robustness20-set1-ende.de)r"r)r'r)r(r*zwmt20/robust/set2zWMT20 robustness task, set 2z-robustness20-3-sets/robustness20-set2-enja.enz-robustness20-3-sets/robustness20-set2-enja.jaz-robustness20-3-sets/robustness20-set2-jaen.jaz-robustness20-3-sets/robustness20-set2-jaen.en)r"r1zwmt20/robust/set3zWMT20 robustness task, set 3z-robustness20-3-sets/robustness20-set3-deen.dez-robustness20-3-sets/robustness20-set3-deen.enz7https://data.statmt.org/wmt19/translation-task/test.tgzzOfficial evaluation data. 84de7162d158e28403103b01aeefc39aa@proceedings{ws-2019-machine, title = "Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)", editor = "Bojar, Ond{\v{r}}ej and Chatterjee, Rajen and Federmann, Christian and Fishel, Mark and Graham, Yvette and Haddow, Barry and Huck, Matthias and Yepes, Antonio Jimeno and Koehn, Philipp and Martins, Andr{\'e} and Monz, Christof and Negri, Matteo and N{\'e}v{\'e}ol, Aur{\'e}lie and Neves, Mariana and Post, Matt and Turchi, Marco and Verspoor, Karin", month = aug, year = "2019", address = "Florence, Italy", publisher = "Association for Computational Linguistics", url = "https://www.aclweb.org/anthology/W19-5200", }zcs-dez sgm/newstest2019-csde-src.cs.sgmz sgm/newstest2019-csde-ref.de.sgmzde-csz sgm/newstest2019-decs-src.de.sgmz sgm/newstest2019-decs-ref.cs.sgmz sgm/newstest2019-deen-src.de.sgmz sgm/newstest2019-deen-ref.en.sgmz sgm/newstest2019-defr-src.de.sgmz sgm/newstest2019-defr-ref.fr.sgmz sgm/newstest2019-encs-src.en.sgmz sgm/newstest2019-encs-ref.cs.sgmz sgm/newstest2019-ende-src.en.sgmz sgm/newstest2019-ende-ref.de.sgmen-fiz sgm/newstest2019-enfi-src.en.sgmz sgm/newstest2019-enfi-ref.fi.sgmen-guz sgm/newstest2019-engu-src.en.sgmz sgm/newstest2019-engu-ref.gu.sgmen-kkz sgm/newstest2019-enkk-src.en.sgmz sgm/newstest2019-enkk-ref.kk.sgmen-ltz sgm/newstest2019-enlt-src.en.sgmz sgm/newstest2019-enlt-ref.lt.sgmz sgm/newstest2019-enru-src.en.sgmz sgm/newstest2019-enru-ref.ru.sgmz sgm/newstest2019-enzh-src.en.sgmz sgm/newstest2019-enzh-ref.zh.sgmfi-enz sgm/newstest2019-fien-src.fi.sgmz sgm/newstest2019-fien-ref.en.sgmz sgm/newstest2019-frde-src.fr.sgmz sgm/newstest2019-frde-ref.de.sgmgu-enz sgm/newstest2019-guen-src.gu.sgmz sgm/newstest2019-guen-ref.en.sgmkk-enz sgm/newstest2019-kken-src.kk.sgmz sgm/newstest2019-kken-ref.en.sgmlt-enz sgm/newstest2019-lten-src.lt.sgmz sgm/newstest2019-lten-ref.en.sgmz sgm/newstest2019-ruen-src.ru.sgmz sgm/newstest2019-ruen-ref.en.sgmz sgm/newstest2019-zhen-src.zh.sgmz sgm/newstest2019-zhen-ref.en.sgm)r2r4)r'r(r)citationr*z wmt19/devz6https://data.statmt.org/wmt19/translation-task/dev.tgzz'Development data for tasks new to 2019. f2ec7af5947c19e0cacb3882eb208002zdev/newsdev2019-lten-src.lt.sgmzdev/newsdev2019-lten-ref.en.sgmzdev/newsdev2019-enlt-src.en.sgmzdev/newsdev2019-enlt-ref.lt.sgmzdev/newsdev2019-guen-src.gu.sgmzdev/newsdev2019-guen-ref.en.sgmzdev/newsdev2019-engu-src.en.sgmzdev/newsdev2019-engu-ref.gu.sgmzdev/newsdev2019-kken-src.kk.sgmzdev/newsdev2019-kken-ref.en.sgmzdev/newsdev2019-enkk-src.en.sgmzdev/newsdev2019-enkk-ref.kk.sgm)r`r\r^rZr_r[zwmt19/google/arzihttps://raw.githubusercontent.com/google/wmt19-paraphrased-references/master/wmt19/ende/wmt19-ende-ar.refz2Additional high-quality reference for WMT19/en-de. d66d9e91548ced0ac476f2390e32e2dez@misc{freitag2020bleu, title={{BLEU} might be Guilty but References are not Innocent}, author={Markus Freitag and David Grangier and Isaac Caswell}, year={2020}, eprint={2004.06063}, archivePrefix={arXiv}, primaryClass={cs.CL}z!wmt19_google_ar.wmt19-ende-ar.refzwmt19/google/arpzjhttps://raw.githubusercontent.com/google/wmt19-paraphrased-references/master/wmt19/ende/wmt19-ende-arp.refz)Additional paraphrase of wmt19/google/ar. c70ea808cf2bff621ad7a8fddd4deca9z#wmt19_google_arp.wmt19-ende-arp.refzwmt19/google/wmtpzkhttps://raw.githubusercontent.com/google/wmt19-paraphrased-references/master/wmt19/ende/wmt19-ende-wmtp.refz6Additional paraphrase of the official WMT19 reference. 587c660ee5fd44727f0db025b71c6a82z%wmt19_google_wmtp.wmt19-ende-wmtp.refzwmt19/google/hqrzjhttps://raw.githubusercontent.com/google/wmt19-paraphrased-references/master/wmt19/ende/wmt19-ende-hqr.refz@Best human selected-reference between wmt19 and wmt19/google/ar. d9221135f62d7152de041f5bfc8efaeaz#wmt19_google_hqr.wmt19-ende-hqr.refzwmt19/google/hqpzjhttps://raw.githubusercontent.com/google/wmt19-paraphrased-references/master/wmt19/ende/wmt19-ende-hqp.refzMBest human-selected reference between wmt19/google/arp and wmt19/google/wmtp. b7c3a07a59c8eccea5367e9ec5417a8az#wmt19_google_hqp.wmt19-ende-hqp.refzwmt19/google/hqallzlhttps://raw.githubusercontent.com/google/wmt19-paraphrased-references/master/wmt19/ende/wmt19-ende-hqall.refziBest human-selected reference among original official reference and the Google reference and paraphrases. edecf10ced59e10b703a6fbcf1fa9dfaz'wmt19_google_hqall.wmt19-ende-hqall.refrz7https://data.statmt.org/wmt18/translation-task/test.tgz f996c245ecffea23d0006fa4c34e9064aq@inproceedings{bojar-etal-2018-findings, title = "Findings of the 2018 Conference on Machine Translation ({WMT}18)", author = "Bojar, Ond{ {r}}ej and Federmann, Christian and Fishel, Mark and Graham, Yvette and Haddow, Barry and Koehn, Philipp and Monz, Christof", booktitle = "Proceedings of the Third Conference on Machine Translation: Shared Task Papers", month = oct, year = "2018", address = "Belgium, Brussels", publisher = "Association for Computational Linguistics", url = "https://www.aclweb.org/anthology/W18-6401", pages = "272--303", }z!test/newstest2018-csen-src.cs.sgmz!test/newstest2018-csen-ref.en.sgmz!test/newstest2018-deen-src.de.sgmz!test/newstest2018-deen-ref.en.sgmz!test/newstest2018-encs-src.en.sgmz!test/newstest2018-encs-ref.cs.sgmz!test/newstest2018-ende-src.en.sgmz!test/newstest2018-ende-ref.de.sgmz!test/newstest2018-enet-src.en.sgmz!test/newstest2018-enet-ref.et.sgmz!test/newstest2018-enfi-src.en.sgmz!test/newstest2018-enfi-ref.fi.sgmz!test/newstest2018-enru-src.en.sgmz!test/newstest2018-enru-ref.ru.sgmz!test/newstest2018-eten-src.et.sgmz!test/newstest2018-eten-ref.en.sgmz!test/newstest2018-fien-src.fi.sgmz!test/newstest2018-fien-ref.en.sgmz!test/newstest2018-ruen-src.ru.sgmz!test/newstest2018-ruen-ref.en.sgmz!test/newstest2018-entr-src.en.sgmz!test/newstest2018-entr-ref.tr.sgmz!test/newstest2018-tren-src.tr.sgmz!test/newstest2018-tren-ref.en.sgmz!test/newstest2018-enzh-src.en.sgmz!test/newstest2018-enzh-ref.zh.sgmz!test/newstest2018-zhen-src.zh.sgmz!test/newstest2018-zhen-ref.en.sgm)r7r0rren-etrYr#et-enr]r2en-trtr-enr%r4)r'r)r(rar*z wmt18/test-tsz:https://data.statmt.org/wmt18/translation-task/test-ts.tgz 5c621a34d512cc2dd74162ae7d00b320z=Official evaluation sources with extra test sets interleaved.z'test-ts/newstest2018-csen-src-ts.cs.sgmz'test-ts/newstest2018-csen-ref-ts.en.sgmz'test-ts/newstest2018-deen-src-ts.de.sgmz'test-ts/newstest2018-deen-ref-ts.en.sgmz'test-ts/newstest2018-encs-src-ts.en.sgmz'test-ts/newstest2018-encs-ref-ts.cs.sgmz'test-ts/newstest2018-ende-src-ts.en.sgmz'test-ts/newstest2018-ende-ref-ts.de.sgmz'test-ts/newstest2018-enet-src-ts.en.sgmz'test-ts/newstest2018-enet-ref-ts.et.sgmz'test-ts/newstest2018-enfi-src-ts.en.sgmz'test-ts/newstest2018-enfi-ref-ts.fi.sgmz'test-ts/newstest2018-enru-src-ts.en.sgmz'test-ts/newstest2018-enru-ref-ts.ru.sgmz'test-ts/newstest2018-eten-src-ts.et.sgmz'test-ts/newstest2018-eten-ref-ts.en.sgmz'test-ts/newstest2018-fien-src-ts.fi.sgmz'test-ts/newstest2018-fien-ref-ts.en.sgmz'test-ts/newstest2018-ruen-src-ts.ru.sgmz'test-ts/newstest2018-ruen-ref-ts.en.sgmz'test-ts/newstest2018-entr-src-ts.en.sgmz'test-ts/newstest2018-entr-ref-ts.tr.sgmz'test-ts/newstest2018-tren-src-ts.tr.sgmz'test-ts/newstest2018-tren-ref-ts.en.sgmz'test-ts/newstest2018-enzh-src-ts.en.sgmz'test-ts/newstest2018-enzh-ref-ts.zh.sgmz'test-ts/newstest2018-zhen-src-ts.zh.sgmz'test-ts/newstest2018-zhen-ref-ts.en.sgmz wmt18/devz6https://data.statmt.org/wmt18/translation-task/dev.tgz 486f391da54a7a3247f02ebd25996f24z%Development data (Estonian<>English).zdev/newsdev2018-eten-src.et.sgmzdev/newsdev2018-eten-ref.en.sgmzdev/newsdev2018-enet-src.en.sgmzdev/newsdev2018-enet-ref.et.sgm)rkrjwmt17z7https://data.statmt.org/wmt17/translation-task/test.tgz 86a1724c276004aa25455ae2a04cef26a#@InProceedings{bojar-EtAl:2017:WMT1, author = {Bojar, Ond\v{r}ej and Chatterjee, Rajen and Federmann, Christian and Graham, Yvette and Haddow, Barry and Huang, Shujian and Huck, Matthias and Koehn, Philipp and Liu, Qun and Logacheva, Varvara and Monz, Christof and Negri, Matteo and Post, Matt and Rubino, Raphael and Specia, Lucia and Turchi, Marco}, title = {Findings of the 2017 Conference on Machine Translation (WMT17)}, booktitle = {Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers}, month = {September}, year = {2017}, address = {Copenhagen, Denmark}, publisher = {Association for Computational Linguistics}, pages = {169--214}, url = {http://www.aclweb.org/anthology/W17-4717} }z!test/newstest2017-csen-src.cs.sgmz!test/newstest2017-csen-ref.en.sgmz!test/newstest2017-deen-src.de.sgmz!test/newstest2017-deen-ref.en.sgmz!test/newstest2017-encs-src.en.sgmz!test/newstest2017-encs-ref.cs.sgmz!test/newstest2017-ende-src.en.sgmz!test/newstest2017-ende-ref.de.sgm!test/newstest2017-enfi-src.en.sgm!test/newstest2017-enfi-ref.fi.sgmz!test/newstest2017-enlv-src.en.sgmz!test/newstest2017-enlv-ref.lv.sgmz!test/newstest2017-enru-src.en.sgmz!test/newstest2017-enru-ref.ru.sgmz!test/newstest2017-entr-src.en.sgmz!test/newstest2017-entr-ref.tr.sgmz!test/newstest2017-enzh-src.en.sgmz!test/newstest2017-enzh-ref.zh.sgmz!test/newstest2017-fien-src.fi.sgmz!test/newstest2017-fien-ref.en.sgmz!test/newstest2017-lven-src.lv.sgmz!test/newstest2017-lven-ref.en.sgmz!test/newstest2017-ruen-src.ru.sgmz!test/newstest2017-ruen-ref.en.sgmz!test/newstest2017-tren-src.tr.sgmz!test/newstest2017-tren-ref.en.sgmz!test/newstest2017-zhen-src.zh.sgmz!test/newstest2017-zhen-ref.en.sgm)r7r0rrrYen-lvr#rlr%r]lv-enr2rmr4zwmt17/Bz)Additional reference for EN-FI and FI-EN.z"test/newstestB2017-enfi-src.en.sgm"test/newstestB2017-enfi-ref.fi.sgmz wmt17/tworefszSystems with two references.)rrrsrvzwmt17/improvedz@https://data.statmt.org/wmt17/translation-task/test-update-1.tgz 91dbfd5af99bc6891a637a68e04dfd41z&Improved zh-en and en-zh translations.znewstest2017-enzh-src.en.sgmznewstest2017-enzh-ref.zh.sgmnewstest2017-zhen-src.zh.sgmnewstest2017-zhen-ref.en.sgm)r%r4z wmt17/devz6https://data.statmt.org/wmt17/translation-task/dev.tgz 9b1aa63c1cf49dccdd20b962fe313989z4Development sets released for new languages in 2017.zdev/newsdev2017-enlv-src.en.sgmzdev/newsdev2017-enlv-ref.lv.sgmzdev/newsdev2017-enzh-src.en.sgmzdev/newsdev2017-enzh-ref.zh.sgmzdev/newsdev2017-lven-src.lv.sgmzdev/newsdev2017-lven-ref.en.sgmzdev/newsdev2017-zhen-src.zh.sgmzdev/newsdev2017-zhen-ref.en.sgm)rtr%rur4zwmt17/mszThttps://github.com/MicrosoftTranslator/Translator-HumanParityData/archive/master.zip 18fdaa7a3c84cf6ef688da1f6a5fa96fz>Additional Chinese-English references from Microsoft Research.u@inproceedings{achieving-human-parity-on-automatic-chinese-to-english-news-translation, author = {Hassan Awadalla, Hany and Aue, Anthony and Chen, Chang and Chowdhary, Vishal and Clark, Jonathan and Federmann, Christian and Huang, Xuedong and Junczys-Dowmunt, Marcin and Lewis, Will and Li, Mu and Liu, Shujie and Liu, Tie-Yan and Luo, Renqian and Menezes, Arul and Qin, Tao and Seide, Frank and Tan, Xu and Tian, Fei and Wu, Lijun and Wu, Shuangzhi and Xia, Yingce and Zhang, Dongdong and Zhang, Zhirui and Zhou, Ming}, title = {Achieving Human Parity on Automatic Chinese to English News Translation}, booktitle = {}, year = {2018}, month = {March}, abstract = {Machine translation has made rapid advances in recent years. Millions of people are using it today in online translation systems and mobile applications in order to communicate across language barriers. The question naturally arises whether such systems can approach or achieve parity with human translations. In this paper, we first address the problem of how to define and accurately measure human parity in translation. We then describe Microsoft’s machine translation system and measure the quality of its translations on the widely used WMT 2017 news translation task from Chinese to English. We find that our latest neural machine translation system has reached a new state-of-the-art, and that the translation quality is at human parity when compared to professional human translations. We also find that it significantly exceeds the quality of crowd-sourced non-professional translations.}, publisher = {}, url = {https://www.microsoft.com/en-us/research/publication/achieving-human-parity-on-automatic-chinese-to-english-news-translation/}, address = {}, pages = {}, journal = {}, volume = {}, chapter = {}, isbn = {}, })rxryzsTranslator-HumanParityData-master/Translator-HumanParityData/References/Translator-HumanParityData-Reference-HT.txtzsTranslator-HumanParityData-master/Translator-HumanParityData/References/Translator-HumanParityData-Reference-PE.txtwmt16z7https://data.statmt.org/wmt16/translation-task/test.tgz 3d809cd0c2c86adb2c67034d15c4e446aw@InProceedings{bojar-EtAl:2016:WMT1, author = {Bojar, Ond\v{r}ej and Chatterjee, Rajen and Federmann, Christian and Graham, Yvette and Haddow, Barry and Huck, Matthias and Jimeno Yepes, Antonio and Koehn, Philipp and Logacheva, Varvara and Monz, Christof and Negri, Matteo and Neveol, Aurelie and Neves, Mariana and Popel, Martin and Post, Matt and Rubino, Raphael and Scarton, Carolina and Specia, Lucia and Turchi, Marco and Verspoor, Karin and Zampieri, Marcos}, title = {Findings of the 2016 Conference on Machine Translation}, booktitle = {Proceedings of the First Conference on Machine Translation}, month = {August}, year = {2016}, address = {Berlin, Germany}, publisher = {Association for Computational Linguistics}, pages = {131--198}, url = {http://www.aclweb.org/anthology/W/W16/W16-2301} }z!test/newstest2016-csen-src.cs.sgmz!test/newstest2016-csen-ref.en.sgmz!test/newstest2016-deen-src.de.sgmz!test/newstest2016-deen-ref.en.sgmz!test/newstest2016-encs-src.en.sgmz!test/newstest2016-encs-ref.cs.sgmz!test/newstest2016-ende-src.en.sgmz!test/newstest2016-ende-ref.de.sgm!test/newstest2016-enfi-src.en.sgm!test/newstest2016-enfi-ref.fi.sgmz!test/newstest2016-enro-src.en.sgmz!test/newstest2016-enro-ref.ro.sgmz!test/newstest2016-enru-src.en.sgmz!test/newstest2016-enru-ref.ru.sgmz!test/newstest2016-entr-src.en.sgmz!test/newstest2016-entr-ref.tr.sgmz!test/newstest2016-fien-src.fi.sgmz!test/newstest2016-fien-ref.en.sgmz!test/newstest2016-roen-src.ro.sgmz!test/newstest2016-roen-ref.en.sgmz!test/newstest2016-ruen-src.ru.sgmz!test/newstest2016-ruen-ref.en.sgmz!test/newstest2016-tren-src.tr.sgmz!test/newstest2016-tren-ref.en.sgm) r7r0rrrYen-ror#rlr]ro-enr2rmzwmt16/BzAdditional reference for EN-FI."test/newstestB2016-enfi-ref.fi.sgmz wmt16/tworefszEN-FI with two references.)r~rrz wmt16/devz6https://data.statmt.org/wmt16/translation-task/dev.tgz 4a3dc2760bb077f4308cce96b06e6af6z4Development sets released for new languages in 2016.zdev/newsdev2016-enro-src.en.sgmzdev/newsdev2016-enro-ref.ro.sgmzdev/newsdev2016-entr-src.en.sgmzdev/newsdev2016-entr-ref.tr.sgmzdev/newsdev2016-roen-src.ro.sgmzdev/newsdev2016-roen-ref.en.sgmzdev/newsdev2016-tren-src.tr.sgmzdev/newsdev2016-tren-ref.en.sgm)rrlrrmwmt15z!https://statmt.org/wmt15/test.tgz 67e3beca15e69fe3d36de149da0a96dfa@InProceedings{bojar-EtAl:2015:WMT, author = {Bojar, Ond\v{r}ej and Chatterjee, Rajen and Federmann, Christian and Haddow, Barry and Huck, Matthias and Hokamp, Chris and Koehn, Philipp and Logacheva, Varvara and Monz, Christof and Negri, Matteo and Post, Matt and Scarton, Carolina and Specia, Lucia and Turchi, Marco}, title = {Findings of the 2015 Workshop on Statistical Machine Translation}, booktitle = {Proceedings of the Tenth Workshop on Statistical Machine Translation}, month = {September}, year = {2015}, address = {Lisbon, Portugal}, publisher = {Association for Computational Linguistics}, pages = {1--46}, url = {http://aclweb.org/anthology/W15-3001} }z(test/newsdiscusstest2015-enfr-src.en.sgmz(test/newsdiscusstest2015-enfr-ref.fr.sgmz(test/newsdiscusstest2015-fren-src.fr.sgmz(test/newsdiscusstest2015-fren-ref.en.sgmz!test/newstest2015-csen-src.cs.sgmz!test/newstest2015-csen-ref.en.sgmz!test/newstest2015-deen-src.de.sgmz!test/newstest2015-deen-ref.en.sgmz!test/newstest2015-encs-src.en.sgmz!test/newstest2015-encs-ref.cs.sgmz!test/newstest2015-ende-src.en.sgmz!test/newstest2015-ende-ref.de.sgmz!test/newstest2015-enfi-src.en.sgmz!test/newstest2015-enfi-ref.fi.sgmz!test/newstest2015-enru-src.en.sgmz!test/newstest2015-enru-ref.ru.sgmz!test/newstest2015-fien-src.fi.sgmz!test/newstest2015-fien-ref.en.sgmz!test/newstest2015-ruen-src.ru.sgmz!test/newstest2015-ruen-ref.en.sgm) en-frfr-enr7r0rrrYr#r]r2wmt14z*https://statmt.org/wmt14/test-filtered.tgz 84c597844c1542e29c2aff23aaee4310a@InProceedings{bojar-EtAl:2014:W14-33, author = {Bojar, Ondrej and Buck, Christian and Federmann, Christian and Haddow, Barry and Koehn, Philipp and Leveling, Johannes and Monz, Christof and Pecina, Pavel and Post, Matt and Saint-Amand, Herve and Soricut, Radu and Specia, Lucia and Tamchyna, Ale\v{s}}, title = {Findings of the 2014 Workshop on Statistical Machine Translation}, booktitle = {Proceedings of the Ninth Workshop on Statistical Machine Translation}, month = {June}, year = {2014}, address = {Baltimore, Maryland, USA}, publisher = {Association for Computational Linguistics}, pages = {12--58}, url = {http://www.aclweb.org/anthology/W/W14/W14-3302} }z!test/newstest2014-csen-src.cs.sgmz!test/newstest2014-csen-ref.en.sgmz!test/newstest2014-csen-src.en.sgmz!test/newstest2014-csen-ref.cs.sgmz!test/newstest2014-deen-src.de.sgmz!test/newstest2014-deen-ref.en.sgmz!test/newstest2014-deen-src.en.sgmz!test/newstest2014-deen-ref.de.sgmz!test/newstest2014-fren-src.en.sgmz!test/newstest2014-fren-ref.fr.sgmz!test/newstest2014-fren-src.fr.sgmz!test/newstest2014-fren-ref.en.sgmz!test/newstest2014-hien-src.en.sgmz!test/newstest2014-hien-ref.hi.sgmz!test/newstest2014-hien-src.hi.sgmz!test/newstest2014-hien-ref.en.sgmz!test/newstest2014-ruen-src.en.sgmz!test/newstest2014-ruen-ref.ru.sgmz!test/newstest2014-ruen-src.ru.sgmz!test/newstest2014-ruen-ref.en.sgm) r7rr0rrrr zhi-enr#r2z wmt14/fullz&https://statmt.org/wmt14/test-full.tgz a8cd784e006feb32ac6f3d9ec7eb389azHEvaluation data released after official evaluation for further research.z&test-full/newstest2014-csen-src.cs.sgmz&test-full/newstest2014-csen-ref.en.sgmz&test-full/newstest2014-csen-src.en.sgmz&test-full/newstest2014-csen-ref.cs.sgmz&test-full/newstest2014-deen-src.de.sgmz&test-full/newstest2014-deen-ref.en.sgmz&test-full/newstest2014-deen-src.en.sgmz&test-full/newstest2014-deen-ref.de.sgmz&test-full/newstest2014-fren-src.en.sgmz&test-full/newstest2014-fren-ref.fr.sgmz&test-full/newstest2014-fren-src.fr.sgmz&test-full/newstest2014-fren-ref.en.sgmz&test-full/newstest2014-hien-src.en.sgmz&test-full/newstest2014-hien-ref.hi.sgmz&test-full/newstest2014-hien-src.hi.sgmz&test-full/newstest2014-hien-ref.en.sgmz&test-full/newstest2014-ruen-src.en.sgmz&test-full/newstest2014-ruen-ref.ru.sgmz&test-full/newstest2014-ruen-src.ru.sgmz&test-full/newstest2014-ruen-ref.en.sgmwmt13z!https://statmt.org/wmt13/test.tgz 48eca5d02f637af44e85186847141f67a@InProceedings{bojar-EtAl:2013:WMT, author = {Bojar, Ond\v{r}ej and Buck, Christian and Callison-Burch, Chris and Federmann, Christian and Haddow, Barry and Koehn, Philipp and Monz, Christof and Post, Matt and Soricut, Radu and Specia, Lucia}, title = {Findings of the 2013 {Workshop on Statistical Machine Translation}}, booktitle = {Proceedings of the Eighth Workshop on Statistical Machine Translation}, month = {August}, year = {2013}, address = {Sofia, Bulgaria}, publisher = {Association for Computational Linguistics}, pages = {1--44}, url = {http://www.aclweb.org/anthology/W13-2201} }ztest/newstest2013-src.cs.sgmztest/newstest2013-src.en.sgmztest/newstest2013-src.de.sgmztest/newstest2013-src.es.sgmztest/newstest2013-src.fr.sgmztest/newstest2013-src.ru.sgm) r7rr0res-enrrrr2r#wmt12z!https://statmt.org/wmt12/test.tgz 608232d34ebc4ba2ff70fead45674e47a>@InProceedings{callisonburch-EtAl:2012:WMT, author = {Callison-Burch, Chris and Koehn, Philipp and Monz, Christof and Post, Matt and Soricut, Radu and Specia, Lucia}, title = {Findings of the 2012 Workshop on Statistical Machine Translation}, booktitle = {Proceedings of the Seventh Workshop on Statistical Machine Translation}, month = {June}, year = {2012}, address = {Montr{'e}al, Canada}, publisher = {Association for Computational Linguistics}, pages = {10--51}, url = {http://www.aclweb.org/anthology/W12-3102} }ztest/newstest2012-src.cs.sgmztest/newstest2012-src.en.sgmztest/newstest2012-src.de.sgmztest/newstest2012-src.es.sgmztest/newstest2012-src.fr.sgm)r7rr0rrrrrwmt11z!https://statmt.org/wmt11/test.tgz b0c9680adf32d394aefc2b24e3a5937ea@InProceedings{callisonburch-EtAl:2011:WMT, author = {Callison-Burch, Chris and Koehn, Philipp and Monz, Christof and Zaidan, Omar}, title = {Findings of the 2011 Workshop on Statistical Machine Translation}, booktitle = {Proceedings of the Sixth Workshop on Statistical Machine Translation}, month = {July}, year = {2011}, address = {Edinburgh, Scotland}, publisher = {Association for Computational Linguistics}, pages = {22--64}, url = {http://www.aclweb.org/anthology/W11-2103} }znewstest2011-src.cs.sgmznewstest2011-src.en.sgmznewstest2011-src.de.sgmznewstest2011-src.fr.sgmznewstest2011-src.es.sgm)r7rr0rrrrrwmt10z!https://statmt.org/wmt10/test.tgz 491cb885a355da5a23ea66e7b3024d5ca@InProceedings{callisonburch-EtAl:2010:WMT, author = {Callison-Burch, Chris and Koehn, Philipp and Monz, Christof and Peterson, Kay and Przybocki, Mark and Zaidan, Omar}, title = {Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation}, booktitle = {Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR}, month = {July}, year = {2010}, address = {Uppsala, Sweden}, publisher = {Association for Computational Linguistics}, pages = {17--53}, note = {Revised August 2010}, url = {http://www.aclweb.org/anthology/W10-1703} }ztest/newstest2010-src.cz.sgmztest/newstest2010-src.en.sgmztest/newstest2010-src.de.sgmztest/newstest2010-src.es.sgmztest/newstest2010-src.fr.sgmwmt09z!https://statmt.org/wmt09/test.tgz da227abfbd7b666ec175b742a0d27b37a&@InProceedings{callisonburch-EtAl:2009:WMT-09, author = {Callison-Burch, Chris and Koehn, Philipp and Monz, Christof and Schroeder, Josh}, title = {Findings of the 2009 {W}orkshop on {S}tatistical {M}achine {T}ranslation}, booktitle = {Proceedings of the Fourth Workshop on Statistical Machine Translation}, month = {March}, year = {2009}, address = {Athens, Greece}, publisher = {Association for Computational Linguistics}, pages = {1--28}, url = {http://www.aclweb.org/anthology/W/W09/W09-0401} }ztest/newstest2009-src.cz.sgmztest/newstest2009-src.en.sgmztest/newstest2009-src.de.sgmztest/newstest2009-src.es.sgmztest/newstest2009-src.fr.sgmztest/newstest2009-src.hu.sgmztest/newstest2009-src.it.sgm) r7rr0rrrrrhu-enen-huit-enzen-itwmt08z!https://statmt.org/wmt08/test.tgz 0582e4e894a3342044059c894e1aea3da @InProceedings{callisonburch-EtAl:2008:WMT, author = {Callison-Burch, Chris and Fordyce, Cameron and Koehn, Philipp and Monz, Christof and Schroeder, Josh}, title = {Further Meta-Evaluation of Machine Translation}, booktitle = {Proceedings of the Third Workshop on Statistical Machine Translation}, month = {June}, year = {2008}, address = {Columbus, Ohio}, publisher = {Association for Computational Linguistics}, pages = {70--106}, url = {http://www.aclweb.org/anthology/W/W08/W08-0309} }ztest/newstest2008-src.cz.sgmztest/newstest2008-src.en.sgmztest/newstest2008-src.de.sgmztest/newstest2008-src.es.sgmztest/newstest2008-src.fr.sgmztest/newstest2008-src.hu.sgm) r7rr0rrrrrrrzwmt08/ncz+Official evaluation data (news commentary).ztest/nc-test2008-src.cz.sgmztest/nc-test2008-src.en.sgm)r7rzwmt08/europarlz$Official evaluation data (Europarl).ztest/test2008-src.de.sgmztest/test2008-src.en.sgmztest/test2008-src.es.sgmztest/test2008-src.fr.sgm)r0rrrrriwslt17) chttps://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/fr/en-fr.tgzchttps://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/fr/en/fr-en.tgzchttps://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/de/en-de.tgzchttps://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/de/en/de-en.tgzzchttps://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/ar/en-ar.tgzzchttps://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/ar/en/ar-en.tgzzchttps://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/ja/en-ja.tgzzchttps://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/ja/en/ja-en.tgzzchttps://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/ko/en-ko.tgzzchttps://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/ko/en/ko-en.tgzchttps://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/en/zh/en-zh.tgzchttps://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-ted-test/texts/zh/en/zh-en.tgz) 1849bcc3b006dc0642a8843b11aa7192 79bf7a2ef02d226875f55fb076e7e473 b68e7097b179491f6c466ef41ad72b9b e3f5b2a075a2da1a395c8b60bf1e9be1 ecdc6bc4ab4c8984e919444f3c05183a 4b5141d14b98706c081371e2f8afe0ca d957ee79de1f33c89077d37c5a2c5b06 c213e8bb918ebf843543fe9fd2e33db2 59f6a81c707378176e9ad8bb8d811f5f 7e580af973bb389ec1d1378a1850742f 975a858783a0ebec8c57d83ddd5bd381 cc51d9b7fe1ff2af858c6a0dd80b8815z#Official evaluation data for IWSLT.u@InProceedings{iwslt2017, author = {Cettolo, Mauro and Federico, Marcello and Bentivogli, Luisa and Niehues, Jan and Stüker, Sebastian and Sudoh, Katsuitho and Yoshino, Koichiro and Federmann, Christian}, title = {Overview of the IWSLT 2017 Evaluation Campaign}, booktitle = {14th International Workshop on Spoken Language Translation}, month = {December}, year = {2017}, address = {Tokyo, Japan}, pages = {2--14}, url = {http://workshop2017.iwslt.org/downloads/iwslt2017_proceeding_v2.pdf} }z&en-fr/IWSLT17.TED.tst2017.en-fr.en.xmlz&fr-en/IWSLT17.TED.tst2017.fr-en.fr.xmlz&en-de/IWSLT17.TED.tst2017.en-de.en.xmlz&de-en/IWSLT17.TED.tst2017.de-en.de.xmlz&en-zh/IWSLT17.TED.tst2017.en-zh.en.xmlz&zh-en/IWSLT17.TED.tst2017.zh-en.zh.xmlz&en-ar/IWSLT17.TED.tst2017.en-ar.en.xmlz&ar-en/IWSLT17.TED.tst2017.ar-en.ar.xmlz&en-ja/IWSLT17.TED.tst2017.en-ja.en.xmlz&ja-en/IWSLT17.TED.tst2017.ja-en.ja.xmlz&en-ko/IWSLT17.TED.tst2017.en-ko.en.xmlz&ko-en/IWSLT17.TED.tst2017.ko-en.ko.xml) rrrr0r%r4zen-arzar-enr"r1zen-kozko-enziwslt17/tst2016)rrrrrr)rrrrrrz Development data for IWSLT 2017.z&en-fr/IWSLT17.TED.tst2016.en-fr.en.xmlz&fr-en/IWSLT17.TED.tst2016.fr-en.fr.xmlz&en-de/IWSLT17.TED.tst2016.en-de.en.xmlz&de-en/IWSLT17.TED.tst2016.de-en.de.xmlz&en-zh/IWSLT17.TED.tst2016.en-zh.en.xmlz&zh-en/IWSLT17.TED.tst2016.zh-en.zh.xml)rrrr0r%r4ziwslt17/tst2015)zahttps://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/de/en-de.tgzzahttps://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/de/en/de-en.tgzzahttps://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/fr/en-fr.tgzzahttps://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/fr/en/fr-en.tgzzahttps://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/en/zh/en-zh.tgzzahttps://raw.githubusercontent.com/hlt-mt/WIT3/master/archive/2017-01-trnted/texts/zh/en/zh-en.tgz) d8a32cfc002a4f12b17429cfa78050e6 ca2b94d694150d4d6c5dc64c200fa589 3cf07ebe305312b12f7f1a4d5f8f8377 19927da9de0f40348cad9c0fc61642ac 575b788dad6c5b9c5cee636f9ac1094a 1c0ae40171d52593df8a6963d3828116z&en-fr/IWSLT17.TED.tst2015.en-fr.en.xmlz&fr-en/IWSLT17.TED.tst2015.fr-en.fr.xmlz&en-de/IWSLT17.TED.tst2015.en-de.en.xmlz&de-en/IWSLT17.TED.tst2015.de-en.de.xmlz&en-zh/IWSLT17.TED.tst2015.en-zh.en.xmlz&zh-en/IWSLT17.TED.tst2015.zh-en.zh.xmlziwslt17/tst2014z&en-fr/IWSLT17.TED.tst2014.en-fr.en.xmlz&fr-en/IWSLT17.TED.tst2014.fr-en.fr.xmlz&en-de/IWSLT17.TED.tst2014.en-de.en.xmlz&de-en/IWSLT17.TED.tst2014.de-en.de.xmlz&en-zh/IWSLT17.TED.tst2014.en-zh.en.xmlz&zh-en/IWSLT17.TED.tst2014.zh-en.zh.xmlziwslt17/tst2013z&en-fr/IWSLT17.TED.tst2013.en-fr.en.xmlz&fr-en/IWSLT17.TED.tst2013.fr-en.fr.xmlz&en-de/IWSLT17.TED.tst2013.en-de.en.xmlz&de-en/IWSLT17.TED.tst2013.de-en.de.xmlz&en-zh/IWSLT17.TED.tst2013.en-zh.en.xmlz&zh-en/IWSLT17.TED.tst2013.zh-en.zh.xmlziwslt17/tst2012z&en-fr/IWSLT17.TED.tst2012.en-fr.en.xmlz&fr-en/IWSLT17.TED.tst2012.fr-en.fr.xmlz&en-de/IWSLT17.TED.tst2012.en-de.en.xmlz&de-en/IWSLT17.TED.tst2012.de-en.de.xmlz&en-zh/IWSLT17.TED.tst2012.en-zh.en.xmlz&zh-en/IWSLT17.TED.tst2012.zh-en.zh.xmlziwslt17/tst2011z&en-fr/IWSLT17.TED.tst2011.en-fr.en.xmlz&fr-en/IWSLT17.TED.tst2011.fr-en.fr.xmlz&en-de/IWSLT17.TED.tst2011.en-de.en.xmlz&de-en/IWSLT17.TED.tst2011.de-en.de.xmlz&en-zh/IWSLT17.TED.tst2011.en-zh.en.xmlz&zh-en/IWSLT17.TED.tst2011.zh-en.zh.xmlziwslt17/tst2010z&en-fr/IWSLT17.TED.tst2010.en-fr.en.xmlz&fr-en/IWSLT17.TED.tst2010.fr-en.fr.xmlz&en-de/IWSLT17.TED.tst2010.en-de.en.xmlz&de-en/IWSLT17.TED.tst2010.de-en.de.xmlz&en-zh/IWSLT17.TED.tst2010.en-zh.en.xmlz&zh-en/IWSLT17.TED.tst2010.zh-en.zh.xmlziwslt17/dev2010z&en-fr/IWSLT17.TED.dev2010.en-fr.en.xmlz&fr-en/IWSLT17.TED.dev2010.fr-en.fr.xmlz&en-de/IWSLT17.TED.dev2010.en-de.en.xmlz&de-en/IWSLT17.TED.dev2010.de-en.de.xmlz&en-zh/IWSLT17.TED.dev2010.en-zh.en.xmlz&zh-en/IWSLT17.TED.dev2010.zh-en.zh.xmlz mtedx/validzGhttps://raw.githubusercontent.com/esalesky/mtedx-eval/main/valid.tar.gzz4mTEDx evaluation data, valid: http://openslr.org/100a|@misc{salesky2021multilingual, title={The Multilingual TEDx Corpus for Speech Recognition and Translation}, author={Elizabeth Salesky and Matthew Wiesner and Jacob Bremerman and Roldano Cattoni and Matteo Negri and Marco Turchi and Douglas W. Oard and Matt Post}, year={2021}, eprint={2102.01757}, archivePrefix={arXiv}, primaryClass={cs.CL} } 40618171614c50e6cbb5e5bbceee0635zvalid/mtedx-valid-elen.elzvalid/mtedx-valid-elen.enzvalid/mtedx-valid-esen.eszvalid/mtedx-valid-esen.enzvalid/mtedx-valid-esfr.eszvalid/mtedx-valid-esfr.frzvalid/mtedx-valid-esit.eszvalid/mtedx-valid-esit.itzvalid/mtedx-valid-espt.eszvalid/mtedx-valid-espt.ptzvalid/mtedx-valid-fren.frzvalid/mtedx-valid-fren.enzvalid/mtedx-valid-fres.frzvalid/mtedx-valid-fres.eszvalid/mtedx-valid-frpt.frzvalid/mtedx-valid-frpt.ptzvalid/mtedx-valid-iten.itzvalid/mtedx-valid-iten.enzvalid/mtedx-valid-ites.itzvalid/mtedx-valid-ites.eszvalid/mtedx-valid-pten.ptzvalid/mtedx-valid-pten.enzvalid/mtedx-valid-ptes.ptzvalid/mtedx-valid-ptes.eszvalid/mtedx-valid-ruen.ruzvalid/mtedx-valid-ruen.en) zel-enrzes-frzes-itzes-ptrzfr-eszfr-ptrzit-eszpt-enzpt-esr2)r'r(rar)r*z mtedx/testzFhttps://raw.githubusercontent.com/esalesky/mtedx-eval/main/test.tar.gzz3mTEDx evaluation data, test: http://openslr.org/100 fa4cb1548c210ec424d7d6bc9a3675a7ztest/mtedx-test-elen.elztest/mtedx-test-elen.enztest/mtedx-test-esen.esztest/mtedx-test-esen.enztest/mtedx-test-esfr.esztest/mtedx-test-esfr.frztest/mtedx-test-esit.esztest/mtedx-test-esit.itztest/mtedx-test-espt.esztest/mtedx-test-espt.ptztest/mtedx-test-fren.frztest/mtedx-test-fren.enztest/mtedx-test-fres.frztest/mtedx-test-fres.esztest/mtedx-test-frpt.frztest/mtedx-test-frpt.ptztest/mtedx-test-iten.itztest/mtedx-test-iten.enztest/mtedx-test-ites.itztest/mtedx-test-ites.esztest/mtedx-test-pten.ptztest/mtedx-test-pten.enztest/mtedx-test-ptes.ptztest/mtedx-test-ptes.esztest/mtedx-test-ruen.ruztest/mtedx-test-ruen.enz multi30k/2016ghttps://raw.githubusercontent.com/multi30k/dataset/master/data/task1/multi30k_test_sets_d3ec2a38.tar.gz 9cf8f22d57fee2ca2af3c682dfdc525bz(2016 flickr test set of Multi30k dataseta@InProceedings{elliott-etal-2016-multi30k, title = "{M}ulti30{K}: Multilingual {E}nglish-{G}erman Image Descriptions", author = "Elliott, Desmond and Frank, Stella and Sima{'}an, Khalil and Specia, Lucia", booktitle = "Proceedings of the 5th Workshop on Vision and Language", month = aug, year = "2016", address = "Berlin, Germany", publisher = "Association for Computational Linguistics", url = "https://www.aclweb.org/anthology/W16-3210", doi = "10.18653/v1/W16-3210", pages = "70--74", }ztest_2016_flickr.enztest_2016_flickr.frztest_2016_flickr.deztest_2016_flickr.cs)rrrz multi30k/2017z(2017 flickr test set of Multi30k datasetav@InProceedings{elliott-etal-2016-multi30k, title = "{M}ulti30{K}: Multilingual {E}nglish-{G}erman Image Descriptions", author = "Elliott, Desmond and Frank, Stella and Sima{'}an, Khalil and Specia, Lucia", booktitle = "Proceedings of the 5th Workshop on Vision and Language", month = aug, year = "2016", address = "Berlin, Germany", publisher = "Association for Computational Linguistics", url = "https://www.aclweb.org/anthology/W16-3210", doi = "10.18653/v1/W16-3210", pages = "70--74", } @InProceedings{elliott-etal-2017-findings, title = "Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description", author = {Elliott, Desmond and Frank, Stella and Barrault, Lo{\"\i}c and Bougares, Fethi and Specia, Lucia}, booktitle = "Proceedings of the Second Conference on Machine Translation", month = sep, year = "2017", address = "Copenhagen, Denmark", publisher = "Association for Computational Linguistics", url = "https://www.aclweb.org/anthology/W17-4718", doi = "10.18653/v1/W17-4718", pages = "215--233", } ztest_2017_flickr.enztest_2017_flickr.frztest_2017_flickr.de)rrz multi30k/2018)rz_https://raw.githubusercontent.com/multi30k/dataset/master/data/task1/raw/test_2018_flickr.cs.gzz_https://raw.githubusercontent.com/multi30k/dataset/master/data/task1/raw/test_2018_flickr.de.gzz_https://raw.githubusercontent.com/multi30k/dataset/master/data/task1/raw/test_2018_flickr.fr.gz)r 4c6b6490e58107b2e397c5e3e1690abc 87e00327083dd69feaa029a8f7c1a047 a64563e986438ed731a6713027c36bfdzq2018 flickr test set of Multi30k dataset. See https://competitions.codalab.org/competitions/19917 for evaluation.aw@InProceedings{elliott-etal-2016-multi30k, title = "{M}ulti30{K}: Multilingual {E}nglish-{G}erman Image Descriptions", author = "Elliott, Desmond and Frank, Stella and Sima{'}an, Khalil and Specia, Lucia", booktitle = "Proceedings of the 5th Workshop on Vision and Language", month = aug, year = "2016", address = "Berlin, Germany", publisher = "Association for Computational Linguistics", url = "https://www.aclweb.org/anthology/W16-3210", doi = "10.18653/v1/W16-3210", pages = "70--74", } @InProceedings{barrault-etal-2018-findings, title = "Findings of the Third Shared Task on Multimodal Machine Translation", author = {Barrault, Lo{\"\i}c and Bougares, Fethi and Specia, Lucia and Lala, Chiraag and Elliott, Desmond and Frank, Stella}, booktitle = "Proceedings of the Third Conference on Machine Translation: Shared Task Papers", month = oct, year = "2018", address = "Belgium, Brussels", publisher = "Association for Computational Linguistics", url = "https://www.aclweb.org/anthology/W18-6402", doi = "10.18653/v1/W18-6402", pages = "304--323", } ztest_2018_flickr.enz$multi30k_2018.test_2018_flickr.fr.gzz$multi30k_2018.test_2018_flickr.de.gzz$multi30k_2018.test_2018_flickr.cs.gzmtnt2019z6https://pmichel31415.github.io/hosting/MTNT2019.tar.gzz.Test set for the WMT 19 robustness shared task 78a672e1931f106a8549023c0e8af8f6z2:MTNT2019/en-fr.final.tsvz3:MTNT2019/en-fr.final.tsvz2:MTNT2019/fr-en.final.tsvz3:MTNT2019/fr-en.final.tsvz2:MTNT2019/en-ja.final.tsvz3:MTNT2019/en-ja.final.tsvz2:MTNT2019/ja-en.final.tsvz3:MTNT2019/ja-en.final.tsv)rrr"r1z mtnt1.1/testzKhttps://github.com/pmichel31415/mtnt/releases/download/v1.1/MTNT.1.1.tar.gzz_Test data for the Machine Translation of Noisy Text task: http://www.cs.cmu.edu/~pmichel1/mtnt/a@InProceedings{michel2018a:mtnt, author = "Michel, Paul and Neubig, Graham", title = "MTNT: A Testbed for Machine Translation of Noisy Text", booktitle = "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing", year = "2018", publisher = "Association for Computational Linguistics", pages = "543--553", location = "Brussels, Belgium", url = "http://aclweb.org/anthology/D18-1050" } 8ce1831ac584979ba8cdcd9d4be43e1dz1:MTNT/test/test.en-fr.tsvz2:MTNT/test/test.en-fr.tsvz1:MTNT/test/test.fr-en.tsvz2:MTNT/test/test.fr-en.tsvz1:MTNT/test/test.en-ja.tsvz2:MTNT/test/test.en-ja.tsvz1:MTNT/test/test.ja-en.tsvz2:MTNT/test/test.ja-en.tsvz mtnt1.1/validzeValidation data for the Machine Translation of Noisy Text task: http://www.cs.cmu.edu/~pmichel1/mtnt/z1:MTNT/valid/valid.en-fr.tsvz2:MTNT/valid/valid.en-fr.tsvz1:MTNT/valid/valid.fr-en.tsvz2:MTNT/valid/valid.fr-en.tsvz1:MTNT/valid/valid.en-ja.tsvz2:MTNT/valid/valid.en-ja.tsvz1:MTNT/valid/valid.ja-en.tsvz2:MTNT/valid/valid.ja-en.tsvz mtnt1.1/trainz1:MTNT/train/train.en-fr.tsvz2:MTNT/train/train.en-fr.tsvz1:MTNT/train/train.fr-en.tsvz2:MTNT/train/train.fr-en.tsvz1:MTNT/train/train.en-ja.tsvz2:MTNT/train/train.en-ja.tsvz1:MTNT/train/train.ja-en.tsvz2:MTNT/train/train.ja-en.tsvN) fake_sgmlrr iwslt_xmlr plain_textrtsvrwmt_xmlr_SUBSETSitemsSUBSETSsortedlistvalues COUNTRIESDOMAINSDATASETSrrrrs     # 3   &`     ~      - >IT_m  !%)-159=AFo(8D  !%)-159=AF2R  _  l  y      !b              x JV c  m  Q] j4e          )        :        K        \            q                  q N|+Y6e             }                #  0   G    S    b    q