o wiF` @sddlZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z mZmZmZddlmZddlmZddlZejdZejdejedZed ZGd d d Z dPdede de!dee"e e"ffddZ#dededefddZ$de e"defddZ% dQdee"deee"dee"fddZ&dRd"d#Z'd$e(de(fd%d&Z)d'd(Z*dSd*e"d+e!fd,d-Z+dTd.d/Z,de"d0e"de"fd1d2Z-de"d0e"de e"fd3d4Z.de e"fd5d6Z/d7d8Z0d9d:Z1dTd;d<Z2dQd=d>Z3d?e"de e"fd@dAZ4de e"fdBdCZ5d0e"de e"fdDdEZ6de e"fdFdGZ7de e"fdHdIZ8dQdJdKZ9dLdMZ:dNdOl;m._color_p_value.fNTz (u ± r(u (μ ± 95% CI)) popfloatitems isinstancerscoremeancip_valueappend) r!r"r# new_scoresp_val_break_char is_bootstrapr+metricvalsnew_valsresult_strrrr_format_score_lines3s*   .   r?results signaturesargsc Csf|jdkrag}t|}tt|dD]@}i}|d||d<tdt|D]'}t||||trB|||||||<q(||||j|||<q(||qt t j |dddS|j}|dvrkd }n|d krqd }t ||j |d kd }i} d } d} |D]0} || } | dds| r| ddr| d} d} t| dd| d<|| | t| d<qt| d|dddd|j dd}t |t |jp|j}|r|jrdnd}|jr|jn|j}|jrdnd}d|d|d |}td!d}t| d}td"d#}td$d%}t d&t|d'|d'd&t|t d(|d|jr5t d)nt d*t t d+|d,|d-t d.|d/t d0t d1|d2t t d3|d4t d5t d6|dt d7|dt t d8|d9t d:t t d;t d<t d;|D]\} }t d+| d=d |qdS)>zEPrints out a nicely formatted table for multi-system evaluation mode.jsonr$system)indentNtext fancy_gridlatexlatex_booktabs)r#Frz Baseline:Tyellowcyankeys)rightcenterr,r-)headerstablefmtcolalignstralignnumalignfloatfmtzbootstrap resamplingzapproximate randomizationzresampling trialstrialszPaired z test with r&baselinezNull hypothesisgreenzhighlighted in redr)-r%z' - Each system is pairwise compared to z] Actual system score / bootstrap estimated true mean / 95% CI are provided for each metric.z3 Actual system score is provided for each metric.z - z: the system and the z translations are essentiallyzH generated by the same underlying process. For a given system and the ,zR the p-value is roughly the probability of the absolute score difference (delta)zD or higher occurring due to chance, under the assumption that the z is correct.z2 - Assuming a significance threshold of 0.05, the z can be rejectedz` for p-values < 0.05 (marked with "*"). This means that the delta is unlikely to be attributedzE to chance, hence the system is significantly "different" than the z Otherwise, the p-values are zJ - NOTE: Significance does not tell whether a system is "better" than the z but ratherzY emphasizes the "difference" of the systems in terms of the replicability of the delta.z-----------------zMetric signaturesz<10)rlistrOrangelenr1r__dict__r6printrCdumpsr?r" startswithr r paired_bs paired_ar paired_bs_n paired_ar_nlowerr0)r@rArB proper_json dict_keysivaluejrSnew_dict has_baseline baseline_namenamevaltable is_paired test_typen_samples_or_trialstest_sample_typer bline bline_namenull_hyp pval_colorsigrrrprint_results_table\s           ( r}c CsB|jdkr"t|dkrdd|d}t|dSt|ddSd|dvrBtd }tt|D]}|d d ||||<q3t|dkrPt|ddSg}|D]}z || d dWqTt yot|YqTwt|dkrt |}t ||D]\}}|d|||d} } t| d || qdSdS)z/Re-process metric strings to align them nicely.rCrEz[ z, z ]rNuμu(\(μ = [0-9\.]+ ± [0-9\.]+\))cSst|dS)NrN)r rgroup)mrrrsz&print_single_results..=>) rr_joinrarecompiler^subr6index ValueErrormaxzip) r@rBricolor_reidxlenslinew_lenleftrPrrrprint_single_resultss<           rrDrefstest_setcstt|tfdd|Dr8td|r1tdtdtdtd|dtd td dSdS) Nc3s|] }t|kVqdSNr_).0 ref_streamn_hypsrr sz'sanity_check_lengths..z4System and reference streams have different lengths.zeThis could be an issue with your system output or with sacreBLEU's reference database if -t is given.z6For the latter, try cleaning out the cache by typing: z rm -r /r%zDThe test sets will be re-downloaded the next time you run sacreBLEU.rE)r_any sacreloggererror SACREBLEU_DIRsysexit)rDrrrrrsanity_check_lengthss    rrtutf-8cCs,|drtj|||ddSt|||ddS)zConvenience function for reading compressed or plain text files. :param file: The file to read. :param mode: The file mode (read, write). :param encoding: The file encoding. z.gzr%)modeencodingnewline)endswithgzipopen)filerrrrr smart_opens rnumcCs|dkrdSt|S)zp Floors the log function :param num: the number :return: log(num) floored to a very low number lc( )mathlog)rrrrmy_logs rcCsnt|dkr |dSt|d}t|ddd}|g|}|D]}t|D] }||||7<q'q!|S)z,Aggregates list of numeric lists by summing.rErr)r_typer^)listssizeinit_valtotalllrkrrr sum_of_lists s    rFprefix strip_prefixcCsJ|d7}i}|jD]\}}||r"|r||dn|}|||<q |S)zbFilters argparse's `Namespace` into dictionary with arguments beginning with the given prefix._rL)r`r0rcreplace)rBrrdkvrrr args_to_dicts rc s|tvr td|t||}t||}d|vr.t|dkr.tdtdnd|vr4|}d|vrNd|vrNt dd|Dfdd|D}g}|D]M}||vrtd |d |d |d td |d|dd |d|vrt dd|D} td| dtd| |} | || qRdd|D} t | ||||} t| D]} td tdd| qdS)aPrints to STDOUT the specified side of the specified test set. :param test_set: the test set to print :param langpair: the language pair :param requested_fields: the fields to print :param origlang: print only sentences with a given original language (2-char ISO639-1 code), "non-" prefix means negation :param subset: print only sentences whose document annotation matches a given regex No such test set allrEz'Cannot use --echo all with other fieldsrefcSg|] }|dr|qSrrcrr-rrr >z"print_test_set..csg|] }|dkr |nqSrrrreplacement_refrrr?szNo such field z in test set z for language pair r,zavailable fields for rz: z, cSrrrrrrrrGrz<'ref' also allowed for backwards compatibility (will return r(cSg|]}t|qSr)r)rrrrrrM cSs|Sr)rstrip)xrrrrPsz print_test_set..N)DATASETS Exception fieldnames get_filesr_rrrrminrrr6 filter_subsetrramap) rlangpairrequested_fieldsoriglangsubsetr all_filesfilesfieldsubrefrstreamslinesrrrprint_test_set's8   "   rrcC$|tvr td|t||S)a Returns the source file for a given testset/langpair. Downloads it first if it is not already local. :param test_set: The test set (e.g., "wmt19") :param langpair: The language pair (e.g., "de-en") :return: the path to the requested source file r)rrget_source_filerrrrrrSs rcCr)a@ Returns a list of one or more reference file paths for the given testset/langpair. Downloads the references first if they are not already local. :param test_set: The test set (e.g., "wmt19") :param langpair: The language pair (e.g., "de-en") :return: a list of one or more reference file paths r)rrget_reference_filesrrrrrbs rcCr)a_ Returns the path of the source file and all reference files for the provided test set / language pair. Downloads the references first if they are not already local. :param test_set: The test set (e.g., "wmt19") :param langpair: The language pair (e.g., "de-en") :return: a list of the source file and all reference files rrrrrrrrrps rcCstd|d||ds|dr7ddl}||}|j|dWddS1s0wYdS|dr_ddl}||d}|j|dWddS1sXwYdSdS) Nz Extracting  to z.tar.gzz.tgzr)pathz.zipr)rinfortarfiler extractallzipfileZipFile)filepathdestdirrtarrrrrextract_tarballs " "rcCsTt}t|d}|D]}||q Wd|S1s!wY|S)Nrb)hashlibmd5rupdate hexdigest) dest_pathrinfilerrrr get_md5sums   rc Csddl}ddl}tj|}tj|dd|d}tj|ddtj|r0tj |dkrt d|d |z4|j |$}t|d } | |Wdn1sZwYWdn1siwYWn|jyt d td Ynw|durt|} | |krt d | d|dt d|dt dtd |durt||WddSWddSWddS1swYdS)aoDownloading utility. Downloads the specified test to the system location specified by the SACREBLEU environment variable. :param source_path: the remote uri to download :param dest_path: where to save the file :param extract_to: for tarballs, where to extract to :param expected_md5: the MD5 sum :return: the set of processed file names rNT)exist_okz.lock<)timeoutz Downloading rwbzAn SSL error was encountered in downloading the files. If you're on a Mac, you may need to run the "Install Certificates.command" file located in the "Python 3" folder, often found under /ApplicationsrEz5Fatal: MD5 sum of downloaded file was incorrect (got z , expected z).zPlease manually delete z and rerun the command.znIf the problem persists, the tarball may have changed, in which case, please contact the SacreBLEU maintainer.)urllib.requestsslosrdirnamemakedirs portalockerLockexistsgetsizerrrequesturlopenrwritereadSSLErrorrrrrr) source_pathr extract_to expected_md5urllibroutdirlockfiler-outcur_md5rrr download_filesB       "rcCs,|tvr td|t|}||}|S)aDownloads the specified test to the system location specified by the SACREBLEU environment variable. :param test_set: the test set to download :param langpair: the language pair (needed for some datasets) :return: the set of processed file names rr)rrdataset file_pathsrrrdownload_test_sets  rtestsetcCs|tvrgStt|jS)z5Return a list of language pairs for a given test set.)rr] langpairsrO)rrrrget_langpairs_for_testsetsrcCsttddS)z%Return a list of available test sets.T)reverse)sortedrrOrrrrget_available_testsetssr cCsd|d}|d}|d}g}tD]}|d||jvs)|d||jvr/||jq|S)z>Return a list of available test sets for a given language pairr[rrE)splitrvaluesrr6rq)rpartssrclangtrglangtestsetsrrrr#get_available_testsets_for_langpairs   r'c Cs|durgSt}|dD]]}t|}tjt|d|j|d}ddlm }t ||r<| |dD]}| |q4| drkt|}|D]} | d r[td d | } | | qHWdn1sfwYqtt|S) zDReturn a list of origlang values according to the raw XML/SGM files.Nr\rawrrE WMTXMLDatasetr.sgm|t|dO}q|t vra|tdd t | DO}|td d t | DO}qtt|S) zkReturn a list of domain values according to the raw XML files and domain/country values from the SGM files.Nr\rEr)r(rdomaincss"|] }d|ddVqdS)country:r[rNr!rrrrrrs z(get_available_subsets..css|] }|ddVqdS)r[rENr;r<rrrrs)r/r!rr0r*r1rrrrrr1SUBSETSr"rr])r3rsubsetsrrr*r5fieldsrrrget_available_subsetss"    r@c sb|dur |dur |S|dus|durtd|dur%|dr%|dd}td}td}g|dD]}t|}tjt |d|j |d } d d l m } t || r|| } d | vrb| d ntd} t| d | D]2\} }|duryd}n|dr| |ddk}n| |k}|dur|dust||sd}|qnq6| dr!i}|dur|tvrtd|t|}t| W}d}|D]J}|dr|durd}n|d|} |dr| |ddk}n| |k}|dur|d|}t|||dsd}|dr|qWdn 1swYq6td| fdd|DS)zRFilter sentences with a given origlang (or subset) according to the raw SGM files.NzPFiltering for --origlang or --subset needs a test (-t) and a language pair (-l).r:r-z.* docid="([^"]+)".*\nr\r(rrEr)r9rTznon-rFFr+z,No subset annotation available for test set r,r.rLz..)r)rrindices_to_keeprrrOs z!filter_subset..)rrcrrr!rrrrrrr0r*r1r1 itertoolsrepeatrsearchr6rr=rrrget)systemsr3rrr re_origlangre_idrrr5r*r?domainsr7 doc_domain include_doc doc_to_tagsr6rdoc_idrrDrrsl                  rc Cs|j}|jr |jnt|j|j}t|dkrtddStt}|D]s}dg}|j dur3||j g7}n |t |j|j7}|D]V} t |g||j|j|| ^} } t| dkrVq>d|} | durd| d7} n| dru| d| dd7} n| d| 7} | D]} | | | }|| t| |fqq>q"td d |Dd }td d t| dD}|D]1\} }t| d |d} |D]\}}t| d|dd|jd |d|jd|dqqdS)Nrz@No subset information found. Consider using --origlang argument.z origlang=z domain=ALLr:z country=rAz domain=cSrrr)rrrrrrwrz(print_subset_results..rEcSsg|] }t|djqS)rE)r_rq)rrrrrrrxrrrDrkeyr:r2max_left_widthmax_metric_widthr!n_systemrrrprint_subset_resultsRsL           8r[rE)rr=)r Tr)rr)F)NN)>rFrCrrrrrrloggingr collectionsrtypingrrrrargparserrrr expanduserUSERHOMEenvironrIrr getLoggerrr dictintboolrr?r}rrrr/rrrrrrrrrrrrr r'r8r@rr[rrr=rrrrsp      )c(     , . >-