o 5tiG@sddlZddlZddlZddlmZmZmZmZm Z m Z m Z m Z ddl ZddlmZmZmZejdkZedZGdddZd ejd eeeffd d Z d%de e e eefdeded eee effddZdejded efddZ   d&deeeejeffdedeedeeeedeeefdededeed eeeeefffdd Z  d'deeeejeffdedeedeeeedeeefdededeed eeeeefffd!d"Z Gd#d$d$Z!dS)(N)SequenceDictOptionalTupleListUnionAnyMapping)MetricScore Signaturent sacrebleuc @sDeZdZdZ  d dedeedeedeefddZd d ZdS) Resulta=A container to represent results from a particular statistical significance test. :param score: The floating point score for the system at hand. :param p_value: If exists, represents the p-value when the system at hand is compared to a baseline using a paired test. :param mean: When paired bootstrap test is applied, this represents the true mean score estimated from bootstrap resamples of the system. :param ci: When paired bootstrap test is applied, this represents the 95% confidence interval around the true mean score `sys_mean`. Nscorep_valuemeancicCs||_||_||_||_dS)N)rrrr)selfrrrrrJ/home/ubuntu/.local/lib/python3.10/site-packages/sacrebleu/significance.py__init__s zResult.__init__cCsddd|jDS)N,cSs"g|] \}}|dt|qS)=)str).0kvrrr #s"z#Result.__repr__..)join__dict__itemsrrrr__repr__"szResult.__repr__)NNN)__name__ __module__ __qualname____doc__floatrrr$rrrrrs  rscoresreturncCsPt|}t|}|d}||d}||||}}d||}||fS)zTakes a list of scores and returns mean and 95% confidence interval around the mean. :param scores: A list of floating point scores. :return: A tuple of mean and the 95% CI. (r g?)npsortlenr)r*n lower_idx upper_idxlowerupperrrrr estimate_ci&s    r5statsmetric n_samplesc stjdd}|dkrdnt|}tj|}|jt ||t |fdd}tj |dd}fd d ||D}t ||fS) aPerforms bootstrap resampling for a single system to estimate a confidence interval around the true mean. :param stats: A list of statistics extracted from the system's hypotheses. :param metric: The `Metric` instance to be used for score computation. :n_samples: Number of bootstrap resamples to use. :return: A tuple of the seed choice as string and the list of `Score` instances for all bootstrap resamples. SACREBLEU_SEED12345noneNTsizereplacefloat32dtypecsg|] }|dqSr)_compute_score_from_statssumr_sr8rrrSsz'_bootstrap_resample..) osenvirongetr3intr-random default_rngchoicer/arrayr) r7r8r9seed_seedrngidxsstats_npr*rrHr_bootstrap_resample9s  rVreal_differencecCs*t||k}|dt|d}|S)zComputes the p-value given the sample statistics and the real statistic. :param stats: A numpy array with the sample statistics. :real_difference: The real statistic. :return: The p-value. r )r-rEitemr/)r7rWcprrr_compute_p_valueYs r[' baseline_infosys_name hypotheses referencesmetricsn_ar_confidencerQcstj|}|jd|t|ftd} | } |dkr'|jt||t|fdd} i} |D]\} || \}}t d| d|d ||} |}t |j |j }t d |d | || |}| || |}tfd d |d d d fD}tfdd |d d d fD}tt t|t||}t|j |}|dkrt d|d tj|dd}tfdd || D}t|\|_|_|| | <q-|| fS)a>Paired two-sided approximate randomization (AR) test for MT evaluation. :param baseline_info: A dictionary with `Metric` instances as the keys, that contains sufficient statistics and a `Result` instance for the baseline system. :param sys_name: The name of the system to be evaluated. :param hypotheses: A sequence of string hypotheses for the system. :param references: A sequence of reference documents with document being defined as a sequence of reference strings. If `None`, references will be used through each metric's internal cache. :param metrics: A dictionary of `Metric` instances that will be computed for each system. :param n_samples: The number of AR trials. :param n_ar_confidence: The number of bootstrap resamples to use for confidence estimation. A value of -1 disables confidence estimation. :param seed: The seed value for the RNG. If `None`, the RNG will not be fixed to a particular seed. :return: A tuple with first element being the system name and the second being a `Result` namedtuple. )r>rBrTr= Computing  for % and extracting sufficient statisticsz8 > Performing approximate randomization test (# trials: )cg|]}|jqSr_aggregate_and_computerrxrHrrrz#_paired_ar_test..NcrirrjrlrHrrrrnzI > Performing bootstrap resampling for confidence interval (# resamples: r@rAcg|] }|djqSrCrDrErrFrHrrrs)r-rMrNintegersr/boolrOr" sacreloggerinfo_extract_corpus_statisticsrkabsrrPr[rr5rr)r^r_r`rarbr9rcrQrSpos_selneg_selbs_idxsresultsnamebl_stats bl_result sys_stats sys_scorediffshuf_ashuf_bscores_ascores_brZres sys_scoresrrHr_paired_ar_testpsH      rcs:tj|}i} |jt||t|fdd} |D]~\} || \} } td| d|d||} |}tj |dd}tj | dd} t | j |j }td|d t fd d | | D}t fd d || D}t |\}}t ||}||}t||}t|j |||| | <q|| fS) aPaired bootstrap resampling test for MT evaluation. This function replicates the behavior of the Moses script called `bootstrap-hypothesis-difference-significance.pl`. :param baseline_info: A dictionary with `Metric` instances as the keys, that contains sufficient statistics and a `Result` instance for the baseline system. :param sys_name: The name of the system to be evaluated. :param hypotheses: A sequence of string hypotheses for the system. :param references: A sequence of reference documents with document being defined as a sequence of reference strings. If `None`, references will be used through each metric's internal cache. :param metrics: A dictionary of `Metric` instances that will be computed for each system. :param n_samples: The number of bootstrap resamples. :param n_ar_confidence: This parameter is not used for this function but is there for signature compatibility in the API. :param seed: The seed value for the RNG. If `None`, the RNG will not be fixed to a particular seed. :return: A tuple with first element being the system name and the second being a `Result` namedtuple. Tr=rerfrgr@rAz= > Performing paired bootstrap resampling test (# resamples: rhcrorCrprFrHrrrz#_paired_bs_test..crorCrprFrHrrrr)r-rMrNrOr/r"rsrtrurkrPrvrr5rr[r)r^r_r`rarbr9rcrQrSrzrTr{r|r}r~rr scores_bl scores_syssys_meansys_ci sample_diffsr7rZrrHr_paired_bs_tests4       rc@seZdZdZdddZ    dd eeeeefd e ee fd e eeed ed e de de fddZ deeeefeeeeeeffffddZdS) PairedTesta This is the manager class that will call the actual standalone implementation for approximate randomization or paired bootstrap resampling, based on the `test_type` argument. :param named_systems: A lisf of (system_name, system_hypotheses) tuples on which the test will be applied. :param metrics: A dictionary of `Metric` instances that will be computed for each system. :param references: A sequence of reference documents with document being defined as a sequence of reference strings. If `None`, already cached references will be used through each metric's internal cache. :param test_type: `ar` for approximate randomization, `bs` for paired bootstrap. :param n_samples: The number of AR trials (for `ar`) or bootstrap resamples (for `bs`). The defaults (10000 or 1000 respectively) will be used if 0 is passed. :param n_ar_confidence: If `approximate randomization` is selected, the number of bootstrap resamples to use for confidence estimation. A value of -1 disables confidence estimation. 0 will use the default of 1000. :param n_jobs: If 0, a worker process will be spawned for each system variant. If > 0, the number of workers will be set accordingly. The default of 1 does not use multi-processing. r\r6arbsrrr]r named_systemsrbra test_typer9rcn_jobscCs8|dvs Jd|||_|jdkrt|_n|jdkrt|_tjdd}|dkr.dnt||_ ||_ ||_ ||_ |dkrB|n|j d|_|dkrN|n|j |j|_t|d |_trgtd d |_ n|j dkrtd } | dkrzd |_ nt| |j|_ i|_i|_i|_|j d\} } |D]\} } td | d | | | |j }| |}|jdkr|jn|j}d\}}|dkrt|| |\}}t t!"dd|D\}}t#|j$||d}||f|j|j%<| |j|j%<| &}|'dt(|j |'|j|j|jdkr|'d|j||j|j%<qdS)NrzUnknown test type rrr:r;r<rr z,Parallel tests are not supported on Windows.rdzPre-computing z statistics for )NNcSsg|]}|jqSr)rrlrrrrvsz'PairedTest.__init__..)rrrQ))rr_fnrrIrJrKr3rLrRrrar_DEFAULT_SAMPLESrcr9r/ n_systems IS_WINDOWSrswarningmp cpu_countmin _signatures_baseline_inforbr"rtrurkrVr5r-rPrrr{ get_signatureupdater)rrrbrarr9rcrrQ n_max_jobsbl_namebl_hypsr{r8r|bl_score confidence_nbl_meanbl_ci_ bl_scoresresultsigrrrr/sf           zPairedTest.__init__r+c sPg}i}ddjD|d<jD] \}\}}|g||<qtjddD] \}\}}jr4jnd} |j||jjjj | fq(j dkrXfdd|D} n4t dj dt d j fd d|D} d d| D} Wdn1swY| D]\} } | D] \}}|||qqj|fS) zCRuns the paired test either on single or multiple worker processes.cSsg|]}|dqSrCr)rnsrrrrz'PairedTest.__call__..Systemr Ncsg|]}j|qSr)rrargsr#rrrsz Launching z parallel workers.forkcsg|] }j|qSr) apply_asyncrrpoolrrrrscSsg|]}|qSr)rK)rjrrrrr)rrr" enumeraterRappendrarbr9rcrrsrtr get_contextPoolr)rtasksr*r8rridxr{hypsrQrzjobsr_ sys_results_resultrrr__call__s0     zPairedTest.__call__N)rrr]r )r%r&r'r(rrrrrr r rrLrrr rrrrrrrrs.  6Yr)r6)r\r]N)r6r]N)"rIloggingmultiprocessingrtypingrrrrrrrr numpyr- metrics.baser r r r{r getLoggerrsrndarrayr)r5rLrrVr[rrrrrrrsx(      ]  L