o Mi.@sddlmZddlZddlZddlmZmZddlZddlZe e Z e dddgZ ddd d Zd d ZGd ddeZGdddeZGdddeZdS))print_functionN)chainproductEvaluationConfig num_samples sample_sizezFilename : {name} Num samples: {samplesize_count} Sample size: {samplesize_avg} F-score : {fscore_avg:.3} Precision : {precision_avg:.3} Recall : {recall_avg:.3}z?{name:10} {precision_avg:6.3} {recall_avg:6.3} {fscore_avg:6.3}zB{name} & {precision_avg:.3} & {recall_avg:.3} & {fscore_avg:.3} \\)defaulttablelatexcCst|||S)zNCreate a specific size sample from the compound list using a specific seed)randomRandomsample) compound_listsizeseedrH/home/ubuntu/.local/lib/python3.10/site-packages/morfessor/evaluation.py_samplesrc@s^eZdZdZddeeeedZdddZ dd Z d d Z d d Z ddZ ddZddZdS)MorfessorEvaluationResultaA MorfessorEvaluationResult is returned by a MorfessorEvaluation object. It's purpose is to store the evaluation data and provide nice formatting options. Each MorfessorEvaluationResult contains the data of 1 evaluation (which can have multiple samples). cCst|t|SN)sumlenxrrr*sz"MorfessorEvaluationResult.)avgminmaxvaluescountNcCs(||_g|_g|_g|_g|_d|_dSr) meta_data precisionrecallfscore samplesize_cache)selfr rrr__init__1s  z"MorfessorEvaluationResult.__init__cCs|jdur ||j|S)zRProvide dict style interface for all values (standard values and metadata)Nr% _fill_cache)r&itemrrr __getitem__;s  z%MorfessorEvaluationResult.__getitem__cCs:|j||j||j||j|d|_dS)z^Method used by MorfessorEvaluation to add the results of a single sample to the objectN)r!appendr"r#r$r%)r&r!r"f_scorerrrradd_data_pointCs     z(MorfessorEvaluationResult.add_data_pointcCs|tdS)z Method for default visualizationr)formatFORMAT_STRINGSr&rrr__str__Nsz!MorfessorEvaluationResult.__str__cs&fdddD_jjdS)zQ Pre calculate all variable / function combinations and put them in cachecs8i|]}jD]\}}d|||t|q qS)z{}_{})print_functionsitemsr/getattr).0val func_namefuncr1rr Us z9MorfessorEvaluationResult._fill_cache..)r!r"r#r$N)r%updater r1rr1rr)Rs z%MorfessorEvaluationResult._fill_cachecCs|jdur ||jS)z, Fill the cache (if necessary) and return itNr(r1rrr _get_cache[s z$MorfessorEvaluationResult._get_cachecCs|jdi|S)z Format this object. The format string can contain all variables, e.g. fscore_avg, precision_values or any item from metadataNr)r/r<)r& format_stringrrrr/asz MorfessorEvaluationResult.formatr)__name__ __module__ __qualname____doc__rrlistrr3r'r+r.r2r)r<r/rrrrr s     rc@sxeZdZdZddZeddfddZeddfdd Zd d Ze d d Z edddfddZ edddfddZ dS)MorfessorEvaluationa Do the evaluation of one model, on one testset. The basic procedure is to create, in a stable manner, a number of samples and evaluate them independently. The stable selection of samples makes it possible to use the resulting values for Pair-wise statistical significance testing. reference_annotations is a standard annotation dictionary: {compound => ([annoation1],.. ) } cs>i_|D]\}}tfdd|Dj|<qi_dS)Nc3s|] }t|VqdSrtuple_segmentation_indices)r6ar1rr ts z/MorfessorEvaluation.__init__..) referencer4rB_samples)r&reference_annotationscompoundanalysesrr1rr'ps   zMorfessorEvaluation.__init__ icsRt|jjjkrtdt|jfddtjD|j <dS)zbCreate, in a stable manner, n testsets of size x as defined in test_configuration z.The test set is too small for this sample sizecsg|] }tj|qSr)rr)r6ir configurationrr sz7MorfessorEvaluation._create_samples..N) rrIrr_loggerwarningsortedkeysrangerJr&rQrrPr_create_samplesys   z#MorfessorEvaluation._create_samplescCs||jvr |||j|S)aGet a list of samples. A sample is a list of compounds. This method is stable, so each time it is called with a specific test_set and configuration it will return the same samples. Also this method caches the samples in the _samples variable. )rJrYrXrrr get_sampless   zMorfessorEvaluation.get_samplesc sddtt|t|j@}d}d}|D]3}t|dkr"q|tfddt|||j|D7}|tfddt|||j|D7}q|t|}|t|}dd |d |}|||t|fS) z9Helper method to get the precision and recall of 1 samplecSs<t|dkrdStt|t|}t||tt|S)Nr?)rsetfloat)refpreddiffrrrcalc_prop_distances z9MorfessorEvaluation._evaluate..calc_prop_distancegc3s|] \}}||VqdSrrr6prrarrrHz0MorfessorEvaluation._evaluate..c3s|] \}}||VqdSrrrcrfrrrHrgg@r[)rUr\rVrIrrr) r& predictionwordlist recall_sum precis_sumwordr!r"r-rrfr _evaluates*    zMorfessorEvaluation._evaluateccs.d}|ddD] }|t|7}|Vq dS)z>Method to transform a annotation into a tuple of split indicesrNr) annotationcur_lenrGrrrrFs  z)MorfessorEvaluation._segmentation_indicesNc Cs||durddi}t|}t||D](\}}td|i}|D]}t|||dg||<q!|j| |q|S)zGet the prediction of the test samples from the model and do the evaluation The meta_data object has preferably at least the key 'name'. NnameUNKNOWNEvaluating sample %sr) r enumeraterZrSdebugrErFviterbi_segmentr.rm) r&modelrQr merrOr rhrLrrrevaluate_models  z"MorfessorEvaluation.evaluate_modelcsddfdd|D}|durddi}t|}t|D]\}td|fd d|D}|j|q!|S) z.Method for evaluating an existing segmentationcSs,|d}tdt|D]}|||}q |S)Nr)rWr) constructionsrLrOrrrmerge_constructionsszFMorfessorEvaluation.evaluate_segmentation..merge_constructionscs*i|]}|dt|dgqS)r{rDr6r)r}r&rrr:s  z=MorfessorEvaluation.evaluate_segmentation..Nrrrsrtcsi|] \}}|vr||qSrr)r6kv)r rrr:s)rrurZrSrvr4r.rm)r& segmentationrQr ryrOrhr)r}r r&revaluate_segmentations  z)MorfessorEvaluation.evaluate_segmentation) r>r?r@rAr'rrYrZrm staticmethodrFrzrrrrrrCgs    rCc@sPeZdZdZedddZeddZedd Z dd d ZeddZ dS)WilcoxonSignedRankzClass for doing statistical signficance testing with the Wilcoxon Signed-Rank test It implements the Pratt method for handling zero-differences and applies a 0.5 continuity correction for the z-statistic. prattTc s(|dvrt|dkrttddt}tddD}tddt|D}td dt|D}t||}||d d }||d d |d } fd dt |D} | dtddt |  D8} t | d} |r||kr~dnd}nd}|||| } d tt| S)N)wilcoxrrcSs|dkSNrr)rGrrrrz.WilcoxonSignedRank._wilcoxon..cSsg|]}t|qSr)absr6rrrrrRz0WilcoxonSignedRank._wilcoxon..css |] \}}|dkr|VqdSrNrr6rerrrrrHz/WilcoxonSignedRank._wilcoxon..css |] \}}|dkr|VqdSrrrrrrrHrr{g?rbcs g|] \}}|dkr|qS)rr)r6rOredrrrR s ?css |] }|||dVqdS)r{Nrr~rrrrH rg8@gr) ValueErrorrBfilterrr _rankdatarziprru collectionsCounterrmathsqrt _norm_cum_pdfr) rmethod correctionrranks rank_sum_pos rank_sum_negtestmeanstdev no_zero_rankszrrr _wilcoxons*    zWilcoxonSignedRank._wilcoxoncszt}|D] }||d7<qid}t|dddD]\}}|||dd|<||7}qfdd|DS)Nr{cSs|dSrrrrrrr!rz.WilcoxonSignedRank._rankdata..)keyrbcsg|]}|qSrrr rank_dictrrrR%rz0WilcoxonSignedRank._rankdata..)rrrUr4)rodrcur_rankr7rrrrrs zWilcoxonSignedRank._rankdatacCsddt|tdS)z@Pure python implementation of the normal cumulative pdf functionrrb)rerfr)rrrrr'sz WilcoxonSignedRank._norm_cum_pdf fscore_valuesrrcsfdd|D}tdd|DrtdiSi}t||D]\}}|ddt||||D|||f<q'|S)zTakes a set of evaluations (which should have the same test-configuration) and calculates the p-value for the Wilcoxon signed rank test Returns a dictionary with (name1,name2) keys and p-values as values. csi|] }||qSrrr6re name_property val_propertyrrr:4sz8WilcoxonSignedRank.significance_test..css|] }t|dkVqdS)rNNror~rrrrH5sz7WilcoxonSignedRank.significance_test..z1Too small number of samples for the Wilcoxon testcSsg|]\}}||qSrr)r6v1v2rrrrR:s z8WilcoxonSignedRank.significance_test..)anyrrSerrorrrVrr)r& evaluationsrrresultsrdr1r2rrrsignificance_test,s  z$WilcoxonSignedRank.significance_testcCsttdd|D}ttdd|Dd}tdg|D] }tdj||ddd qt|D]$}tdj||ddd |D]}td j|||f|ddd qAtq2d S) z>Nicely format a results table as returned by significance_testcss|]}|dVqdSrrrrrrrHCz1WilcoxonSignedRank.print_table..css|]}t|VqdSrro)r6nrrrrHErz {:{width}})width|)endz {:{width}.5}N)rUr\rVrrprintr/)rnames col_widthhrrname2rrr print_table@szWilcoxonSignedRank.print_tableN)rT)rrr) r>r?r@rArrrrrrrrrrrs    r) __future__rrlogging itertoolsrrrr getLoggerr>rS namedtuplerr0robjectrrCrrrrrs&   G