o MiVz@s&ddlZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z ddl m Z mZmZmZmZddlmZddlmZddlmZmZmZmZejdd kZer^eZnd d Zee Z!d Z"d dZ#ddZ$e j%e"dddZ&e j%e"dddZ'ddZ(ddZ)ddZ*dS)N) get_version)utils) BaselineModelAnnotationCorpusWeightMorphLengthCorpusWeightNumMorphCorpusWeightFixedCorpusWeight)ArgumentException) MorfessorIO)MorfessorEvaluationEvaluationConfigWilcoxonSignedRankFORMAT_STRINGScCst|tdS)Nencoding)unicodelocalegetpreferredencodingxrA/home/ubuntu/.local/lib/python3.10/site-packages/morfessor/cmd.pysri@Bc CsNddl}|jddtd|jdd}|dj}|dd d dd d d |ddddd dd |ddddgd dd|ddddgd dd|dj}|dddd d d!d |d"d#d$dd d%d |d&d'd(dd d)d |d*d+dd d,d |d-d.d/dd d0d |d1d2d3td4d5d6|d7j}|d8d9d:d;dd?dd@dAdB|dCdDdd@dEdB|dFdGtddHdIdJ|dKdLtdMdHdNdJ|dOdPtdQdRdSdJ|dTdUtdVdWdXdJ|dYdZtd[dRd\dJ|d]d^dd@d_dB|d`j}|dadbdcdddegdfdgdh|didjdkdldmdldngdodh|dpdqdrtdsdtgdudvdw|dxdydztd gd{d|dJ|d}d~dt ddddJ|ddddddd |ddddt ddd6|dddd@ddB|ddtd3d4ddJ|ddtdd4ddJ|ddtddHddJ|ddtdd4ddJ|dddt ddd6|dddtd4dd6| j}|ddddd dd |dddt ddd6|dddt ddd6|dj}|dddt ddddJ|ddddt dd|ddd@ddd|ddddd dd |dddt ddddJ|dġj}|ddddd dd |dɡj}|dddtd3d4ddJ|ddd dd=|dddd@ddB|dԡj}|dddddٍ|dddtddލ|S)Nrz morfessor.pyuq Morfessor %s Copyright (c) 2012-2019, Sami Virpioja, Peter Smit, and Stig-Arne Grönroos. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Command-line arguments: z Simple usage examples (training and testing): %(prog)s -t training_corpus.txt -s model.pickled %(prog)s -l model.pickled -T test_corpus.txt -o test_corpus.segmented Interactive use (read corpus from user): %(prog)s -m online -v 2 -t - F)prog descriptionepilogformatter_classadd_helpzinput data filesz-lz--loadloadfilez4load existing model from file (pickled model object))destdefaultmetavarhelpz-Lz--load-segmentation loadsegfilezAload existing model from segmentation file (Morfessor 1.0 format)-tz --traindata trainfilesappendzinput corpus file(s) for training (text or bz2/gzipped text; use '-' for standard input; add several times in order to append multiple files))r"actionr#r$r%z-Tz --testdata testfileszinput corpus file(s) to analyze (text or bz2/gzipped text; use '-' for standard input; add several times in order to append multiple files)zoutput data filesz-oz--outputoutfile-zWoutput file for test data results (for standard output, use '-'; default '%(default)s')z-sz--savesavefilez/save final model to file (pickled model object)z-Sz--save-segmentation savesegfilez7save model segmentations to file (Morfessor 1.0 format)z--save-reduced savereducedzsave final model to file in reduced form (pickled model object). A model in reduced form can only be used for segmentation of new words.z-xz --lexiconlexfilez"output final lexicon to given filez--nbestnbestrzoutput n-best viterbi results)r"r#typer$r%zdata format options-e --encodingr bencoding of input and output files (if none is given, both the local encoding and UTF-8 are tried)r"r$r%z --lowercase lowercase store_truezlowercase input datar"r#r*r%z--traindata-listlistz`input file(s) for batch training are lists (one compound per line, optionally count as a prefix)z--atom-separator separatorz+atom separator regexp (default %(default)s)r"r4r#r$r%z--compound-separator cseparatorz\s+z1compound separator regexp (default '%(default)s')z--analysis-separatoranalysisseparator,zziseparator for different analyses in an annotation file. Use NONE for only allowing one analysis per linez--output-format outputformatz {analysis}\naiformat string for --output file (default: '%(default)s'). Valid keywords are: {analysis} = constructions of the compound, {compound} = compound string, {count} = count of the compound (currently always 1), {logprob} = log-probability of the analysis, and {clogprob} = log-probability of the compound. Valid escape sequences are '\n' (newline) and '\t' (tabular)z--output-format-separatoroutputformatseparator zMconstruction separator for analysis in --output file (default: '%(default)s')z--output-newlinesoutputnewlineszRfor each newline in input, print newline in --output file (default: '%(default)s')z!training and segmentation optionsz-mz--mode trainmode init+batchz)nonebatchinitrJonline online+batchzitraining mode ('none', 'init', 'batch', 'init+batch', 'online', or 'online+batch'; default '%(default)s'))r"r#r$choicesr%z-az --algorithm algorithm recursivez viterbiz>algorithm type ('recursive', 'viterbi'; default '%(default)s')z-dz --dampening dampeningonesz)rKlogrUzWfrequency dampening for training data ('none', 'log', or 'ones'; default '%(default)s'))r"r4r#r$rPr%z-fz --forcesplit forcesplitzztforce split on given atoms (default '-'). The list argument is a string of characthers, use '' for no forced splits.z-Fz--finish-thresholdfinish_thresholdg{Gzt?zzStopping threshold. Training stops when the improvement of the last iteration issmaller then finish_threshold * #boundaries; (default '%(default)s')z-rz --randseedrandseedzz seed for random number generatorz-Rz --randsplit splitprobzainitialize new words by random splitting using the given split probability (default no splitting)z--skipsskipszCuse random skips for frequently seen compounds to speed up trainingz--batch-minfreq freqthresholdzEcompound frequency threshold for batch training (default %(default)s)z --max-epochs maxepochsz"hard maximum of epochs in trainingz --nosplit-renosplitzfif the expression matches the two surrounding characters, do not allow splitting (default %(default)s)z--online-epochint epochinterval'z8epoch interval for online training (default %(default)s)z--viterbi-smoothing viterbismoothzXadditive smoothing parameter for Viterbi training and segmentation (default %(default)s)z--viterbi-maxlen viterbimaxlenzVmaximum construction length in Viterbi training and segmentation (default %(default)s)z-Dz --develset develfilez:load annotated data for tuning the corpus weight parameterz--morph-length morphlengthz@tune the corpusweight to obtain the desired average morph lengthz--num-morph-types morphtypeszAtune the corpusweight to obtain the desired number of morph typesz semi-supervised training optionsz-wz--corpusweight corpusweightg?zfcorpus weight parameter (default %(default)s); sets the initial value if other tuning options are usedz--weight-threshold thresholdg{Gz?z7percentual stopping threshold for corpusweight updaters)r"r#r$r4r%z--full-retrain fullretrainz2do a full retrain after any weights have converged)r"r*r#r%z-Az --annotationsannofilez0load annotated data for semi-supervised learningz-Wz--annotationweightannotationweightzcorpus weight parameter for annotated data (if unset, the weight is set to balance the number of tokens in annotated and unannotated data sets)zEvaluation optionsz-Gz--goldstandard goldstandardz9If provided, evaluate the model against the gold standardlogging options-v --verboseverbosefverbose level; controls what is written to the standard error stream or log file (default %(default)s) --logfilelog_file?write log messages to file in addition to standard error streamz --progressbarprogresszcForce the progressbar to be displayed (possibly lowers the log level for the standard error stream) other options-h--helpr%show this help message and exitr*r% --versionversion %(prog)s show version number and exitr*r|r%) argparseArgumentParserrRawDescriptionHelpFormatteradd_argument_group add_argumentint_strr=floatadd_mutually_exclusive_group)rparseradd_argrrrget_default_argparser!s !  2                                   rc Cs|jdkr tj}n |jdkrtj}ntj}t}|tjd}d}d}|}|jdus4t|dr:|j r:t |tj}t }||| t ||||jdurot|jd}||| t |||||S) z-Initialize loggers based on command line argsrz%%(asctime)s %(levelname)s:%(message)sz%Y-%m-%d %H:%M:%Sz %(message)sNruw)rploggingDEBUGINFOWARNING getLoggersetLevelrshasattrrumax StreamHandler setFormatter Formatter addHandler FileHandler) argsloglevel rootloggerlogfile_format date_formatconsole_format console_levelchfhrrrinitialize_logging$s.         r)maxsizecCs||||SN)viterbi_segment)modelatomssmoothmaxlenrrr_viterbi_segmentJsrcCs|||||Sr) viterbi_nbest)rrr2rrrrr_viterbi_nbestOsrc Cst|}|tjksttjdrtjsdt_|j rdt_|j dur2|j dur2t |j dkr2td|jdur=t|jt|j|j|j|jd}|j durU||j }n t|j|j|j|jd}|j duro|||j |jdkrw|jnd}|j dur|j!|j |d }|"||j#|j$dur|j!|j$|d }t%||j&}|'||j(durt)|j(|j&}|'||j*durt+|j*|j&}|'||,}|j-d krd} n|j-d krd d } n|j-dkrdd } ntd|j-|j.dkr|j/|j0f} nd} |j1d krn{|j1dkrRt |2dkrt34dnet |j dkr't34dt55} |6|j.| |j7|j8\} } t55}t39d| t39d| t39d|| n-t |j dkrzt55} |j1dkr|j:ro|;|j }n|<|j }|=||j>| |j?} n|j1dkr|j:r|;|j }n|<|j }|=||j>| |j?} |6|j.| |j7|j8\} } t39d| |j@rtA|,|dkr|'tB|,|C|6|j.| |j7|j8\} } t39d| n|j1dkr|<|j }|D|| |jE|j.| |j?|j8\} } t39d| n_|j1dkra|<|j }|D|| |jE|j.| |j?|j8\} } |6|j.| |j7|j8| \} } t39d| |j@r`tA|,|dkr`|C|6|j.| |j7|j8\} } t39d| ntd|j1t55}t39d| t39d|| nt34d |jFdur|G|jF||jHdur|I|jH|J|jKdur|L|jK|M|jNdur|O|G|jN|t |jPdkrt39d!|jQ}|jR}|Sd"d#}|Sd$d%}d&d'tTUV|D}|W|jX}|<|jP}d}|D]\}}|jYdurd(Z|}n|jYZ|}t |dkr|j[r|\d#qd)|vr*|]|}nd}|j^d*krZt_|||j^|j/|j0}|D]\}}|j`||d+}|\|ja|||||d,q?ntb|||j/|j0\}}|j`||d+}|\|ja|||||d,|d*7}|d-dkrtj\d.qtj\d#Wdn 1swYt39d/|jcdurt39d0td|!|jc} | je|d1d2id3}tf|atgd4t39d5dSdS)6NisattyFTrz4either model file or training data should be defined)rcompound_separatoratom_separatorr:)forcesplit_listrg use_skips nosplit_reNONE) analysis_seprKrVcSsttt|ddS)Nrr)rroundmathrVrrrrrszmain..rUcSsdS)Nrrrrrrrszunknown dampening type '%s'rSrrLzVModel contains no compounds for batch training. Use 'init+batch' mode to add new data.z`Training mode 'batch' ignores new data files. Use 'init+batch' or 'online' to add new compounds.z Epochs: %szFinal cost: %szTraining time: %.3fsrMrJg?zRetrain Epochs: %srNrOzunknown training mode '%s'z!No training data files specified.zSegmenting test data...z\n z\t cSsg|]}|dqS)rr).0rrrr szmain..clogprobr)csep)analysiscompoundcountlogprobrr`.zDone.zEvaluating ModelnameMODEL) meta_datar#Done)hrrrrsysstderrrrshow_progress_barrur r&lenr(r rYrandomseedr rrAr>r:read_binary_model_filerrWrgr[r^load_segmentationsread_segmentation_filerBrjread_annotations_fileset_annotationsrkrdrrhset_corpus_weight_updaterrerrfrget_corpus_coding_weightrTrQrarbrI get_compounds_loggerwarningtime train_batchrXr]infor=read_corpus_list_filesread_corpus_files load_datar\rZriabsr clear_segmentation train_onliner_r.write_binary_model_filer/write_segmentation_fileget_segmentationsr1write_lexicon_fileget_constructionsr0make_segment_onlyr+rDrFreplacestringrparse_open_text_file_writer,rjoinrHwriteforward_logprobr2rformat_constructionsformatrrlr evaluate_modelprintr)rriorr annotations develannotsupdaterstart_corpus_weightdampfunc algparamstsectedata outformatrkeywordsfobjtestdatairrrr nbestlist constructionslogprresultrrrmainTs                                                                              (  rc Cs\ddl}t}|jdd|j|jdd}|dj}|ddtd d d d |d dtd ddd |dj}|ddddd|dddddd|dj}|ddtd d!d"d#|d$d%d&d'd(d|d)j}|d*d+d,td-d d.d#|d/d0d1d2d|d3j}|d4d5d6d7d8|d9d:d;t dd?d-d@dA|dBdCdDdEdA|dFdGdHgdIdJdK|S)LNrzmorfessor-evaluatez>Simple usage example: %(prog)s gold_standard model1 model2 F)rrrrrzevaluation optionsz --num-samples numsamplesr3 z%number of samples to take for testing)r"r4r$r#r%z --sample-size samplesizeizsize of each testing sampleszformatting optionsz--format-string formatstringrEaPython new style format string used to report evaluation results. The following variables are a value and and action separated with and underscore. E.g. fscore_avg for the average f-score. The available values are "precision", "recall", "fscore", "samplesize" and the available actions: "avg", "max", "min", "values", "count". A last meta-data variable (without action) is "name", the filename of the model See also the format-template option for predefined stringsr9z--format-templatetemplatez