o Mi@sddlZddlZddlZddlZddlZddlZddlZddlmZm Z ddl m Z m Z e eZddZedgdZGd d d eZGd d d eZGd ddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZdS)N) _progress _is_string)MorfessorExceptionSegmentOnlyModelExceptioncCs*t|dr d|Sdtdd|S)z5Return a readable string for a list of constructions.rz + cSs d|S)N )join)xr F/home/ubuntu/.local/lib/python3.10/site-packages/morfessor/baseline.pys z'_constructions_to_str..)rrmap) constructionsr r r _constructions_to_strs  r ConstrNode)rcountcountsplitlocc@s|eZdZdZdZ  dZddZddZd d Zed d Z ed dZ ddZ ddZ ddZ d[ddZddZddZddZddZd\d"d#Zd$d%Zd&d'Zd(d)Zd*d+Zed,d-Zed.d/Zed0d1Zd2d3Zd4d5Zd6d7Zd8d9Z  d]d;d<Z d=d>Z!d^d?d@Z"dAdBZ# D d_dFdGZ$ H D d`dIdJZ%dadLdMZ&dNdOZ'dadPdQZ(dRdSZ)dTdUZ*dVdWZ+dXdYZ,dS)b BaselineModelaMorfessor Baseline model class. Implements training of and segmenting with a Morfessor model. The model is complete agnostic to whether it is used with lists of strings (finding phrases in sentences) or strings of characters (finding morphs in words). 3333NFcCsi|_d|_t|_t|j|_d|_||||_d|_ t |_ |dur+g|_ n||_ |dur6d|_nt|tj|_d|_dS)aInitialize a new model instance. Arguments: forcesplit_list: force segmentations on the characters in the given list corpusweight: weight for the corpus cost use_skips: randomly skip frequently occurring constructions to speed up training nosplit_re: regular expression string for preventing splitting in certain contexts FN) _analyses _segment_onlyLexiconEncoding_lexicon_codingCorpusEncoding_corpus_coding _annot_codingset_corpus_weight_updater _use_skips _supervised collectionsCounter_counterforcesplit_list nosplit_rerecompileUNICODE annotations)selfr# corpusweight use_skipsr$r r r __init__,s     zBaselineModel.__init__cCsD|dur td|_nt|tjrt||_n||_|j|ddS)N?r)FixedCorpusWeight_corpus_weight_updater isinstancenumbersNumberupdate)r) corpus_weightr r r r]s    z'BaselineModel.set_corpus_weight_updatercCs|jrtdSN)rrr)r r r _check_segment_onlygsz!BaselineModel._check_segment_onlycC|jjS)z)Return the number of construction tokens.)rtokensr6r r r r9kszBaselineModel.tokenscCs |jjdS)z(Return the number of construction types.r)rtypesr6r r r r:ps zBaselineModel.typescCsH|jj|7_||||j|j}|j|j||d|j|<dS)z"Add compound with count c to data.rN)r boundaries_modify_construction_countrr_replace)r)compoundcoldrcr r r _add_compoundus    zBaselineModel._add_compoundcCs&|j|\}}}||| ||fS)zRemove construction from model.)rr=)r) constructionrrrr r r _remove}szBaselineModel._removecs,tfddtdt|D}|||S)zReturn a random split for compound. Arguments: compound: compound to split threshold: probability of splitting at each position c3s |] }tkr|VqdSr5)random).0i thresholdr r s  z.BaselineModel._random_split..r)tuplerangelen_splitloc_to_segmentation)r)r?rIrr rHr _random_splits  zBaselineModel._random_splitrbranchc CsFt|dkr||\}}t|dt|j|<|||dS|dkrE||\}}||}t||||j|<|D]}|||q:dS|dkr|}tt|D]I} ||\}}|| } | t|dkrvt|dd|j|<|||qQ||| dd} t||t| |j|<|| ||| || }qQdSt d|)aSet analysis of compound to according to given segmentation. Arguments: compound: compound to split parts: desired constructions of the compound ptype: type of the parse tree to use If ptype is 'rbranch', the analysis is stored internally as a right-branching tree. If ptype is 'flat', the analysis is stored directly to the compound's node. rrflatrPNzUnknown parse type '%s') rMrDrrKrr=segmentation_to_splitlocrL_join_constructionsr) r)r?partsptyperrrconstrrCpprefixsuffixr r r _set_compound_analysiss<      z$BaselineModel._set_compound_analysisc Cs|jsdSt}|jD]'\}}||jvr||d||\}}|D]}|||j|j7<q&q|j ||D]\}}d}||jvrW|j|j sW|j|j }|j ||q@dS)zUpdate the selection of alternative analyses in annotations. For semi-supervised models, select the most likely alternative analyses included in the annotations of the compounds. Nrr)rr r!r(itemsrrB_best_analysisrrset_constructionsrr set_count) r)rr? alternativesanalysiscostmfrr r r _update_annotation_choicess"    z(BaselineModel._update_annotation_choicescCsd}d}|D]8}d}|D]%}||jvr,|j|js,|t|jjt|j|j7}q ||j8}q |dus:||kr>|}|}q||fS)z2Select the best analysis out of the given choices.N)rrmathlogrr9rpenalty)r)choicesbestcost bestanalysisr`rarbr r r r\s zBaselineModel._best_analysiscCst|jdkr |gSt|}d}g}td|D]+}|||jvrBt|||dkr3||||||||d|d}q||krP|||ddd|DS)z$Return forced split of the compound.rrNcSsg|] }t|dkr|qSr)rM)rFrWr r r sz.BaselineModel._force_split..)rMr#rLappend)r)r?clenjrTrGr r r _force_splitszBaselineModel._force_splitcCsD||jvr|j|}tdtd|krdS|j|d7<dS)z.Return true if construction should be skipped.r-rTF)r"rEmax)r)rCtr r r _test_skips  zBaselineModel._test_skiprcCstt|}|dkr |gS|jr||r||S||}g}|D]}||j|||dd7}q!|j||dd|S)a%Optimize segmentation of the compound using the Viterbi algorithm. Arguments: compound: compound to optimize addcount: constant for additive smoothing of Viterbi probs maxlen: maximum length for a construction Returns list of segments. r)addcountmaxlenrrQ)rU)rMrrtsegmentrqviterbi_segmentrZ)r)r?rvrwrorTrpartr r r _viterbi_optimizes    zBaselineModel._viterbi_optimizecCsxt|dkr |gS|jr||r||S||}t|dkr&||S|||g}|D] }|||7}q0|S)zmOptimize segmentation of the compound using recursive splitting. Returns list of segments. r)rMrrtrxrq_recursive_splitrZ)r)r?rTrrzr r r _recursive_optimizes      z!BaselineModel._recursive_optimizec Cst|dkr |gS|jr||r||S||\}}||||}||| d}tdt|D]F}|jrK|j ||d|drKq7|d|}||d}|||||||} ||| ||| | |kr}| }|}q7|rt ||||j |<|d|}||d}||||||| |} ||kr| | |S| | St |dt |j |<||||gS)znOptimize segmentation of the construction by recursive splitting. Returns list of segments. rrN)rMrrtrxrDr=get_costrLr$matchrrr|rK) r)rCrrmincostrrGrXrYralpr r r r|1sP             zBaselineModel._recursive_splitc Cs||jvr|j|\}}}nd\}}}||}|dkr |j|=n t||||j|<|r>|||}|D]}|||q3dS|j||||jrQ|j||||dkra|dkra|j |dS|dkrq|dkrs|j |dSdSdS)zModify the count of construction by dcount. For virtual constructions, recurses to child nodes in the tree. For real constructions, adds/removes construction to/from the lexicon whenever necessary. )rrrrN) rrrNr=r update_countrrraddremove) r)rCdcountrrrnewcountchildrenchildr r r r=ds,     z(BaselineModel._modify_construction_countcCsFd}|j||r |d7}|jrt|_|jr!||j |S)amDo model updates that are necessary between training epochs. The argument is the number of training epochs finished. In practice, this does two things: - If random skipping is in use, reset construction counters. - If semi-supervised learning is in use and there are alternative analyses in the annotated data, select the annotations that are most likely given the model parameters. If not hand-set, update the weight of the annotated corpus. This method should also be run prior to training (with the epoch number argument as 0). r) r/r3rr r!r"rrdr update_weight)r) epoch_num forced_epochsr r r _epoch_updates  zBaselineModel._epoch_updatecCs8g}d}|D] }|t|7}||qt|ddS)z:Return a list of split locations for a segmented compound.rN)rMrnrK)rrrGr@r r r rRs   z&BaselineModel.segmentation_to_splitloccCsvt|tjr|d|||dgSg}d}d}tt|D]}||}|||||}q|||d|S)zAReturn segmentation corresponding to the list of split locations.Nr)r0r1r2rLrMrn)r?rrTstartposendposrGr r r rNs z'BaselineModel._splitloc_to_segmentationcCs$t|d}|D]}||7}q |S)z`Append the constructions after each other by addition. Works for both lists and strings r)type)rresultr@r r r rSs z!BaselineModel._join_constructionscCs|dd|jDS)z.Return the compound types stored by the model.cSsg|] \}}|jdkr|qSrlr;)rFwnoder r r rms z/BaselineModel.get_compounds..)r7rr[r6r r r get_compoundsszBaselineModel.get_compoundscCstdd|jDS)z.)sortedrr[r6r r r get_constructionsszBaselineModel.get_constructionscCs,|j|j}|jr||jS|S)z#Return current model encoding cost.)rr~rrr)r)rar r r r~szBaselineModel.get_costccsH|t|jD]}|j|j}|dkr!||||fVq dS)z>Retrieve segmentations for all compounds encoded by the model.rN)r7rrkeysrrx)r)rr@r r r get_segmentationss zBaselineModel.get_segmentationsrc Cs|t}|D]\}}t|dkr|||7<q |D]0\}}||kr*q!|dur7||||n||||durQ|dkrQ|||}|||q!|S)aLoad data to initialize the model for batch training. Arguments: data: iterator of (count, compound_atoms) tuples freqthreshold: discard compounds that occur less than given times in the corpus (default 1) count_modifier: function for adjusting the counts of each compound init_rand_split: If given, random split the word with init_rand_split as the probability for each split Adds the compounds in the corpus to the model lexicon. Returns the total cost. rN) r7r r!rMr[rBrOrZr~) r)data freqthresholdcount_modifierinit_rand_split totalcountratomsrTr r r load_datas"     zBaselineModel.load_datacCs4||D]\}}}||||||qdS)zLoad model from existing segmentations. The argument should be an iterator providing a count, a compound, and its segmentation. N)r7rBrZ)r) segmentationsrr? segmentationr r r load_segmentationss  z BaselineModel.load_segmentationscCs6|d|_||_t|j|d|_t|j|j_dS)zVPrepare model for semi-supervised learning with given annotations. TweightN)r7rr(AnnotatedCorpusEncodingrrrMr<)r)r(annotatedcorpusweightr r r set_annotationsszBaselineModel.set_annotationscCsR||j|\}}}g}|r"|||D] }|||7}q|S|||S)zSegment the compound by looking it up in the model analyses. Raises KeyError if compound is not present in the training data. For segmenting new words, use viterbi_segment(compound). )r7rrNrxrn)r)r?rrrrrr r r rxs zBaselineModel.segment recursiver {Gzt?c Csfd}td||}|}t|}tdt||jj tdtd|| t |t |D]-} |dkrE|j | g|R} n|dkrS|j| g|R} ntd |td | t| q5|d7}td |t|||}|} |}td|||dkr|| ||jj krn|dkr|d8}|d ur||krtd nq,td||fS)aTrain the model in batch fashion. The model is trained with the data already loaded into the model (by using an existing model or calling one of the load_ methods). In each iteration (epoch) all compounds in the training data are optimized once, in a random order. If applicable, corpus weight, annotation cost, and random split counters are recalculated after each iteration. Arguments: algorithm: string in ('recursive', 'viterbi') that indicates the splitting algorithm used. algorithm_params: parameters passed to the splitting algorithm. finish_threshold: the stopping threshold. Training stops when the improvement of the last iteration is smaller then finish_threshold * #boundaries max_epochs: maximum number of epochs to train rrz0Compounds in training data: %s types / %s tokenszStarting batch trainingzEpochs: %s Cost: %sTrviterbiunknown algorithm '%s'z #%s -> %szCost before epoch update: %sN+Max number of epochs reached, stop trainingzDone.)rrrr~listr_loggerinforMrr<rEshufflerr}r{rdebugr) r) algorithmalgorithm_paramsfinish_threshold max_epochsepochsrnewcost compoundsrsegmentsoldcostr r r train_batch0sN       zBaselineModel.train_batch'c Cs||dur i}tdd} d} d} | r|| |} td| | tt|D]} zt|\} }Wn tyCd} Ynwt |dkrKq-|dur{||vr\d}d||<d}n||}|d||<||d||}|dkrz| ||n| |d|dur|dkr| ||}| |||dkr|j |g|R}n|d kr|j|g|R}ntd |td | |t|| d7} q-| d7} |dur| |krtd n| s|| |} td| | | | fS) aTrain the model in online fashion. The model is trained with the data provided in the data argument. As example the data could come from a generator linked to standard in for live monitoring of the splitting. All compounds from data are only optimized once. After online training, batch training could be used for further optimization. Epochs are defined as a fixed number of compounds. After each epoch ( like in batch training), the annotation cost, and random split counters are recalculated if applicable. Arguments: data: iterator of (_, compound_atoms) tuples. The first argument is ignored, as every occurence of the compound is taken with count 1 count_modifier: function for adjusting the counts of each compound epoch_interval: number of compounds to process before starting a new epoch algorithm: string in ('recursive', 'viterbi') that indicates the splitting algorithm used. algorithm_params: parameters passed to the splitting algorithm. init_rand_split: probability for random splitting a compound to at any point for initializing the model. None or 0 means no random splitting. max_epochs: maximum number of epochs to train NzStarting online trainingrTzTokens processed: %s Cost: %sFrrrrz #%s: %s -> %sr)r7rrrr~rrLnext StopIterationrMrBrOrZr}r{rrr)r)rrepoch_intervalrrrrcountsrrG more_tokensr_rr@addcrTrr r r train_onlineqsj!            /zBaselineModel.train_onliner-c Cst|}dg}|jj|jj|dkr!t|jj|jj|}nd}||d}td|dD]}d} d} |jrW||krW|j||d|drW| |||dfq0tt d|||D]} || ddurlqa|| d} || |} | |j vr|j | j s|j | j dkrtd| |j | j f| |t|j | j |7} ne|dkr|jjdkr| |t||j| |jj7} nG| |t||jj|t|jj||jjt|jj|j| |jj7} nt| dkr| |7} n|jr | t| |7} nqa| dus| | kr| } | } qa| | | fq0g}|d\} }|d}|durL|}| |||||d}|}|dus4|| t|jj|jjt|jj7} || fS)a Find optimal segmentation using the Viterbi algorithm. Arguments: compound: compound to be segmented addcount: constant for additive smoothing (0 = no smoothing) maxlen: maximum length for the constructions If additive smoothing is applied, new complex construction types can be selected during the search. Without smoothing, only new single-atom constructions can be selected. Returns the most probable segmentation and its log-probability. )reNrr-rN Construction count of '%s' is %sr)rMrr9r<rfrgrLr$rrnrrrrrrrget_codelengthrreverse)r)r?rvrwrogrid logtokens badlikelihoodrsbestpathrjptrarCrpathltr r r rys                 zBaselineModel.viterbi_segmentc CsDt|}dg}|jj|jjdkrt|jj|jj}nd}td|dD]a}d}td|D]B}||}|||} | |jvrh|j| jsh|j| j dkrZt d| |j| j f||t|j| j 7}nq/|t | 7}q/|dkr| t| q&| |j q&|d}|t|jj|jjt|jj7}|S)aWFind log-probability of a compound using the forward algorithm. Arguments: compound: compound to process Returns the (negative) log-probability of the compound. If the probability is zero, returns a number that is larger than the value defined by the penalty attribute of the model object. rerrrr)rMrr9r<rfrgrLrrrrexprnrh) r)r?rorrrspsumrrarCr r r forward_logprob,sL        zBaselineModel.forward_logprobc Cs\t|}dgg}|jj|jj|dkr"t|jj|jj|}nd}||d}td|dD]} g} |jrZ| |krZ|j|| d| drZ| | || ddfgq1tt d| || D]} tt|| D]} || | ddur{qn|| | d} || | }||j vr|j |j s|j |j dkrtd||j |j f| |t|j |j |8} ng|dkr|jjdkr| |t||j||jj8} nH| |t||jj|t|jj||jjt|jj|j||jj8} nt|dkr| |8} n|jr| t||8} nqnt| |kr0t| | | | fqnt| | | | fqnqd| | q1g}tt|dD]Y} g}|d| \} }}|d}|dur|} | || ||| |d}|| |d}| }|dusa|| t|jj|jjt|jj8} | | |fqKd d t|DS) aPFind top-n optimal segmentations using the Viterbi algorithm. Arguments: compound: compound to be segmented n: how many segmentations to return addcount: constant for additive smoothing (0 = no smoothing) maxlen: maximum length for the constructions If additive smoothing is applied, new complex construction types can be selected during the search. Without smoothing, only new single-atom constructions can be selected. Returns the n most probable segmentations and their log-probabilities. )reNNrr-rrNrrcSsg|]\}}||fqSr r )rFrarVr r r rmsz/BaselineModel.viterbi_nbest..)rMrr9r<rfrgrLr$rrnrrrrrrrrrheapqheappush heappushpoprr)r)r?nrvrwrorrrrsbestnrkrarCresultsrrkirr r r viterbi_nbest]s            -    zBaselineModel.viterbi_nbestcCr8r5)rrr6r r r get_corpus_coding_weightsz&BaselineModel.get_corpus_coding_weightcCs|||j_dSr5)r7rrr)rr r r set_corpus_coding_weights z&BaselineModel.set_corpus_coding_weightcCs d|_dd|jD|_dS)aReduce the size of this model by removing all non-morphs from the analyses. After calling this method it is not possible anymore to call any other method that would change the state of the model. Anyway doing so would throw an exception. TcSsi|] \}}|js||qSr )r)rFrvr r r s  z3BaselineModel.make_segment_only..N)rrr[r6r r r make_segment_onlyszBaselineModel.make_segment_onlycCs$t|D] }|||gqdSr5)rrrZ)r)r?r r r clear_segmentationsz BaselineModel.clear_segmentation)NNFN)rP)rru)rNNr5)rr rN)Nrrr NN)r-ru)-__name__ __module__ __qualname____doc__rhr,rr7propertyr9r:rBrDrOrZrdr\rqrtr{r}r|r=r staticmethodrRrNrSrrr~rrrrrxrrryrrrrrrr r r r r!sn 1    * 3"    &   A ^] 1e rc@seZdZeddZdS) CorpusWeightcCs\|dkr,|}|dkr|dd|9}n |ddd|9}||td|dSdS)Nrr@r-zCorpus weight set to %sTF)rrrr)clsmodel directionepochrr r r move_directions  zCorpusWeight.move_directionN)rrr classmethodrr r r r rsrc@seZdZddZddZdS)r.cCs ||_dSr5rrr r r r,s zFixedCorpusWeight.__init__cCs||jdS)NF)rr)r)rrr r r r3s zFixedCorpusWeight.updateNrrrr,r3r r r r r.s r.c@sBeZdZdZdddZddZeddZed d Zd d Z d S)AnnotationCorpusWeightzcClass for using development annotations to update the corpus weight during batch training {Gz?cC||_||_dSr5)rrI)r) devel_setrIr r r r, zAnnotationCorpusWeight.__init__csN|dkrdS|j}t|\}}fdd|D}|||}|||S)zuTune model corpus weight based on the precision and recall of the development data, trying to keep them equalrFcsg|] }|dqSrl)ry)rFrrr r rmsz1AnnotationCorpusWeight.update..)rr[zip_estimate_segmentation_dirr)r)rrtmpwlistr(rdr rr r3s   zAnnotationCorpusWeight.updatec Csd}d}t||D]G\}}d}|D]2}tt|} t| dkr$d}n |D]} tt| } t| | tt| } | |krB| }q&q|dkrP||7}|d7}q ||fS)z:Calculate average boundary recall for given segmentations.rrerr-r)rsetrrRrM intersectionfloat) r prediction reference rec_totalrec_sumpre_listref_listbestrefref_bprepre_brr r r _boundary_recalls* z'AnnotationCorpusWeight._boundary_recallc CsN|||\}}|||\}}||}||}d||||} ||| fS)zAReturn boundary precision, recall, and F-score for segmentations.r)r ) rrrrec_srec_tpre_spre_trecr rcr r r _bpr_evaluations  z&AnnotationCorpusWeight._bpr_evaluationcCsP|dd|D|\}}}td||t|||jkr dS||kr&dSdS)aEstimate if the given compounds are under- or oversegmented. The decision is based on the difference between boundary precision and recall values for the given sample of segmented data. Arguments: segments: list of predicted segmentations annotations: list of reference segmentations Return 1 in the case of oversegmentation, -1 in the case of undersegmentation, and 0 if no changes are required. cSsg|]}|gqSr r )rFr r r r rm6szEAnnotationCorpusWeight._estimate_segmentation_dir..z0Boundary evaluation: precision %.4f; recall %.4frrr)rrrabsrI)r)rr(r rrcr r r r(sz1AnnotationCorpusWeight._estimate_segmentation_dirNr) rrrrr,r3rr rrr r r r rs    rc@s*eZdZd ddZddZeddZdS) MorphLengthCorpusWeightrcCrr5) morph_lengthrI)r) morph_lenghtrIr r r r,Arz MorphLengthCorpusWeight.__init__cCsf|dkrdS||}td|t|j||j|jkr1t|j||j|}||||SdS)NrFzCurrent morph-length: %s)calc_morph_lengthrrrrrIr)r)rr cur_lengthrr r r r3Es  zMorphLengthCorpusWeight.updatecCsVd}d}|D]}||}|D] }|d7}|t|7}qq|dkr)t||SdS)Nrrre)rrxrMr)rrtotal_constructions total_atomsr?rrCr r r rSs   z)MorphLengthCorpusWeight.calc_morph_lengthNr)rrrr,r3rrr r r r r@s  rc@seZdZdddZddZdS)NumMorphCorpusWeightrcCrr5)num_morph_typesrI)r)rrIr r r r,crzNumMorphCorpusWeight.__init__cCsd|dkrdS|jj}td|t|j||j|jkr0t|j||j|}||||SdS)NrFzNumber of morph types: %s)rr<rrrrrIr)r)rrcur_morph_typesrr r r r3gs  zNumMorphCorpusWeight.updateNrrr r r r rbs  rc@sbeZdZdZdddZedejZe ddZ e dd Z d d Z d d ZddZddZdS)EncodingzBase class for calculating the entropy (encoding length) of a corpus or lexicon. Commonly subclassed to redefine specific methods. r-cCsd|_d|_d|_||_dS)z`Initizalize class Arguments: weight: weight used for this encoding rerN) logtokensumr9r<rrr r r r,}s zEncoding.__init__rcCsdS)zxDefine number of types as 0. types is made a property method to ensure easy redefinition in subclasses rr r6r r r r:szEncoding.typescCsH|dkrdS|dkrtt|St|}|||d||jS)zaCalculate logarithm of n!. For large n (n > 20), use Stirling's approximation. rreg?)rfrg factorial_log2pi)rrlognr r r _logfactorials  zEncoding._logfactorialcCsH|jdkrdS|j|j}||d||jd|||jS)z~Calculate -log[(u - 1)! (v - u)! / (v - 1)!] v is the number of tokens+boundaries and u the number of types rrer)r:r9r<r%r)r9r r r frequency_distribution_costs   z$Encoding.frequency_distribution_costcCs||j S)z'The permutations cost for the encoding.)r%r<r6r r r permutations_costszEncoding.permutations_costcCsZ|j||7_|dkr|j|t|8_|dkr+|j|t|7_dSdS)z"Update the counts in the encoding.rN)r9r rfrg)r)rC old_count new_countr r r rs zEncoding.update_countcCsV|jdkrdS|j|j}|t||jt|j|j||j|S)z2Calculate the cost for encoding the corpus/lexiconrre)r<r9rfrgr r(rr'r)rr r r r~s   zEncoding.get_costNr-)rrrrr,rfrgpir#rr:rr%r'r(rr~r r r r rvs       rcs>eZdZdZd fdd ZeddZddZd d ZZ S) rzEncoding the corpus class The basic difference to a normal encoding is that the number of types is not stored directly but fetched from the lexicon encoding. Also does the cost function not contain any permutation cost. r-cstt||||_dSr5)superrr,lexicon_encoding)r)r/r __class__r r r,s zCorpusEncoding.__init__cCs |jjdS)z}Return the number of types of the corpus, which is the same as the number of boundaries in the lexicon + 1 r)r/r<r6r r r r:s zCorpusEncoding.typescCsF|jdkrdS|j}||d||jd|||jdS)zXCalculate -log[(M - 1)! (N - M)! / (N - 1)!] for M types and N tokens. rrer)r:r9r%r&r r r r's  z*CorpusEncoding.frequency_distribution_costcCsN|jdkrdS|j|j}|t||jt|j|j|j|S)zhOverride for the Encoding get_cost function. A corpus does not have a permutation cost rre)r<r9rfrgr rr'r+r r r r~s   zCorpusEncoding.get_costr,) rrrrr,rr:r'r~ __classcell__r r r0r rs  rcsJeZdZdZdfdd ZddZdd Zd d Zd d ZddZ Z S)rzsEncoding the cost of an Annotated Corpus. In this encoding constructions that are missing are penalized. NrcsHtt|d|_d|_|durd|_||_||_||_t|_ dS)a Initialize encoding with appropriate meta data Arguments: corpus_coding: CorpusEncoding instance used for retrieving the number of tokens and boundaries in the corpus weight: The weight of this encoding. If the weight is None, it is updated automatically to be in balance with the corpus penalty: log penalty used for missing constructions Tr-NF) r.rr,do_update_weightr corpus_codingrhr r!r)r)r4rrhr0r r r,s z AnnotatedCorpusEncoding.__init__cCs||_t||_d|_dS)zMethod for re-initializing the constructions. The count of the constructions must still be set with a call to set_count reN)rsumvaluesr9r )r)rr r r r]s z)AnnotatedCorpusEncoding.set_constructionscCsF|j|}|dkr|j|t|7_dS|j||j7_dS)z`Set an initial count for each construction. Missing constructions are penalized rNrr rfrgrh)r)rCr annot_countr r r r^s z!AnnotatedCorpusEncoding.set_countcCs||jvrC|j|}|dkr|j|t|8_n |j||j8_|dkr7|j|t|7_dS|j||j7_dSdS)zoUpdate the counts in the Encoding, setting (or removing) a penalty for missing constructions rNr7)r)rCr)r*r8r r r r&s  z$AnnotatedCorpusEncoding.update_countcCsL|jsdS|j}|jjt|jj|j|_|j|kr$td|jdSdS)zxUpdate the weight of the Encoding by taking the ratio of the corpus boundaries and annotated boundaries Nz)Corpus weight of annotated data set to %s)r3rr4rr<rr)r)oldr r r r6s  z%AnnotatedCorpusEncoding.update_weightcCsT|jdkrdS|j|j}|t|jj|jj|jt|jj|j|jS)z)Return the cost of the Annotation Corpus.rre)r<r9rfrgr4r rr+r r r r~Bs   z AnnotatedCorpusEncoding.get_cost)Nr) rrrrr,r]r^rrr~r2r r r0r rs   rcsDeZdZdZfddZeddZddZdd Zd d Z Z S) rz7Class for calculating the encoding cost for the Lexiconcstt|t|_dS)zInitialize Lexcion EncodingN)r.rr,r r!rr6r0r r r,PszLexiconEncoding.__init__cCst|jdS)zdReturn the number of different atoms in the lexicon + 1 for the compound-end-token r)rMrr6r r r r:UszLexiconEncoding.typescCsF|jd7_|D]}|j|}|d|j|<||||dq dS)zcAdd a construction to the lexicon, updating automatically the count for its atoms rNr<rrr)rCatomr@r r r r]  zLexiconEncoding.addcCsF|jd8_|D]}|j|}|d|j|<||||dq dS)zfRemove construction from the lexicon, updating automatically the count for its atoms rNr:r;r r r rhr=zLexiconEncoding.removecCsjt|d}|t|j|}|t|jd8}|D]}||jvr)|j|}nd}|t|8}q|S)z6Return an approximate codelength for new construction.r)rMrfrgr9r<r)r)rClrar<r@r r r rss   zLexiconEncoding.get_codelength) rrrrr,rr:rrrr2r r r0r rMs    r)r rloggingrfr1rEr%utilsrr exceptionrr getLoggerrrr namedtuplerobjectrrr.rrrrrrrr r r r s> ? O"R.W