o Mih@sdZddlZddlZddlZddlZddlZejddkZddZer*ddl m Z nzddl m Z Wne yBe deZ YnwdZd Zd Zd d ZGd ddeZdddZddZddZdddZddZddZddZdS) zqData structures and functions of general utility, shared between different modules and variants of the software. NcOsddS)NcS|SN)funcrrC/home/ubuntu/.local/lib/python3.10/site-packages/morfessor/utils.pysz"_dummy_lru_cache..rargskwargsrrr_dummy_lru_cachesr ) lru_cachezDLRU cache disabled, install backports.functools_lru_cache to enable.i@BiTcsttsSzddlmWntyGddd}|Ynwttjr.fdd}|Stdr8SS)aDecorator/function for displaying a progress bar when iterating through a list. iter_func can be both a function providing a iterator (for decorator style use) or an iterator itself. No progressbar is displayed when the show_progress_bar variable is set to false. If the progressbar module is available a fancy percentage style progressbar is displayed. Otherwise 60 dots are printed as indicator. r) ProgressBarc@s8eZdZdZdZddZddZddZd d ZeZ d S) z$_progress..SimpleProgressBarzlCreate a simple progress bar that prints 60 dots on a single line, proportional to the progress <cSsd|_d|_d|_dS)Ndr)itdotfreqiselfrrr__init__Fs z-_progress..SimpleProgressBar.__init__cSs>t||_d|_t||jd|j|_|jdkrd|_|S)Nr)iterrrlenNUM_DOTSr)rrrrr__call__Ks  z-_progress..SimpleProgressBar.__call__cSrrrrrrr__iter__Vsz-_progress..SimpleProgressBar.__iter__cSs^|jd7_|j|jdkrtjdtjzt|jWSty.tjdw)Nrr. ) rrsysstderrwriteflushnextr StopIterationrrrr__next__Ys     z-_progress..SimpleProgressBar.__next__N) __name__ __module__ __qualname____doc__rrrrr%r#rrrrSimpleProgressBarAs  r*cs4tttjr|i|S|i|Sr)logging getLoggerr& isEnabledForINFOr r iter_funcrrrlsz_progress..ir)show_progress_bar progressbarr ImportError isinstancetypes FunctionTypehasattr)r0r*rrr/r _progress+s & +  r8c@s(eZdZdZddZddZddZdS) SparseaA defaultdict-like data structure, which tries to remain as sparse as possible. If a value becomes equal to the default value, it (and the key associated with it) are transparently removed. Only supports immutable values, e.g. namedtuples. cOs(|d|_tj|g|Ri|dS)zCreate a new Sparse datastructure. Keyword arguments: default: Default value. Unlike defaultdict this should be a prototype immutable, not a factory. defaultN)pop_defaultdictr)rpargsr rrrrs zSparse.__init__cCs(zt||WSty|jYSwr)r= __getitem__KeyErrorr<)rkeyrrrr?s   zSparse.__getitem__cCs:d|vr||jkr||vr||=dSdSt|||dS)Nr<)r<r= __setitem__)rrAvaluerrrrBs  zSparse.__setitem__N)r&r'r(r)rr?rBrrrrr9{s  r9ccsNg}|D]}||t||kr|| d}t||kr$t|VqdS)zReturns all ngram tokens in an input sequence, for a specified n. E.g. ngrams(['A', 'B', 'A', 'B', 'D'], n=2) yields ('A', 'B'), ('B', 'A'), ('A', 'B'), ('B', 'D') N)appendrtuple)sequencenwindowitemrrrngramss    rKcCs:d}t|D]\}}|ddus||dkr||f}q|S)zaReturns the minimum value and the first index at which it can be found in the input sequence.)NNrN) enumerate)rGbestrrCrrr minargmins rNcCs$|dksJ|dkr tSt| S)z>Logarithm which uses constant value for log(0) instead of -infgr) LOGPROB_ZEROmathlog)xrrrzlogs  rScCs|gt|j}||S)zsConvenience function to return a namedtuple initialized to zeros, without needing to know the number of fields.)r_fields) constructorzerozerosrrr _nt_zerossrXc stdd|Dtfddt|D}g}tdd|D}d}d}|D]}||kr;t|\}}||7}||ks-||q'|S)a=Samples with replacement from the data set so that the probability of each data point being selected is proportional to the occurrence count. Arguments: data: A list of tuples (weight, ...) num_samples: The number of samples to return Returns: a sorted list of indices to data cs|]}|dVqdSrNr.0rRrrr z"weighted_sample..csg|] }tddqS)rr)randomrandint)r\_tokensrr sz#weighted_sample..csrYrZrr[rrrr]r^r)sumsortedrangerLr#rE) data num_samples token_indices data_indicesdditisample_token_indexweightrrbrweighted_samples   rrcCstdkr|Sdd}||S)aPrints a progress bar for visualizing flow through a generator. The length of a generator is not known in advance, so the bar has no fixed length. GENERATOR_DOT_FREQ controls the frequency of dots. This function wraps the argument generator, returning a new generator. rcssLt|D]\}}|tdkrtjdtj|VqtjddS)Nrrr)rLGENERATOR_DOT_FREQrr r!r") generatorrrRrrr_progress_wrappers   z._generator_progress.._progress_wrapper)rs)rtrurrr_generator_progresssrvcCs*zt|tWStyt|tYSwr)r4 basestring NameErrorstr)objrrr _is_strings   r{)rD)r)r)r+rPr_rr5 version_infoPY3r functoolsr backports.functools_lru_cacher3warningrOrsr1r8r=r9rKrNrSrXrrrvr{rrrrs< P !