o Qi@sxddlZddlmZddlmZddlmZddlmZddlm Z ddl m Z dd Z d d Z dddZdddZdS)N) csr_matrix)diags) dok_matrix)pairwise_distances)get_process_memory)sent_to_word_contexts_matrixcs8t|d}tfdd|jdD|jd<|S)Nrc$g|]}|dkr dnd|qSr.0valphar D/home/ubuntu/.local/lib/python3.10/site-packages/soynlp/word/_pmi.py $z_as_diag..)rtolistnpasarraydata)pxrpx_diagr rr_as_diag s$rc Csl|j\}}|\}}|j}t||kd}||}||}||}t|}t|||ff||fd}|S)Nr)shape)rnonzerorrwherelogr) exp_pmi min_exp_pminmrowscolsrindicesexp_pmi_r r r_logarithm_and_ppmis   r'r c Csd|kr dksJJt|jdd|d}|dur2t|jdd|d}|dkr@||}||}||}t|d}t||}|||} |dkr^dnt|} t| | } | ||fS)az :param X: scipy.sparse.csr_matrix (word, contexts) sparse matrix :param py: numpy.ndarray (1, word) shape, probability of context words. :param min_pmi: float Minimum value of pmi. all the values that smaller than min_pmi are reset to zero. Default is zero. :param alpha: float Smoothing factor. pmi(x,y; alpha) = p_xy /(p_x * (p_y + alpha)) Default is 0.0 :param beta: float Smoothing factor. pmi(x,y) = log ( Pxy / (Px x Py^beta) ) Default is 1.0 Returns ---------- pmi : scipy.sparse.csr_matrix (word, contexts) pmi value sparse matrix px : numpy.ndarray Probability of rows (items) py : numpy.ndarray Probability of columns (features) Usage ----- >>> pmi, px, py = pmi_memory_friendly(X, py=None, min_pmi=0, alpha=0, beta=1.0) rr axisN)rrsumreshaperdotexpr') Xpymin_pmirbetarpxyrpy_diagrr pmir r rr6!s        r6?Fcsd|kr dksJJ|jdd|d}|dur,|jdd|d}||}|jd|jdks>J|dkrL||}||}t|d}t|d} tdd|jdD|jd<tfdd| jdD| jd<||| } |dkrdnt |} t | j| kd} t | j} | \}}| j}t | D]0\}}|r|d dkrtd d || jdd td ddt||| ||||f<q|rtddd d| ||fS)a :param X: scipy.sparse.csr_matrix (word, contexts) sparse matrix :param py: numpy.ndarray (1, word) shape, probability of context words. :param min_pmi: float Minimum value of pmi. all the values that smaller than min_pmi are reset to zero. Default is zero. :param alpha: float Smoothing factor. pmi(x,y; alpha) = p_xy /(p_x * (p_y + alpha)) Default is 0.0 :param beta: float Smoothing factor. pmi(x,y) = log ( Pxy / (Px x Py^beta) ) Default is 1.0 :param verbose: Boolean If True, verbose mode on Returns ---------- pmi : scipy.sparse.dok_matrix (word, contexts) pmi value sparse matrix px : numpy.ndarray Probability of rows (items) py : numpy.ndarray Probability of columns (features) Usage ----- >>> pmi, px, py = pmi_memory_friendly(X, py=None, min_pmi=0, alpha=0, beta=1.0) rr r)r+NcSs g|] }|dkr dnd|qSr r r r r rrs z'pmi_memory_friendly..crr r r rr rrri'z% computing pmi {:.3} % mem={} Gb dz%.3fT)flushendz computing pmi was done{}z )r:)r,r-rrrrrrr.r/rrr enumerateprintformatrr)r0r1r2rr3verboserr4rr5rr r%pmi_dokr#r$r_n_idxidxr rrpmi_memory_friendlyXs<!   $    rC)Nrr(r )Nrr(r7F)numpyr scipy.sparserrrsklearn.metricsr soynlp.utilsrsoynlp.vectorizerrrr'r6rCr r r rs       7