o }oi-@s ddlmZddlZddlZddlZddlZddlmZe dej Z e e Z e dej Ze dZd/d d Zd/d d Zedd0ddZd1ddZedd0ddZd1ddZd2ddZd3ddZd4d d!Zd5d$d%Z & d6d7d+d,Z &d8d9d-d.ZdS):) annotationsNerflog_p np.ndarraylog_qreturncCs t||SN)np logaddexprr rS/home/ubuntu/.local/lib/python3.10/site-packages/optuna/samplers/_tpe/_truncnorm.py_log_sum3s rcCs|tt|| Sr )r log1pexprrrr _log_diff7sriafloatcCsX|d}|dkrdt| }|S|dkr!ddt|}|Sddt|}|S)N;f?g;f?g;f??)matherfcr)rxyrrr _ndtr_single;srcCsddt|dS)Nrrrrrrr_ndtrIsr c Cs|dkr t|  S|dkrtt|Sd|dt| dtdtj}d}d}d}d}d|d}d}d }t||tjjkrn|d7}|}| }||9}|d|d9}||||7}t||tjjksG|t|S) Nirrgrr)rrlogpiabssys float_infoepsilon) rlog_LHS last_totalright_hand_side numerator denom_factor denom_conssignirrr_log_ndtr_singleNs* , r2cCsttdd|tS)Nr#)r frompyfuncr2astyperrrrr _log_ndtrisr5rcCs|d dtS)Nrg@)_norm_pdf_logC)rrrr _norm_logpdfmr7bc s|dk}|dk}||B}ddddfdd }dd d }tj|tjtjd }||}jr8|||||<||} jrH|| ||||<||} jrX|| ||||<t|S)z3Log of Gaussian probability mass within an intervalrrrr9r cSstt|t|Sr )rr5rr9rrrmass_case_leftzr8z'_log_gauss_mass..mass_case_leftcs| | Sr rr:r;rrmass_case_right}sz(_log_gauss_mass..mass_case_rightcSstt| t| Sr )r rr r:rrrmass_case_centrals z*_log_gauss_mass..mass_case_central) fill_valuedtypeNrrr9rr r)r full_likenan complex128sizereal) rr9 case_left case_right case_centralr=r>outa_lefta_right a_centralrr<r_log_gauss_massqs    rNrc Cs|dk}|}tt|| ||<t|}t|dk}djr4td||t ||<t|dk}djrPt tt|| ||<t dD].}t |}d|dt}||t ||} || 8}t t| dt|krnqT||d 9<|S) a\ Use the Newton method to efficiently find the root. `ndtri_exp(y)` returns `x` such that `y = log_ndtr(x)`, meaning that the Newton method is supposed to find the root of `f(x) := log_ndtr(x) - y = 0`. Since `df/dx = d log_ndtr(x)/dx = (ndtr(x))'/ndtr(x) = norm_pdf(x)/ndtr(x)`, the Newton update is x[n + 1] := x[n] - (log_ndtr(x) - y) * ndtr(x) / norm_pdf(x). As an initial guess, we use the Gaussian tail asymptotic approximation: 1 - ndtr(x) \simeq norm_pdf(x) / x --> log(norm_pdf(x) / x) = -1/2 * x**2 - 1/2 * log(2pi) - log(x) First recall that y is a non-positive value and y = log_ndtr(inf) = 0 and y = log_ndtr(-inf) = -inf. If abs(y) is very small, we first derive -x such that z = log_ndtr(-x) and then flip the sign. Please note that the following holds: z = log_ndtr(-x) --> z = log(1 - ndtr(x)) = log(1 - exp(y)) = log(-expm1(y)). Recall that as long as ndtr(x) = exp(y) > 0.5 --> y > -log(2) = -0.693..., x becomes positive. ndtr(x) = exp(y) \simeq 1 + y --> -y \simeq 1 - ndtr(x). From this, we can calculate: log(1 - ndtr(x)) \simeq log(-y) \simeq -1/2 * x**2 - 1/2 * log(2pi) - log(x). Because x**2 >> log(x), we can ignore the second and third terms, leading to: log(-y) \simeq -1/2 * x**2 --> x \simeq sqrt(-2 log(-y)). where we take the positive `x` as abs(y) becomes very small only if x >> 0. The second order approximation version is sqrt(-2 log(-y) - log(-2 log(-y))). If abs(y) is very large, we use log_ndtr(x) \simeq -1/2 * x**2 - 1/2 * log(2pi) - log(x). To solve this equation analytically, we ignore the log term, yielding: log_ndtr(x) = y \simeq -1/2 * x**2 - 1/2 * log(2pi) --> y + 1/2 * log(2pi) = -1/2 * x**2 --> x**2 = -2 * (y + 1/2 * log(2pi)) --> x = sqrt(-2 * (y + 1/2 * log(2pi)) For the moderate y, we use Eq. (13), i.e., standard logistic CDF, in the following paper: - `Approximating the Cumulative Distribution Function of the Normal Distribution `__ The standard logistic CDF approximates ndtr(x) with: exp(y) = ndtr(x) \simeq 1 / (1 + exp(-pi * x / sqrt(3))) --> exp(-y) \simeq 1 + exp(-pi * x / sqrt(3)) --> log(exp(-y) - 1) \simeq -pi * x / sqrt(3) --> x \simeq -sqrt(3) / pi * log(exp(-y) - 1). g{Gzrgdr"rg:0yE>)copyr r$expm1 empty_likenonzerorEsqrtr6_ndtri_exp_approx_Cranger5rallr&) rflippedzr small_inds moderate_inds_ log_ndtr_xlog_norm_pdf_xdxrrr _ndtri_exps$/   rbqnp.ndarray | floatc Cst|||\}}}t|||\}}}|dk}|}t||}ddd }dd d }t|}||} jrE|| ||||||||<||} jr[|| ||||||||<||dk||dk<||d k||d k<tj|||k<|S)a= Compute the percent point function (inverse of cdf) at q of the given truncated Gaussian. Namely, this function returns the value `c` such that: q = \int_{a}^{c} f(x) dx where `f(x)` is the probability density function of the truncated normal distribution with the lower limit `a` and the upper limit `b`. More precisely, this function returns `c` such that: ndtr(c) = ndtr(a) + q * (ndtr(b) - ndtr(a)) for the case where `a < 0`, i.e., `case_left`. For `case_right`, we flip the sign for the better numerical stability. rrcrrr9log_massr cSs tt|t||}t|Sr )rr5r r$rbrcrr9re log_Phi_xrrrppf_leftszppf..ppf_leftcSs&tt| t| |}t| Sr )rr5r rrbrfrrr ppf_rights zppf..ppf_rightr#N) rcrrrr9rrerr r)r atleast_1dbroadcast_arraysrNrTrErrC) rcrr9rGrHrerhrirJq_leftq_rightrrrppfs     rnr#locscale random_statenp.random.RandomState | NonecCsD|ptj}t||||j}|jdd|d}t|||||S)z This function generates random variates from a truncated normal distribution defined between `a` and `b` with the mean of `loc` and the standard deviation of `scale`. rr#)lowhighrE)r random RandomState broadcastshapeuniformrn)rr9rorprqrE quantilesrrrrvs s r{cCs~|||}t|||\}}}t|t||t|}t|||\}}}tj||k||k||kBgtjtj g|dS)N)default) r rjr7rNr$rkselectrCinf)rrr9rorprJrrrlogpdfs .r)rrr rr r)rrr r)rrr r)rrr rrA)rrr r)rcrrrdr9rdr r)rr#N) rrr9rrordrprdrqrrr r)rr#) rrrrdr9rdrordrprdr r) __future__r functoolsrr'numpyr optuna.samplers._tpe._erfrrVr% _norm_pdf_Cr$r6rW_log_2rr lru_cacherr r2r5r7rNrbrnr{rrrrrs: !            ' H0