o `۷i@sddlZddlmZddlmZddlmZmZddl m Z ddl m Z GdddeZ d d Zd d Zdd dZddZddZdS)N)minimize)GaussianProcessRegressor)HyperparameterKernel)pairwise_distances)euclidean_distancesc@seZdZdZ      dddZedd Zed d Zed d ZdddZ ddZ ddZ eddZ e j ddZ eddZdS) TV_SquaredExpzTime varying squared exponential kernel. For more info see the TV-GP-UCB paper: http://proceedings.mlr.press/v51/bogunovic16.pdf ?皙?h㈵>gj@r ?cCs(||_||_||_||_||_||_dSN)variance lengthscaleepsilonvariance_boundslengthscale_boundsepsilon_bounds)selfrrrrrrrS/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/tune/schedulers/pb2_utils.py__init__s   zTV_SquaredExp.__init__cCtdd|jS)Nrnumeric)rrrrrrhyperparameter_variancez%TV_SquaredExp.hyperparameter_variancecCr)Nrr)rrrrrrhyperparameter_lengthscale#rz(TV_SquaredExp.hyperparameter_lengthscalecCr)Nrr)rrrrrrhyperparameter_epsilon'rz$TV_SquaredExp.hyperparameter_epsilonNFcCs"t|}|dur |}t|jdd}|dddfdd}|dddfdd}t||d}d|d|}|ddddf} |ddddf} |jttt | |  |j } | |} |r| } tt | | }| ||j }|d}| ||d|}| t | ||gfS| S)Nr rr cityblock) np atleast_2dcliprreshaperrexpsquarerrdstack)rXY eval_gradientrT1T2dists timekernel X_spatial Y_spatialrbfKK_gradient_variancedist2K_gradient_lengthscalenK_gradient_epsilonrrr__call__+s0  zTV_SquaredExp.__call__cCstj|jd|jtjdS)Nr)dtype)r%fullshaperfloat64)rr,rrrdiagMszTV_SquaredExp.diagcCsdSNFrrrrr is_stationaryPszTV_SquaredExp.is_stationarycCst|j|j|jgSr)r%logrrrrrrrthetaSszTV_SquaredExp.thetacCs4t|d|_t|d|_t|d|_dS)Nrr"r$)r%r)rrr)rrErrrrEWscCs"tt|jt|jt|jgSr)r%rDlistrrrrrrrbounds]s zTV_SquaredExp.bounds)r r r r r r rB)__name__ __module__ __qualname____doc__rpropertyrrr r<rArCrEsetterrGrrrrr s0     "  rcCs2|tj|ddtj|ddtj|dddS)zgNormalize data to be in range (0,1), with respect to (wrt) boundaries, which can be specified. raxis:0yE>)r%minmax)datawrtrrr normalizehsrUcCs2|tj|ddtj|ddd}t|ddS)z5Standardize to be Gaussian N(0,1). Clip final values.rrNrPr$)r%meanstdr')rSrrr standardizeqs$rYc Csd}d}|tdt||jjd}|durt|n|}t|ddt|ddfj }z | |d} Wn t yFd} Ynwz|j |dd \} } | dd } Wn t ycd} Ynw| || S) aPUCB acquisition function. Interesting points to note: 1) We concat with the fixed points, because we are not optimizing wrt these. This is the Reward and Time, which we can't change. We want to find the best hyperparameters *given* the reward and time. 2) We use m to get the mean and m1 to get the variance. If we already have trials running, then m1 contains this information. This reduces the variance at points currently running, even if we don't have their label. Ref: https://jmlr.org/papers/volume15/desautels14a/desautels14a.pdf g?g?rNr!r"iT) return_stdr$) rRr%rDX_train_r?sqrt concatenater(arrayTpredict ValueError) mm1xfixedkappac1c2beta_txtestrW_rXvarrrrUCBws"  &   rmcsdddd}d}d}jdddf}dd tjjd |D} t|D]1} tjdd jjd |} tfd d | | d |d} | j} | |krW| }| j}q&t|dd S)zOptimize acquisition function.F)maxitermaxfundisp irNcSsg|]}dqS))rr"r).0rkrrr z optimize_acq..r"cs| Srr)rdrefuncrbrcrrruzoptimize_acq..zL-BFGS-B)rGmethodoptions) r[ranger?r%randomuniformrrdr')rwrbrcrenum_foptsr_ best_value best_thetarGiix0resvalrrvr optimize_acqs(  rcCsRd}|jd|kr|jdS|d}g}|d|jdkr|d7}tt|j}|| dddf}|| d} |ddd|f} ttj| ddtj| ddf d| jd} tj| |fdd} t || } t |  | j d}t ddd d }t|d d }|| |||j|d|jdkst|}|t|dd}|S) z?Select the number of datapoints to keep, using cross validationrnrrrNrNr$r"r r )rrr fmin_l_bfgs_b)kernel optimizer)r?r%r^rFvaluesr_r]rRrQr(rUrYsizerrfitappendlog_marginal_likelihood_value_argmaxint)XrawyrawrGr~min_lenlengthscores base_valsX_leny_len oldpointsold_limslimitsr,yrrbidxrrr select_lengths6      rr)numpyr%scipy.optimizersklearn.gaussian_processr sklearn.gaussian_process.kernelsrrsklearn.metricsrsklearn.metrics.pairwiserrrUrYrmrrrrrrs    _