o XεiM@sddlZddlmZddlmZmZmZddlm Z ddl Z ddl m Z ddl mZmZddlZddlZddlmZzddlZWneyPeded dwz ddlZdd lmZWneymed ed dwd Zz!ddlZddlmZddlmmZddlZddl Zddl!Z!WneyeddZYnwGdddeZ"ddZ# d3ddZ$d4ddZ% d5ddZ& d6ddZ' d6dd Z(d!d"Z)d7d#d$Z*d8d%d&Z+d8d'd(Z,Gd)d*d*ej-j.Z/d+d,Z0Gd-d.d.ej1Z2erGd/d0d0ej3Z4d1d2Z5dS dS)9N)UMAP)warncatch_warningsfilterwarnings) TypingError)spectral_layout)check_random_state check_array)KDTreeaThe umap.parametric_umap package requires Tensorflow > 2.0 to be installed. You can install Tensorflow at https://www.tensorflow.org/install or you can install the CPU version of Tensorflow using pip install umap-learn[parametric_umap] z/umap.parametric_umap requires Tensorflow >= 2.0)opszEThe umap.parametric_umap package requires Keras >= 3 to be installed.z#umap.parametric_umap requires KerasTz7Torch and ONNX required for exporting to those formats.Fc seZdZddddddddddddif fdd Zd fdd Z d fd d Zd!d d Zfd dZddZd!ddZ ddZ d"ddZ    d#ddZ ddZ ddZZS)$ParametricUMAPNF?rc stjd i|||_||_||_||_||_||_||_||_ d|_ | |_ | |_ | |_ d|_d|_| |_| |_d|_t|jtrItj|jd|_tjjddd|_|jdurt|jdjd|jkrvt d !|jdjd|jdSdS) af Parametric UMAP subclassing UMAP-learn, based on keras/tensorflow. There is also a non-parametric implementation contained within to compare with the base non-parametric implementation. Parameters ---------- batch_size : int, optional size of batch used for batch training, by default None dims : tuple, optional dimensionality of data, if not flat (e.g. (32x32x3 images for ConvNet), by default None encoder : keras.Sequential, optional The encoder Keras network decoder : keras.Sequential, optional the decoder Keras network parametric_reconstruction : bool, optional Whether the decoder is parametric or non-parametric, by default False parametric_reconstruction_loss_fcn : bool, optional What loss function to use for parametric reconstruction, by default keras.losses.BinaryCrossentropy parametric_reconstruction_loss_weight : float, optional How to weight the parametric reconstruction loss relative to umap loss, by default 1.0 autoencoder_loss : bool, optional [description], by default False reconstruction_validation : array, optional validation X data for reconstruction loss, by default None global_correlation_loss_weight : float, optional Whether to additionally train on correlation of global pairwise relationships (>0), by default 0 landmark_loss_fn : callable, optional The function to use for landmark loss, by default the euclidean distance landmark_loss_weight : float, optional How to weight the landmark loss relative to umap loss, by default 1.0 keras_fit_kwargs : dict, optional additional arguments for model.fit (like callbacks), by default {} NMbP?@ clipvaluerzNDimensionality of embedder network output ({}) doesnot match n_components ({}))"super__init__dimsencoderdecoderparametric_reconstruction%parametric_reconstruction_loss_weight"parametric_reconstruction_loss_fcnautoencoder_loss batch_sizeloss_report_frequencyglobal_correlation_loss_weightlandmark_loss_fnlandmark_loss_weight prev_epoch_X window_valsreconstruction_validationkeras_fit_kwargsparametric_model isinstance random_stateintkerasutilsset_random_seedn_training_epochs optimizersAdam optimizeroutputsshape n_components ValueErrorformat)selfrrrrrrrrr&r!r"r#r'kwargs __class__rH/home/ubuntu/.local/lib/python3.10/site-packages/umap/parametric_umap.pyr0sD4  zParametricUMAP.__init__c|jdu|du@r+tttjtjgg|jdt||j}t||jf}|durFt |}t |}||krFt d|d|d|j dkr_|durSt d||_ t j|||dSt j|||dS aFit X into an embedded space. Optionally use a precomputed distance matrix, y for supervised dimension reduction, or landmarked positions. Parameters ---------- X : array, shape (n_samples, n_features) Contains a sample per row. If the method is 'exact', X may be a sparse matrix of type 'csr', 'csc' or 'coo'. Unlike UMAP, ParametricUMAP requires precomputed distances to be passed seperately. y : array, shape (n_samples) A target array for supervised dimension reduction. How this is handled is determined by parameters UMAP was instantiated with. The relevant attributes are ``target_metric`` and ``target_metric_kwds``. precomputed_distances : array, shape (n_samples, n_samples), optional A precomputed a square distance matrix. Unlike UMAP, ParametricUMAP still requires X to be passed seperately for training. landmark_positions : array, shape (n_samples, n_components), optional The desired position in low-dimensional space of each sample in X. Points that are not landmarks should have nan coordinates. NrzLength of x = z5, length of landmark_positions = z, while it must be equal. precomputedzTPrecomputed distances must be supplied if metric is precomputed.landmark_positions)r$npstackarraynanr4list transform concatenatelenr6metric_Xrfitr8Xyprecomputed_distancesrAlen_Xlen_landr:rr<rLs8 zParametricUMAP.fitcr=r>)r$rBrCrDrEr4rFrGrHrIr6rJrKr fit_transformrMr:rr<rSs8 zParametricUMAP.fit_transformcCs(|r|n|j}|jjt|||jdS)aTransform X into the existing embedded space and return that transformed output. Parameters ---------- X : array, shape (n_samples, n_features) New data to be transformed. batch_size : int, optional Batch size for inference, defaults to the self.batch_size used in training. Returns ------- X_new : array, shape (n_samples, n_components) Embedding of the new data in low-dimensional space. rverbose)rrpredictrB asanyarrayrU)r8rNrrrr<rGszParametricUMAP.transformcs.|jr|jjt||j|jdSt|S)aTransform X in the existing embedded space back into the input data space and return that transformed output. Parameters ---------- X : array, shape (n_samples, n_components) New points to be inverse transformed. Returns ------- X_new : array, shape (n_samples, n_features) Generated data points new data in data space. rT) rrrVrBrWrrUrinverse_transform)r8rNr:rr<rX2s   z ParametricUMAP.inverse_transformcCsF|j}t|j|j|j|j|j|j|j||j |j |j |j |j d |_dS)zDefine the model in keras) negative_sample_raterr!parametric_reconstruction_loss_fnrrr!rr"r#r2N)r UMAPModel_a_brYrrrrr!rr"r#r2r()r8prlwrrr< _define_modelFs  zParametricUMAP._define_modelc Cs:|jdkr|j}|jdurt|dg|_nt|jdkr,t|t|gt|j}|jrAt |dks=t |dkrAt d|durMt |tj dd}t||j|j|j|j|j|d \}|_}}} |_tt|tjd |_tt| tjd |_|jdurd} t|} t|j|j|j|j| |j| \|_|_| t!||j|j"} |jr|j#durt|jdkrt|j#t|j#gt|j|_#|j#t$|j#fd |j#if} nd} |jj%|f|j"|j&| | d |j'}t(|d s|j)|_*n|j)+D]}|j*||j)|7<q|jj,||j-d}|ifS)Nr?rrr zMData should be scaled to the range 0-1 for cross-entropy reconstruction loss.z allow-nan)dtypeforce_all_finiter@rreconstruction)epochssteps_per_epochvalidation_data_history)rU).rJrKrrBr4rIreshaperFrmaxminrr float32construct_edge_datasetgraph_n_epochsrr! edge_weightr rD expand_dimsastypeint64headtailr(prepare_networksrrr5r_r+r r& zeros_likerLr/r'hasattrhistoryrgkeysrVrU)r8rNrninitr*rA edge_datasetn_edgesrsrtinit_embeddingn_datarerfrxkey embeddingrrr<_fit_embed_dataYs  "         zParametricUMAP._fit_embed_datacCstdd|jDS)Ncss.|]\}}t||r|dvr||fVqdS))r2rrr(N) should_pickle).0kvrrr< sz.ParametricUMAP.__getstate__..)dict__dict__itemsr8rrr< __getstate__szParametricUMAP.__getstate__Tc Cs@|jdurtj|d}|j||rtd||jdur6tj|d}|j||r6td||jdurQtj|d}|j||rQtd|t Bt dtj|d}t |d }t ||t jWdn1sxwY|rtd |WddSWddS1swYdS) N encoder.keraszKeras encoder model saved to {} decoder.keraszKeras decoder model saved to {}zparametric_model.keraszKeras full model saved to {}ignore model.pklwbz*Pickle of ParametricUMAP model saved to {})rospathjoinsaveprintr7rr(rropenpickledumpHIGHEST_PROTOCOL)r8 save_locationrUencoder_outputdecoder_outputparametric_model_output model_outputoutputrrr<rs8       "zParametricUMAP.save{Gz?uniformcCs||_||_||_|jdkr.ttjjt|jdt |jd|dd|_ ||j |_ dS|jdkrF|dur;t d||_ ||j |_ dSt d)aAdd some points from a dataset X as "landmarks." Parameters ---------- X : array, shape (n_samples, n_features) Old data to be retained. sample_pct : float, optional Percentage of old data to use as landmarks. sample_mode : str, optional Method for sampling points. Allows "uniform" and "predefined." landmark_loss_weight : float, optional Multiplier for landmark loss function. rrF)replace predeterminedNz'Choice of sample_mode is not supported.) sample_pct sample_moder#rFrBrandomchoiceranger4r+prev_epoch_idxr$r6)r8rNrrr#idxrrr< add_landmarkss(  zParametricUMAP.add_landmarkscCs d|_dSN)r$rrrr<remove_landmarks! zParametricUMAP.remove_landmarkscCsD|j}t|jd|j}t||}td|jd}tj|||S)z(Exports trained parametric UMAP as ONNX.rr) rPumapNetrr5 weight_copiertorchrandnonnxexport)r8rkmpm dummy_inputrrr<to_ONNX$s  zParametricUMAP.to_ONNX)NNNrT)rrrN)__name__ __module__ __qualname__rrLrSrGrXr_rrrrrr __classcell__rrr:r<r /s>gA E  n % /r cCs|}||jd}|dur|jddkrd}nd}d|j|j|jt|k<|||j}|j}|j}|j}||||||fS)a< gets elements of graphs, weights, and number of epochs per edge Parameters ---------- graph_ : scipy.sparse.csr.csr_matrix umap graph of probabilities n_epochs : int maximum number of epochs per edge Returns ------- graph scipy.sparse.csr.csr_matrix umap graph epochs_per_sample np.array number of epochs to train each sample for head np.array edge head tail np.array edge tail weight np.array edge weight n_vertices int number of vertices in graph rNr'ir`) tocoosum_duplicatesr4datarifloateliminate_zerosrowcol)rmrngraph n_verticesepochs_per_samplersrtweightrrr<get_graph_elements2s  rspectralcCsF|durtd}t|tr$|dkr$|jdd|jd|fdtj}|St|trZ|dkrZt||||||d}dt | } || tj|j d |jd|gd tj}|St |} t | jd krtj| dd jd| jdkrt| } | j| d d \} } t| dddf}| |j d|| jd tj}|S| }|S)a*Initialize embedding using graph. This is for direct embeddings. Parameters ---------- init : str, optional Type of initialization to use. Either random, or spectral, by default "spectral" Returns ------- embedding : np.array the initialized embedding Nrg$g$@r)lowhighsizer)rJ metric_kwds-C6?)scaleraxis)rrr)rr)strrr4rqrBrkrabsrinormalrDrIuniquer querymean) _raw_datarr5r*rJ _metric_kwdsrzrinitialisation expansion init_datatreedistindnndistrrr<init_embedding_from_graphhsX$  rr cCst||d| S)a convert distance representation into log probability, as a function of a, b params Parameters ---------- distances : array euclidean distance between two points in embedding a : float, optional parameter based on min_dist, by default 1.0 b : float, optional parameter based on min_dist, by default 1.0 Returns ------- float log probability in embedding space r)r log1p) distancesabrrr<#convert_distance_to_log_probabilitysrrcCs>| t|}d| t|||}||}|||fS)a Compute cross entropy between low and high probability Parameters ---------- probabilities_graph : array high dimensional probabilities log_probabilities_distance : array low dimensional log probabilities EPS : float, optional offset to ensure log is taken of a positive number, by default 1e-4 repulsion_strength : float, optional strength of repulsion between negative samples, by default 1.0 Returns ------- attraction_term: float attraction term for cross entropy loss repellant_term: float repellent term for cross entropy loss cross_entropy: float cross entropy umap loss r )r log_sigmoid)probabilities_graphlog_probabilities_distanceEPSrepulsion_strengthattraction_termrepellant_termCErrr<compute_cross_entropys  rc Cs|dur/ttjj|dtjtjjdddtjjdddtjjdddtjj|ddg}|durf|rfttjj|fdtjjdddtjjdddtjjdddtjjt|ddd tj|g}||fS) a` Generates a set of keras networks for the encoder and decoder if one has not already been predefined. Parameters ---------- encoder : keras.Sequential The encoder Keras network decoder : keras.Sequential the decoder Keras network n_components : int the dimensionality of the latent space dims : tuple of shape (dim1, dim2, dim3...) dimensionality of data n_data : number of elements in dataset # of elements in training dataset parametric_reconstruction : bool Whether the decoder is parametric or non-parametric init_embedding : array (optional, default None) The initial embedding, for nonparametric embeddings Returns ------- encoder: keras.Sequential encoder keras network decoder: keras.Sequential decoder keras network N)r4drelu)units activationz)rnamerecon)rrr) r, SequentiallayersInputFlattenDenserBproductReshape)rrr5rr~rr}rrr<rus0&     rucsdddjddkr dnddurjddkrdndfdd }fd d }t||\} } } } } }durItt|d gt| | d t| | d }}tjt t |}||tj }||tj }t j j||f}|}|d}|jdd}|j|t j jjd}|j|t j jjd}|d}|t || | | fS)a Construct a tf.data.Dataset of edges, sampled by edge weight. Parameters ---------- X : array, shape (n_samples, n_features) New data to be transformed. graph_ : scipy.sparse.csr.csr_matrix Generated UMAP graph n_epochs : int # of epochs to train each edge batch_size : int batch size parametric_reconstruction : bool Whether the decoder is parametric or non-parametric landmark_positions : array, shape (n_samples, n_components), optional The desired position in low-dimensional space of each sample in X. Points that are not landmarks should have nan coordinates. cSs||Srr)tensorindexrrr< gather_indexDsz,construct_edge_dataset..gather_indexg& .>g?TFNcs^rt|gtjgd}t|gtjgd}n t|}t|}||||fS)Nr)tf py_functionrkgather)edge_to edge_from edge_to_batchedge_from_batch)rNrgather_indices_in_pythonrr<gather_XOs   z(construct_edge_dataset..gather_Xcstdtdi}dkr||d<r||d<dur4r,t|gtjgd|d<nt||d<||f|fS)Numaprglobal_correlationrc landmark_to)r repeatrrrkr)rr r r r3)rr!gather_landmark_indices_in_pythonr!rArrr< get_outputs[s  z+construct_edge_dataset..get_outputsir+r)drop_remainder)num_parallel_callsr)nbytesrr+rBrjrrqr permutationrrIrrrrDatasetfrom_tensor_slicesshufflebatchmap experimentalAUTOTUNEprefetch)rNrmrnrrr!rAr r_rrsrtrr edges_to_expedges_from_exp shuffle_maskr{r)rNrrr rr!rArr<rl's<     rlcCsztt|d}tt|d}WdStjtjj t tjj tjj t ttfyC}ztd||WYd}~dSd}~wtyb}ztd|d|d|WYd}~dSd}~ww) a Checks if a dictionary item can be pickled Parameters ---------- key : try key for dictionary element val : None element of dictionary Returns ------- picklable: bool whether the dictionary item can be pickled base64zDid not pickle {}: {}NFzFailed at pickling :z due to T)codecsencoderdumpsdecodeloads PicklingErrorrerrorsInvalidArgumentError TypeError InternalError NotFoundError OverflowErrorrAttributeErrorrr7r6)rvalpickledr errr<rs,  rcCstj|d}tt|d}|rtd|tj|d}tj|r5t j ||_ |r5td|tj|d}tj|rPt j ||_ td|tj|d}tj|rkt j ||_td ||S) a Load a parametric UMAP model consisting of a umap-learn UMAP object and corresponding keras models. Parameters ---------- save_location : str the folder that the model was saved in verbose : bool, optional Whether to print the loading steps, by default True Returns ------- parametric_umap.ParametricUMAP Parametric UMAP objects rrbz-Pickle of ParametricUMAP model loaded from {}rz"Keras encoder model loaded from {}rz"Keras decoder model loaded from {}r(zKeras full model loaded from {})rrrrloadrrr7existsr,models load_modelrrr()rrUrmodelrrrrrr<load_ParametricUMAPs$   r<c Cst|}|tj|ddd}|dur#|}tj|t|d|d}ntj||jd}|tj|ddd}t|jdg}dg}tj|dd}tj|dd}t|}t|}t|d}t|d}t |||f} t |||f} t |||f} t |||f} t | t| t||j} t | |ddf} t| } t | t| dd||f} |stj | dd } | S) zAdapted from TF Probability.rTrkeepdimsNrarint32rr) r convert_to_tensorrconjrarIr4cast transposerhmatmulsqueeze) xrOr> event_axis sample_axisx_permedy_permedn_events n_samples x_permed_flat y_permed_flatcovrrr< covariancesD     rQcCs>|tj|ddd}|dur|tj|ddd}t|||dS)NrTr=rGrOr>)r stdrQrRrrr< correlation$srTc@seZdZddZdS) StopGradientcCs t|Sr)r stop_gradientr8rGrrr<call,rzStopGradient.callN)rrrrXrrrr<rU+s rUcCs tjttj||ddS)Nrr)r, activationsrr rnorm)rOy_predrrr<_default_landmark_loss0s r\csdeZdZ         dfdd Zdd Zdd d Zdd d ZddZddZddZ Z S)r[NFr r` umap_modelcstj|d||_||_||_| |_| |_||_||_||_ | |_ | |_ | |_ |p1t jjddd}|j|dt j|_t j|_|durQt jjdd|_n||_| dur]t|_ dS| |_ dS)N)rrrr)r2T) from_logits)rrrrrr!rrY umap_loss_a umap_loss_brr"r#r,r0r1compilerrflattenr SeedGeneratorseed_generatorlossesBinaryCrossentropyrZr\)r8r_r`rYrrr2rZrrr!rr"r#rr:rr<r7s2      zUMAPModel.__init__cCsZ|\}}||}||}||d}|jr+|jr||}n|t|}||d<|S)N) embedding_toembedding_fromrc)rrrrr rV)r8inputsto_xfrom_xrgrhr[embedding_to_reconrrr<rXis   zUMAPModel.callcKsg}|jD]}|tj|tjdq||||jdkr+|| |||j r7|| ||d|vrD|| ||t |S)Nr?rr)reappendr rCr,backendfloatx _umap_lossr!_global_correlation_lossr_parametric_reconstruction_loss_landmark_losssum)r8rGrOr[ sample_weightr9relossrrr< compute_loss|s   zUMAPModel.compute_losscCs|d}|d}tj||jdd}tj||jdd}t|d}tjjt||jd}tj dkr;t ||} n||} tj tj||ddtj|| ddgdd} t| |j|j} t|d} tj t| ft| |jfgdd} t| | |d\}}}t|S) Nrgrhrrseed tensorflowr)r)r rrYr4r,rrarangerdconfigrnrrrHrZrr_r`oneszerosrr)r8r[rrgrhembedding_neg_to repeat_negrepeat_neg_batch_dimshuffled_indicesembedding_neg_fromdistance_embeddingrrrattraction_lossrepellant_lossce_lossrrr<rpsB     zUMAPModel._umap_lossc Cs||d}||d}dd}||}||}t|dd}t|dd}tj|dd|dddd }tj|dd|dddd }|tjj|j|jd d }t t t |dt |dd }| |j S) NrrgcSs|t|t|Sr)r rrS)rGrrr<z_scoresz3UMAPModel._global_correlation_loss..z_scoreirrrrrxg|=)rGrO) rbr cliprZr,rrr4rdrFrTrpr!) r8rOr[rGz_xrdxdzcorr_drrr<rqs"" z"UMAPModel._global_correlation_losscCs||d|d}||jS)Nrc)rZr)r8rOr[rvrrr<rrs  z)UMAPModel._parametric_reconstruction_losscCsZ|d}tjt|t|d|dd}tjt|t||d}||||jS)Nrrg)x1x2)r whereisnanrvr"r#)r8rOr[y_toclean_y_pred_to clean_y_torrr<rss  zUMAPModel._landmark_loss) NNFr r`FNr r])NNNN)r ) rrrrrXrwrprqrrrsrrrr:r<r[6s"2  4r[cs$eZdZfddZddZZS)rcsJtt|t|d|_tdd|_tdd|_td||_dS)Nr) rrrnnLineardense1dense2dense3dense4)r8indimoutdimr:rr<rs zPumapNet.__init__cCsT||}t|}||}t|}||}t|}||}t|}|Sr)rFrrrrrWrrr<forwards        zPumapNet.forward)rrrrrrrrr:r<rs rcs|}tt|d}dd|DfddtdD}|}t|D]}|d|d|||d<t|d||||d<q*|D] }t ||||<qN| ||S) a4Copies weights from a parametric UMAP encoder to pytorch. Parameters ---------- km : encoder extracted from parametric UMAP. pm: a PumapNet object. Will be overwritten. Returns ------- pm : PumapNet Object. Net with copied weights. rcSsg|]}|qSrr)rrGrrr< 6sz!weight_copier..cs"g|] }d|ddqS)r.r)split)riall_keysrr<r7s"rz.biasz.weight) get_weightsr+rI state_dictryrrBrDr from_numpyload_state_dict)rrkweightsn_layerspm_namespyt_state_dictrrrrr<r's     r)r)r r )rr rr)NF)6numpyrBrrwarningsrrrnumbarr umap.spectralr sklearn.utilsrr r&rsklearn.neighborsr rzr ImportErrorr,r torch_importedrtorch.nnrtorch.nn.functional functionalr torch.onnx torchvisionr rrrrrurlrr<rQrTrLayerrUr\Modelr[Modulerrrrrr<s~             7 : 2 K k ( / ;K %!