o Ni6@s&ddlmZddlZGdddZdS))OptionalNc@seZdZdZ        d(deedeedeed eed ed eed ed eefddZddZ ddZ ddZ de j fddZedee j e j e j ffddZed)dee j e j ffddZd)ddZd)dd Zde j fd!d"Zed*d$eded%efd&d'ZdS)+IncrementalPCAay An implementation of Incremental Principal Components Analysis (IPCA) that leverages PyTorch for GPU acceleration. Adapted from https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/decomposition/_incremental_pca.py This class provides methods to fit the model on data incrementally in batches, and to transform new data based on the principal components learned during the fitting process. Args: n_components (int, optional): Number of components to keep. If `None`, it's set to the minimum of the number of samples and features. Defaults to None. copy (bool): If False, input data will be overwritten. Defaults to True. batch_size (int, optional): The number of samples to use for each batch. Only needed if self.fit is called. If `None`, it's inferred from the data and set to `5 * n_features`. Defaults to None. svd_driver (str, optional): name of the cuSOLVER method to be used for torch.linalg.svd. This keyword argument only works on CUDA inputs. Available options are: None, gesvd, gesvdj, and gesvda. Defaults to None. lowrank (bool, optional): Whether to use torch.svd_lowrank instead of torch.linalg.svd which can be faster. Defaults to False. lowrank_q (int, optional): For an adequate approximation of n_components, this parameter defaults to n_components * 2. lowrank_niter (int, optional): Number of subspace iterations to conduct for torch.svd_lowrank. Defaults to 4. lowrank_seed (int, optional): Seed for making results of torch.svd_lowrank reproducible. NTF n_componentscopy batch_size svd_driverlowrank lowrank_q lowrank_niter lowrank_seedc CsL||_||_||_||_||_||_||_||_d|_|jr$| dSdS)N) rrrrr r r r n_features__validate_lowrank_params) selfrrrrr r r r rN/home/ubuntu/.local/lib/python3.10/site-packages/peft/utils/incremental_pca.py__init__.s  zIncrementalPCA.__init__cCsD|jdur|jdurtd|jd|_dS|j|jkr tddS)NzKn_components must be specified when using lowrank mode with lowrank_q=None.z8lowrank_q must be greater than or equal to n_components.)r r ValueError)rrrrrGs   z'IncrementalPCA._validate_lowrank_paramscCstjj|d|jdS)NF) full_matricesdriver)torchlinalgsvdrrXrrr _svd_fn_fullOszIncrementalPCA._svd_fn_fullcCsr|jdu}tjj|d#|rt|jtj||j|jd\}}}|||jfWdS1s2wYdS)N)enabled)qniter) r rrandomfork_rng manual_seed svd_lowrankr r mH)rr seed_enabledUSVrrr_svd_fn_lowrankRs   $zIncrementalPCA._svd_fn_lowrankreturncCstjtjg}t|tjstj|tjd}n|jr|}|j\}}|j dur'n!|j |kr8t d|j d|d|j |krHt d|j d||j |vrS| tj}|S)z Validates and converts the input data `X` to the appropriate tensor format. Args: X (torch.Tensor): Input data. Returns: torch.Tensor: Converted to appropriate format. )dtypeNz n_components=z invalid for n_features=z<, need more rows than columns for IncrementalPCA processing.z6 must be less or equal to the batch number of samples ) rfloat32float64 isinstanceTensortensorrcloneshaperrr+to)rr valid_dtypes n_samples n_featuresrrr_validate_dataZs&       zIncrementalPCA._validate_datacCsH|jddkr |||fS|dkr |durtd|dur tdtj|jdg|jd}||}|durBtj|jdtj|jd}n||}|jdtjd}|||}||} || } | jdtjd} | | jdtjd} | | |8} |dur| |} n||}| |}|| |||||}||} || |fS) aj Computes the incremental mean and variance for the data `X`. Args: X (torch.Tensor): The batch input data tensor with shape (n_samples, n_features). last_mean (torch.Tensor): The previous mean tensor with shape (n_features,). last_variance (torch.Tensor): The previous variance tensor with shape (n_features,). last_sample_count (torch.Tensor): The count tensor of samples processed before the current batch. Returns: Tuple[torch.Tensor, torch.Tensor, torch.Tensor]: Updated mean, variance tensors, and total sample count. rNz6last_mean should not be None if last_sample_count > 0.z:last_variance should not be None if last_sample_count > 0.device)r+r9)dimr+) r2rrr0r9zerosr-sumsquaresquare_double)r last_mean last_variancelast_sample_countnew_sample_countupdated_sample_countlast_sumnew_sum updated_meanTtemp correctionnew_unnormalized_varianceupdated_variancelast_unnormalized_variancelast_over_new_countupdated_unnormalized_variancerrr_incremental_mean_and_var}s@      z(IncrementalPCA._incremental_mean_and_varcCs|rtjt|dd}t||t|jdf}ntjt|dd}t|t|jd|f}||d|jddd9}||dd9}||fS)a[ Adjusts the signs of the singular vectors from the SVD decomposition for deterministic output. This method ensures that the output remains consistent across different runs. Args: u (torch.Tensor): Left singular vectors tensor. v (torch.Tensor): Right singular vectors tensor. u_based_decision (bool, optional): If True, uses the left singular vectors to determine the sign flipping. Defaults to True. Returns: Tuple[torch.Tensor, torch.Tensor]: Adjusted left and right singular vectors tensors. rr;r:N)rargmaxabssignranger2view)uvu_based_decision max_abs_colssigns max_abs_rowsrrr _svd_flipszIncrementalPCA._svd_flipcCs`|r||}|j\}}|jdurd||_|j||j|jpddD] }|j||ddq"|S)a\ Fits the model with data `X` using minibatches of size `batch_size`. Args: X (torch.Tensor): The input data tensor with shape (n_samples, n_features). check_input (bool, optional): If True, validates the input. Defaults to True. Returns: IncrementalPCA: The fitted IPCA model. Nr)min_batch_sizeF) check_input)r7r2r gen_batchesr partial_fit)rrrbr5r6batchrrrfits    zIncrementalPCA.fitcCst|d }|r ||}|j\}}|r0d|_d|_tjdg|jd|_||_ |j s0t |||_ ||j kr9t d| ||j|j|j\}}}|rN||8}n-tj|dd} || 8}t|j||} | |j| } t|jd|j|| f}|jr||\} } }n||\} } }|j| |dd \} }| d |d }| d t||}||_|d|j |_| d|j |_||_||_|d|j |_|d|j |_|j ||fvr||j d|_|Stjd |jd|_|S) ak Incrementally fits the model with batch data `X`. Args: X (torch.Tensor): The batch input data tensor with shape (n_samples, n_features). check_input (bool, optional): If True, validates the input. Defaults to True. Returns: IncrementalPCA: The updated IPCA model after processing the batch. components_Nrr8z]Number of features of the new batch does not match the number of features of the first batch.rR)rSr:F)r[rr:g)hasattrr7r2mean_var_rr0r9n_samples_seen_r rminrrQmeansqrtr@vstacksingular_values_rXrgr r)rr_r=explained_variance_explained_variance_ratio_noise_variance_)rrrb first_passr5r6col_meancol_varn_total_samplescol_batch_meanmean_correction_factormean_correctionr&r'Vtexplained_varianceexplained_variance_ratiorrrrds`       zIncrementalPCA.partial_fitcCs&||j}t||jj|jS)a Applies dimensionality reduction to `X`. The input data `X` is projected on the first principal components previously extracted from a training set. Args: X (torch.Tensor): New data tensor with shape (n_samples, n_features) to be transformed. Returns: torch.Tensor: Transformed data tensor with shape (n_samples, n_components). )rirmmr@rgrIr3r+rrrr transform-s zIncrementalPCA.transformrnraccs^d}tt||D]}||}|||krq t||V|}q ||kr-t||VdSdS)aGenerator to create slices containing `batch_size` elements from 0 to `n`. The last slice may contain less than `batch_size` elements, when `batch_size` does not divide `n`. Args: n (int): Size of the sequence. batch_size (int): Number of elements in each batch. min_batch_size (int, optional): Minimum number of elements in each batch. Defaults to 0. Yields: slice: A slice of `batch_size` elements. rN)rWintslice)rrrastart_endrrrrc<s  zIncrementalPCA.gen_batches)NTNNFNrN)T)r)__name__ __module__ __qualname____doc__rrboolstrrrrr)rr/r7 staticmethodtuplerQr_rfrdrrcrrrrrsT # 8  Gr)typingrrrrrrrs