o $i4I@s$ddlmZmZmZmZmZmZddlZddl Z ddl m Z m Z mZmZmZmZddlmZmZddlmZddlmZerDddlmZedd ed d d Gd ddeZedd ed dd GdddeZedd ed dd GdddeZedd ed dd GdddeZdS)) TYPE_CHECKINGAnyDictListOptionalTupleN)AbsMaxApproximateQuantileMaxMeanMinStd) PreprocessorSerializablePreprocessorBase)SerializablePreprocessor) PublicAPI)Datasetalpha) stabilityz$io.ray.preprocessors.standard_scaler)version identifierceZdZdZddeedeeeffdd Zddd efd d Z d e j fd dZ d e eeffddZde eefdefddZddZZS)StandardScalerav Translate and scale each column by its mean and standard deviation, respectively. The general formula is given by .. math:: x' = \frac{x - \bar{x}}{s} where :math:`x` is the column, :math:`x'` is the transformed column, :math:`\bar{x}` is the column average, and :math:`s` is the column's sample standard deviation. If :math:`s = 0` (i.e., the column is constant-valued), then the transformed column will contain zeros. .. warning:: :class:`StandardScaler` works best when your data is normal. If your data isn't approximately normal, then the transformed features won't be meaningful. Examples: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import StandardScaler >>> >>> df = pd.DataFrame({"X1": [-2, 0, 2], "X2": [-3, -3, 3], "X3": [1, 1, 1]}) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> ds.to_pandas() # doctest: +SKIP X1 X2 X3 0 -2 -3 1 1 0 -3 1 2 2 3 1 Columns are scaled separately. >>> preprocessor = StandardScaler(columns=["X1", "X2"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 0 -1.224745 -0.707107 1 1 0.000000 -0.707107 1 2 1.224745 1.414214 1 Constant-valued columns get filled with zeros. >>> preprocessor = StandardScaler(columns=["X3"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 0 -2 -3 0.0 1 0 -3 0.0 2 2 3 0.0 >>> preprocessor = StandardScaler( ... columns=["X1", "X2"], ... output_columns=["X1_scaled", "X2_scaled"] ... ) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 X1_scaled X2_scaled 0 -2 -3 1 -1.224745 -0.707107 1 0 -3 1 0.000000 -0.707107 2 2 3 1 1.224745 1.414214 Args: columns: The columns to separately scale. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. Ncolumnsoutput_columnsc"t||_t|||_dSNsuper__init__rr#_derive_and_validate_output_columnsrselfrr __class__Z/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/ray/data/preprocessors/scaler.pyr U  zStandardScaler.__init__datasetrreturncCs,|jjt|jd|jjdd|jd|S)N) aggregator_fnrcSs t|ddS)Nr)ddof)r )colr&r&r'bs z%StandardScaler._fit..)stat_computation_planadd_aggregatorr r)r#r)r&r&r'_fit\szStandardScaler._fitdfc.dtjffdd }|j||j<|S)Nscsbjd|jd}jd|jd}|dus|dur%tj|dd<|S|dkr+d}|||S)Nzmean()zstd(rr)stats_namenpnan)r4s_means_stdr#r&r'column_standard_scalerhs z@StandardScaler._transform_pandas..column_standard_scalerpdSeriesr transformr)r#r2r=r&r<r'_transform_pandasgsz StandardScaler._transform_pandascC|j|jt|dddSN_fitted)rrrErrgetattrr<r&r&r'_get_serializable_fieldsz z'StandardScaler._get_serializable_fieldsfieldsrcC$|d|_|d|_|d|_dSNrrrErrgetrEr#rJrr&r&r'_set_serializable_fields  z'StandardScaler._set_serializable_fieldscC|jjd|jd|jdSN (columns=z, output_columns=r5r%__name__rrr<r&r&r'__repr__zStandardScaler.__repr__rrV __module__ __qualname____doc__rstrrr rr1r? DataFramerBrrrHintrPrW __classcell__r&r&r$r'rs$C rz#io.ray.preprocessors.min_max_scalercr) MinMaxScaleraScale each column by its range. The general formula is given by .. math:: x' = \frac{x - \min(x)}{\max{x} - \min{x}} where :math:`x` is the column and :math:`x'` is the transformed column. If :math:`\max{x} - \min{x} = 0` (i.e., the column is constant-valued), then the transformed column will get filled with zeros. Transformed values are always in the range :math:`[0, 1]`. .. tip:: This can be used as an alternative to :py:class:`StandardScaler`. Examples: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import MinMaxScaler >>> >>> df = pd.DataFrame({"X1": [-2, 0, 2], "X2": [-3, -3, 3], "X3": [1, 1, 1]}) # noqa: E501 >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> ds.to_pandas() # doctest: +SKIP X1 X2 X3 0 -2 -3 1 1 0 -3 1 2 2 3 1 Columns are scaled separately. >>> preprocessor = MinMaxScaler(columns=["X1", "X2"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 0 0.0 0.0 1 1 0.5 0.0 1 2 1.0 1.0 1 Constant-valued columns get filled with zeros. >>> preprocessor = MinMaxScaler(columns=["X3"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 0 -2 -3 0.0 1 0 -3 0.0 2 2 3 0.0 >>> preprocessor = MinMaxScaler(columns=["X1", "X2"], output_columns=["X1_scaled", "X2_scaled"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 X1_scaled X2_scaled 0 -2 -3 1 0.0 0.0 1 0 -3 1 0.5 0.0 2 2 3 1 1.0 1.0 Args: columns: The columns to separately scale. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. Nrrcrrrr"r$r&r'r r(zMinMaxScaler.__init__r)rr*cs&fddttfD}|j|_S)Ncs g|] }jD]}||qqSr&)r).0Aggr-r<r&r' s z%MinMaxScaler._fit..)r r aggregater6r#r) aggregatesr&r<r'r1s zMinMaxScaler._fitr2cr3)Nr4csHjd|jd}jd|jd}||}|dkrd}|||S)Nzmin(r5zmax(rrr6r7)r4s_mins_maxdiffr<r&r'column_min_max_scalers  z=MinMaxScaler._transform_pandas..column_min_max_scalerr>)r#r2rlr&r<r'rBs zMinMaxScaler._transform_pandascCrCrDrFr<r&r&r'rHrIz%MinMaxScaler._get_serializable_fieldsrJrcCrKrLrMrOr&r&r'rPrQz%MinMaxScaler._set_serializable_fieldscCrRrSrUr<r&r&r'rWrXzMinMaxScaler.__repr__rrYr&r&r$r'ras$?raz#io.ray.preprocessors.max_abs_scalercr) MaxAbsScalera@Scale each column by its absolute max value. The general formula is given by .. math:: x' = \frac{x}{\max{\vert x \vert}} where :math:`x` is the column and :math:`x'` is the transformed column. If :math:`\max{\vert x \vert} = 0` (i.e., the column contains all zeros), then the column is unmodified. .. tip:: This is the recommended way to scale sparse data. If you data isn't sparse, you can use :class:`MinMaxScaler` or :class:`StandardScaler` instead. Examples: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import MaxAbsScaler >>> >>> df = pd.DataFrame({"X1": [-6, 3], "X2": [2, -4], "X3": [0, 0]}) # noqa: E501 >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> ds.to_pandas() # doctest: +SKIP X1 X2 X3 0 -6 2 0 1 3 -4 0 Columns are scaled separately. >>> preprocessor = MaxAbsScaler(columns=["X1", "X2"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 0 -1.0 0.5 0 1 0.5 -1.0 0 Zero-valued columns aren't scaled. >>> preprocessor = MaxAbsScaler(columns=["X3"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 0 -6 2 0.0 1 3 -4 0.0 >>> preprocessor = MaxAbsScaler(columns=["X1", "X2"], output_columns=["X1_scaled", "X2_scaled"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 X1_scaled X2_scaled 0 -2 -3 1 -1.0 -1.0 1 0 -3 1 0.0 -1.0 2 2 3 1 1.0 1.0 Args: columns: The columns to separately scale. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. Nrrcrrrr"r$r&r'r :r(zMaxAbsScaler.__init__r)rr*cCs dd|jD}|j||_|S)NcSsg|]}t|qSr&)rrbr-r&r&r'rdBsz%MaxAbsScaler._fit..)rrer6rfr&r&r'r1As zMaxAbsScaler._fitr2cr3)Nr4cs(jd|jd}|dkrd}||S)Nzabs_max(r5rrrh)r4 s_abs_maxr<r&r'column_abs_max_scalerGsz=MaxAbsScaler._transform_pandas..column_abs_max_scalerr>)r#r2rpr&r<r'rBFs zMaxAbsScaler._transform_pandascCrCrDrFr<r&r&r'rHTrIz%MaxAbsScaler._get_serializable_fieldsrJrcCrKrLrMrOr&r&r'rP[rQz%MaxAbsScaler._set_serializable_fieldscCrRrSrUr<r&r&r'rWbrXzMaxAbsScaler.__repr__rrYr&r&r$r'rms$;rmz"io.ray.preprocessors.robust_scalerc seZdZdZdZddefdeedeeefde eede ffd d Z d d d e fddZ dejfddZd eeeffddZdeeefde fddZddZZS) RobustScaleraI Scale and translate each column using approximate quantiles. The general formula is given by .. math:: x' = \frac{x - \mu_{1/2}}{\mu_h - \mu_l} where :math:`x` is the column, :math:`x'` is the transformed column, :math:`\mu_{1/2}` is the column median. :math:`\mu_{h}` and :math:`\mu_{l}` are the high and low quantiles, respectively. By default, :math:`\mu_{h}` is the third quartile and :math:`\mu_{l}` is the first quartile. Internally, the `ApproximateQuantile` aggregator is used to calculate the approximate quantiles. .. tip:: This scaler works well when your data contains many outliers. Examples: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import RobustScaler >>> >>> df = pd.DataFrame({ ... "X1": [1, 2, 3, 4, 5], ... "X2": [13, 5, 14, 2, 8], ... "X3": [1, 2, 2, 2, 3], ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> ds.to_pandas() # doctest: +SKIP X1 X2 X3 0 1 13 1 1 2 5 2 2 3 14 2 3 4 2 2 4 5 8 3 :class:`RobustScaler` separately scales each column. >>> preprocessor = RobustScaler(columns=["X1", "X2"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 0 -1.0 0.625 1 1 -0.5 -0.375 2 2 0.0 0.750 2 3 0.5 -0.750 2 4 1.0 0.000 3 >>> preprocessor = RobustScaler( ... columns=["X1", "X2"], ... output_columns=["X1_scaled", "X2_scaled"] ... ) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 X1_scaled X2_scaled 0 1 13 1 -1.0 0.625 1 2 5 2 -0.5 -0.375 2 3 14 2 0.0 0.750 3 4 2 2 0.5 -0.750 4 5 8 3 1.0 0.000 Args: columns: The columns to separately scale. quantile_range: A tuple that defines the lower and upper quantiles. Values must be between 0 and 1. Defaults to the 1st and 3rd quartiles: ``(0.25, 0.75)``. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. quantile_precision: Controls the accuracy and memory footprint of the sketch (K in KLL); higher values yield lower error but use more memory. Defaults to 800. See https://datasketches.apache.org/docs/KLL/KLLAccuracyAndSize.html for details on accuracy and size. i )g?g?Nrquantile_rangerquantile_precisioncs.t||_||_||_t|||_dSr)rr rrrrsrr!r)r#rrrrrsr$r&r'r s  zRobustScaler.__init__r)rr*csjddjdgfddjD}|j|}i_jD](}|d|d\}}}|jd|d<|jd |d<|jd |d<q!S) Nrg?rcsg|] }t|jdqS))on quantilesrs)r rsrnrur#r&r'rdsz%RobustScaler._fit..zapprox_quantile(r5 low_quantile(median(high_quantile()rrrrer6)r#r)rg aggregatedr-low_qmed_qhigh_qr&rvr'r1s   zRobustScaler._fitr2cr3)Nr4csbjd|jd}jd|jd}jd|jd}||}|dkr+t|S|||S)Nrwr5rxryr)r6r7r8 zeros_like)r4s_low_qs_medians_high_qrkr<r&r'column_robust_scalers  z.column_robust_scalerr>)r#r2rr&r<r'rBs zRobustScaler._transform_pandascCs |j|j|j|jt|dddS)NrE)rrrrrsrE)rrrrrsrGr<r&r&r'rHs  z%RobustScaler._get_serializable_fieldsrJrcCs8|d|_|d|_|d|_|d|_|d|_dS)NrrrrrsrE)rrrrrsrNrErOr&r&r'rPs    z%RobustScaler._set_serializable_fieldscCs&|jjd|jd|jd|jdS)NrTz, quantile_range=z), output_columns=r5)r%rVrrrrr<r&r&r'rWs zRobustScaler.__repr__)rVrZr[r\DEFAULT_QUANTILE_PRECISIONrr]rfloatrr_r rr1r?r^rBrrrHrPrWr`r&r&r$r'rqfs(K    rq)typingrrrrrrnumpyr8pandasr?ray.data.aggregaterr r r r r ray.data.preprocessorrr&ray.data.preprocessors.version_supportrray.util.annotationsrray.data.datasetrrrarmrqr&r&r&r's*      { n h