o ci3=@sddlmZmZmZddlZddlZddlm Z ddl m Z m Z m Z mZmZddlmZddlmZeddGd d d eZeddGd d d eZeddGd ddeZeddGdddeZdS))ListOptionalTupleN)Dataset)AbsMaxMaxMeanMinStd) Preprocessor) PublicAPIalpha) stabilityc@ZeZdZdZddeedeeefddZdede fd d Z d e j fd d Z ddZdS)StandardScalerav Translate and scale each column by its mean and standard deviation, respectively. The general formula is given by .. math:: x' = \frac{x - \bar{x}}{s} where :math:`x` is the column, :math:`x'` is the transformed column, :math:`\bar{x}` is the column average, and :math:`s` is the column's sample standard deviation. If :math:`s = 0` (i.e., the column is constant-valued), then the transformed column will contain zeros. .. warning:: :class:`StandardScaler` works best when your data is normal. If your data isn't approximately normal, then the transformed features won't be meaningful. Examples: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import StandardScaler >>> >>> df = pd.DataFrame({"X1": [-2, 0, 2], "X2": [-3, -3, 3], "X3": [1, 1, 1]}) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> ds.to_pandas() # doctest: +SKIP X1 X2 X3 0 -2 -3 1 1 0 -3 1 2 2 3 1 Columns are scaled separately. >>> preprocessor = StandardScaler(columns=["X1", "X2"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 0 -1.224745 -0.707107 1 1 0.000000 -0.707107 1 2 1.224745 1.414214 1 Constant-valued columns get filled with zeros. >>> preprocessor = StandardScaler(columns=["X3"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 0 -2 -3 0.0 1 0 -3 0.0 2 2 3 0.0 >>> preprocessor = StandardScaler( ... columns=["X1", "X2"], ... output_columns=["X1_scaled", "X2_scaled"] ... ) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 X1_scaled X2_scaled 0 -2 -3 1 -1.224745 -0.707107 1 0 -3 1 0.000000 -0.707107 2 2 3 1 1.224745 1.414214 Args: columns: The columns to separately scale. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. Ncolumnsoutput_columnscC||_t|||_dSNrr #_derive_and_validate_output_columnsrselfrrrQ/home/ubuntu/.local/lib/python3.10/site-packages/ray/data/preprocessors/scaler.py__init__Q zStandardScaler.__init__datasetreturncCs:dd|jD}dd|jD}|jg||R|_|S)NcSg|]}t|qSr)r.0colrrr Xz'StandardScaler._fit..cSsg|]}t|ddqS)r)ddof)r r rrrr#Yr aggregatestats_)rrmean_aggregatesstd_aggregatesrrr_fitWszStandardScaler._fitdfc.dtjffdd }|j||j<|S)Nscsbjd|jd}jd|jd}|dus|dur%tj|dd<|S|dkr+d}|||S)Nzmean()zstd(r)r)namenpnan)r/s_means_stdrrrcolumn_standard_scaler^s z@StandardScaler._transform_pandas..column_standard_scalerpdSeriesr transformr)rr-r8rr7r_transform_pandas]sz StandardScaler._transform_pandascC|jjd|jd|jdSN (columns=z, output_columns=r0 __class____name__rrr7rrr__repr__pzStandardScaler.__repr__rrC __module__ __qualname____doc__rstrrrrr r,r: DataFramer=rDrrrrr s  C rc@r) MinMaxScaleraScale each column by its range. The general formula is given by .. math:: x' = \frac{x - \min(x)}{\max{x} - \min{x}} where :math:`x` is the column and :math:`x'` is the transformed column. If :math:`\max{x} - \min{x} = 0` (i.e., the column is constant-valued), then the transformed column will get filled with zeros. Transformed values are always in the range :math:`[0, 1]`. .. tip:: This can be used as an alternative to :py:class:`StandardScaler`. Examples: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import MinMaxScaler >>> >>> df = pd.DataFrame({"X1": [-2, 0, 2], "X2": [-3, -3, 3], "X3": [1, 1, 1]}) # noqa: E501 >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> ds.to_pandas() # doctest: +SKIP X1 X2 X3 0 -2 -3 1 1 0 -3 1 2 2 3 1 Columns are scaled separately. >>> preprocessor = MinMaxScaler(columns=["X1", "X2"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 0 0.0 0.0 1 1 0.5 0.0 1 2 1.0 1.0 1 Constant-valued columns get filled with zeros. >>> preprocessor = MinMaxScaler(columns=["X3"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 0 -2 -3 0.0 1 0 -3 0.0 2 2 3 0.0 >>> preprocessor = MinMaxScaler(columns=["X1", "X2"], output_columns=["X1_scaled", "X2_scaled"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 X1_scaled X2_scaled 0 -2 -3 1 0.0 0.0 1 0 -3 1 0.5 0.0 2 2 3 1 1.0 1.0 Args: columns: The columns to separately scale. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. NrrcCrrrrrrrrrzMinMaxScaler.__init__rrcs&fddttfD}|j|_S)Ncs g|] }jD]}||qqSr)r)r!Aggr"r7rrr#s z%MinMaxScaler._fit..)r rr(r)rr aggregatesrr7rr,s zMinMaxScaler._fitr-cr.)Nr/csHjd|jd}jd|jd}||}|dkrd}|||S)Nzmin(r0zmax(rr1r)r2)r/s_mins_maxdiffr7rrcolumn_min_max_scalers  z=MinMaxScaler._transform_pandas..column_min_max_scalerr9)rr-rTrr7rr=s zMinMaxScaler._transform_pandascCr>r?rAr7rrrrDrEzMinMaxScaler.__repr__rrFrrrrrLts  ? rLc@r) MaxAbsScalera@Scale each column by its absolute max value. The general formula is given by .. math:: x' = \frac{x}{\max{\vert x \vert}} where :math:`x` is the column and :math:`x'` is the transformed column. If :math:`\max{\vert x \vert} = 0` (i.e., the column contains all zeros), then the column is unmodified. .. tip:: This is the recommended way to scale sparse data. If you data isn't sparse, you can use :class:`MinMaxScaler` or :class:`StandardScaler` instead. Examples: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import MaxAbsScaler >>> >>> df = pd.DataFrame({"X1": [-6, 3], "X2": [2, -4], "X3": [0, 0]}) # noqa: E501 >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> ds.to_pandas() # doctest: +SKIP X1 X2 X3 0 -6 2 0 1 3 -4 0 Columns are scaled separately. >>> preprocessor = MaxAbsScaler(columns=["X1", "X2"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 0 -1.0 0.5 0 1 0.5 -1.0 0 Zero-valued columns aren't scaled. >>> preprocessor = MaxAbsScaler(columns=["X3"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 0 -6 2 0.0 1 3 -4 0.0 >>> preprocessor = MaxAbsScaler(columns=["X1", "X2"], output_columns=["X1_scaled", "X2_scaled"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 X1_scaled X2_scaled 0 -2 -3 1 -1.0 -1.0 1 0 -3 1 0.0 -1.0 2 2 3 1 1.0 1.0 Args: columns: The columns to separately scale. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. NrrcCrrrrrrrrrzMaxAbsScaler.__init__rrcCs dd|jD}|j||_|S)NcSrr)rr rrrr#r$z%MaxAbsScaler._fit..r'rNrrrr,s zMaxAbsScaler._fitr-cr.)Nr/cs(jd|jd}|dkrd}||S)Nzabs_max(r0rr1rP)r/ s_abs_maxr7rrcolumn_abs_max_scalersz=MaxAbsScaler._transform_pandas..column_abs_max_scalerr9)rr-rWrr7rr=s zMaxAbsScaler._transform_pandascCr>r?rAr7rrrrD*rEzMaxAbsScaler.__repr__rrFrrrrrUs  ; rUc @sjeZdZdZ  ddeedeeefdeeefddZ d e d e fd d Z d e jfddZddZdS) RobustScalera Scale and translate each column using quantiles. The general formula is given by .. math:: x' = \frac{x - \mu_{1/2}}{\mu_h - \mu_l} where :math:`x` is the column, :math:`x'` is the transformed column, :math:`\mu_{1/2}` is the column median. :math:`\mu_{h}` and :math:`\mu_{l}` are the high and low quantiles, respectively. By default, :math:`\mu_{h}` is the third quartile and :math:`\mu_{l}` is the first quartile. .. tip:: This scaler works well when your data contains many outliers. Examples: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import RobustScaler >>> >>> df = pd.DataFrame({ ... "X1": [1, 2, 3, 4, 5], ... "X2": [13, 5, 14, 2, 8], ... "X3": [1, 2, 2, 2, 3], ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> ds.to_pandas() # doctest: +SKIP X1 X2 X3 0 1 13 1 1 2 5 2 2 3 14 2 3 4 2 2 4 5 8 3 :class:`RobustScaler` separately scales each column. >>> preprocessor = RobustScaler(columns=["X1", "X2"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 0 -1.0 0.625 1 1 -0.5 -0.375 2 2 0.0 0.750 2 3 0.5 -0.750 2 4 1.0 0.000 3 >>> preprocessor = RobustScaler( ... columns=["X1", "X2"], ... output_columns=["X1_scaled", "X2_scaled"] ... ) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 X1_scaled X2_scaled 0 1 13 1 -1.0 0.625 1 2 5 2 -0.5 -0.375 2 3 14 2 0.0 0.750 3 4 2 2 0.5 -0.750 4 5 8 3 1.0 0.000 Args: columns: The columns to separately scale. quantile_range: A tuple that defines the lower and upper quantiles. Values must be between 0 and 1. Defaults to the 1st and 3rd quartiles: ``(0.25, 0.75)``. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. g?g?Nrquantile_rangercCs||_||_t|||_dSr)rrZr rr)rrrZrrrrrts  zRobustScaler.__init__rrcs|jd}d}|jd}|}|dfdd|||fD}i|_|jD]N|jfdddd }|}||\} }}}d td tfd d } | |} | |} | |} | |jdd<| |jdd<| |jdd<q&|S)Nrg?r1csg|]}t|qSr)int)r! percentile) max_indexrrr#r&z%RobustScaler._fit..cs |gSrr)r-)r"rrs z#RobustScaler._fit..pandas) batch_formatdsccSs|dd|S)Nr1r)take)rarbrrr_get_first_valuesz+RobustScaler._fit.._get_first_value low_quantile(r0median(high_quantile() rZcountr)r map_batchessortsplit_at_indicesrrJ)rrlowmedhigh num_records split_indicesfiltered_datasetsorted_dataset_rdlow_valmed_valhigh_valr)r"r]rr,s*        zRobustScaler._fitr-cr.)Nr/csbjd|jd}jd|jd}jd|jd}||}|dkr+t|S|||S)Nrer0rfrgr)r)r2r3 zeros_like)r/s_low_qs_medians_high_qrSr7rrcolumn_robust_scalers  z.column_robust_scalerr9)rr-r{rr7rr=s zRobustScaler._transform_pandascCs&|jjd|jd|jd|jdS)Nr@z, quantile_range=z), output_columns=r0)rBrCrrZrr7rrrrDs zRobustScaler.__repr__)rYN)rCrGrHrIrrJrfloatrrrr r,r:rKr=rDrrrrrX.sG    " rX)typingrrrnumpyr3r_r:ray.datarray.data.aggregaterrrr r ray.data.preprocessorr ray.util.annotationsr rrLrUrXrrrrs   g_Y