o ci2? @sddlmZmZmZmZmZmZddlZddl Z ddl m Z ddl mZmZddlmZddlmZGdddeZed d Gd d d eZed d Gd ddeZdededededeef ddZdS))DictIterableListOptionalTypeUnionN)Dataset)MaxMin) Preprocessor) PublicAPIc@s0eZdZdZdejfddZddZddZd S) _AbstractKBinsDiscretizerzAbstract base class for all KBinsDiscretizers. Essentially a thin wraper around ``pd.cut``. Expects either ``self.stats_`` or ``self.bins`` to be set and contain {column:list_of_bin_intervals}. dfcs<dtjdtjffdd }|j|dd}|j|j<|S)Nsreturnc s|jjvr|Sjrj|jnd}d}|r)t|tjr'|j}t|j }nd}j r/j nj }tj |t|tr?||jn|j||djjdS)NFT)rightlabelsorderedretbinsinclude_lowest duplicates)namecolumnsdtypesget isinstancepdCategoricalDtyperlist categories _is_fittablestats_binscutdictrrr)rrrr"selfV/home/ubuntu/.local/lib/python3.10/site-packages/ray/data/preprocessors/discretizer.py bin_valuess(   z?_AbstractKBinsDiscretizer._transform_pandas..bin_valuesr)axis)rSeriesapplyroutput_columns)r&rr) binned_dfr'r%r(_transform_pandassz+_AbstractKBinsDiscretizer._transform_pandascs4tjtrtfddjDstddSdS)Nc3s|]}|jvVqdSN)r").0colr%r'r( 3s  zC_AbstractKBinsDiscretizer._validate_bins_columns..zKIf `bins` is a dictionary, all elements of `columns` must be present in it.)rr"r$allr ValueErrorr%r'r%r(_validate_bins_columns2sz0_AbstractKBinsDiscretizer._validate_bins_columnscCs0dddt|D}|jjd|dS)Nz, cSs(g|]\}}|ds|d|qS)_=) startswith)r1 attr_name attr_valuer'r'r( =s  z6_AbstractKBinsDiscretizer.__repr__..())joinvarsitems __class____name__)r&attr_strr'r'r(__repr__;s  z"_AbstractKBinsDiscretizer.__repr__N) rC __module__ __qualname____doc__r DataFramer/r6rEr'r'r'r(r s  r alpha) stabilityc@seZdZdZdddddddeedeeee j e eeeee j fffd e d e d ed e e eee jeejffd e eefddZdZdS)CustomKBinsDiscretizerakBin values into discrete intervals using custom bin edges. Columns must contain numerical values. Examples: Use :class:`CustomKBinsDiscretizer` to bin continuous features. >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import CustomKBinsDiscretizer >>> df = pd.DataFrame({ ... "value_1": [0.2, 1.4, 2.5, 6.2, 9.7, 2.1], ... "value_2": [10, 15, 13, 12, 23, 25], ... }) >>> ds = ray.data.from_pandas(df) >>> discretizer = CustomKBinsDiscretizer( ... columns=["value_1", "value_2"], ... bins=[0, 1, 4, 10, 25] ... ) >>> discretizer.transform(ds).to_pandas() value_1 value_2 0 0 2 1 1 3 2 1 3 3 2 3 4 2 3 5 1 3 :class:`CustomKBinsDiscretizer` can also be used in append mode by providing the name of the output_columns that should hold the encoded values. >>> discretizer = CustomKBinsDiscretizer( ... columns=["value_1", "value_2"], ... bins=[0, 1, 4, 10, 25], ... output_columns=["value_1_discretized", "value_2_discretized"] ... ) >>> discretizer.fit_transform(ds).to_pandas() # doctest: +SKIP value_1 value_2 value_1_discretized value_2_discretized 0 0.2 10 0 2 1 1.4 15 1 3 2 2.5 13 1 3 3 6.2 12 2 3 4 9.7 23 2 3 5 2.1 25 1 3 You can also specify different bin edges per column. >>> discretizer = CustomKBinsDiscretizer( ... columns=["value_1", "value_2"], ... bins={"value_1": [0, 1, 4], "value_2": [0, 18, 35, 70]}, ... ) >>> discretizer.transform(ds).to_pandas() value_1 value_2 0 0.0 0 1 1.0 0 2 1.0 0 3 NaN 0 4 NaN 1 5 1.0 1 Args: columns: The columns to discretize. bins: Defines custom bin edges. Can be an iterable of numbers, a ``pd.IntervalIndex``, or a dict mapping columns to either of them. Note that ``pd.IntervalIndex`` for bins must be non-overlapping. right: Indicates whether bins include the rightmost edge. include_lowest: Indicates whether the first interval should be left-inclusive. duplicates: Can be either 'raise' or 'drop'. If bin edges are not unique, raise ``ValueError`` or drop non-uniques. dtypes: An optional dictionary that maps columns to ``pd.CategoricalDtype`` objects or ``np.integer`` types. If you don't include a column in ``dtypes`` or specify it as an integer dtype, the outputted column will consist of ordered integers corresponding to bins. If you use a ``pd.CategoricalDtype``, the outputted column will be a ``pd.CategoricalDtype`` with the categories being mapped to bins. You can use ``pd.CategoricalDtype(categories, ordered=True)`` to preserve information about bin order. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. .. seealso:: :class:`UniformKBinsDiscretizer` If you want to bin data into uniform width bins. TFraiseNrrrrr-rr"rrrrr-cCs>||_||_||_||_||_||_t|||_| dSr0) rr"rrrrr #_derive_and_validate_output_columnsr-r6r&rr"rrrrr-r'r'r(__init__s zCustomKBinsDiscretizer.__init__)rCrFrGrHrstrrrfloatr IntervalIndexrboolrrrnpintegerrQr r'r'r'r(rLFs<b      rLc@seZdZdZdddddddeedeeeeeffd e d e d ed e eeee j e ejffd e eefddZdedefddZddZdefddZdS)UniformKBinsDiscretizerarBin values into discrete intervals (bins) of uniform width. Columns must contain numerical values. Examples: Use :class:`UniformKBinsDiscretizer` to bin continuous features. >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import UniformKBinsDiscretizer >>> df = pd.DataFrame({ ... "value_1": [0.2, 1.4, 2.5, 6.2, 9.7, 2.1], ... "value_2": [10, 15, 13, 12, 23, 25], ... }) >>> ds = ray.data.from_pandas(df) >>> discretizer = UniformKBinsDiscretizer( ... columns=["value_1", "value_2"], bins=4 ... ) >>> discretizer.fit_transform(ds).to_pandas() value_1 value_2 0 0 0 1 0 1 2 0 0 3 2 0 4 3 3 5 0 3 :class:`UniformKBinsDiscretizer` can also be used in append mode by providing the name of the output_columns that should hold the encoded values. >>> discretizer = UniformKBinsDiscretizer( ... columns=["value_1", "value_2"], ... bins=4, ... output_columns=["value_1_discretized", "value_2_discretized"] ... ) >>> discretizer.fit_transform(ds).to_pandas() # doctest: +SKIP value_1 value_2 value_1_discretized value_2_discretized 0 0.2 10 0 0 1 1.4 15 0 1 2 2.5 13 0 0 3 6.2 12 2 0 4 9.7 23 3 3 5 2.1 25 0 3 You can also specify different number of bins per column. >>> discretizer = UniformKBinsDiscretizer( ... columns=["value_1", "value_2"], bins={"value_1": 4, "value_2": 3} ... ) >>> discretizer.fit_transform(ds).to_pandas() value_1 value_2 0 0 0 1 0 0 2 0 0 3 2 0 4 3 2 5 0 2 Args: columns: The columns to discretize. bins: Defines the number of equal-width bins. Can be either an integer (which will be applied to all columns), or a dict that maps columns to integers. The range is extended by .1% on each side to include the minimum and maximum values. right: Indicates whether bins includes the rightmost edge or not. include_lowest: Whether the first interval should be left-inclusive or not. duplicates: Can be either 'raise' or 'drop'. If bin edges are not unique, raise ``ValueError`` or drop non-uniques. dtypes: An optional dictionary that maps columns to ``pd.CategoricalDtype`` objects or ``np.integer`` types. If you don't include a column in ``dtypes`` or specify it as an integer dtype, the outputted column will consist of ordered integers corresponding to bins. If you use a ``pd.CategoricalDtype``, the outputted column will be a ``pd.CategoricalDtype`` with the categories being mapped to bins. You can use ``pd.CategoricalDtype(categories, ordered=True)`` to preserve information about bin order. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. .. seealso:: :class:`CustomKBinsDiscretizer` If you want to specify your own bin edges. TFrMNrNrr"rrrrr-cCs6||_||_||_||_||_||_t|||_dSr0) rr"rrrrr rOr-rPr'r'r(rQs  z UniformKBinsDiscretizer.__init__datasetrc Cs|i}g}t|jtr|j}n|j}|D] }|||q|j|}i}i}| D]\} } | dd} | drD| || <| drM| || <q1|D]}t|jtr_|j|n|j} t ||||| |j ||<qR||_ |S)Nminmax)_validate_on_fitrr"r$keysrextend._fit_uniform_covert_bin_to_aggregate_if_needed aggregaterAr9._translate_min_max_number_of_bins_to_bin_edgesrr!) r&rYstats aggregatesrcolumnaggregate_statsminsmaxeskeyvalue column_namer"r'r'r(_fit4s6        zUniformKBinsDiscretizer._fitcCs |dSr0)r6r%r'r'r(r^Us z(UniformKBinsDiscretizer._validate_on_fitrfcCsDt|jtr |j|n|j}t|trt|t|fStd|)Nz5`bins` must be an integer or a dict of integers, got )rr"r$intr r TypeError)r&rfr"r'r'r(raXs  zFUniformKBinsDiscretizer._fit_uniform_covert_bin_to_aggregate_if_needed)rCrFrGrHrrRrrnrrUrrrrrVrWrQrr rmr^rar'r'r'r(rXs6_ !rXmnmxr"rrcCs||f}dd|D\}}t|st|rtd||krH||dkr*dt|nd8}||dkr8dt|nd7}tj|||ddd}|Stj|||ddd}||d}|re|d|8<|S|d |7<|S) zETranslates a range and desired number of bins into list of bin edges.css|]}|dVqdS)gNr')r1mir'r'r(r3szA_translate_min_max_number_of_bins_to_bin_edges..z@Cannot specify integer `bins` when input data contains infinity.rgMbP?T)endpointr[)rVisinfr5abslinspace)rprqr"rrngadjr'r'r(rcs$ rc)typingrrrrrrnumpyrVpandasrray.datarray.data.aggregater r ray.data.preprocessorr ray.util.annotationsr r rLrXrSrnrUrcr'r'r'r(s0    :zD