o `۷i@ @sddlmZmZmZmZmZmZmZddlZ ddl Z ddl m Z mZddlmZddlmZer6ddlmZGdddeZed d Gd d d eZed d Gd ddeZdedeeefdefddZdededededeef ddZdS)) TYPE_CHECKINGDictIterableListOptionalTypeUnionN)MaxMin) Preprocessor) PublicAPI)Datasetc@s0eZdZdZdejfddZddZddZd S) _AbstractKBinsDiscretizerzAbstract base class for all KBinsDiscretizers. Essentially a thin wraper around ``pd.cut``. Expects either ``self.stats_`` or ``self.bins`` to be set and contain {column:list_of_bin_intervals}. dfcs<dtjdtjffdd }|j|dd}|j|j<|S)Nsreturnc s|jjvr|Sjrj|jnd}d}|r)t|tjr'|j}t|j }nd}j r/j nj }tj |t|tr?||jn|j||djjdS)NFT)rightlabelsorderedretbinsinclude_lowest duplicates)namecolumnsdtypesget isinstancepdCategoricalDtyperlist categories _is_fittablestats_binscutdictrrr)rrrr#selfX/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/data/preprocessors/discretizer.py bin_valuess(   z?_AbstractKBinsDiscretizer._transform_pandas..bin_valuesr)axis)rSeriesapplyroutput_columns)r'rr* binned_dfr(r&r)_transform_pandassz+_AbstractKBinsDiscretizer._transform_pandascs4tjtrtfddjDstddSdS)Nc3s|]}|jvVqdSN)r#).0colr&r(r) 5s  zC_AbstractKBinsDiscretizer._validate_bins_columns..zKIf `bins` is a dictionary, all elements of `columns` must be present in it.)rr#r%allr ValueErrorr&r(r&r)_validate_bins_columns4sz0_AbstractKBinsDiscretizer._validate_bins_columnscCs0dddt|D}|jjd|dS)Nz, cSs(g|]\}}|ds|d|qS)_=) startswith)r2 attr_name attr_valuer(r(r) ?s  z6_AbstractKBinsDiscretizer.__repr__..())joinvarsitems __class____name__)r'attr_strr(r(r)__repr__=s  z"_AbstractKBinsDiscretizer.__repr__N) rD __module__ __qualname____doc__r DataFramer0r7rFr(r(r(r)rs  ralpha) stabilityc@seZdZdZdddddddeedeeee j e eeeee j fffd e d e d ed e e eee jeejffd e eefddZdZdS)CustomKBinsDiscretizerakBin values into discrete intervals using custom bin edges. Columns must contain numerical values. Examples: Use :class:`CustomKBinsDiscretizer` to bin continuous features. >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import CustomKBinsDiscretizer >>> df = pd.DataFrame({ ... "value_1": [0.2, 1.4, 2.5, 6.2, 9.7, 2.1], ... "value_2": [10, 15, 13, 12, 23, 25], ... }) >>> ds = ray.data.from_pandas(df) >>> discretizer = CustomKBinsDiscretizer( ... columns=["value_1", "value_2"], ... bins=[0, 1, 4, 10, 25] ... ) >>> discretizer.transform(ds).to_pandas() value_1 value_2 0 0 2 1 1 3 2 1 3 3 2 3 4 2 3 5 1 3 :class:`CustomKBinsDiscretizer` can also be used in append mode by providing the name of the output_columns that should hold the encoded values. >>> discretizer = CustomKBinsDiscretizer( ... columns=["value_1", "value_2"], ... bins=[0, 1, 4, 10, 25], ... output_columns=["value_1_discretized", "value_2_discretized"] ... ) >>> discretizer.fit_transform(ds).to_pandas() # doctest: +SKIP value_1 value_2 value_1_discretized value_2_discretized 0 0.2 10 0 2 1 1.4 15 1 3 2 2.5 13 1 3 3 6.2 12 2 3 4 9.7 23 2 3 5 2.1 25 1 3 You can also specify different bin edges per column. >>> discretizer = CustomKBinsDiscretizer( ... columns=["value_1", "value_2"], ... bins={"value_1": [0, 1, 4], "value_2": [0, 18, 35, 70]}, ... ) >>> discretizer.transform(ds).to_pandas() value_1 value_2 0 0.0 0 1 1.0 0 2 1.0 0 3 NaN 0 4 NaN 1 5 1.0 1 Args: columns: The columns to discretize. bins: Defines custom bin edges. Can be an iterable of numbers, a ``pd.IntervalIndex``, or a dict mapping columns to either of them. Note that ``pd.IntervalIndex`` for bins must be non-overlapping. right: Indicates whether bins include the rightmost edge. include_lowest: Indicates whether the first interval should be left-inclusive. duplicates: Can be either 'raise' or 'drop'. If bin edges are not unique, raise ``ValueError`` or drop non-uniques. dtypes: An optional dictionary that maps columns to ``pd.CategoricalDtype`` objects or ``np.integer`` types. If you don't include a column in ``dtypes`` or specify it as an integer dtype, the outputted column will consist of ordered integers corresponding to bins. If you use a ``pd.CategoricalDtype``, the outputted column will be a ``pd.CategoricalDtype`` with the categories being mapped to bins. You can use ``pd.CategoricalDtype(categories, ordered=True)`` to preserve information about bin order. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. .. seealso:: :class:`UniformKBinsDiscretizer` If you want to bin data into uniform width bins. TFraiseNrrrrr.rr#rrrrr.cCs>||_||_||_||_||_||_t|||_| dSr1) rr#rrrrr #_derive_and_validate_output_columnsr.r7r'rr#rrrrr.r(r(r)__init__s zCustomKBinsDiscretizer.__init__)rDrGrHrIrstrrrfloatr IntervalIndexrboolrrrnpintegerrRr!r(r(r(r)rMHs<b      rMcseZdZdZdddddddeedeeeeeffd e d e d ed e eeee j e ejffd e eeffddZdddefddZddZdddZZS)UniformKBinsDiscretizerarBin values into discrete intervals (bins) of uniform width. Columns must contain numerical values. Examples: Use :class:`UniformKBinsDiscretizer` to bin continuous features. >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import UniformKBinsDiscretizer >>> df = pd.DataFrame({ ... "value_1": [0.2, 1.4, 2.5, 6.2, 9.7, 2.1], ... "value_2": [10, 15, 13, 12, 23, 25], ... }) >>> ds = ray.data.from_pandas(df) >>> discretizer = UniformKBinsDiscretizer( ... columns=["value_1", "value_2"], bins=4 ... ) >>> discretizer.fit_transform(ds).to_pandas() value_1 value_2 0 0 0 1 0 1 2 0 0 3 2 0 4 3 3 5 0 3 :class:`UniformKBinsDiscretizer` can also be used in append mode by providing the name of the output_columns that should hold the encoded values. >>> discretizer = UniformKBinsDiscretizer( ... columns=["value_1", "value_2"], ... bins=4, ... output_columns=["value_1_discretized", "value_2_discretized"] ... ) >>> discretizer.fit_transform(ds).to_pandas() # doctest: +SKIP value_1 value_2 value_1_discretized value_2_discretized 0 0.2 10 0 0 1 1.4 15 0 1 2 2.5 13 0 0 3 6.2 12 2 0 4 9.7 23 3 3 5 2.1 25 0 3 You can also specify different number of bins per column. >>> discretizer = UniformKBinsDiscretizer( ... columns=["value_1", "value_2"], bins={"value_1": 4, "value_2": 3} ... ) >>> discretizer.fit_transform(ds).to_pandas() value_1 value_2 0 0 0 1 0 0 2 0 0 3 2 0 4 3 2 5 0 2 Args: columns: The columns to discretize. bins: Defines the number of equal-width bins. Can be either an integer (which will be applied to all columns), or a dict that maps columns to integers. The range is extended by .1% on each side to include the minimum and maximum values. right: Indicates whether bins includes the rightmost edge or not. include_lowest: Whether the first interval should be left-inclusive or not. duplicates: Can be either 'raise' or 'drop'. If bin edges are not unique, raise ``ValueError`` or drop non-uniques. dtypes: An optional dictionary that maps columns to ``pd.CategoricalDtype`` objects or ``np.integer`` types. If you don't include a column in ``dtypes`` or specify it as an integer dtype, the outputted column will consist of ordered integers corresponding to bins. If you use a ``pd.CategoricalDtype``, the outputted column will be a ``pd.CategoricalDtype`` with the categories being mapped to bins. You can use ``pd.CategoricalDtype(categories, ordered=True)`` to preserve information about bin order. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. .. seealso:: :class:`CustomKBinsDiscretizer` If you want to specify your own bin edges. TFrNNrOrr#rrrrr.cs@t||_||_||_||_||_||_t |||_ dSr1) superrRrr#rrrrr rPr.rQrCr(r)rRs  z UniformKBinsDiscretizer.__init__datasetr rcCs|t|jtr|j}n|j}|D]}t|jtr"|j|n|j}t|ts1td|q|jj t |d|jj t |d|S)Nz5`bins` must be an integer or a dict of integers, got ) aggregator_fnr) _validate_on_fitrr#r%keysrint TypeErrorstat_computation_planadd_aggregatorr r )r'r\rcolumnr#r(r(r)_fit7s(   zUniformKBinsDiscretizer._fitcCs |dSr1)r7r&r(r(r)r^Qs z(UniformKBinsDiscretizer._validate_on_fitcCs"|j|}t||j|j|_|Sr1)rbcomputepost_fit_processorr#rr")r'r\statsr(r(r) _fit_executeTs z$UniformKBinsDiscretizer._fit_execute)r\r )rDrGrHrIrrSrr`rrVrrrrrWrXrRr rer^ri __classcell__r(r(r[r)rYs6_ rYaggregate_statsr#rc Csiii}}}|D]\}}|dd}|dr|||<|dr(|||<q |D]} t|| || t|tr?|| n||d|| <q-|S)Nminmax)mnmxr#r)rBr:r_._translate_min_max_number_of_bins_to_bin_edgesrr%) rkr#rminsmaxesrhkeyvalue column_namerdr(r(r)rgZs      rgrprqrcCs||f}dd|D\}}t|st|rtd||krH||dkr*dt|nd8}||dkr8dt|nd7}tj|||ddd}|Stj|||ddd}||d}|re|d|8<|S|d |7<|S) zETranslates a range and desired number of bins into list of bin edges.css|]}|dVqdS)gNr()r2mir(r(r)r4szA_translate_min_max_number_of_bins_to_bin_edges..z@Cannot specify integer `bins` when input data contains infinity.rgMbP?T)endpointrm)rWisinfr6abslinspace)rprqr#rrngadjr(r(r)rrs$ rr)typingrrrrrrrnumpyrWpandasrray.data.aggregater r ray.data.preprocessorr ray.util.annotationsr ray.data.datasetr rrMrYr%rSrVrgrTr`rrr(r(r(r)s4$   :z7