o $i@sddlZddlmZddlmZddlmZmZmZm Z m Z m Z m Z m Z ddlZddlZddlZddlmZddlmZddlmZmZmZddlmZdd lmZdd lm Z er`dd l!m"Z"e#e$Z%e d d edddGdddeZ&e d d edddGdddeZ'e d d edddGdddeZ(e d d edddGdddeZ)e d d edddGdddeZ*ddd d!d"d#e e+d$ed%e,d&e e e+e-ff d'd(Z.d4d*e,d+ee ge e+e-fffd,d-Z/d.ej0d#e+d+dfd/d0Z1d1ej2d+e,fd2d3Z3dS)5N)Counter)partial) TYPE_CHECKINGAnyCallableDictHashableListOptionalSet) BatchFormatis_null) PreprocessorPreprocessorNotFittedExceptionSerializablePreprocessorBase)make_post_processor)SerializablePreprocessor) PublicAPI)Datasetalpha) stabilityz$io.ray.preprocessors.ordinal_encoder)version identifierc seZdZdZddddeededeeeffdd Zd d d e fd dZ de defddZ de jfddZd eeeffddZdeeefdefddZddZZS)OrdinalEncodera Encode values within columns as ordered integer values. :class:`OrdinalEncoder` encodes categorical features as integers that range from :math:`0` to :math:`n - 1`, where :math:`n` is the number of categories. If you transform a value that isn't in the fitted datset, then the value is encoded as ``float("nan")``. Columns must contain either hashable values or lists of hashable values. Also, you can't have both scalars and lists in the same column. Examples: Use :class:`OrdinalEncoder` to encode categorical features as integers. >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import OrdinalEncoder >>> df = pd.DataFrame({ ... "sex": ["male", "female", "male", "female"], ... "level": ["L4", "L5", "L3", "L4"], ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> encoder = OrdinalEncoder(columns=["sex", "level"]) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP sex level 0 1 1 1 0 2 2 1 0 3 0 1 :class:`OrdinalEncoder` can also be used in append mode by providing the name of the output_columns that should hold the encoded values. >>> encoder = OrdinalEncoder(columns=["sex", "level"], output_columns=["sex_encoded", "level_encoded"]) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP sex level sex_encoded level_encoded 0 male L4 1 1 1 female L5 0 2 2 male L3 1 0 3 female L4 0 1 If you transform a value not present in the original dataset, then the value is encoded as ``float("nan")``. >>> df = pd.DataFrame({"sex": ["female"], "level": ["L6"]}) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> encoder.transform(ds).to_pandas() # doctest: +SKIP sex level 0 0 NaN :class:`OrdinalEncoder` can also encode categories in a list. >>> df = pd.DataFrame({ ... "name": ["Shaolin Soccer", "Moana", "The Smartest Guys in the Room"], ... "genre": [ ... ["comedy", "action", "sports"], ... ["animation", "comedy", "action"], ... ["documentary"], ... ], ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> encoder = OrdinalEncoder(columns=["genre"]) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP name genre 0 Shaolin Soccer [2, 0, 4] 1 Moana [1, 2, 0] 2 The Smartest Guys in the Room [3] Args: columns: The columns to separately encode. encode_lists: If ``True``, encode list elements. If ``False``, encode whole lists (i.e., replace each list with an integer). ``True`` by default. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. .. seealso:: :class:`OneHotEncoder` Another preprocessor that encodes categorical data. TN) encode_listsoutput_columnscolumnsrrcs(t||_||_t|||_dSN)super__init__rrr#_derive_and_validate_output_columnsr)selfrrr __class__[/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/ray/data/preprocessors/encoder.pyr!ts  zOrdinalEncoder.__init__datasetrreturnc0jjfddtddddjdS)Ncstjj|dS)N)r(rrkey_gen)compute_unique_value_indicesrrr+r(r#r&r's z%OrdinalEncoder._fit..cS d|dSNzunique()r&colr&r&r'r/ cSr0Nunique_values(r2r&r3r&r&r'r/r5stat_fnpost_process_fn stat_key_fn post_key_fnrstat_computation_planadd_callable_statunique_post_fnrr#r(r&r.r'_fits  zOrdinalEncoder._fitelement column_namecs8|jd|d|jrfdd|DSt|S)Nr7r2csg|]}|qSr&get.0x ordinal_mapr&r' sz7OrdinalEncoder._encode_list_element..)stats_rrFtuple)r#rCrDr&rJr'_encode_list_elementsz#OrdinalEncoder._encode_list_elementdfcs@t|gjRdtjffdd }|j||j<|S)Nscs:trfddSjdjd}|S)Ncsj|jdS)N)rD)rOname)elem)rQr#r&r'r/szROrdinalEncoder._transform_pandas..column_ordinal_encoder..r7r2)_is_series_composed_of_listsmaprMrRrQs_valuesr#)rQr'column_ordinal_encoders   z@OrdinalEncoder._transform_pandas..column_ordinal_encoder) _validate_dfrpdSeriesapplyr)r#rPrYr&rXr'_transform_pandass z OrdinalEncoder._transform_pandascC|j|j|jt|dddS)N_fitted)rrrr`)rrrgetattrrXr&r&r'_get_serializable_fields  z'OrdinalEncoder._get_serializable_fieldsfieldsrcC.|d|_|d|_|d|_|d|_dS)Nrrrr`)rrrrFr`r#rdrr&r&r'_set_serializable_fields   z'OrdinalEncoder._set_serializable_fieldscC&|jjd|jd|jd|jdS)N (columns=z, encode_lists=, output_columns=r2)r%__name__rrrrXr&r&r'__repr__ zOrdinalEncoder.__repr__)rl __module__ __qualname____doc__r strboolr r!rrBlistrOr[ DataFramer^rrrbintrgrm __classcell__r&r&r$r'rs"Y  rz$io.ray.preprocessors.one_hot_encoderc seZdZdZddddeedeeeefdeeeffddZ d d d e fd d Z de deeeffddZ dejfddZd eee ffddZdeee fdefddZddZZS) OneHotEncodera- `One-hot encode `_ categorical data. This preprocessor transforms each specified column into a one-hot encoded vector. Each element in the vector corresponds to a unique category in the column, with a value of 1 if the category matches and 0 otherwise. If a category is infrequent (based on ``max_categories``) or not present in the fitted dataset, it is encoded as all 0s. Columns must contain hashable objects or lists of hashable objects. .. note:: Lists are treated as categories. If you want to encode individual list elements, use :class:`MultiHotEncoder`. Example: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import OneHotEncoder >>> >>> df = pd.DataFrame({"color": ["red", "green", "red", "red", "blue", "green"]}) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> encoder = OneHotEncoder(columns=["color"]) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP color 0 [0, 0, 1] 1 [0, 1, 0] 2 [0, 0, 1] 3 [0, 0, 1] 4 [1, 0, 0] 5 [0, 1, 0] OneHotEncoder can also be used in append mode by providing the name of the output_columns that should hold the encoded values. >>> encoder = OneHotEncoder(columns=["color"], output_columns=["color_encoded"]) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP color color_encoded 0 red [0, 0, 1] 1 green [0, 1, 0] 2 red [0, 0, 1] 3 red [0, 0, 1] 4 blue [1, 0, 0] 5 green [0, 1, 0] If you one-hot encode a value that isn't in the fitted dataset, then the value is encoded with zeros. >>> df = pd.DataFrame({"color": ["yellow"]}) >>> batch = ray.data.from_pandas(df) # doctest: +SKIP >>> encoder.transform(batch).to_pandas() # doctest: +SKIP color color_encoded 0 yellow [0, 0, 0] Likewise, if you one-hot encode an infrequent value, then the value is encoded with zeros. >>> encoder = OneHotEncoder(columns=["color"], max_categories={"color": 2}) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP color 0 [1, 0] 1 [0, 1] 2 [1, 0] 3 [1, 0] 4 [0, 0] 5 [0, 1] Args: columns: The columns to separately encode. max_categories: The maximum number of features to create for each column. If a value isn't specified for a column, then a feature is created for every category in that column. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. .. seealso:: :class:`MultiHotEncoder` If you want to encode individual list elements, use :class:`MultiHotEncoder`. :class:`OrdinalEncoder` If your categories are ordered, you may want to use :class:`OrdinalEncoder`. Nmax_categoriesrrrzrc,t||_|p i|_t|||_dSrr r!rrzrr"rr#rrzrr$r&r'r!   zOneHotEncoder.__init__r(rr)cr*)Nctjd|jdS)NFr(rrr+rzr,rrzr-r.r&r'r//z$OneHotEncoder._fit..cSr0r1r&r3r&r&r'r/7r5cSr0r6r&r3r&r&r'r/8r5r8r=rAr&r.r'rB-  zOneHotEncoder._fitvstatscCs2t|ttjfr t|}t|tr||dSdS)N) isinstancertnpndarrayrNrrF)r#rrr&r&r'safe_get=s   zOneHotEncoder.safe_getrPc st|gjRtjjD]D\}}jd|dt}tjt||ftjd}|| fdd }|dk}t |d}d||||f<| ||<q|S) Nr7r2)dtypecs |Sr)r)rr#rr&r'r/Nr5z1OneHotEncoder._transform_pandas..rrr) rZrziprrMlenrzerosuint8r]to_numpynonzerotolist) r#rPcolumn output_columnnum_categoriesone_hotcodesvalid_category_masknon_zero_indicesr&rr'r^Es zOneHotEncoder._transform_pandascCr_Nr`)rrrzr`rrrzrarXr&r&r'rb\rcz&OneHotEncoder._get_serializable_fieldsrdrcCreNrrrzr`rrrzrFr`rfr&r&r'rgdrhz&OneHotEncoder._set_serializable_fieldscCriNrjz, max_categories=rkr2r%rlrrzrrXr&r&r'rmlrnzOneHotEncoder.__repr__)rlrorprqr rrr rrvr!rrBrrr[rur^rbrgrmrwr&r&r$r'rxs"] rxz&io.ray.preprocessors.multi_hot_encoderc seZdZdZddddeedeeeefdeeeffddZ d d d e fd d Z de j fddZd eeeffddZdeeefdefddZddZZS)MultiHotEncoderaMulti-hot encode categorical data. This preprocessor replaces each list of categories with an :math:`m`-length binary list, where :math:`m` is the number of unique categories in the column or the value specified in ``max_categories``. The :math:`i\\text{-th}` element of the binary list is :math:`1` if category :math:`i` is in the input list and :math:`0` otherwise. Columns must contain hashable objects or lists of hashable objects. Also, you can't have both types in the same column. .. note:: The logic is similar to scikit-learn's [MultiLabelBinarizer][1] Examples: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import MultiHotEncoder >>> >>> df = pd.DataFrame({ ... "name": ["Shaolin Soccer", "Moana", "The Smartest Guys in the Room"], ... "genre": [ ... ["comedy", "action", "sports"], ... ["animation", "comedy", "action"], ... ["documentary"], ... ], ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> >>> encoder = MultiHotEncoder(columns=["genre"]) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP name genre 0 Shaolin Soccer [1, 0, 1, 0, 1] 1 Moana [1, 1, 1, 0, 0] 2 The Smartest Guys in the Room [0, 0, 0, 1, 0] :class:`MultiHotEncoder` can also be used in append mode by providing the name of the output_columns that should hold the encoded values. >>> encoder = MultiHotEncoder(columns=["genre"], output_columns=["genre_encoded"]) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP name genre genre_encoded 0 Shaolin Soccer [comedy, action, sports] [1, 0, 1, 0, 1] 1 Moana [animation, comedy, action] [1, 1, 1, 0, 0] 2 The Smartest Guys in the Room [documentary] [0, 0, 0, 1, 0] If you specify ``max_categories``, then :class:`MultiHotEncoder` creates features for only the most frequent categories. >>> encoder = MultiHotEncoder(columns=["genre"], max_categories={"genre": 3}) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP name genre 0 Shaolin Soccer [1, 1, 1] 1 Moana [1, 1, 0] 2 The Smartest Guys in the Room [0, 0, 0] >>> encoder.stats_ # doctest: +SKIP OrderedDict([('unique_values(genre)', {'comedy': 0, 'action': 1, 'sports': 2})]) Args: columns: The columns to separately encode. max_categories: The maximum number of features to create for each column. If a value isn't specified for a column, then a feature is created for every unique category in that column. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. .. seealso:: :class:`OneHotEncoder` If you're encoding individual categories instead of lists of categories, use :class:`OneHotEncoder`. :class:`OrdinalEncoder` If your categories are ordered, you may want to use :class:`OrdinalEncoder`. [1]: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MultiLabelBinarizer.html Nryrrzrcr{rr|r}r$r&r'r!r~zMultiHotEncoder.__init__r(rr)cr*)Ncr)NTrrr-r.r&r'r/rz&MultiHotEncoder._fit..cSr0r1r&r3r&r&r'r/r5cSr0r6r&r3r&r&r'r/r5r8r=rAr&r.r'rBrzMultiHotEncoder._fitrPcs^t|gjRdtdtffdd }tjjD]\}}||t||d||<q|S)NrCrRcsRt|tjr |}nt|ts|g}jd|d}t|fdd|DS)Nr7r2csg|]}|dqS)rrErGcounterr&r'rLzJMultiHotEncoder._transform_pandas..encode_list..)rrrrrtrMr)rCrRrrXrr' encode_lists   z6MultiHotEncoder._transform_pandas..encode_list)rR)rZrrtrrrrrUr)r#rPrrrr&rXr'r^s  z!MultiHotEncoder._transform_pandascCr_rrrXr&r&r'rbrcz(MultiHotEncoder._get_serializable_fieldsrdrcCrerrrfr&r&r'rgrhz(MultiHotEncoder._set_serializable_fieldscCs&|jjd|jd|jd|jdSrrrXr&r&r'rm rnzMultiHotEncoder.__repr__)rlrorprqr rrr rrvr!rrBr[rur^rrbrgrmrwr&r&r$r'rts T rz"io.ray.preprocessors.label_encodercseZdZdZdddedeeffddZdd d efd d Zd e j fddZ d!ddZ d e j fddZ d eefddZd eefddZd eeeffddZdeeefdefddZdd ZZS)" LabelEncodera Encode labels as integer targets. :class:`LabelEncoder` encodes labels as integer targets that range from :math:`0` to :math:`n - 1`, where :math:`n` is the number of unique labels. If you transform a label that isn't in the fitted datset, then the label is encoded as ``float("nan")``. Examples: >>> import pandas as pd >>> import ray >>> df = pd.DataFrame({ ... "sepal_width": [5.1, 7, 4.9, 6.2], ... "sepal_height": [3.5, 3.2, 3, 3.4], ... "species": ["setosa", "versicolor", "setosa", "virginica"] ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> >>> from ray.data.preprocessors import LabelEncoder >>> encoder = LabelEncoder(label_column="species") >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP sepal_width sepal_height species 0 5.1 3.5 0 1 7.0 3.2 1 2 4.9 3.0 0 3 6.2 3.4 2 You can also provide the name of the output column that should hold the encoded labels if you want to use :class:`LabelEncoder` in append mode. >>> encoder = LabelEncoder(label_column="species", output_column="species_encoded") >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP sepal_width sepal_height species species_encoded 0 5.1 3.5 setosa 0 1 7.0 3.2 versicolor 1 2 4.9 3.0 setosa 0 3 6.2 3.4 virginica 2 If you transform a label not present in the original dataset, then the new label is encoded as ``float("nan")``. >>> df = pd.DataFrame({ ... "sepal_width": [4.2], ... "sepal_height": [2.7], ... "species": ["bracteata"] ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> encoder.transform(ds).to_pandas() # doctest: +SKIP sepal_width sepal_height species 0 4.2 2.7 NaN Args: label_column: A column containing labels that you want to encode. output_column: The name of the column that will contain the encoded labels. If None, the output column will have the same name as the input column. .. seealso:: :class:`OrdinalEncoder` If you're encoding ordered features, use :class:`OrdinalEncoder` instead of :class:`LabelEncoder`. Nr label_columnrcst||_|p ||_dSr)r r!rr)r#rrr$r&r'r!Ts zLabelEncoder.__init__r(rr)cs2jjfddtddddjgdS)Ncstjg|dSN)r(rr+)r,rr-r.r&r'r/[s z#LabelEncoder._fit..cSr0r1r&r3r&r&r'r/ar5cSr0r6r&r3r&r&r'r/br5r8)r>r?r@rrAr&r.r'rBYs  zLabelEncoder._fitrPcs:t|jdtjffdd }|j||j<|S)NrQcsjd|jd}||Sr6)rMrRrUrVrXr&r'column_label_encoderjs z.column_label_encoder)rZrr[r\ transformr)r#rPrr&rXr'r^gs zLabelEncoder._transform_pandasdscCsF|}|tjjtjjfvrtd|}|j|jfdt j i|S)a/Inverse transform the given dataset. Args: ds: Input Dataset that has been fitted and/or transformed. Returns: ray.data.Dataset: The inverse transformed Dataset. Raises: PreprocessorNotFittedException: if ``fit`` is not called yet. z1`fit` must be called before `inverse_transform`, batch_format) fit_statusr FitStatusPARTIALLY_FITTED NOT_FITTEDr_get_transform_config map_batches_inverse_transform_pandasr PANDAS)r#rrkwargsr&r&r'inverse_transformqs  zLabelEncoder.inverse_transformcs.dtjffdd }|j||j<|S)NrQcs,ddjdjdD}||S)NcSsi|]\}}||qSr&r&)rHkeyvaluer&r&r' szXLabelEncoder._inverse_transform_pandas..column_label_decoder..r7r2)rMritemsrU)rQinverse_valuesrXr&r'column_label_decoders  zDLabelEncoder._inverse_transform_pandas..column_label_decoder)r[r\rrr)r#rPrr&rXr'rs z&LabelEncoder._inverse_transform_pandascC|jgSr)rrXr&r&r'get_input_columnszLabelEncoder.get_input_columnscCrrrrXr&r&r'get_output_columnsrzLabelEncoder.get_output_columnscCs|j|jt|dddS)Nr`)rrr`)rrrarXr&r&r'rbs z%LabelEncoder._get_serializable_fieldsrdrcCs$|d|_|d|_|d|_dS)Nrrr`)rrrFr`rfr&r&r'rgs  z%LabelEncoder._set_serializable_fieldscCs|jjd|jd|jdS)Nz(label_column=z, output_column=r2)r%rlrrrXr&r&r'rmszLabelEncoder.__repr__)rrr)r)rlrorprqrrr r!rrBr[rur^rrr rrrrrbrvrgrmrwr&r&r$r'rs @  rz io.ray.preprocessors.categorizerc seZdZdZ  ddeedeeeej fdeeeffdd Z dd d e fd d Z d ej fddZd eeeffddZdeeefdefddZddZZS) Categorizera^ Convert columns to ``pd.CategoricalDtype``. Use this preprocessor with frameworks that have built-in support for ``pd.CategoricalDtype`` like LightGBM. .. warning:: If you don't specify ``dtypes``, fit this preprocessor before splitting your dataset into train and test splits. This ensures categories are consistent across splits. Examples: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import Categorizer >>> >>> df = pd.DataFrame( ... { ... "sex": ["male", "female", "male", "female"], ... "level": ["L4", "L5", "L3", "L4"], ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> categorizer = Categorizer(columns=["sex", "level"]) >>> categorizer.fit_transform(ds).schema().types # doctest: +SKIP [CategoricalDtype(categories=['female', 'male'], ordered=False), CategoricalDtype(categories=['L3', 'L4', 'L5'], ordered=False)] :class:`Categorizer` can also be used in append mode by providing the name of the output_columns that should hold the categorized values. >>> categorizer = Categorizer(columns=["sex", "level"], output_columns=["sex_cat", "level_cat"]) >>> categorizer.fit_transform(ds).to_pandas() # doctest: +SKIP sex level sex_cat level_cat 0 male L4 male L4 1 female L5 female L5 2 male L3 male L3 3 female L4 female L4 If you know the categories in advance, you can specify the categories with the ``dtypes`` parameter. >>> categorizer = Categorizer( ... columns=["sex", "level"], ... dtypes={"level": pd.CategoricalDtype(["L3", "L4", "L5", "L6"], ordered=True)}, ... ) >>> categorizer.fit_transform(ds).schema().types # doctest: +SKIP [CategoricalDtype(categories=['female', 'male'], ordered=False), CategoricalDtype(categories=['L3', 'L4', 'L5', 'L6'], ordered=True)] Args: columns: The columns to convert to ``pd.CategoricalDtype``. dtypes: An optional dictionary that maps columns to ``pd.CategoricalDtype`` objects. If you don't include a column in ``dtypes``, the categories are inferred. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. Nrdtypesrcs0t|s i}||_||_t|||_dSr)r r!rrrr"r)r#rrrr$r&r'r!s  zCategorizer.__init__r(rr)csfddjDjjO_sSdtttfdtjfdd}jjfddt t d d |gd d dd ddS)Ncsg|] }|jvr|qSr&)rrHrrXr&r'rLsz$Categorizer._fit..unique_indicesr)cSst|Sr)r[CategoricalDtypekeys)rr&r&r'callback sz"Categorizer._fit..callbackcst|dSr)r,r-)columns_to_getr(r&r'r/ s z"Categorizer._fit..Tdrop_na_values)base_fn callbackscSr0r1r&r3r&r&r'r/r5cSs|Srr&r3r&r&r'r/sr8) rrMrrrrr[rr>r?rr@)r#r(rr&)rr(r#r'rBs$  zCategorizer._fitrPcCs||j|j||j<|Sr)rastyperMr)r#rPr&r&r'r^szCategorizer._transform_pandascCsB|j|jt|ddt|dr|jrdd|jDdSddS)Nr`rcSs$i|]\}}|t|j|jdqS) categoriesordered)rtrr)rHr4rr&r&r'r%sz8Categorizer._get_serializable_fields..)rrr`r)rrrahasattrrrrXr&r&r'rb s  z$Categorizer._get_serializable_fieldsrdrcCsJ|drdd|dDni|_|d|_|d|_|d|_dS)NrcSs(i|]\}}|tj|d|ddqS)rrr)r[r)rHr4 dtype_datar&r&r'r1s  z8Categorizer._set_serializable_fields..rrr`)rFrrrrr`rfr&r&r'rg-s   z$Categorizer._set_serializable_fieldscCri)Nrjz , dtypes=rkr2)r%rlrrrrXr&r&r'rm@s zCategorizer.__repr__)NN)rlrorprqr rrr rr[rr!rrBrur^rrbrvrgrmrwr&r&r$r'rs >  rT)rrzr(rrr+rrzcs|duri}t}|D]}||vrtd|ddq dtjdtffdd dtjdttttfffd d }|j|d d }fd dD} |j ddD]3} | D],\} } | D]%} dd| D} | |vryt t |  || } | | | q_qYqS| S)NzYou set `max_categories` for z, which is not present in .r4r)csNt|rrtfdd}||S|dd}t|jddS)Ncs||Sr)update)rCrr&r'update_counter`s z\compute_unique_value_indices..get_pd_value_counts_per_column..update_countercSst|Sr)rN)rIr&r&r'r/hszVcompute_unique_value_indices..get_pd_value_counts_per_column..F)dropna)rTrrU value_countsto_dict)r4r)rrr'get_pd_value_counts_per_columnYs  zDcompute_unique_value_indices..get_pd_value_counts_per_columnrPcsJ|j}i}D]}||vr||g||<q td|d||S)NzColumn 'z2' does not exist in DataFrame, which has columns: )rr ValueError)rP df_columnsresultr4)rrr&r'get_pd_value_countsks z9compute_unique_value_indices..get_pd_value_countspandas)rcsi|]}|tqSr&)set)rHr4r-r&r'ryrz0compute_unique_value_indices..) batch_sizecSsi|] \}}|dur||qSrr&)rHkrr&r&r'r}s)rrr[r\rrurrr r iter_batchesrdictr most_commonrr)r(rr+rrz columns_setrrvalue_counts_dsunique_values_by_colbatchr4countersrr&)rrrr+r'r,Gs:&  r,Frr)cs"dtdtttfffdd }|S)a Returns a post-processing function that generates an encoding map by sorting the unique values produced during aggregation or stats computation. Args: drop_na_values: If True, NA/null values will be silently dropped from the encoding map. If False, raises an error if any NA/null values are present. Returns: A callable that takes a set of unique values and returns a dictionary mapping each value to a unique integer index. valuesr)csBtdd|Drstddd|D}ddtt|DS)Ncss|]}t|VqdSrr rHrr&r&r' sz:unique_post_fn..gen_value_index..z]Unable to fit column because it contains null values. Consider imputing missing values first.cSsg|]}t|s|qSr&r rr&r&r'rLrz;unique_post_fn..gen_value_index..cSs(i|]\}}t|ts |nt||qSr&)rrtrN)rHirr&r&r'rsz;unique_post_fn..gen_value_index..)anyr enumeratesorted)rnon_null_valuesrr&r'gen_value_indexs z'unique_post_fn..gen_value_index)r rrrv)rrr&rr'r@sr@rPcs*fdd|D}|rtd|ddS)Ncs"g|] }|jr|qSr&)isnullrrrrPr&r'rLs"z _validate_df..zUnable to transform columns zJ because they contain null values. Consider imputing missing values first.)r)rPr null_columnsr&rr'rZs  rZseriescCs4tdd|Dd}tjj|jot|ttj fS)Ncss|] }|dur|VqdSrr&)rHrCr&r&r'rsz/_is_series_composed_of_lists..) nextrapitypesis_object_dtyperrrtrr)rfirst_not_none_elementr&r&r'rTs  rT)F)4logging collectionsr functoolsrtypingrrrrrr r r numpyrrr[pandas.api.types"ray.air.util.data_batch_conversionr ray.data._internal.utilrray.data.preprocessorrrrray.data.preprocessors.utilsr&ray.data.preprocessors.version_supportrray.util.annotationsrray.data.datasetr getLoggerrlloggerrrxrrrrrrsrvr,r@rurZr\rTr&r&r&r'sl  (        % 1 !  &C"