o `۷i=@svddlZddlmZddlmZddlmZmZmZm Z m Z m Z m Z m Z mZmZddlZddlZddlZddlZddlmZddlmZddlmZddlmZm Z m!Z!ddl"m#Z#dd l$m%Z%dd l&m'Z'dd l(m)Z)m*Z*ervdd l+m,Z,e-e.Z/d e e0efde0deej1ej1ffddZ2e*dde%dddGddde!Z3e*dde%dddGddde!Z4e*dde%dddGddde!Z5e*dde%dddGd d!d!e!Z6e*dde%dd"dGd#d$d$e!Z7d%dd&d'd(d)e e0d*ed+e8d,e e e0e9ff d-d.Z: d=d0e8d1e'defd2d3Z;d4ejd)e0ddfd8d9Z?d:ej@de8fd;d<ZAdS)>N)Counter)partial) TYPE_CHECKINGAnyCallableDictHashableListOptionalSetTupleUnionis_null) BlockAccessor) PreprocessorPreprocessorNotFittedExceptionSerializablePreprocessorBase)make_post_processor)SerializablePreprocessor) BatchFormat) DeveloperAPI PublicAPI)Datasetstats input_colreturncsh|d|dttr,t}t|}tjfdd|Dtd}||fS\}}||fS)aGet Arrow arrays for keys and values from encoder stats. Args: stats: The encoder's stats_ dictionary. input_col: The name of the column to get arrays for. Returns: Tuple of (keys_array, values_array) for the column's ordinal mapping. unique_values()csg|]}|qSr).0k stat_valuerT/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/data/preprocessors/encoder.py =sz2_get_unique_value_arrow_arrays..type) isinstancedictsortedkeyspaarrayint64)rr sorted_keys keys_array values_arrayrr"r$_get_unique_value_arrow_arrays,s    r2alpha) stabilityz$io.ray.preprocessors.ordinal_encoder)version identifierc s,eZdZdZddddeededeeeffdd Zd d d e fd dZ ded e e e ffddZded eejejffddZdedefddZdejfddZdejd ejfddZdejded ejfdd Zeed efd!d"Zd e ee ffd#d$Z d%e ee fd&e fd'd(Z!d)d*Z"Z#S)+OrdinalEncodera Encode values within columns as ordered integer values. :class:`OrdinalEncoder` encodes categorical features as integers that range from :math:`0` to :math:`n - 1`, where :math:`n` is the number of categories. If you transform a value that isn't in the fitted datset, then the value is encoded as ``float("nan")``. Columns must contain either hashable values or lists of hashable values. Also, you can't have both scalars and lists in the same column. Examples: Use :class:`OrdinalEncoder` to encode categorical features as integers. >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import OrdinalEncoder >>> df = pd.DataFrame({ ... "sex": ["male", "female", "male", "female"], ... "level": ["L4", "L5", "L3", "L4"], ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> encoder = OrdinalEncoder(columns=["sex", "level"]) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP sex level 0 1 1 1 0 2 2 1 0 3 0 1 :class:`OrdinalEncoder` can also be used in append mode by providing the name of the output_columns that should hold the encoded values. >>> encoder = OrdinalEncoder(columns=["sex", "level"], output_columns=["sex_encoded", "level_encoded"]) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP sex level sex_encoded level_encoded 0 male L4 1 1 1 female L5 0 2 2 male L3 1 0 3 female L4 0 1 If you transform a value not present in the original dataset, then the value is encoded as ``float("nan")``. >>> df = pd.DataFrame({"sex": ["female"], "level": ["L6"]}) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> encoder.transform(ds).to_pandas() # doctest: +SKIP sex level 0 0 NaN :class:`OrdinalEncoder` can also encode categories in a list. >>> df = pd.DataFrame({ ... "name": ["Shaolin Soccer", "Moana", "The Smartest Guys in the Room"], ... "genre": [ ... ["comedy", "action", "sports"], ... ["animation", "comedy", "action"], ... ["documentary"], ... ], ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> encoder = OrdinalEncoder(columns=["genre"]) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP name genre 0 Shaolin Soccer [2, 0, 4] 1 Moana [1, 2, 0] 2 The Smartest Guys in the Room [3] Args: columns: The columns to separately encode. encode_lists: If ``True``, encode list elements. If ``False``, encode whole lists (i.e., replace each list with an integer). ``True`` by default. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. .. seealso:: :class:`OneHotEncoder` Another preprocessor that encodes categorical data. TN) encode_listsoutput_columnscolumnsr9r:cs(t||_||_t|||_dSN)super__init__r;r9r#_derive_and_validate_output_columnsr:)selfr;r9r: __class__rr$r>s  zOrdinalEncoder.__init__datasetrrc0jjfddtddddjdS)Ncstjj|dS)N)rCr;r9key_gen)compute_unique_value_indicesr;r9rErCr@rr$s z%OrdinalEncoder._fit..cS d|dSNzunique(rrcolrrr$rI cSrJNrrrrLrrr$rIrNstat_fnpost_process_fn stat_key_fn post_key_fnr;stat_computation_planadd_callable_statunique_post_fnr;r@rCrrHr$_fits  zOrdinalEncoder._fit column_namecCs<|jd|d}t|tr|S|\}}ddt||DS)aGet the ordinal mapping for a column as a dict. Stats can be stored in either: - Dict format: {value: index} (from pandas-style processing) - Arrow format: (keys_array, values_array) tuple This method returns a dict in either case. rrcSsi|] \}}||qSr)as_pyr r!vrrr$ sz3OrdinalEncoder._get_ordinal_map..)stats_r(r)zip)r@r[r#r0r1rrr$_get_ordinal_maps  zOrdinalEncoder._get_ordinal_maprcC t|j|Sz%Get Arrow arrays for keys and values.r2r`r@rrrr$_get_arrow_arrays z OrdinalEncoder._get_arrow_arrayselementcs0|||jrfdd|DSt|S)Ncsg|]}|qSrgetr x ordinal_maprr$r%sz7OrdinalEncoder._encode_list_element..)rbr9rktuple)r@rir[rrnr$_encode_list_elements z#OrdinalEncoder._encode_list_elementdfcs@t|gjRdtjffdd }|j||j<|S)Nscs2trfddSj}|S)Ncsj|jdS)N)r[)rqname)elem)rsr@rr$rIszROrdinalEncoder._transform_pandas..column_ordinal_encoder..)_is_series_composed_of_listsmaprbrtrss_valuesr@)rsr$column_ordinal_encoders    z@OrdinalEncoder._transform_pandas..column_ordinal_encoder) _validate_dfr;pdSeriesapplyr:)r@rrr{rrzr$_transform_pandass z OrdinalEncoder._transform_pandastablec Ct|g|jR|jD](}|j|j}tj|s!tj|r4| }| |}tj j |ddSq t |j|jD]\}}||}|||} t||| }q<|SaTransform using fast native PyArrow operations for scalar columns. List-type columns are preferably handled by _transform_pandas, which is selected via _determine_transform_to_use when a PyArrow schema is available. However, for pandas-backed datasets (PandasBlockSchema), we can't detect list columns until runtime, so we fall back to pandas here if list columns are found. F)preserve_index)_validate_arrowr;schemafieldr'r,typesis_list is_large_list to_pandasrTable from_pandasrar:column_encode_column_vectorizedr for_block upsert_column r@rcol_namecol_typerr result_dfr output_colrencoded_columnrrr$_transform_arrow     zOrdinalEncoder._transform_arrowrcCs@||\}}|j|jkrt||j}t||}t||S)aEncode column using PyArrow's vectorized pc.index_in. Unseen categories are encoded as null in the output, which becomes NaN when converted to pandas. Null values should be validated before calling this method via _validate_arrow. )rgr'pccastindex_intake)r@rrr0r1indicesrrr$rs    z(OrdinalEncoder._encode_column_vectorizedcCtjSr<rARROWclsrrr$preferred_batch_formatz%OrdinalEncoder.preferred_batch_formatcC|j|j|jt|dddS)N_fitted)r;r:r9r)r;r:r9getattrrzrrr$_get_serializable_fields  z'OrdinalEncoder._get_serializable_fieldsfieldsr6cC.|d|_|d|_|d|_|d|_dS)Nr;r:r9r)r;r:r9rkrr@rr6rrr$_set_serializable_fields%   z'OrdinalEncoder._set_serializable_fieldscC&|jjd|jd|jd|jdS)N (columns=z, encode_lists=, output_columns=r)rB__name__r;r9r:rzrrr$__repr__- zOrdinalEncoder.__repr__)$r __module__ __qualname____doc__r strboolr r>rrZrrintrbr r,Arrayrglistrqr} DataFramerrr ChunkedArrayr classmethodrrrrrr __classcell__rrrAr$r8Ds<Y   r8z$io.ray.preprocessors.one_hot_encoderc s&eZdZdZddddeedeeeefdeeeffddZ d d d e fd d Z e e d efddZdedeeeffddZdejfddZdejd ejfddZded eejejffddZdejded ejfddZd eeeffd d!Zd"eeefd#efd$d%Zd&d'Z Z!S)( OneHotEncodera- `One-hot encode `_ categorical data. This preprocessor transforms each specified column into a one-hot encoded vector. Each element in the vector corresponds to a unique category in the column, with a value of 1 if the category matches and 0 otherwise. If a category is infrequent (based on ``max_categories``) or not present in the fitted dataset, it is encoded as all 0s. Columns must contain hashable objects or lists of hashable objects. .. note:: Lists are treated as categories. If you want to encode individual list elements, use :class:`MultiHotEncoder`. Example: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import OneHotEncoder >>> >>> df = pd.DataFrame({"color": ["red", "green", "red", "red", "blue", "green"]}) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> encoder = OneHotEncoder(columns=["color"]) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP color 0 [0, 0, 1] 1 [0, 1, 0] 2 [0, 0, 1] 3 [0, 0, 1] 4 [1, 0, 0] 5 [0, 1, 0] OneHotEncoder can also be used in append mode by providing the name of the output_columns that should hold the encoded values. >>> encoder = OneHotEncoder(columns=["color"], output_columns=["color_encoded"]) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP color color_encoded 0 red [0, 0, 1] 1 green [0, 1, 0] 2 red [0, 0, 1] 3 red [0, 0, 1] 4 blue [1, 0, 0] 5 green [0, 1, 0] If you one-hot encode a value that isn't in the fitted dataset, then the value is encoded with zeros. >>> df = pd.DataFrame({"color": ["yellow"]}) >>> batch = ray.data.from_pandas(df) # doctest: +SKIP >>> encoder.transform(batch).to_pandas() # doctest: +SKIP color color_encoded 0 yellow [0, 0, 0] Likewise, if you one-hot encode an infrequent value, then the value is encoded with zeros. >>> encoder = OneHotEncoder(columns=["color"], max_categories={"color": 2}) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP color 0 [1, 0] 1 [0, 1] 2 [1, 0] 3 [1, 0] 4 [0, 0] 5 [0, 1] Args: columns: The columns to separately encode. max_categories: The maximum number of features to create for each column. If a value isn't specified for a column, then a feature is created for every category in that column. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. .. seealso:: :class:`MultiHotEncoder` If you want to encode individual list elements, use :class:`MultiHotEncoder`. :class:`OrdinalEncoder` If your categories are ordered, you may want to use :class:`OrdinalEncoder`. Nmax_categoriesr:r;rr:c,t||_|p i|_t|||_dSr<r=r>r;rrr?r:r@r;rr:rArr$r>   zOneHotEncoder.__init__rCrrcrD)Nctjd|jdS)NFrCr;r9rErrFr;rrGrHrr$rIz$OneHotEncoder._fit..cSrJrKrrLrrr$rIrNcSrJrOrrLrrr$rIrNrPrUrYrrHr$rZ  zOneHotEncoder._fitcCrr<rrrrr$rrz$OneHotEncoder.preferred_batch_formatr^rcCs2t|ttjfr t|}t|tr||dSdS)N)r(rnpndarrayrprrk)r@r^rrrr$safe_gets   zOneHotEncoder.safe_getrrc st|gjRtjjD]D\}}jd|dt}tjt||ftjd}|| fdd }|dk}t |d}d||||f<| ||<q|S) Nrrdtypecs |Sr<)r)r^r@rrr$rIrNz1OneHotEncoder._transform_pandas..rrr5) r|r;rar:r`lenrzerosuint8rto_numpynonzerotolist) r@rrr output_columnnum_categoriesone_hotcodesvalid_category_masknon_zero_indicesrrr$rs zOneHotEncoder._transform_pandasrc Crr)rr;rrr'r,rrrrrrrrar:r_encode_column_one_hotrrrrrrr$rrzOneHotEncoder._transform_arrowrcCrcrdrerfrrr$rgrhzOneHotEncoder._get_arrow_arraysrc Cs||\}}t|}|j|jkrt||j}t||}t|d}t|}|} tj ||ftj d} | dk} t | d} t| dkrOd| | | | f<t j | |S)zEncode a column to one-hot vectors using Arrow arrays. Unseen categories are encoded as all-zeros vectors, matching the pandas behavior. Null values should be validated before calling this method via _validate_arrow. rrrr5)rgrr'rrr fill_nullrrrrrr,FixedSizeListArray from_arraysravel) r@rrr0_rrindices_fillednum_rows indices_npone_hot_matrix valid_mask valid_indicesrrr$rs     z$OneHotEncoder._encode_column_one_hotcCrNr)r;r:rrr;r:rrrzrrr$rrz&OneHotEncoder._get_serializable_fieldsrr6cCrNr;r:rrr;r:rrkrrrrr$r$rz&OneHotEncoder._set_serializable_fieldscCrNrz, max_categories=rrrBrr;rr:rzrrr$r,rzOneHotEncoder.__repr__)"rrrrr rr rrr>rrZrrrrrrr}rrr,rrr rrgrrrrrrrrrrAr$r5s:]  &rz&io.ray.preprocessors.multi_hot_encoderc seZdZdZddddeedeeeefdeeeffddZ d d d e fd d Z de j fddZd eeeffddZdeeefdefddZddZZS)MultiHotEncoderaMulti-hot encode categorical data. This preprocessor replaces each list of categories with an :math:`m`-length binary list, where :math:`m` is the number of unique categories in the column or the value specified in ``max_categories``. The :math:`i\\text{-th}` element of the binary list is :math:`1` if category :math:`i` is in the input list and :math:`0` otherwise. Columns must contain hashable objects or lists of hashable objects. Also, you can't have both types in the same column. .. note:: The logic is similar to scikit-learn's [MultiLabelBinarizer][1] Examples: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import MultiHotEncoder >>> >>> df = pd.DataFrame({ ... "name": ["Shaolin Soccer", "Moana", "The Smartest Guys in the Room"], ... "genre": [ ... ["comedy", "action", "sports"], ... ["animation", "comedy", "action"], ... ["documentary"], ... ], ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> >>> encoder = MultiHotEncoder(columns=["genre"]) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP name genre 0 Shaolin Soccer [1, 0, 1, 0, 1] 1 Moana [1, 1, 1, 0, 0] 2 The Smartest Guys in the Room [0, 0, 0, 1, 0] :class:`MultiHotEncoder` can also be used in append mode by providing the name of the output_columns that should hold the encoded values. >>> encoder = MultiHotEncoder(columns=["genre"], output_columns=["genre_encoded"]) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP name genre genre_encoded 0 Shaolin Soccer [comedy, action, sports] [1, 0, 1, 0, 1] 1 Moana [animation, comedy, action] [1, 1, 1, 0, 0] 2 The Smartest Guys in the Room [documentary] [0, 0, 0, 1, 0] If you specify ``max_categories``, then :class:`MultiHotEncoder` creates features for only the most frequent categories. >>> encoder = MultiHotEncoder(columns=["genre"], max_categories={"genre": 3}) >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP name genre 0 Shaolin Soccer [1, 1, 1] 1 Moana [1, 1, 0] 2 The Smartest Guys in the Room [0, 0, 0] >>> encoder.stats_ # doctest: +SKIP OrderedDict([('unique_values(genre)', {'comedy': 0, 'action': 1, 'sports': 2})]) Args: columns: The columns to separately encode. max_categories: The maximum number of features to create for each column. If a value isn't specified for a column, then a feature is created for every unique category in that column. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. .. seealso:: :class:`OneHotEncoder` If you're encoding individual categories instead of lists of categories, use :class:`OneHotEncoder`. :class:`OrdinalEncoder` If your categories are ordered, you may want to use :class:`OrdinalEncoder`. [1]: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MultiLabelBinarizer.html Nrr;rr:crr<rrrArr$r>rzMultiHotEncoder.__init__rCrrcrD)Ncr)NTrrrGrHrr$rIrz&MultiHotEncoder._fit..cSrJrKrrLrrr$rIrNcSrJrOrrLrrr$rIrNrPrUrYrrHr$rZrzMultiHotEncoder._fitrrcs^t|gjRdtdtffdd }tjjD]\}}||t||d||<q|S)NrirtcsRt|tjr |}nt|ts|g}jd|d}t|fdd|DS)Nrrcsg|]}|dqS)rrjrlcounterrr$r%zJMultiHotEncoder._transform_pandas..encode_list..)r(rrrrr`r)rirtrrzrr$ encode_lists   z6MultiHotEncoder._transform_pandas..encode_list)rt)r|r;rrrar:rwr)r@rrrrrrrzr$rs  z!MultiHotEncoder._transform_pandascCrrrrzrrr$rrz(MultiHotEncoder._get_serializable_fieldsrr6cCrrrrrrr$rrz(MultiHotEncoder._set_serializable_fieldscCs&|jjd|jd|jd|jdSrrrzrrr$rrzMultiHotEncoder.__repr__)rrrrr rr rrr>rrZr}rrrrrrrrrrAr$r4s T rz"io.ray.preprocessors.label_encodercseZdZdZdddedeeffddZdd d efd d Zd e j fddZ d!ddZ d e j fddZ d eefddZd eefddZd eeeffddZdeeefdefddZdd ZZS)" LabelEncodera Encode labels as integer targets. :class:`LabelEncoder` encodes labels as integer targets that range from :math:`0` to :math:`n - 1`, where :math:`n` is the number of unique labels. If you transform a label that isn't in the fitted datset, then the label is encoded as ``float("nan")``. Examples: >>> import pandas as pd >>> import ray >>> df = pd.DataFrame({ ... "sepal_width": [5.1, 7, 4.9, 6.2], ... "sepal_height": [3.5, 3.2, 3, 3.4], ... "species": ["setosa", "versicolor", "setosa", "virginica"] ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> >>> from ray.data.preprocessors import LabelEncoder >>> encoder = LabelEncoder(label_column="species") >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP sepal_width sepal_height species 0 5.1 3.5 0 1 7.0 3.2 1 2 4.9 3.0 0 3 6.2 3.4 2 You can also provide the name of the output column that should hold the encoded labels if you want to use :class:`LabelEncoder` in append mode. >>> encoder = LabelEncoder(label_column="species", output_column="species_encoded") >>> encoder.fit_transform(ds).to_pandas() # doctest: +SKIP sepal_width sepal_height species species_encoded 0 5.1 3.5 setosa 0 1 7.0 3.2 versicolor 1 2 4.9 3.0 setosa 0 3 6.2 3.4 virginica 2 If you transform a label not present in the original dataset, then the new label is encoded as ``float("nan")``. >>> df = pd.DataFrame({ ... "sepal_width": [4.2], ... "sepal_height": [2.7], ... "species": ["bracteata"] ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> encoder.transform(ds).to_pandas() # doctest: +SKIP sepal_width sepal_height species 0 4.2 2.7 NaN Args: label_column: A column containing labels that you want to encode. output_column: The name of the column that will contain the encoded labels. If None, the output column will have the same name as the input column. .. seealso:: :class:`OrdinalEncoder` If you're encoding ordered features, use :class:`OrdinalEncoder` instead of :class:`LabelEncoder`. Nr label_columnrcst||_|p ||_dSr<)r=r>rr)r@rrrArr$r>s zLabelEncoder.__init__rCrrcs2jjfddtddddjgdS)Ncstjg|dSN)rCr;rE)rFrrGrHrr$rIs z#LabelEncoder._fit..cSrJrKrrLrrr$rI!rNcSrJrOrrLrrr$rI"rNrP)rVrWrXrrYrrHr$rZs  zLabelEncoder._fitrrcs:t|jdtjffdd }|j||j<|S)Nrscsjd|jd}||SrO)r`rtrwrxrzrr$column_label_encoder*s z.column_label_encoder)r|rr}r~ transformr)r@rrrrrzr$r's zLabelEncoder._transform_pandasdscCsF|}|tjjtjjfvrtd|}|j|jfdt j i|S)a/Inverse transform the given dataset. Args: ds: Input Dataset that has been fitted and/or transformed. Returns: ray.data.Dataset: The inverse transformed Dataset. Raises: PreprocessorNotFittedException: if ``fit`` is not called yet. z1`fit` must be called before `inverse_transform`, batch_format) fit_statusr FitStatusPARTIALLY_FITTED NOT_FITTEDr_get_transform_config map_batches_inverse_transform_pandasrPANDAS)r@rrkwargsrrr$inverse_transform1s  zLabelEncoder.inverse_transformcs.dtjffdd }|j||j<|S)Nrscs,ddjdjdD}||S)NcSsi|]\}}||qSrr)r keyvaluerrr$r_PszXLabelEncoder._inverse_transform_pandas..column_label_decoder..rr)r`ritemsrw)rsinverse_valuesrzrr$column_label_decoderOs  zDLabelEncoder._inverse_transform_pandas..column_label_decoder)r}r~rrr)r@rrrrrzr$rNs z&LabelEncoder._inverse_transform_pandascC|jgSr<)rrzrrr$get_input_columns[zLabelEncoder.get_input_columnscCrr<rrzrrr$get_output_columns^rzLabelEncoder.get_output_columnscCs|j|jt|dddS)Nr)rrr)rrrrzrrr$ras z%LabelEncoder._get_serializable_fieldsrr6cCs$|d|_|d|_|d|_dS)Nrrr)rrrkrrrrr$rhs  z%LabelEncoder._set_serializable_fieldscCs|jjd|jd|jdS)Nz(label_column=z, output_column=r)rBrrrrzrrr$roszLabelEncoder.__repr__)rrrr)rrrrrr r>rrZr}rrr rr rrrrrrrrrrrrAr$rs @  rz io.ray.preprocessors.categorizerc seZdZdZ  ddeedeeeej fdeeeffdd Z dd d e fd d Z d ej fddZd eeeffddZdeeefdefddZddZZS) Categorizera^ Convert columns to ``pd.CategoricalDtype``. Use this preprocessor with frameworks that have built-in support for ``pd.CategoricalDtype`` like LightGBM. .. warning:: If you don't specify ``dtypes``, fit this preprocessor before splitting your dataset into train and test splits. This ensures categories are consistent across splits. Examples: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import Categorizer >>> >>> df = pd.DataFrame( ... { ... "sex": ["male", "female", "male", "female"], ... "level": ["L4", "L5", "L3", "L4"], ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> categorizer = Categorizer(columns=["sex", "level"]) >>> categorizer.fit_transform(ds).schema().types # doctest: +SKIP [CategoricalDtype(categories=['female', 'male'], ordered=False), CategoricalDtype(categories=['L3', 'L4', 'L5'], ordered=False)] :class:`Categorizer` can also be used in append mode by providing the name of the output_columns that should hold the categorized values. >>> categorizer = Categorizer(columns=["sex", "level"], output_columns=["sex_cat", "level_cat"]) >>> categorizer.fit_transform(ds).to_pandas() # doctest: +SKIP sex level sex_cat level_cat 0 male L4 male L4 1 female L5 female L5 2 male L3 male L3 3 female L4 female L4 If you know the categories in advance, you can specify the categories with the ``dtypes`` parameter. >>> categorizer = Categorizer( ... columns=["sex", "level"], ... dtypes={"level": pd.CategoricalDtype(["L3", "L4", "L5", "L6"], ordered=True)}, ... ) >>> categorizer.fit_transform(ds).schema().types # doctest: +SKIP [CategoricalDtype(categories=['female', 'male'], ordered=False), CategoricalDtype(categories=['L3', 'L4', 'L5', 'L6'], ordered=True)] Args: columns: The columns to convert to ``pd.CategoricalDtype``. dtypes: An optional dictionary that maps columns to ``pd.CategoricalDtype`` objects. If you don't include a column in ``dtypes``, the categories are inferred. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. Nr;dtypesr:cs0t|s i}||_||_t|||_dSr<)r=r>r;rrr?r:)r@r;rr:rArr$r>s  zCategorizer.__init__rCrrcsfddjDjjO_sSdtttfdtjfdd}jjfddt t d d |gd d dd ddS)Ncsg|] }|jvr|qSr)rr rrzrr$r%sz$Categorizer._fit..unique_indicesrcSst|Sr<)r}CategoricalDtyper+)rrrr$callbacksz"Categorizer._fit..callbackcst|dSr)rFrG)columns_to_getrCrr$rIs z"Categorizer._fit..Tdrop_na_values)base_fn callbackscSrJrKrrLrrr$rIrNcSs|Sr<rrLrrr$rIsrP) r;r`rrrr}rrVrWrrX)r@rCrr)rrCr@r$rZs$  zCategorizer._fitrrcCs||j|j||j<|Sr<)r;astyper`r:)r@rrrrr$rszCategorizer._transform_pandascCsB|j|jt|ddt|dr|jrdd|jDdSddS)NrrcSs$i|]\}}|t|j|jdqS) categoriesordered)rr r!)r rMrrrr$r_sz8Categorizer._get_serializable_fields..)r;r:rr)r;r:rhasattrrr rzrrr$rs  z$Categorizer._get_serializable_fieldsrr6cCsJ|drdd|dDni|_|d|_|d|_|d|_dS)NrcSs(i|]\}}|tj|d|ddqS)r r!r)r}r)r rM dtype_datarrr$r_s  z8Categorizer._set_serializable_fields..r;r:r)rkr rr;r:rrrrr$rs   z$Categorizer._set_serializable_fieldscCr)Nrz , dtypes=rr)rBrr;rr:rzrrr$rs zCategorizer.__repr__)NN)rrrrr rr rr}rr>rrZrrrrrrrrrrrAr$rss >  rT)r9rrCrr;rEr9rcs|duri}t}|D]}||vrtd|ddq dtjdtffdd dtjdttttfffd d }|j|d d }fd dD} |j ddD]3} | D],\} } | D]%} dd| D} | |vryt t |  || } | | | q_qYqS| S)NzYou set `max_categories` for z, which is not present in .rMrcsNt|rrtfdd}||S|dd}t|jddS)Ncs||Sr<)update)rirrr$update_counter!s z\compute_unique_value_indices..get_pd_value_counts_per_column..update_countercSst|Sr<)rp)rmrrr$rI)szVcompute_unique_value_indices..get_pd_value_counts_per_column..F)dropna)rvrrw value_countsto_dict)rMr&)r9rr$get_pd_value_counts_per_columns  zDcompute_unique_value_indices..get_pd_value_counts_per_columnrrcsJ|j}i}D]}||vr||g||<q td|d||S)NzColumn 'z2' does not exist in DataFrame, which has columns: )r;r ValueError)rr df_columnsresultrM)r;r*rr$get_pd_value_counts,s z9compute_unique_value_indices..get_pd_value_countspandas)rcsi|]}|tqSr)set)r rMrGrr$r_:rz0compute_unique_value_indices..) batch_sizecSsi|] \}}|dur||qSr<rr]rrr$r_>s)r0r+r}r~rrrr r iter_batchesr r)r most_commonr%r+)rCr;rEr9r columns_setrr.value_counts_dsunique_values_by_colbatchrMcountersrr)r;r9r*rEr$rFs:&  rFFrrcs`dtdtttfffdd dtddttdtttffffdd }|tjkr.|SS) a Returns a post-processing function that generates an encoding map by sorting the unique values produced during aggregation or stats computation. Args: drop_na_values: If True, NA/null values will be silently dropped from the encoding map. If False, raises an error if any NA/null values are present. batch_format: Determines the output format of the encoding map. - If BatchFormat.ARROW: Returns Arrow format (tuple of arrays) for scalar types, or dict format for list types that PyArrow can't sort. - Otherwise: Returns pandas dict format {value: index}. Returns: A callable that takes unique values and returns an encoding map. The map format depends on batch_format and input types: - Dict format: {value: int} - used for pandas path or list-type data - Arrow format: (keys_array, values_array) - used for Arrow path with scalar data valuesrcsBtdd|Drstddd|D}ddtt|DS)a Generate an encoding map from a list of unique values using Python sorting. Args: values: List of unique values to encode (can include lists/tuples). Returns: Dict mapping each value to a unique integer index. List values are converted to tuples for hashability. Raises: ValueError: If null values are present and drop_na_values is False. css|]}t|VqdSr<rr r^rrr$ rsz:unique_post_fn..gen_value_index..]Unable to fit column because it contains null values. Consider imputing missing values first.cSsg|]}t|s|qSrrr:rrr$r%xrz;unique_post_fn..gen_value_index..cSs(i|]\}}t|ts |nt||qSr)r(rrp)r ir^rrr$r_zsz;unique_post_fn..gen_value_index..)anyr+ enumerater*)r9non_null_valuesrrr$gen_value_indexbs z'unique_post_fn..gen_value_index)z pa.ListScalarpa.Array)rBrBcst|tjr |j}tj|jstj|jr|Sr%t |}nt t | r3tdt |}t ||}tjtt|td}||fS)aGenerate an encoding map from unique values using Arrow-native operations. Args: values: The aggregation result as a pa.ListScalar (list of unique values) or a pa.Array of values directly. Returns: For scalar types that PyArrow can sort natively, returns a tuple of (sorted_keys, indices) as pa.Array. For list types that require fallback, returns a dict mapping {value: index}. Note: PyArrow's sort_indices doesn't support list types, so we fall back to dict format for columns containing lists. The _transform_arrow method handles this by detecting dict-format stats and converting as needed. r<r&)r(r, ListScalarr9rrr'r to_pylistr drop_nullr>rr\r+ sort_indicesrr-rangerr.)r9sorted_indices sorted_valuesr1rrArr$ gen_value_index_arrow_from_arrows     z8unique_post_fn..gen_value_index_arrow_from_arrow)r rrrr r rr)rrrKrrJr$rXLs 1rXrrcs*fdd|D}|rtd|ddS)Ncs"g|] }|jr|qSr)isnullr9r>rrrrr$r%s"z _validate_df..Unable to transform columns J because they contain null values. Consider imputing missing values first.r+)rrr; null_columnsrrMr$r|s  r|rcs*fdd|D}|rtd|ddS)aValidate that specified columns in an Arrow table do not contain null values. Args: table: The Arrow table to validate. *columns: Column names to check for null values. Raises: ValueError: If any of the specified columns contain null values. c s*g|]}tt|r|qSr)rr>rrr\rrrr$r%s z#_validate_arrow..rNrONrP)rr;rQrrRr$rs  rseriescCs4tdd|Dd}tjj|jot|ttj fS)Ncss|] }|dur|VqdSr<r)r rirrr$r;sz/_is_series_composed_of_lists..) nextr/apiris_object_dtyperr(rrr)rSfirst_not_none_elementrrr$rvs  rv)FN)Blogging collectionsr functoolsrtypingrrrrrr r r r r numpyrr/r}pandas.api.typespyarrowr,pyarrow.computecomputerray.data._internal.utilrray.data.blockrray.data.preprocessorrrrray.data.preprocessors.utilsr&ray.data.preprocessors.version_supportr#ray.data.util.data_batch_conversionrray.util.annotationsrrray.data.datasetr getLoggerrloggerrrr2r8rrrrrrrFrXrr|rrr~rvrrrr$s  0            p ~ !  E j