o `۷i@s^ddlZddlmZddlZddlmZddlmZddl m Z e ddGdd d eZ dS) N)List) Preprocessor) simple_hash) PublicAPIalpha) stabilitycsreZdZdZdZdeededeffdd Zde j fd d Z d eefd d Z d eefddZ ddZZS) FeatureHashera Apply the `hashing trick `_ to a table that describes token frequencies. :class:`FeatureHasher` creates ``num_features`` columns named ``hash_{index}``, where ``index`` ranges from :math:`0` to ``num_features``:math:`- 1`. The column ``hash_{index}`` describes the frequency of tokens that hash to ``index``. Distinct tokens can correspond to the same index. However, if ``num_features`` is large enough, then columns probably correspond to a unique token. This preprocessor is memory efficient and quick to pickle. However, given a transformed column, you can't know which tokens correspond to it. This might make it hard to determine which tokens are important to your model. .. warning:: Sparse matrices aren't supported. If you use a large ``num_features``, this preprocessor might behave poorly. Examples: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import FeatureHasher The data below describes the frequencies of tokens in ``"I like Python"`` and ``"I dislike Python"``. >>> df = pd.DataFrame({ ... "I": [1, 1], ... "like": [1, 0], ... "dislike": [0, 1], ... "Python": [1, 1] ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP :class:`FeatureHasher` hashes each token to determine its index. For example, the index of ``"I"`` is :math:`hash(\\texttt{"I"}) \pmod 8 = 5`. >>> hasher = FeatureHasher(columns=["I", "like", "dislike", "Python"], num_features=8, output_column = "hashed") >>> hasher.fit_transform(ds)["hashed"].to_pandas().to_numpy() # doctest: +SKIP array([[0, 0, 0, 2, 0, 1, 0, 0], [0, 0, 0, 1, 0, 1, 1, 0]]) Notice the hash collision: both ``"like"`` and ``"Python"`` correspond to index :math:`3`. You can avoid hash collisions like these by increasing ``num_features``. Args: columns: The columns to apply the hashing trick to. Each column should describe the frequency of a token. num_features: The number of features used to represent the vocabulary. You should choose a value large enough to prevent hash collisions between distinct tokens. output_column: The name of the column that contains the hashed features. .. seealso:: :class:`~ray.data.preprocessors.CountVectorizer` Use this preprocessor to generate inputs for :class:`FeatureHasher`. :class:`ray.data.preprocessors.HashingVectorizer` If your input data describes documents rather than token frequencies, use :class:`~ray.data.preprocessors.HashingVectorizer`. Fcolumns num_features output_columncs t||_||_||_dSN)super__init__r r r )selfr r r  __class__S/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/data/preprocessors/hasher.pyrOs  zFeatureHasher.__init__dfcsnfdd}|jddjfj|ddd}ddtjD}||}tt||jddj f<|S)NcsNttjD]}t|j}|||7<qfddtjDS)Ncsi|] }d||qShash_r.0i hash_countsrr cszOFeatureHasher._transform_pandas..row_feature_hasher..) collections defaultdictintr rr range)rowcolumn hashed_valuerrrrow_feature_hasher^s   z;FeatureHasher._transform_pandas..row_feature_hasherexpand)axis result_typecSsg|]}d|qSrrrrrr jsz3FeatureHasher._transform_pandas..) locr applyr r to_numpypdSerieslistr )rrr%feature_columns hash_columns concatenatedrr$r_transform_pandas\s  zFeatureHasher._transform_pandasreturncCs|jSr )r r$rrrget_input_columnsrszFeatureHasher.get_input_columnscCs|jgSr )r r$rrrget_output_columnsusz FeatureHasher.get_output_columnscCs&|jjd|jd|jd|jdS)Nz (columns=z, num_features=z, output_column=))r__name__r r r r$rrr__repr__xs zFeatureHasher.__repr__)r9 __module__ __qualname____doc__ _is_fittablerstrrrr. DataFramer4r6r7r: __classcell__rrrrr s@ r) rtypingrpandasr.ray.data.preprocessorrray.data.preprocessors.utilsrray.util.annotationsrrrrrrs