o cid@s^ddlZddlmZddlZddlmZddlmZddl m Z e ddGdd d eZ dS) N)List) Preprocessor) simple_hash) PublicAPIalpha) stabilityc@sFeZdZdZdZdeededefddZde j fd d Z d d Z d S) FeatureHashera Apply the `hashing trick `_ to a table that describes token frequencies. :class:`FeatureHasher` creates ``num_features`` columns named ``hash_{index}``, where ``index`` ranges from :math:`0` to ``num_features``:math:`- 1`. The column ``hash_{index}`` describes the frequency of tokens that hash to ``index``. Distinct tokens can correspond to the same index. However, if ``num_features`` is large enough, then columns probably correspond to a unique token. This preprocessor is memory efficient and quick to pickle. However, given a transformed column, you can't know which tokens correspond to it. This might make it hard to determine which tokens are important to your model. .. warning:: Sparse matrices aren't supported. If you use a large ``num_features``, this preprocessor might behave poorly. Examples: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import FeatureHasher The data below describes the frequencies of tokens in ``"I like Python"`` and ``"I dislike Python"``. >>> df = pd.DataFrame({ ... "I": [1, 1], ... "like": [1, 0], ... "dislike": [0, 1], ... "Python": [1, 1] ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP :class:`FeatureHasher` hashes each token to determine its index. For example, the index of ``"I"`` is :math:`hash(\\texttt{"I"}) \pmod 8 = 5`. >>> hasher = FeatureHasher(columns=["I", "like", "dislike", "Python"], num_features=8, output_column = "hashed") >>> hasher.fit_transform(ds)["hashed"].to_pandas().to_numpy() # doctest: +SKIP array([[0, 0, 0, 2, 0, 1, 0, 0], [0, 0, 0, 1, 0, 1, 1, 0]]) Notice the hash collision: both ``"like"`` and ``"Python"`` correspond to index :math:`3`. You can avoid hash collisions like these by increasing ``num_features``. Args: columns: The columns to apply the hashing trick to. Each column should describe the frequency of a token. num_features: The number of features used to represent the vocabulary. You should choose a value large enough to prevent hash collisions between distinct tokens. output_column: The name of the column that contains the hashed features. .. seealso:: :class:`~ray.data.preprocessors.CountVectorizer` Use this preprocessor to generate inputs for :class:`FeatureHasher`. :class:`ray.data.preprocessors.HashingVectorizer` If your input data describes documents rather than token frequencies, use :class:`~ray.data.preprocessors.HashingVectorizer`. Fcolumns num_features output_columncCs||_||_||_dS)N)r r r )selfr r r r Q/home/ubuntu/.local/lib/python3.10/site-packages/ray/data/preprocessors/hasher.py__init__Os zFeatureHasher.__init__dfcsnfdd}|jddjfj|ddd}ddtjD}||}tt||jddj f<|S)NcsNttjD]}t|j}|||7<qfddtjDS)Ncsi|] }d||qShash_r .0i hash_countsr r bszOFeatureHasher._transform_pandas..row_feature_hasher..) collections defaultdictintr rr range)rowcolumn hashed_valuer rrrow_feature_hasher]s   z;FeatureHasher._transform_pandas..row_feature_hasherexpand)axis result_typecSsg|]}d|qSrr rr r r isz3FeatureHasher._transform_pandas..) locr applyrr to_numpypdSerieslistr )r rr!feature_columns hash_columns concatenatedr r r_transform_pandas[s  zFeatureHasher._transform_pandascCs&|jjd|jd|jd|jdS)Nz (columns=z, num_features=z, output_column=)) __class____name__r r r r r r r__repr__qs zFeatureHasher.__repr__N) r3 __module__ __qualname____doc__ _is_fittablerstrrrr* DataFramer0r4r r r rr s@  r) rtypingrpandasr*ray.data.preprocessorrray.data.preprocessors.utilsrray.util.annotationsrrr r r rs