o $i @s^ddlmZmZmZddlZddlmZddlm Z ddl m Z e ddGdd d eZ dS) )CallableListOptionalN) Preprocessor)simple_split_tokenizer) PublicAPIalpha) stabilityc sneZdZdZdZ  ddeedeeegeefdeeeffdd Z d e j fd d Z d d Z ZS) TokenizeraReplace each string with a list of tokens. Examples: >>> import pandas as pd >>> import ray >>> df = pd.DataFrame({"text": ["Hello, world!", "foo bar\nbaz"]}) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP The default ``tokenization_fn`` delimits strings using the space character. >>> from ray.data.preprocessors import Tokenizer >>> tokenizer = Tokenizer(columns=["text"]) >>> tokenizer.transform(ds).to_pandas() # doctest: +SKIP text 0 [Hello,, world!] 1 [foo, bar\nbaz] If the default logic isn't adequate for your use case, you can specify a custom ``tokenization_fn``. >>> import string >>> def tokenization_fn(s): ... for character in string.punctuation: ... s = s.replace(character, "") ... return s.split() >>> tokenizer = Tokenizer(columns=["text"], tokenization_fn=tokenization_fn) >>> tokenizer.transform(ds).to_pandas() # doctest: +SKIP text 0 [Hello, world] 1 [foo, bar, baz] :class:`Tokenizer` can also be used in append mode by providing the name of the output_columns that should hold the tokenized values. >>> tokenizer = Tokenizer(columns=["text"], output_columns=["text_tokenized"]) >>> tokenizer.transform(ds).to_pandas() # doctest: +SKIP text text_tokenized 0 Hello, world! [Hello,, world!] 1 foo bar\nbaz [foo, bar\nbaz] Args: columns: The columns to tokenize. tokenization_fn: The function used to generate tokens. This function should accept a string as input and return a list of tokens as output. If unspecified, the tokenizer uses a function equivalent to ``lambda s: s.split(" ")``. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. FNcolumnstokenization_fnoutput_columnscs,t||_|p t|_t|||_dSN)super__init__r rr r#_derive_and_validate_output_columnsr )selfr r r  __class__]/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/ray/data/preprocessors/tokenizer.pyrBs   zTokenizer.__init__dfcs8dtjffdd }|jddjf||j<|S)Nscs |jSr)mapr )rrrrcolumn_tokenizerQs z5Tokenizer._transform_pandas..column_tokenizer)pdSerieslocr transformr )rrrrrr_transform_pandasPs zTokenizer._transform_pandascCs4t|jd|j}|jjd|jd|d|jdS)N__name__z (columns=z, tokenization_fn=z, output_columns=))getattrr rr!r r )rnamerrr__repr__WszTokenizer.__repr__)NN)r! __module__ __qualname____doc__ _is_fittablerstrrrrr DataFramer r% __classcell__rrrrr s4 r ) typingrrrpandasrray.data.preprocessorrray.data.preprocessors.utilsrray.util.annotationsrr rrrrs