o ci@shddlZddlmZmZddlZddlZddlm Z ddl m Z e e Ze ddGddde ZdS) N)ListOptional) Preprocessor) PublicAPIalpha) stabilityc @sreZdZdZdZ    ddeededeej de d e f d d Z d e j d dfddZd e j fddZddZdS) Concatenatora Combine numeric columns into a column of type :class:`~ray.air.util.tensor_extensions.pandas.TensorDtype`. Only columns specified in ``columns`` will be concatenated. This preprocessor concatenates numeric columns and stores the result in a new column. The new column contains :class:`~ray.air.util.tensor_extensions.pandas.TensorArrayElement` objects of shape :math:`(m,)`, where :math:`m` is the number of columns concatenated. The :math:`m` concatenated columns are dropped after concatenation. The preprocessor preserves the order of the columns provided in the ``colummns`` argument and will use that order when calling ``transform()`` and ``transform_batch()``. Examples: >>> import numpy as np >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import Concatenator :py:class:`Concatenator` combines numeric columns into a column of :py:class:`~ray.air.util.tensor_extensions.pandas.TensorDtype`. >>> df = pd.DataFrame({"X0": [0, 3, 1], "X1": [0.5, 0.2, 0.9]}) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> concatenator = Concatenator(columns=["X0", "X1"]) >>> concatenator.transform(ds).to_pandas() # doctest: +SKIP concat_out 0 [0.0, 0.5] 1 [3.0, 0.2] 2 [1.0, 0.9] By default, the created column is called `"concat_out"`, but you can specify a different name. >>> concatenator = Concatenator(columns=["X0", "X1"], output_column_name="tensor") >>> concatenator.transform(ds).to_pandas() # doctest: +SKIP tensor 0 [0.0, 0.5] 1 [3.0, 0.2] 2 [1.0, 0.9] >>> concatenator = Concatenator(columns=["X0", "X1"], dtype=np.float32) >>> concatenator.transform(ds) # doctest: +SKIP Dataset(num_rows=3, schema={Y: object, concat_out: TensorDtype(shape=(2,), dtype=float32)}) When ``flatten=True``, nested vectors in the columns will be flattened during concatenation: >>> df = pd.DataFrame({"X0": [[1, 2], [3, 4]], "X1": [0.5, 0.2]}) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> concatenator = Concatenator(columns=["X0", "X1"], flatten=True) >>> concatenator.transform(ds).to_pandas() # doctest: +SKIP concat_out 0 [1.0, 2.0, 0.5] 1 [3.0, 4.0, 0.2] Args: columns: A list of columns to concatenate. The provided order of the columns will be retained during concatenation. output_column_name: The desired name for the new column. Defaults to ``"concat_out"``. dtype: The ``dtype`` to convert the output tensors to. If unspecified, the ``dtype`` is determined by standard coercion rules. raise_if_missing: If ``True``, an error is raised if any of the columns in ``columns`` don't exist. Defaults to ``False``. flatten: If ``True``, nested vectors in the columns will be flattened during concatenation. Defaults to ``False``. Raises: ValueError: if `raise_if_missing` is `True` and a column in `columns` or doesn't exist in the dataset. F concat_outNcolumnsoutput_column_namedtyperaise_if_missingflattencCs"||_||_||_||_||_dSN)r r r r r)selfr r r r rrW/home/ubuntu/.local/lib/python3.10/site-packages/ray/data/preprocessors/concatenator.py__init__Ys  zConcatenator.__init__dfreturncCsHt|jt|}|r"d|jd|}|jrt|t|dSdS)NzMissing columns specified in 'z': )setr r ValueErrorloggerwarning)rrmissing_columnsmessagerrr _validatehszConcatenator._validatecsv|jr|j}fdd|D}n |jjjd}|jjd}tt||j ddj f<|S)Ncs$g|]}tfdd|DqS)cs2g|]}jdurt|nt|jqSr)r np atleast_1dastype).0elemrrr zs  z=Concatenator._transform_pandas...)r concatenate)r rowr"rrr#xs  z2Concatenator._transform_pandas..)r )r ) rrr to_numpyr droppdSerieslistlocr )rr concatenatedrr"r_transform_pandasss    zConcatenator._transform_pandascCsfdddddd}g}|D]\}}t||}||kr%||d|q|jjdd|dS)Nr F)r r r r r=(z, ))itemsgetattrappend __class____name__join)rdefault_valuesnon_default_arguments parameter default_valuevaluerrr__repr__s zConcatenator.__repr__)r NFF)r5 __module__ __qualname____doc__ _is_fittablerstrrrr boolrr( DataFramerr-r<rrrrr s*H  r)loggingtypingrrnumpyrpandasr(ray.data.preprocessorrray.util.annotationsr getLoggerr5rrrrrrs