o bi0 @sddlZddlmZddlmZmZmZmZddlZ ddl m Z ddl m Z ddlmZmZer4ddlZzddlZWn eyEdZYnwdaddZeGd d d eeZeGd d d eeZ d(de deddfddZ d(dddedede fddZe d(de defddZe d(dddedefddZde dee jeee jfffddZ de jdedee jffdd Z!de jde jfd!d"Z"d)d$d%Z#d)d&d'Z$dS)*N)Enum) TYPE_CHECKINGDictListUnion)TENSOR_COLUMN_NAME) DataBatchType) Deprecated DeveloperAPIcCstdur ddl}|atS)Nr)_pandaspandas)r r V/home/ubuntu/.local/lib/python3.10/site-packages/ray/air/util/data_batch_conversion.py_lazy_import_pandassrc@seZdZdZdZdZdS) BatchFormatr arrownumpyN)__name__ __module__ __qualname__PANDASARROWNUMPYr r r rr"src@seZdZdZdZdZdZdS) BlockFormatz#Internal Dataset block format enum.r rsimpleN)rrr__doc__rrSIMPLEr r r rr*s rFdatacast_tensor_columnsreturn pd.DataFramecCst}t|tjr|tt|i}nPt|trBi}|D]\}}t|tjs5t dt |d|dt|||<q||}n!t durQt|t j rQ| }nt||jsct dt |dt|rit|}|S)a Convert the provided data to a Pandas DataFrame. Args: data: Data of type DataBatchType cast_tensor_columns: Whether tensor columns should be cast to NumPy ndarrays. Returns: A pandas Dataframe representation of the input data. GAll values in the provided dict must be of type np.ndarray. Found type for key instead.NReceived data of type: , but expected it to be one of )r isinstancenpndarray DataFramer_ndarray_to_columndictitems ValueErrortypepyarrowTable to_pandasr _cast_tensor_columns_to_ndarrays)rrpd tensor_dictcol_namecolr r r_convert_batch_type_to_pandas3s6       r7r.cCs|rt|}|tjkr |S|tjkr5t|jdkr$|jdddfSi}|D] }||||<q(|S|tjkrFt s@t dt j |St d|dt )`Convert the provided Pandas dataframe to the provided ``type``. Args: data: A Pandas DataFrame type: The specific ``BatchFormat`` to convert to. cast_tensor_columns: Whether tensor columns should be cast to our tensor extension type. Returns: The input data represented with the provided type. Nrz|Attempted to convert data to Pyarrow Table but Pyarrow is not installed. Please do `pip install pyarrow` to install Pyarrow.zReceived type r%))_cast_ndarray_columns_to_tensor_extensionrrrlencolumnsilocto_numpyrr/r-r0 from_pandasr)rr.r output_dictcolumnr r r_convert_pandas_to_batch_type\s(    rBcCstdtt||dS)a5Convert the provided data to a Pandas DataFrame. This API is deprecated from Ray 2.4. Args: data: Data of type DataBatchType cast_tensor_columns: Whether tensor columns should be cast to NumPy ndarrays. Returns: A pandas Dataframe representation of the input data. z`convert_batch_type_to_pandas` is deprecated as a developer API starting from Ray 2.4. All batch format conversions should be done manually instead of relying on this API.rr)warningswarnPendingDeprecationWarningr7rCr r rconvert_batch_type_to_pandassrGcCstdtt|||dS)r8z`convert_pandas_to_batch_type` is deprecated as a developer API starting from Ray 2.4. All batch format conversions should be done manually instead of relying on this API.rr.r)rDrErFrBrHr r rconvert_pandas_to_batch_typesrIc Cs"t}t|tjr |St|tr.|D]\}}t|tjs+tdt|d|dq|Stduryt|tj ryddl m }ddl m }g}|jD]}||}||j|dd qI|}|jtgkrqt|jjd|rq|dStt|j|St||jrt|tjStd t|d t) zConvert the provided data to a NumPy ndarray or dict of ndarrays. Args: data: Data of type DataBatchType Returns: A numpy representation of the input data. r!r"r#Nr),get_arrow_extension_fixed_shape_tensor_types)transform_pyarrowF)zero_copy_onlyr$r%)rr&r'r(r+r,r-r.r/r0$ray.air.util.tensor_extensions.arrowrJray.data._internal.arrow_opsrKr<combine_chunked_arrayappendr> column_namesrschematypeszipr)rBrrr) rr3r5r6rJrKcolumn_values_ndarrayscombined_arrayarrow_fixed_shape_tensor_typesr r r_convert_batch_type_to_numpysJ            rXarrz pd.SeriescCs.t}z||WStyt|YSw)zConvert a NumPy ndarray into an appropriate column format for insertion into a pandas DataFrame. If conversion to a pandas Series fails (e.g. if the ndarray is multi-dimensional), fall back to a list of NumPy ndarrays. )rSeriesr-list)rYr3r r rr*s    r*cCsB|jjtjurz tdd|D}W|StyY|Sw|S)zUnwrap an object-dtyped NumPy ndarray containing ndarray pointers into a single contiguous ndarray, if needed/possible. cSsg|]}t|qSr )r'asarray).0vr r r sz9_unwrap_ndarray_object_type_if_needed..)dtyper.r'object_array Exception)rYr r r%_unwrap_ndarray_object_type_if_neededs rddfc Cst}z|jjj}Wn ty|jj}Ynwddlm}m}| D]H\}}||rlz*t t j dt dt j d|d||||<Wdn1sPwYWq$tyk}z td|d|d}~wwq$|S)zY Cast all NumPy ndarray columns in df to our tensor extension type, TensorArray. r) TensorArraycolumn_needs_tensor_extensionignorecategoryNzTried to cast column z to the TensorArray tensor extension type but the conversion failed. To disable automatic casting to this tensor extension, set ctx = DataContext.get_current(); ctx.enable_tensor_extension_casting = False.)rcorecommonSettingWithCopyWarningAttributeErrorerrors%ray.air.util.tensor_extensions.pandasrfrgr,rDcatch_warnings simplefilter FutureWarningrcr-)rer3rmrfrgr5r6er r rr:s4     r:c Cst}z|jjj}Wn ty|jj}Ynwddlm}|D]4\}}t |j |rVt t j dtdt j d|dt|||<Wdn1sQwYq"|S)z:Cast all tensor extension columns in df to NumPy ndarrays.r) TensorDtyperhriN)rrkrlrmrnrorprur,r&r`rDrqrrrsr[r>)rer3rmrur5r6r r rr2Is      r2)F)rer rr )%rDenumrtypingrrrrrr'ray.air.constantsrray.air.data_batch_typerray.util.annotationsr r r r3r/ ImportErrorr rstrrrboolr7rBrGrIr(rXr*rdr:r2r r r rs~        , /   "; -