o `۷i) @sXddlZddlmZddlmZmZmZmZddlZ ddl m Z ddl m Z ddlmZer2ddlZzddlZWn eyCdZYnwdaddZeGd d d eeZ d"d e d eddfddZ d"d dded ede fddZd e dee jeee jfffddZde jdedee jffddZde jde jfddZd#ddZd#d d!Z dS)$N)Enum) TYPE_CHECKINGDictListUnion) DataBatchType)TENSOR_COLUMN_NAME) DeveloperAPIcCstdur ddl}|atS)Nr)_pandaspandas)r r Y/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/data/util/data_batch_conversion.py_lazy_import_pandassrc@seZdZdZdZdZdS) BatchFormatr arrownumpyN)__name__ __module__ __qualname__PANDASARROWNUMPYr r r r r"srFdatacast_tensor_columnsreturn pd.DataFramecCst}t|tjr|tt|i}nPt|trBi}|D]\}}t|tjs5t dt |d|dt|||<q||}n!t durQt|t j rQ| }nt||jsct dt |dt|rit|}|S)a Convert the provided data to a Pandas DataFrame. Args: data: Data of type DataBatchType cast_tensor_columns: Whether tensor columns should be cast to NumPy ndarrays. Returns: A pandas Dataframe representation of the input data. GAll values in the provided dict must be of type np.ndarray. Found type for key instead.NReceived data of type: , but expected it to be one of )r isinstancenpndarray DataFramer_ndarray_to_columndictitems ValueErrortypepyarrowTable to_pandasr _cast_tensor_columns_to_ndarrays)rrpd tensor_dictcol_namecolr r r _convert_batch_type_to_pandas*s6       r2r)cCs|rt|}|tjkr |S|tjkr5t|jdkr$|jdddfSi}|D] }||||<q(|S|tjkrFt s@t dt j |St d|dt )a`Convert the provided Pandas dataframe to the provided ``type``. Args: data: A Pandas DataFrame type: The specific ``BatchFormat`` to convert to. cast_tensor_columns: Whether tensor columns should be cast to our tensor extension type. Returns: The input data represented with the provided type. Nrz|Attempted to convert data to Pyarrow Table but Pyarrow is not installed. Please do `pip install pyarrow` to install Pyarrow.zReceived type r ))_cast_ndarray_columns_to_tensor_extensionrrrlencolumnsilocto_numpyrr*r(r+ from_pandasr)rr)r output_dictcolumnr r r _convert_pandas_to_batch_typeSs(    r<c Cs"t}t|tjr |St|tr.|D]\}}t|tjs+tdt|d|dq|Stduryt|tj ryddl m }ddl m }g}|jD]}||}||j|dd qI|}|jtgkrqt|jjd|rq|dStt|j|St||jrt|tjStd t|d t) zConvert the provided data to a NumPy ndarray or dict of ndarrays. Args: data: Data of type DataBatchType Returns: A numpy representation of the input data. rrrNr)transform_pyarrow),get_arrow_extension_fixed_shape_tensor_typesF)zero_copy_onlyrr )rr!r"r#r&r'r(r)r*r+ray.data._internal.arrow_opsr=*ray.data._internal.tensor_extensions.arrowr>r6combine_chunked_arrayappendr8 column_namesrschematypeszipr$r<rrr) rr.r0r1r=r>column_values_ndarrayscombined_arrayarrow_fixed_shape_tensor_typesr r r _convert_batch_type_to_numpysJ            rKarrz pd.SeriescCs.t}z||WStyt|YSw)zConvert a NumPy ndarray into an appropriate column format for insertion into a pandas DataFrame. If conversion to a pandas Series fails (e.g. if the ndarray is multi-dimensional), fall back to a list of NumPy ndarrays. )rSeriesr(list)rLr.r r r r%s    r%cCsB|jjtjurz tdd|D}W|StyY|Sw|S)zUnwrap an object-dtyped NumPy ndarray containing ndarray pointers into a single contiguous ndarray, if needed/possible. cSsg|]}t|qSr )r"asarray).0vr r r sz9_unwrap_ndarray_object_type_if_needed..)dtyper)r"object_array Exception)rLr r r %_unwrap_ndarray_object_type_if_neededs rWdfc Cst}z|jjj}Wn ty|jj}Ynwddlm}m}| D]H\}}||rlz*t t j dt dt j d|d||||<Wdn1sPwYWq$tyk}z td|d|d}~wwq$|S)zY Cast all NumPy ndarray columns in df to our tensor extension type, TensorArray. r) TensorArraycolumn_needs_tensor_extensionignorecategoryNzTried to cast column z to the TensorArray tensor extension type but the conversion failed. To disable automatic casting to this tensor extension, set ctx = DataContext.get_current(); ctx.enable_tensor_extension_casting = False.)rcorecommonSettingWithCopyWarningAttributeErrorerrors+ray.data._internal.tensor_extensions.pandasrYrZr'warningscatch_warnings simplefilter FutureWarningrVr()rXr.r`rYrZr0r1er r r r4s4     r4c Cst}z|jjj}Wn ty|jj}Ynwddlm}|D]4\}}t |j |rVt t j dtdt j d|dt|||<Wdn1sQwYq"|S)z:Cast all tensor extension columns in df to NumPy ndarrays.r) TensorDtyper[r\N)rr^r_r`rarbrcrir'r!rSrdrerfrgrNr8)rXr.r`rir0r1r r r r-s      r-)F)rXrrr)!rdenumrtypingrrrrrr"ray.air.data_batch_typerray.data.constantsrray.util.annotationsr r r.r* ImportErrorr rstrrboolr2r<r#rKr%rWr4r-r r r r sZ         , . "; -