o bi @sddlZddlmZddlmZmZddlZddlmZm Z ddl m Z e e Zdedefdd Zd eedefd d Zd ededejfddZd eedejfddZdedejfddZdS)N)datetime)AnyList)create_ragged_ndarrayis_ndarray_like)_truncated_repr column_valuesreturncCst|tpt|tjpt|S)zxCheck whether a UDF column is valid. Valid columns must either be a list of elements, or an array-like object. ) isinstancelistnpndarrayr)rrT/home/ubuntu/.local/lib/python3.10/site-packages/ray/data/_internal/numpy_support.py_is_valid_column_valuess  r datetime_listcCsgd}d}|D]L}t|dd}|dkrd}n&|jdkr)|jddkr&d}nd}n|jdks8|jdks8|jdkr;d}nd }||}t||}|t|d krT||Sq||S) aDetect the highest precision for a list of datetime objects. Args: datetime_list: List of datetime objects. Returns: A string representing the highest precision among the datetime objects ('D', 's', 'ms', 'us', 'ns'). )Dsmsusnsr nanosecondrrrrr)getattr microsecondsecondminutehourindexmaxlen)rprecision_hierarchyhighest_precision_indexdtrcurrent_precision current_indexrrr"_detect_highest_datetime_precisions(     r'r$ precisioncCs^|dkr#|jd}t|dd}||}t|d}|t||dSt|d|dS)a} Converts a datetime object to a numpy datetime64 object with the specified precision. Args: dt: A datetime object to be converted. precision: The desired precision for the datetime64 conversion. Possible values are 'D', 's', 'ms', 'us', 'ns'. Returns: np.datetime64: A numpy datetime64 object with the specified precision. rrrrrz datetime64[])rrr datetime64 timedelta64astype)r$r(microseconds_as_ns nanosecondstotal_nanosecondsbase_dtrrr_convert_to_datetime64Gs   r1cs t|tfdd|DS)aConvert a list of datetime objects to a NumPy array of datetime64 with nanosecond precision. Args: datetime_list (List[datetime]): A list of `datetime` objects to be converted. Each `datetime` object represents a specific point in time. Returns: np.ndarray: A NumPy array containing the `datetime64` values of the datetime objects from the input list, with the appropriate precision (e.g., nanoseconds, microseconds, milliseconds, etc.). csg|]}t|qSr)r1).0r$r(rr uz4_convert_datetime_to_np_datetime..)r'r asarray)rrr3r _convert_datetime_to_np_datetimecsr7c Cst|tjr|St|trt|dkr$t|dtjr$tj|dddStdd|Dr1t|SzLtdd|DrBdd|D}t}d }|D]!}t|tjr[| |j |j fqIt|t rcd }qIt |sjd }qI|sst|dkrxt|WSt|WSty}ztjd t||d td t|d|d|d}~wwt|rzt|WSty}ztjd t||d td t|d|d|d}~ww|S)aConvert UDF columns (output of map_batches) to numpy, if possible. This includes lists of scalars, objects supporting the array protocol, and lists of objects supporting the array protocol, such as `[1, 2, 3]`, `Tensor([1, 2, 3])`, and `[array(1), array(2), array(3)]`. Returns: The input as an np.ndarray if possible, otherwise the original input. Raises: ValueError: If an input was array-like but we failed to convert it to an array. rr)axiscss|]}t|tVqdSN)r r)r2elemrrr sz#convert_to_numpy..css|]}t|VqdSr9)rr2errrr;scSsg|]}t|qSr)r r6r<rrrr4r5z$convert_to_numpy..FTz0Failed to convert column values to numpy array: )exc_infoz1Failed to convert column values to numpy array: (z): .N)r r r r r! expand_dimsallr7setadddtypeshapebytesisscalarrr6 Exceptionloggererrorr ValueErrorr)rshapes has_objectr=rrrconvert_to_numpyxs~          rN)loggingrtypingrrnumpyr $ray.air.util.tensor_extensions.utilsrrray.data._internal.utilr getLogger__name__rIboolrstrr'r*r1r r7rNrrrrs    *