o
bi
@ sF U d dl Z d dlmZmZmZmZmZ d dlmZ d dl m
Z
d dl m Z m
Z
mZ d dlmZ d dlmZmZ er=d dlZdZee ed< z?d d lmZ d
e jvre rd dlZd dl Z d dl Ze j ej ! d Z"ej#$d
e"Z%ej#&e%Z'e'e je%j(< e%j)*e' W n ey Z+ ze+ZW Y dZ+[+ndZ+[+ww G d d
d
eZ,dS ) N)
TYPE_CHECKINGIterableListOptionalUnion)pyarrow_table_from_pydict)_check_pyarrow_version)Block
BlockAccessor
BlockMetadata)Dataset)
DatasourceReadTaskTRANSFORMERS_IMPORT_ERROR)is_datasets_availabledatasets_modulesz __init__.pyc @ s~ e Zd ZdZ dded defddZeded defd d
Z de
e fd d Z de e
fd
dZdedee fddZdS )HuggingFaceDatasourceah Hugging Face Dataset datasource, for reading from a
`Hugging Face Datasets Dataset `_.
This Datasource implements a streamed read using a
single read task, most beneficial for a
`Hugging Face Datasets IterableDataset `_
or datasets which are too large to fit in-memory.
For an in-memory Hugging Face Dataset (`datasets.Dataset`), use :meth:`~ray.data.from_huggingface`
directly for faster performance.
dataset)zdatasets.Datasetzdatasets.IterableDataset
batch_sizec C s t d urt || _|| _d S N)r _dataset _batch_size)selfr r r h/home/ubuntu/.local/lib/python3.10/site-packages/ray/data/_internal/datasource/huggingface_datasource.py__init__< s
z HuggingFaceDatasource.__init__returnc C s ddl }|jj}|jj}t|j}t||js;ddl m} z||||d}|j |j kr.g W S W n t
y: g Y S w ddl }d| d| d| } | | }
|
j
|jd kr[|
S g S ) zReturn list of Hugging Face hosted parquet file URLs if they
exist for the data (i.e. if the dataset is a public dataset that
has not been transformed) else return an empty list.r N) load_dataset)splitz$https://huggingface.co/api/datasets/z /parquet//ok)datasetsinfo dataset_name config_namestrr
isinstanceIterableDatasetr _fingerprint Exceptionrequestsget status_codecodesjson) clsr r" r$ r%
split_namer dsr+
public_urlrespr r r list_parquet_urls_from_datasetG s2
z4HuggingFaceDatasource.list_parquet_urls_from_datasetc C s | j jS r )r dataset_size)r r r r estimate_inmemory_data_sizev s z1HuggingFaceDatasource.estimate_inmemory_data_sizec c s dd l }dd l}dd l}| jdj| jdD ]5}t||j|j t
|j fs0t dt
| dt||jr:d|i}t|t
rCt|}t| }|V qd S )Nr arrow)r z
Batch format z isn't supported. Only the following batch formats are supported: dict (corresponds to `None` in `dataset.with_format()`), pyarrow.Table, np.array, pd.DataFrame.item)numpypandaspyarrowr with_formatiterr r' Table DataFramedictarray
ValueErrortypendarrayr r
for_block
to_default)r nppdr< batchblockr r r
_read_datasety s$
z#HuggingFaceDatasource._read_dataset parallelismc C s( t td d d d d}t| j|g}|S )N)num_rows
size_bytes input_files
exec_stats)r r r rL )r rM meta
read_tasksr r r get_read_tasks s z$HuggingFaceDatasource.get_read_tasksN)r )__name__
__module__ __qualname____doc__r intr classmethodr r5 r r7 r r rL r r rT r r r r r 1 s*
.$r )-systypingr r r r r $ray.air.util.tensor_extensions.arrowr ray.data._internal.utilr ray.data.blockr r
r ray.data.datasetr ray.data.datasourcer
r r" r ImportError__annotations__transformers.utilsr modules importlibos
datasets.loadpathjoinloadinit_dynamic_modulesdynamic_modules_pathutilspec_from_file_locationspecmodule_from_specr nameloader exec_moduleer r r r r s>