o $iz@sXddlmZmZmZddlmZddlmZeddGdddZGdd d eZ d S) )AnyDictOptional)Block) PublicAPIalpha) stabilityc @sXeZdZdZdededededef ddZd eee fdededed edef d d Z d S)FilenameProvideraGenerates filenames when you write a :class:`~ray.data.Dataset`. Use this class to customize the filenames used when writing a Dataset. Some methods write each row to a separate file, while others write each block to a separate file. For example, :meth:`ray.data.Dataset.write_images` writes individual rows, and :func:`ray.data.Dataset.write_parquet` writes blocks of data. For more information about blocks, see :ref:`Data internals `. If you're writing each row to a separate file, implement :meth:`~FilenameProvider.get_filename_for_row`. Otherwise, implement :meth:`~FilenameProvider.get_filename_for_block`. Example: This snippet shows you how to encode labels in written files. For example, if `"cat"` is a label, you might write a file named `cat_000000_000000_000000.png`. .. testcode:: import ray from ray.data.datasource import FilenameProvider class ImageFilenameProvider(FilenameProvider): def __init__(self, file_format: str): self.file_format = file_format def get_filename_for_row(self, row, write_uuid, task_index, block_index, row_index): return ( f"{row['label']}_{write_uuid}_{task_index:06}_{block_index:06}" f"_{row_index:06}.{self.file_format}" ) ds = ray.data.read_parquet("s3://anonymous@ray-example-data/images.parquet") ds.write_images( "/tmp/results", column="image", filename_provider=ImageFilenameProvider("png") ) block write_uuid task_index block_indexreturncCt)aVGenerate a filename for a block of data. .. note:: Filenames must be unique and deterministic for a given write UUID, and task and block index. A block consists of multiple rows and corresponds to a single output file. Each task might produce a different number of blocks. Args: block: The block that will be written to a file. write_uuid: The UUID of the write operation. task_index: The index of the write task. block_index: The index of the block *within* the write task. NotImplementedError)selfr r r r rb/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/ray/data/datasource/filename_provider.pyget_filename_for_block3sz'FilenameProvider.get_filename_for_blockrow row_indexcCr)aGenerate a filename for a row. .. note:: Filenames must be unique and deterministic for a given write UUID, and task, block, and row index. A block consists of multiple rows, and each row corresponds to a single output file. Each task might produce a different number of blocks, and each block might contain a different number of rows. .. tip:: If you require a contiguous row index into the global dataset, use :meth:`~ray.data.Dataset.iter_rows`. This method is single-threaded and isn't recommended for large datasets. Args: row: The row that will be written to a file. write_uuid: The UUID of the write operation. task_index: The index of the write task. block_index: The index of the block *within* the write task. row_index: The index of the row *within* the block. r)rrr r r rrrrget_filename_for_rowGsz%FilenameProvider.get_filename_for_rowN) __name__ __module__ __qualname____doc__rstrintrrrrrrrrr s4*  r c @seZdZ ddeedeefddZdededed ed ef d d Zd e ee fdeded eded ef ddZ ded efddZ dS)_DefaultFilenameProviderN dataset_uuid file_formatcCs||_||_dS)N _dataset_uuid _file_format)rr r!rrr__init__is z!_DefaultFilenameProvider.__init__r r r r rcCs"|d|dd|d}||SN_06_generate_filename)rr r r r file_idrrrros z/_DefaultFilenameProvider.get_filename_for_blockrrcCs*|d|dd|dd|d}||Sr&r))rrr r r rr+rrrrus  z-_DefaultFilenameProvider.get_filename_for_rowr+cCsDd}|jdur||jd7}||7}|jdur |d|j7}|S)Nr'.r")rr+filenamerrrr*s  z+_DefaultFilenameProvider._generate_filename)NN) rrrrrr%rrrrrrr*rrrrrhs@     rN) typingrrrray.data.blockrray.util.annotationsrr rrrrrs   `