o $iS @sddlmZddlmZmZmZddlZddlmZm Z er$ddl m Z deeededfd d Z gd Z d edefd dZdedefddZde de fddZdS))ThreadPoolExecutor) TYPE_CHECKINGAnyListN) BlockAccessor CallableClass RefBundle simple_datareturnr c Cstddl}ddl}ddlm}g}|D]%}|d|i}||t|t | fgd|j j j |dddq|S) zcCreate ref bundles from a list of block data. One bundle is created for each input block. rNridTF)preserve_index) owns_blocksschema)pandaspyarrow'ray.data._internal.execution.interfacesr DataFrameappendrayputr for_block get_metadatalibSchema from_pandas)r pdpar outputblockr ^/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/ray/data/_internal/execution/util.pymake_ref_bundles s"   r")BKiBMiBGiBTiBPiB num_bytescCsVd}|dkr"|ttdkr"|d}|d7}|dkr"|ttdks|dt|S)zDReturn a human-readable memory string for the given amount of bytes.riz.1f)len memory_units)r)kr r r! memory_string*s r. locality_hitscCs|sdSd|d||dS)z9Return a human-readable string for object locality stats.z[all objects local][/z objects local]r )r/locality_missesr r r!locality_string3sr3 callable_clscCsGddd|}|S)aBReturns a thread-safe CallableClass with the same logic as the provided `callable_cls`. This function allows the usage of concurrent actors by safeguarding user logic behind a separate thread. This allows batch slicing and formatting to occur concurrently, to overlap with the user provided UDF. cs4eZdZfddZfddZfddZZS)zCmake_callable_class_single_threaded.._SingleThreadedWrappercs"tdd|_tj|i|dS)Nr*) max_workers)rthread_pool_executorsuper__init__)selfargskwargs __class__r r!r8Fs zLmake_callable_class_single_threaded.._SingleThreadedWrapper.__init__cs tSN)r7__repr__)r9r<r r!r?Js zLmake_callable_class_single_threaded.._SingleThreadedWrapper.__repr__cs&|jjtjg|Ri|}|Sr>)r6submitr7__call__result)r9r:r;futurer<r r!rAMszLmake_callable_class_single_threaded.._SingleThreadedWrapper.__call__)__name__ __module__ __qualname__r8r?rA __classcell__r r r<r!_SingleThreadedWrapperEs  rHr )r4rHr r r!#make_callable_class_single_threaded:s  rI)concurrent.futuresrtypingrrrrray.data.blockrrrr r"r,floatstrr.intr3rIr r r r!s