o `۷i|@sdZddlZddlZddlZddlmZddlmZmZm Z m Z ddl m Z ddl mZeGdddeZeGd d d ejZeGd d d eZeGd ddeZGdddZdS)z Serialization handlers for preprocessor save/load functionality. This module implements a factory pattern to abstract different serialization formats, making it easier to add new formats and maintain existing ones. N)Enum)AnyDictOptionalUnion) cloudpickle) DeveloperAPIc@seZdZdZdZdZdS)HandlerFormatNamez1Enum for consistent format naming in the factory.rpickleN)__name__ __module__ __qualname____doc__ CLOUDPICKLEPICKLErrc/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/data/preprocessors/serialization_handlers.pyr sr c@seZdZdZejdedeee ffdeee ffddZ ejdeee fde fdd Z ejdeee ffd d Z deee fdeee ffd d ZdS)SerializationHandlerzDAbstract base class for handling preprocessor serialization formats.data PreprocessorreturncCdS)zSerialize preprocessor data to the specific format. Args: data: Dictionary containing preprocessor metadata and stats Returns: Serialized data in format-specific representation Nrselfrrrr serializes zSerializationHandler.serialize serializedcCr)aUDeserialize data from the specific format. Args: serialized: Serialized data in format-specific representation Returns: For structured formats (CloudPickle/JSON/MessagePack): Dictionary containing preprocessor metadata and stats For pickle format: The actual deserialized object Nrrrrrr deserialize,s z SerializationHandler.deserializecCr)z+Get the magic bytes/prefix for this format.Nrrrrrget_magic_bytes9sz$SerializationHandler.get_magic_bytescCs4|}t|ttfr||r|t|dS|S)z(Remove magic bytes from serialized data.N)r isinstancestrbytes startswithlen)rrmagicrrrstrip_magic_bytes>sz&SerializationHandler.strip_magic_bytesN)r r r rabcabstractmethodrrr!rr"rrrr&rrrrrs    &rc@s^eZdZdZdZdedeeeffde fddZ de deeeffd d Z de fd d Z d S)CloudPickleSerializationHandlerz-Handler for CloudPickle serialization format.sCPKL:rrrcCs|jt|S)z2Serialize to CloudPickle format with magic prefix.)MAGIC_CLOUDPICKLErdumpsrrrrrLsz)CloudPickleSerializationHandler.serializercCsRt|tstdt|||jstd|dd||}t|S)z$Deserialize from CloudPickle format.z4Expected bytes for CloudPickle deserialization, got z!Invalid CloudPickle magic bytes: N ) r r" ValueErrortyper#r*r&rloads)rrcloudpickle_datarrrrRs     z+CloudPickleSerializationHandler.deserializecCs|jS)N)r*rrrrr_sz/CloudPickleSerializationHandler.get_magic_bytesN) r r r rr*rrr!rr"rrrrrrrr)Fs  r)c@sReZdZdZdedeeeffdefddZdedefdd Z defd d Z d S) PickleSerializationHandlerz/Handler for legacy Pickle serialization format.rrrcCstt|dS)z Serialize using pickle format (for backward compatibility). data is ignored, but kept for consistency ascii)base64 b64encoder r+decoderrrrrgsz$PickleSerializationHandler.serializercCstt|S)z0Deserialize from pickle format (legacy support).)r r/r3 b64decoderrrrrqsz&PickleSerializationHandler.deserializecCr)Nrrrrrrxsz*PickleSerializationHandler.get_magic_bytesN) r r r rrrr!rrrrrrrrr1cs   r1c @seZdZdZejeejeiZ e dede fddZ e  dde ede eeefd efd d Ze d eeefd efd dZdS)SerializationHandlerFactoryz>Factory class for creating appropriate serialization handlers. format_name handler_classcCs||j|<dS)z%Register a new serialization handler.N) _handlers)clsr9r:rrrregister_handlersz,SerializationHandlerFactory.register_handlerNformat_identifierrrcKsH|s||}||jvrtd|jdt|j|j|}|S)aGet the appropriate serialization handler for a format or serialized data. Args: format_identifier: The format to use for serialization. If None, will detect from data. data: Serialized data to detect format from (used when format_identifier is None). **kwargs: Additional keyword arguments (currently unused). Returns: SerializationHandler instance for the format Raises: ValueError: If format is not supported or cannot be detected z"Unsupported serialization format: z. Supported formats: ) detect_formatr;r-valuelistkeys)r<r>rkwargsr:rrr get_handlers     z'SerializationHandlerFactory.get_handlerrcCsDt|tr|tjrtjSt|trtjSt d|ddd)zDetect the serialization format from the magic bytes. Args: serialized: Serialized data Returns: Format name enum Raises: ValueError: If format cannot be detected z)Cannot detect serialization format from: Nz...) r r"r#r)r*r rr!rr-)r<rrrrr?s z)SerializationHandlerFactory.detect_format)NN)r r r rr rr)rr1r; classmethodr.r=rrr!r"rrDr?rrrrr8|s&  r8)rr'r3r enumrtypingrrrrray.cloudpicklerray.util.annotationsrr ABCrr)r1r8rrrrs"   +