o qoi@y @sUdZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlmZmZmZmZmZmZddlmZddlmZddlmZddlmZddlmZdd lmZd Zd ej fd d Z!ej"ddGdddej#Z$ej"ddGddde$Z%ej"ddGddde$Z&ej"ddGdddej#Z'ej(ddZ)e)j*Z*e)j+Z+ddZ,ddZ-deefddZ.e/e0fD]Z1e*e1d d!e1fd"d!d#d!d$qe*e2d%d!d&d!d'd!d$e*ej3d(d!d)d!d*d!d$d+d,Z4d-ej5d.e6fd/d0Z7d1eej5d.ee6fd2d3Z8Gd4d5d5e9Z:Gd6d7d7ej;d8Zd=e6d>e6d.e6fd?d@Z?dAee6fdBdCZ@dDe6d.e6fdEdFZAej"GdGdHdHZBiZCeeDeBfeEdI<iZFeeeBfeEdJ<d=e6d>e6dKeGfdLdMZHdNeejId.dfdOdPZJdQZKdRZLdSZMdTZNdUZOd1ZPdVZQdWZRdXZSdYZTdZZUd[ZVd\ZWd=ZXdDZYd]d^ZZGd_d`d`Z[Gdadbdbe9Z\GdcddddZ]  dldYedeee<dfeeDd.e6fdgdhZ^ dmdie6deee<d.efdjdkZ_dS)na(Functions to help serialize configuration. This module provides functions that help ensure pickle-compatibility (`clear_argument_history`), as well as functions and associated helper classes used to serialize Fiddle structures into a custom JSON-based representation (`dump_json`, `load_json`). N)AnyDictIterableListOptionalType)daglish)config)reraised_exception)special_overrides)daglish_legacy) lazy_importsz0.0.1 buildablecCs&dtdtjdtfdd}tj||S)a'Creates a copy of a buildable, clearing its history. This can be useful when a Config's history contains a non-picklable value. Args: buildable: A Fiddle configuration DAG whose history will be removed. Returns: A copy of `buildable` with `__argument_history__` set to empty. valuestatereturncSsN||r"t|tjr||}|j}|j|j |S| |St |SN) is_traversable isinstance config_lib Buildableflattened_map_childrenmetadatawithout_historynode_traverser unflattenvalues map_childrencopydeepcopy)rr sub_resultsrr!Z/home/ubuntu/.local/lib/python3.10/site-packages/fiddle/_src/experimental/serialization.pytraverse>s       z(clear_argument_history..traverse)rrStateMemoizedTraversalrun)rr#r!r!r"clear_argument_history2s  r'T)frozenc@seZdZdZdefddZdS)UnreachableElementzRepresents an unreachable element. These are added during the serialization traversal below to provide more informative error messages. containercCstd)Nz'Unreachable elements can't be followed.)NotImplementedErrorselfr*r!r!r"followTszUnreachableElement.followN)__name__ __module__ __qualname____doc__rr.r!r!r!r"r)Lsr)c@eZdZdZeddZdS) SetElementzRepresents an element of a set.cCdS)Nz[]r!r-r!r!r"code\zSetElement.codeNr/r0r1r2propertyr7r!r!r!r"r4Xr4c@r3)MetadataElementz-Represents traversal into a value's metadata.cCr5)Nz .r!r6r!r!r"r7er8zMetadataElement.codeNr9r!r!r!r"r<ar;r<c@s*eZdZdZeddZdefddZdS)IdentityElementz0Represents a path element that is its container.cCr5)Nr!r6r!r!r"r7nr8zIdentityElement.coder*cCs|Srr!r,r!r!r"r.rzIdentityElement.followN)r/r0r1r2r:r7rr.r!r!r!r"r=js  r=) use_fallbackcCs$t|j}|jt||ffSr)tuple__dict__keysrtype)instancerCr!r!r"_flatten_standard_object~srFcCs(|\}}t|}|jt|||Sr)object__new__rBupdatezip)rr object_type dict_keysrEr!r!r"_unflatten_standard_objects rMrKcCst|ttddddS)aNRegisters serialization support for the given (dict-based) `object_type`. This adds serialization support for "dict-based" Python objects, where here dict-based means that we can simply serialize the `object_type.__dict__` attribute, and on deserialization create a new empty instance of `object_type` and restore its `__dict__` contents. If this behavior is insufficient or incorrect for a given type, more explicit handling using `register_node_traverser` will be required to support serialization. Args: object_type: The type to register serialization support for. cSstdd|jDS)Ncss|]}t|VqdSr)rAttr).0kr!r!r" sz?register_dict_based_object....)rArBxr!r!r"z,register_dict_based_object.. flatten_fn unflatten_fnpath_elements_fnN)register_node_traverserrFrM)rKr!r!r"register_dict_based_objects   r[cCs t|dfSrrArRr!r!r"rTs rTcCs||Srr!)r_set_typer!r!r"rTcCstdd|DS)Ncss|]}tVqdSr)r4)rOr]r!r!r"rQsz..r\rRr!r!r"rTsrVcCs|dfdfS)Nraw_unicode_escape)decoderRr!r!r"rTscCs|ddS)Nrr`)encode)rr]r!r!r"rTscCstfSr)r=rRr!r!r"rTr_cCr5)N)r!Nr!r]r!r!r"rTcCstjSr)rNO_VALUE)rrr!r!r"rTscCr5)Nr!r!rcr!r!r"rTrdcCs|tttttdfvS)zDReturns whether `value_type` is a JSON-representable primitive type.N)intfloatboolstrrD) value_typer!r!r" _is_leaf_typesrkpathrcCsdt|S)z!Returns a path string for `path`.z)rpath_str)rlr!r!r"rmrmpathscCsdd|DS)z8Returns a list of path strings corresponding to `paths`.cSsg|]}t|qSr!)rm)rOrlr!r!r" rUzpath_strs..r!)ror!r!r" path_strsrnrqc@eZdZdZdS)UnserializableValueErrorz@An exception thrown when an unserializable value is encountered.Nr/r0r1r2r!r!r!r"rssrsc@s>eZdZdZejdededefddZde defdd Z d S) PyrefPolicyzRepresents a policy for importing references to Python objects. A `PyrefPolicy` provides control over what kind of Python symbols can be loaded during deserialization. modulesymbolrcCr5)aHReturns whether this policy allows importing `symbol` from `module`. This will be called by `import_symbol` before any imports take place. If this returns `True`, `symbol` will be imported from `module`. Args: module: The module to import `symbol` from. symbol: The symbol to import from `module`. Nr!r-rvrwr!r!r" allows_importrdzPyrefPolicy.allows_importrcCr5)zReturns whether this policy allows an imported `value` to be used. This is called after `value` has already been imported, but before it is returned by `import_symbol`. Args: value: An imported value. Nr!r-rr!r!r" allows_valuerdzPyrefPolicy.allows_valueN) r/r0r1r2abcabstractmethodrirhryrr{r!r!r!r"rus  ru) metaclasscs8eZdZdZeZdedededeffdd Z Z S)PyrefPolicyErrorzAAn error arising when an import is disallowed by a `PyrefPolicy`.policyrvrwrcsZ||_||_||_|tjurd|d}nd}d|d||d|d}t|dS)Nz (with value )r>z Importing  from z9 is not permitted by the active Python reference policy (z).)rvrwrr PRE_IMPORTsuper__init__)r-rrvrwr value_strmsg __class__r!r"rs zPyrefPolicyError.__init__) r/r0r1r2rGrrurirr __classcell__r!r!rr"rs&rc@s&eZdZdZddZdefddZdS)DefaultPyrefPolicyaProvides a default policy for loading pyref values. This policy excludes builtins that are not a supported serializable type (e.g., `list`, `tuple`, `dict`, etc are allowed). This policy is subject to change and may become stricter in the future. cCr5)NTr!rxr!r!r"ryr?z DefaultPyrefPolicy.allows_importrcCs<t|to t|du}tt|dd}|tjv}|p| S)Nr/)rrDfind_node_traversergetattrinspect getmodulesysbuiltin_module_names)r-ris_serializable_type module_name is_builtinr!r!r"r{s   zDefaultPyrefPolicy.allows_valueN)r/r0r1r2ryrhr{r!r!r!r"rsrrvrwcCsd|d|dS)Nz9 Fiddle context: Error occurred while importing pyref to r.r!)rvrwr!r!r"_fiddle_pyref_context srrcCstj}|||rDt||}tt||}t |t |}| dD]}t ||}q&Wdn1s8wY||rD|St||||)aReturns the value obtained from importing `symbol` from `module`. Args: policy: The `PyrefPolicy` governing which symbols can be imported. module: The module to import `symbol` from. symbol: The symbol to import from `module`. Raises: ModuleNotFoundError: If `module` can't be found. AttributeError: If `symbol` can't be accessed on `module`. PyrefPolicyError: If importing `symbol` from `module` is disallowed by this `PyrefPolicy`. rN)rrryr ;maybe_get_module_override_for_migrated_serialization_symbol functoolspartialrr try_with_lazy_message importlib import_modulesplitrr{)rrvrwr make_message attr_namer!r!r" import_symbols     rnamecCstdd|S)z:Converts a camel or studly-caps name to a snake_case name.z(?<=[^_A-Z])(?=[A-Z])r])resublower)rr!r!r"_to_snake_case0src@s&eZdZUeed<eed<ddZdS)SerializationConstantrvrwcCsttt|jt|jiSr) _TYPE_KEY _PYREF_TYPE _MODULE_KEYrv _NAME_KEYrwr6r!r!r"to_pyref:szSerializationConstant.to_pyrefN)r/r0r1ri__annotations__rr!r!r!r"r5s  r_serialization_constants_by_id!_serialization_constants_by_valuecompare_by_identitycCstt||}t|ttjfrtd|d|dtt|r+td|d|dtt|dur>td|d|dt ||}|rM|t t |<dS|t |<dS)aRegisters a module-level constant value for serialization. The specified value will be serialized as a pyref (python reference), meaning that its value will be deserialized by reading the specified symbol from the specified module. Args: module: The name of the module containing the constant symbol. symbol: The symbol in `module` that contains a constant value. compare_by_identity: If True, then only use a pyref to serialize a value `x` if `x is .`. If False, then use a pyref to serialize any value `x` where `x == .`. If `compare_by_identity` is False, then the value must be hashable. Raises: ModuleNotFoundError: If `module` can't be found. AttributeError If 'symbol' can't be accessed on `module`. PyrefPolicyError: If importing `symbol` from `module` is disallowed by the `DefaultPyrefPolicy`. ValueError: If registration is unnecessary for `.` (e.g., because it is a `type` or `function`). zCan not register rz_ as a serialization constant because it is a type or function (so registration is unnecessary).zp as a serialization constant because it is a JSON-representable primitive type (so registration is unnecessary).Nzg as a serialization constant because it is a daglish-traversable type (so registration is unnecessary).) rrrrDtypes FunctionType ValueErrorrkrrridr)rvrwrrserialization_constantr!r!r"register_constantFs"   r enum_typecCs,|jD]}t|j|jd|ddqdS)zRegisters all members of the given enum using `register_constant`. Args: enum_type: Any class that inherits from `enum.Enum`. rT)rN) __members__rr0r1)rrr!r!r" register_enumws rrootobjects refcountsversionrDitemsrleafrrefkeypyrefcCsj|j}|d}t|dkr|\}}nt|dkr#|jddd\}}ntd|dttt|t|i}|S)z(Serialize proxy object for lazy imports.:r)maxsplitzInvalid qualname z for ProxyObject.) r1rlenrsplitrrrrr) proxy_objectqualnamepartsr symbol_nameoutputr!r!r"_serialize_lazy_importss    rc @seZdZdZ ddedeefddZede e effdd Z dede fd d Z d d Z dejfddZddZ ddedejdeejde e effddZdS) Serializationa] Maintains state for an in-progress serialization. This class is used to recursively assemble a JSON-serializable dictionary representation of a given value. This class manages the state maintained during this process. The resulting dictionary representation can be obtained from the result property once an instance has been created: dict_representation = Serialization(value, pyref_policy).result A new instance of the class should be created per serialization. The resulting dictionary representation has the following keys at the top-level: * 'root': A dictionary representation of the root object. * 'objects': A dictionary mapping object names to their dictionary representations. * 'refcounts': A dictionary mapping object names to the number of times they are referenced (e.g., via other objects). * 'version': A version string, indicating what version of the serialization format produced this result. Individual object representations (as contained in 'root' and 'objects') may either be a JSON primitive value or a dictionary representation. Dictionary representations will always have a 'type' key, and additional fields which depend on this type: * Traversable structures (those that have either a serialization-specific or daglish traverser available) will have their type information, flattened items, and traversal metadata recursively serialized into a dictionary with keys 'type', 'items', and 'metadata'. The items will be a list of lists, where each sublist contains both a string representation of the item's `daglish.PathElement` (only used for debugging/visualization purposes) as well as the serialized item itself. * Python types and functions are serialized with a special 'pyref' type value, along with fields 'module' and 'name' that are used when importing the type or function during deserialization. * References to other objects have a special 'ref' type, and a 'key' field that identifies an object in the top-level 'objects' dictionary. * Leaf values that need path information will have a 'leaf' type, with an additional 'value' field that is always a JSON primitive value. Any object's dictionary representation may also have a 'paths' field that stores a list of all paths to that object from the root. Nr pyref_policycCsli|_i|_g|_tt|_||_tj |dd|_ |pt |_ t |j|jdddt|jt|jtti|_dS)aYInitializes the serialization. Args: value: The value to serialize. pyref_policy: The `PyrefPolicy` instance that should govern whether a given Python reference is permissible. Providing a policy here can help catch potential errors early, however this will only guarantee successful deserialization if the same policy is passed when creating a `Deserialization` instance. Raises: UnserializableValueError: If an unserializable value is encountered. ModuleNotFoundError: If a module can't be imported in the course of creating a Python reference. AttributeError: If an attribute or symbol can't be accessed in the course of creating a Python reference. PyrefPolicyError: If a Python reference (e.g., function or type) is disallowed by `pyref_policy`. T)memoizable_onlyr!)r! all_pathsN)_memo_objects _ref_values collections defaultdictrf _refcounts_rootr collect_paths_by_id _paths_by_idr _pyref_policy _ROOT_KEY _serialize _OBJECTS_KEY_REFCOUNTS_KEY _VERSION_KEY_VERSION_result)r-rrr!r!r"rs   zSerialization.__init__rcC|jS)z>Returns the serialization result (JSON-compatible dictionary).rr6r!r!r"resultzSerialization.resultc Cszt|}t|tjrt|}t|tjr|j}n|j}Wnt y2t ddt t|}Ynwt |}tdD]}|d|}||jvrM|Sq<|S)z2Creates a unique and informative name for `value`.z[<>()\[\] '\"]r>rr])rDrrr get_callabler ProxyObjectrr/AttributeErrorrrrir itertoolscountr)r-r hint_typehintirr!r!r" _unique_names&     zSerialization._unique_namecCs|j|d7<ttt|iS)z1Creates a reference to another serialized object.r)rr _REF_TYPE_KEY_KEY)r-rr!r!r"_ref/s zSerialization._ref current_pathc Cst|}|durd|d}t||j}t|tjr&|jjd|j }n|j}|t |j ||urGd|d|d|dt |d }t|t tt|t|iS) z2Creates a reference to an importable Python value.NzCouldn't find the module for z5; Are you wrapping the function or using a test mock?rzCouldn't create a pyref for z/; the same value was not obtained by importing r. Error occurred at path )rrrsr/renumEnumrr1rrrrmrrrr)r-rrrvrrrwr!r!r"_pyref4s.   zSerialization._pyrefcCs ttt|iS)zCreates a leaf value.r _LEAF_TYPE _VALUE_KEYrzr!r!r"_leafPszSerialization._leafrcCs^t||jvr||jt|S|dur|jt||}tt|}|durtt|r:|dur4|S||}nt |tt j t j frK|||}nt |tjrVt|}nt|tvrett|}nt |tjjrw|tvrwt|}n{d|dt|dt|d}t|||\}}||} t|t| ksJg} t| |D]%\} } |durt|| nd} |j | || f| d}| |f}| !|qt |t"j#r|$}|j ||t%fdd}t&|t||t'| t(|i}|durt)||t*<t+|r-|t&t,kr-|-|}||j.vsJ||jt|<||j.|<|j/!|||S|S)aAReturns a JSON-serializable dict representation of `value`. This function traverses `value`, recursively building up a JSON-serializable dictionary capturing values and associated metadata, debugging, and visualization info. Args: value: The value to serialize. current_path: The traversal's current path. all_paths: An optional list of all paths that reach `value`. If `None`, no path information will be added to `value`'s representation. Raises: UnserializableValueError: If an unserializable value is encountered. NzUnserializable value z of type rz.")r)0rrrrgetrrDrkrrrrrrrr rrrrrr|Hashablerrmrsflatten path_elementsrrJradd_path_elementrappendrBuildableTraverserMetadatarr<r _ITEMS_KEY _METADATA_KEYrq _PATHS_KEY is_memoizablerrrr)r-rrr traverserrrrrrserialized_items path_element child_value child_pathsserialized_valueserialized_itemserialized_metadatarr!r!r"rWsp                 zSerialization._serializer)r/r0r1r2rrrurr:rrirrrrPathrrPathsrr!r!r!r"rs01 0  rc@rr)DeserializationErrorz7Represents an error that occurs during deserialization.Nrtr!r!r!r"rsrc@sjeZdZdZ ddeeefdeefddZ e defdd Z d d Z d d Z ddZdefddZdS)DeserializationaMaintains state for an in-progress deserialization. This class is used to recursively recreate a Python object from its JSON-serializable dictionary representation (as created by the `Serialization` class above). This class manages the state maintained during this proces. The resulting deserialized value can be obtained from the result() method once an instance has been created: value = Deserialization(dict_representation, pyref_policy).result A new instance of the class should be created per deserialization. Nr rcCs8|t|_i|_|t|_|pt|_||j|_dS)aAInitializes the deserialization. Args: serialized_value: The dictionary representation of the value to deserialize. pyref_policy: The `PyrefPolicy` instance that should govern whether a given Python reference is permissible. Defaults to `DefaultPyrefPolicy` if not specified. N) r_serialized_objects_deserialized_objectsrrrr _deserializer)r-r rr!r!r"rs   zDeserialization.__init__rcCr)z*Returns the result of the deserialization.rr6r!r!r"rrzDeserialization.resultcCsJ|ttksJ|t}||jvr|j|S||j|}||j|<|Sr)rrrrrr)r-rrdeserialized_objectr!r!r"_deserialize_refs   z Deserialization._deserialize_refcCs&|ttksJt|j|t|tSr)rrrrrr)r-rr!r!r"_deserialize_pyrefs z"Deserialization._deserialize_pyrefcCs|ttksJ|tSrr)r-rr!r!r"_deserialize_leafsz!Deserialization._deserialize_leafserialized_objectcst|trfdd|DSt|ts|S|t}|tkr"|S|tkr+|S|tkr4 |St|tr@| ttkrHt d|d |}t |}|dur]t d|d|t}dd |D} |t}|||S)z7Recursively deserializes the given `serialized_object`.csg|]}|qSr!)r)rOrSr6r!r"rpsz0Deserialization._deserialize..zInvalid object type: rNz'No traverser available for object type cSsg|]\}}|qSr!r!)rOr]rr!r!r"rprU)rlistdictrrrrrrrrrrrrrr)r-rrKrr rrr!r6r"rs0         zDeserialization._deserializer)r/r0r1r2rrirrrurr:rrrrrr!r!r!r"rs   rrindentcCstjt||j|dS)aReturns the JSON serialization of `value`. Args: value: The value to serialize. pyref_policy: A `PyrefPolicy` instance that governs which Python references are permissible. Providing a policy here can help catch potential errors early, however this will only guarantee successful deserialization if the same policy is passed when calling `load_json`. Defaults to `DefaultPyrefPolicy`. indent: An optional indentation to use when formatting the JSON. If given, the resulting JSON will be formatted with each new indentation level indented by `indent` spaces from the last for readability. If not provided, the resulting JSON will be compact with minimal whitespace. Raises: UnserializableValueError: If an unserializable value is encountered. PyrefPolicyError: If a Python reference (e.g., function or type) is disallowed by `pyref_policy`. )r)jsondumpsrr)rrrr!r!r" dump_json"sr!r cCstt||jS)aReturns a Python object deserialized from `serialized_value`. Args: serialized_value: A JSON-serialized value. pyref_policy: A `PyrefPolicy` instance that governs which Python references are permissible. Raises: DeserializationError: If an unknown type or other serialization format error is encountered during deserialization. ModuleNotFoundError: If a referenced module can't be imported. AttributeError: If a referenced attribute or symbol can't be accessed. PyrefPolicyError: If a Python reference (e.g., function or type) is disallowed by `pyref_policy`. json.decoder.JSONDecodeError: If Python's `json.loads` encounters an error while decoding the provided JSON. )rrloadsr)r rr!r!r" load_json=sr#)NNr)`r2r|rr dataclassesrrrrrrrrrtypingrrrrrrfiddler fiddle._srcr rr r fiddle._src.experimentalr r rrr' dataclass PathElementr)r4r<r=NodeTraverserRegistry_traverser_registryrZrrFrMr[set frozensetr^bytesNoValuerkrrirmrq ExceptionrsABCMetarurrrrrrrrfrrrhrrrrrrrrrrrrrrrrrrrrrrr!r#r!r!r!r"s               1i