o qoiw @sdZddlmZddlZddlZddlZddlZddlmZm Z m Z m Z m Z m Z mZmZmZmZGdddejdZejdd Gd d d eZejdd Gd d d eZejdd GdddeZdkddZGdddeZejdd GdddeZeedfZeedfZeeee e!e"ej#e$de$e%e$e&f Z'GdddZ(eedfZ)e ege)fZ*eeefZ+e ege e+fZ,e egeeedfeffZ-e e eegefZ.edZ/ejdd GdddZ0Gdd d Z1e1d!d"Z2e2j3Z3e2j4Z4e2j5Z5e3e6d#d$d%d$d&d$d'd(d)Z7d*d+Z8e3ej9e7e8d,d$d'e3e:d-d$d.d$d/d$d0d$d1e3e(d2d$d3d$d4d$d'e3e;d5d$d6d$d7d$d8d$d1dlddodJdKZ?dpdNdOZ@dpdPdQZAdqdTdUZBejGdVdWdWejdZCedXZDejdd GdYdZdZZEejdd Gd[d\d\ZFejGd]d^d^eCZGejGd_d`d`eGZH !dre2dadsdddeZIdde2fdtdidjZJdS)uzLibrary for manipulating DAGs.) annotationsN) AnyCallableDictIterableListOptionalTupleTypeTypeVarUnionc@s>eZdZdZeejdddZejdddZdd d Z d S) PathElementzElement of a path.returnstrcCt)z'Generates code for accessing this path.NotImplementedErrorselfrG/home/ubuntu/.local/lib/python3.10/site-packages/fiddle/_src/daglish.pycodezPathElement.codercCsdS)zBReturns the element of `container` specified by this path element.Nrr containerrrrfollow$szPathElement.followotherboolcCs0t|t|urtt|tt|kStd)z.Define the less than relation for PathElement.z?__lt__ relation should be handled by subclasses of PathElement.)typerrrrrrr__lt__(s zPathElement.__lt__Nrr)rrrr rr) __name__ __module__ __qualname____doc__propertyabcabstractmethodrrr rrrrr s r ) metaclassT)frozencsDeZdZUdZded<edddZdd d Zdfdd ZZ S)Indexz)An index into a sequence (list or tuple).intindexrrcCsd|jdSN[]r.rrrrr7z Index.coder!Union[List[Any], Tuple[Any, ...]]rcC ||jSNr2rrrrr; z Index.followrr rc(t|t|ur|j|jkSt|Sr6)rr.superr r __class__rrr >  z Index.__lt__r!)rr4rrr" r#r$r%r&__annotations__r'rrr __classcell__rrr:rr,2   r,csDeZdZUdZded<edddZdd d Zdfdd ZZ S)Keyz A key of a mapping (e.g., dict).rkeyrrcCsd|jdSr/rBrrrrrJr3zKey.coderDict[Any, Any]cCr5r6rCrrrrrNr7z Key.followrr rcr8r6)rrBr9r rr:rrr Qr<z Key.__lt__r!)rrDrrr"r=rrr:rrAEr@rAcsDeZdZUdZded<edddZdd d Zdfdd ZZ S)AttrzAn attribute of an object.rnamercCs d|jS)N.)rFrrrrr]s z Attr.coderrcCs t||jSr6)getattrrFrrrrra z Attr.followrr rcr8r6)rrFr9r rr:rrr dr<z Attr.__lt__r!)rrrrr"r=rrr:rrEXr@rErBUnion[str, int]rcCst|tr t|St|S)zr[rrrrrSs rSc@s<eZdZdZdddZ ddddZdddZdddZdS) NodeTraverserRegistryzA registry of `NodeTraverser`s. use_fallback$Union['NodeTraverserRegistry', bool]cCs.i|_t|tr |}n|rt}nd}||_dS)aInitializes the instance. Args: use_fallback: Whether to use a fallback registry for lookups if a traverser can't be found in this registry. If `True`, this uses the default daglish registry for fallbacks. If `False`, fallback behavior is disabled. This can also be a `NodeTraverserRegistry` instance, in which case it will be used for fallback lookups. N)_node_traversersrKr\_default_traverser_registryfallback_registry)rr]rarrrrPs  zNodeTraverserRegistry.__init__N node_type Type[Any] flatten_fnrT unflatten_fnrVpath_elements_fnrXflatten_with_paths_fnrZrNonecCsNt|ts td|d||jvrtd|dt||||d|j|<dS)aRegisters a node traverser for `node_type`. Args: node_type: The node type to regiser a traverser for. The traverser will be used *only* for nodes of this type, not subclasses (with the exception of the special-cased `daglish.NamedTupleType`). flatten_fn: A function that flattens values for traversal. This should accept an instance of `node_type`, and return a tuple of `(values, metadata)`, where `values` is a sequence of values and `metadata` is arbitrary traverser-specific data. unflatten_fn: A function that unflattens values, which should accept `values` and `metadata` and return a new instance of `node_type`. path_elements_fn: A function that returns `PathElement` instances for the flattened values returned by `flatten_fn`. This should accept an instance of `node_type`, and return a sequence of `PathElement`s aligned with the values returned by `flatten_fn`. flatten_with_paths_fn: A version of `flatten_fn` that returns an iterable of `(value, path)` pairs, where `value` is a child value and `path` is a `Path` to the value. `node_type` () must be a type.zA node traverser for z has already been registered.)rUrWrYr[N)rKr TypeErrorr_ ValueErrorrS)rrbrdrerfrgrrrregister_node_traversers   z-NodeTraverserRegistry.register_node_traverserOptional[NodeTraverser]cCsPt|ts td|dt|rt}|j|}|dur&|jr&|j|}|S)aEFinds a `NodeTraverser` for the given `node_type`. This simply looks up `node_type`, with one special case: if `node_type` is a `NamedTuple` (as determined by the `is_namedtuple_subclass` function), then `daglish.NamedTupleType` is looked up in `registry` instead. If this node traverser registry doesn't contain a traverser for `node_type`, then it will be looked up in `self.fallback_registry`. Args: node_type: The node type to find a traverser for. Returns: A `NodeTraverser` instance for `node_type`, if it exists, else `None`. rirjN) rKrrkis_namedtuple_subclassrQr_getrafind_node_traverser)rrb traverserrrrrqs   z)NodeTraverserRegistry.find_node_traverserrcCs||duS)z-Returns whether `node_type` can be traversed.N)rqrrbrrris_traversable_typer3z)NodeTraverserRegistry.is_traversable_type)r]r^r6) rbrcrdrTrerVrfrXrgrZrrhrbrcrrn)rbrcrr)r#r$r%r&rPrmrqrtrrrrr\s  (r\F)r]cCst|t|fSr6)tuplevalueskeysxrrrsr{cCstt||Sr6)dictzip)rwrxrrrr{scCsdd|DS)NcSsg|]}t|qSrrA.0rBrrr z..)rxryrrrr{)rdrerfcCst||jt|ffSr6)rvrwdefault_factoryrx)noderrrflatten_defaultdictsrcCs|\}}t|t||Sr6) collections defaultdictr})rwmetadatarrxrrrunflatten_defaultdictsrcCstdd|DS)Ncs|]}t|VqdSr6r~rrrr $..)rvrxryrrrr{$scCs|dfSr6rryrrrr{(cCt|Sr6rvrz_rrrr{)rcCtddtt|DS)Ncsrr6r,rirrrr*rrrvrangelenryrrrr{*cCddt|DS)Ncs |] \}}|t|fVqdSr6rrrrzrrrr+r enumeratexsrrrr{+r)rdrerfrgcCst|t|fSr6)rvrryrrrr{0scCs||Sr6r)rwrbrrrr{1rcCstdd|DS)Ncsrr6)rE)rrFrrrr2rr)rv_asdictrxryrrrr{2rcCs t|dfSr6rryrrrr{6s cCrr6)listrrrrr{7rcCr)Ncsrr6rrrrrr8rrrryrrrr{8rcCr)Ncsrr6rrrrrr9rrrrrrrr{9r prefix_pathPathcontaining_pathcCs||dt|kS)a%Returns `True` if `prefix_path` is a prefix of `containing_path`. Args: prefix_path: the `Path` that may be a prefix of `containing_path`. containing_path: the `Path` that may be prefixed by `prefix_path`. Returns: `True` if `prefix_path` is a prefix of `containing_path`. N)r)rrrrr is_prefix=s rpathrcCsddd|DS)Ncss|]}|jVqdSr6)r)rrzrrrrKszpath_str..)join)rrrrpath_strJsrrootrc Cst|}t|D]1\}}z||}Wqttttfy7}zt|dt|d|d|d||d}~ww|S)aFollows the path from a root item to a contained item, and returns it. Equivalent to `functools.reduce(lambda v, p: p.follow(v), root, path)`, but gives better error messages. Args: root: The starting point for the path. path: A sequence of `PathElement`s, indicating how to get from `root` to the contained item. Returns: The contained item identified by `path`. Raises: ValueError: If `path` is not compatible with `root`. z is not compatible with rootN=z: )rrKeyError IndexErrorrkAttributeErrorrlr)rrvaluerpath_elterrr follow_pathNs"rpathsIterable[Path]elementPathscstfdd|DS)Nc3s|]}|fVqdSr6rrrrrrrjz#add_path_element..r)rrrrradd_path_elementisrrrcCst|t o |dkS)afDetermines what values can be memoized. A primary concern is whether `value` may be subject to Python's interning optimizations, which could lead to confusing results under some circumstances if memoization is allowed. For the purposes of this function then, immutable types that can't contain references (including strings and ints) are excluded from memoization. Instances of such types are guaranteed to maintain an equality relationship with themselves over time. Args: value: A candidate value to check for memoizability. Returns: A bool indicating whether `value` is memoizable. r)rK_IMMUTABLE_NONCONTAINER_TYPESrrrr is_memoizablems rcCs(t| pt|tuotdd|DS)aReturns true if Python can apply an interning optimization to `value`. If this is false, then `x is y` is only true if they point to the same object, created at the same place. If this is true, then `x is y` may be true for unrelated but equal values (i.e., values that were created at different places). The most common examples of values that the interning optimization can apply to are constants, such as booleans, strings, and small integers. The interning optimization may be applied to (nested) tuples whose values are constants. Args: value: any value, it can be a Fiddle buildable or a regular Python value. Returns: A bool indicating whether interning optimization is applicable to `value`. csrr6) is_internable)rrrrrrrz is_internable..)rrrvallrrrrrs rtype_rccCs2t|tot|dot|dotdd|jDS)Nr_fieldscss|]}t|tVqdSr6)rKr)rfrrrrrz)is_namedtuple_subclass..) issubclassrvhasattrrr)rrrrros roc@s~eZdZUdZded<ded<eZded<d"d d Zej d#ddZ ej d$ddZ d%ddZ e d&ddZe d'dd Zd!S)( TraversalzDefines an API that traversers must implement. Please note that users of a traverser will mostly interact with the State API, which bundles a Traverser with the current paths in the traversal. Callable[..., Any] traversal_fnrroot_objr\registryrbrcrrncCs |j|S)z5Uses the configured registry to find a NodeTraverser.)rrqrsrrrrqs zTraversal.find_node_traverserrstateStatecCr)z5Calls the underlying function bound to the traversal.rrrrrrrapplyszTraversal.apply object_idr- allow_cachingr List[Path]cCr)z!Returns all paths to a container.r)rrrrrrall_paths_to_objectrzTraversal.all_paths_to_objectcCst|d|jddS)zReturns an initial state for this traversal. The initial state has a reference to the traversal, the empty path, and no parent states. rN)parent)rrrrrr initial_stateszTraversal.initial_statefncCs|||dS)aCreates a new traversal and returns the initial state. Args: fn: Function which is applied at each node during the traversal. root_obj: Root object being traversed. Returns: The initial state (from `initial_state`) of a new traversal instance. )rrr)clsrrrrrbegins zTraversal.begincCs|j||d}|||S)zCCreates a traversal and state, and then calls/returns `fn` on this.)rr)r)rrrrrrrruns z Traversal.runNru)rrrrrrrr-rrrr)rr)rrrrrr)rrrrrr)r#r$r%r&r>r`rrqr(r)rrr classmethodrrrrrrrs        r_Tc@s:eZdZUdZded<ded<ded<ded <d d Zd S) SubTraversalResultnode_traverserrwrrYrSrz List[Any]rwrr PathElementsrYcCs|j|j|jSr6)rrWrwrrrrrrWszSubTraversalResult.unflattenN)r#r$r% __slots__r>rWrrrrrs  rc@seZdZUdZdZded<ded<ded<d ed <ed3d dZed4ddZeddZ ed5ddZ d6d7ddZ d8ddZ d9d"d#Z d:d%d&Zd;d'd(Z )dd0d1Zd2S)?rasContains a current traversal state. Attributes: traversal: Reference to main traversal object. current_path: A path that can be followed to the current object. In the case of shared objects, there will be other paths to the current object, and often these are determined by a somewhat arbitrary DAG traversal order. parent: The parent state. ) traversal current_path_valuerrrrrrrzOptional[State]rrr-cC t|jSr6)idrrrrr _object_id zState._object_idrcCrr6)rrrrrr_is_memoizable rzState._is_memoizablecCs|jS)a&Original value constructed with this state. Generally please don't use this value, it's much more clear to use the first argument of your `traverse(value, state)` function, especially since for post-order traversals, you'll often write `value = state.map_children(value)`. )rrrrroriginal_value s zState.original_valueIterable[State]ccs(|V|jdur|jjEdHdSdS)z,Gets ancestors, including the current state.N)rancestors_inclusiverrrrrs  zState.ancestors_inclusiveTrrcshdg}d}|jD]}|jr|jj|j|d}n|d7}q|dkr)|j| dndfdd|DS)aGets all paths to the current value. Args: allow_caching: Whether paths can be cached. Most traverers will compute paths to all objects from the root config object. But this can fail to reflect objects that are being added by the traversal, so occasionally one may want to take a big performance hit to reflect any newly added objects. Returns: List of paths. rr)rNcsg|]}|qSrrr path_suffixrrr8rz'State.get_all_paths..)rrrrrr)rr base_paths suffix_lenrrrr get_all_pathss   zState.get_all_pathsrcCs|jt|duS)z'Returns whether a value is traversable.N)rrqr)rrrrris_traversable:szState.is_traversablerrSrcs@||\}}||}fddt||D}t||||dS)z:Shared function for map_children / flattened_map_children.csg|] \}}||qSr)call)rsubvalue path_elementrrrrCs z1State._flattened_map_children..r)rUrYr}r)rrr subvaluesrrY new_subvaluesrrr_flattened_map_children>s  zState._flattened_map_childrenrcCs8|jt|}|dur|S|||}||j|jS)zMaps over children for traversable values, otherwise returns it. Args: value: Value to map over. Non-traversable values are returned unmodified. Returns: A mapped value of the same type. N)rrqrrrWrwr)rrrresultrrr map_childrenMs   zState.map_childrencCs4|jt|}|durtd|d|||S)a)Maps over children for traversable values, but doesn't unflatten results. Unlike `map_children`, we've found that there's not an easy natural choice for handling the case where `value` is not a traversable type. Generally calling code expects a SubTraversalResult, which contains an iterable of values and metadata. The transformation of these values and choice of metadata is non-trivial, and client code would probably have to branch on it anyway. Therefore, we throw an error for non-traversable values; please test whether `value` is traversable beforehand. Args: value: Value to map over. Returns: Sub-traversal results. Raises: ValueError: If `value` is not traversable. Please test beforehand by calling `state.is_traversable()`. NNo node traverser found for L, please register traverser or pre-process the non-traversable values first.)rrqrrlr)rrrrrrflattened_map_children\s   zState.flattened_map_childrenF ignore_leaves Iterable[Any]c cs|jt|}|dur|rdStd|d|jdur1||D] \}}|||Vq#dS||\}}||}t||D] \}} ||| VqBdS)aMaps over children for traversable values, but doesn't unflatten results. This method only returns result values, so use it in place of `state.flattened_map_children(value).values` when the result is consumed once. If you are calling this for an "effectful" computation (including ones that just gather results in `nonlocal` variables) then consider the following pattern, for _ in state.yield_map_child_values(node, ignore_leaves=True): pass # Run lazy iterator. Args: value: Value to map over. ignore_leaves: If True, then this function will return an empty iterable if `value` is not traversable. Otherwise, it will raise a ValueError. Yields: Sub-traversal results, the same type as returned by your _traverse function. Raises: ValueError: If `value` is not traversable and `ignore_leaves` is `False`. Please test beforehand by calling `state.is_traversable()`. Nrr) rrqrrlr[rrUrYr}) rrrrr sub_values unused_metarY sub_valuerrrryield_map_child_valuesys"   zState.yield_map_child_valuesadditional_pathr cGs*t|jg|j|R||}|j||S)aLow-level function to execute a sub-traversal. This creates a new state, and then applies the function bound to the traverser. Args: value: Sub-value to run the traversal on. *additional_path: Additional path elements relating the sub-value to the value referred to by `self`. Returns: Result of sub-traversal (whatever `self.traversal.traversal_fn` returns). )rrrr)rrr new_staterrrrsz State.callN)rr-)rr)rrT)rrrrrrrr)rrrrSrr)rrrr)rrrrF)rrrrrr)rr )r#r$r%r&rr>r'rrrrrrrrrrrrrrrrs.            -rc@s8eZdZUdZejedZded<ddZ dd dZ dS)BasicTraversalzBasic traversal. This traversal will go through shared objects multiple times, for each path that they are reachable from. rDict[int, List[Path]] paths_cachecCs |||Sr6)rrrrrrrIzBasicTraversal.applyrr-rrrrcCs8|r ||jvr |j|St|jd|jd}|_||S)NT)memoizable_onlyr)rcollect_paths_by_idrr)rrr all_pathsrrrrs    z"BasicTraversal.all_paths_to_objectNr) r#r$r%r& dataclassesfieldr|rr>rrrrrrr s r c@s`eZdZUdZdZded<ejedZ ded<ejedZ ded <e ddddZ ddZ dS)MemoizedTraversalaTraversal that memoizes results. Attributes: memoize_internables: Whether to memoize applications of the function on internable (typically, immutable) values. For example, if "4" is a common leaf value, by default the traversal will only visit it once, and reuse the results. However, if you are writing a traversal that cares about paths to the value "4", you might want to set this to False so it is visited multiple times. memo: Memoization dictionary mapping `id(value)` to the tuple `(value, result)`, where `value` is a traversed node and `result` is the value returned by `apply` for `value`. _cycle_start: Dictionary used for cycle detection. When the traversal is applied to a value, `_cycle_start[id(value)]` is temporarily set to the current the `state`. Once the result has been computed, `_cycle_start[id(value)]` is deleted. If a cycle is detected during traversal, `MemoizedTraversal` raises a `ValueError`. Trmemoize_internablesrzDict[int, Tuple[Any, Any]]memozDict[int, State] _cycle_startrrrrrrcCs||||dS)aCreates a new traversal and returns the initial state. Args: fn: Function which is applied at each node during the traversal. root_obj: Root object being traversed. memoize_internables: `True` if internables should be memoized. Something is internable if it's declaration in separate code locations still cause references to it to use the same instance when the value is the same (for example, primitives are internable). This is a result of Python's internal optimization. See also `daglish.is_internable`. Returns: The initial state (from `initial_state`) of a new traversal instance. )rrrr)rrrrrrrrszMemoizedTraversal.begincCst|}|jst|r|||S||jvr|j|dS||jvr8|j|}tdt|jdt|jd||j|<|||}||f|j|<|j|=|S)Nrz8Fiddle detected a cycle while traversing a value: z is z/. Configurations with cycles are not supported.) rrrrrrrlrr)rrrvalue_idoriginal_staterrrrrs&      zMemoizedTraversal.applyNr )rrrrrrrr)r#r$r%r&rr>rrr|rrrrrrrrrrs   rrrrcs@stdidfdd }t|||d}|||S) aReturns a dict mapping id(v)->paths for all `v` traversable from structure. I.e., if `result = collect_paths_by_id(structure)`, then `result[id(v)]` is the list of every path `p` such that `follow_path(structure, p) is v`. This dict only includes values `v` for which `is_memoizable(v)` is true. Args: structure: The structure for which the id->paths mapping should be created. memoizable_only: If true, then only include values `v` for which `is_memoizable(v)` is true. Currently required to be True, to avoid bugs that can result from Python's interning optimizations. registry: Node traversal registry used to collect paths. zIncluding non-memoizable objects when collecting paths by id may cause problems, because of Python's interning optimizations. If you are sure this is what you need, contact the Fiddle team, and we can look into enabling this flag.rrcs<rt|rt|g|j|j|ddD]}qdSNT)r)r setdefaultrappendrr)rrrr paths_by_idrrtraverse4s z%collect_paths_by_id..traverserNrr)rlr r) structurerrr"rrr rrsrmemoizedrIterable[Tuple[Any, Path]]cCs<ddd}|rt||||d}nt|||d}|||S) aIterates through values in a DAG. This API is suitable for read-only access and small/reasonable edits to a configuration DAG. If a transformation seeks to change the type of nodes, then we recommend using the functional traversal API. Often, when mutation can be used, it will preserve history better, so this API is not discouraged for modifying configuration. Args: value: fdl.Buildable or collection of buildables. memoized: Whether to yield shared nodes only once. Defaults to True. With this setting, you will only see one path (which is somewhat arbitrary) to shared nodes. memoize_internables: Whether to memoize Python internable values. Check the docstring of MemoizedTraversal for details. registry: Override to the NodeTraverserRegistry; this is a low-level setting for traversing into custom data types. Returns: Iterable of (value, path) tuples. rrcss0||jfV|j|ddD]}|EdHqdSr)rr)rr sub_resultrrr _traverse\s   ziterate.._traverse)rrrNr#)rr r)rr%rrr(rrrriterate?s r))rBrJrr )rrrr)rrrr)rrrr)rrrr rrr )rrcrrr )rr\rr) rrr%rrrrr\rr&)Kr& __future__rr(rrenumtypingrrrrrrr r r r ABCMetar dataclassr,rArErLrMrNrrrr-floatcomplexrbytesEnumrNotImplementedEllipsisrrQrrX _ValueAndPathOptimizedFlattenFnrTrVrRrSr\r`rmrqrtr|rrrrvrrrrrrrrorrrrr rrr)rrrrs 0           `      7  JH)