o ]i@sdZddlZddlZddlmZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl m Z mZmZmZddlmZddlZddlmZejjejejjdGdd d eZGd d d eZGd d d eZedZddZddZ d=ddZ!ddZ"de#dee#ee$ffddZ%Gddde&dddgZ'ej(Gdd d Z)d>d!d"Z*Gd#d$d$Z+de,fd%d&Z-d>d'd(Z.d)d*Z/d+d,Z0d>d-d.Z1Gd/d0d0e+Z2Gd1d2d2e j3Z4Gd3d4d4e4Z5d5d6Z6d=d7d8Z7d9d:Z8d;d<Z9dS)?aIClasses for defining configurations of experiments and models. This file defines the classes `ConfigDict` and `FrozenConfigDict`, which are "dict-like" data structures with Lua-like access and some other nice features. Together, they are supposed to be used as a main way of expressing configurations of experiments and models. N)abc)AnyMappingOptionalTuple)logging) representer) data_typerc@ eZdZdS)RequiredValueErrorN__name__ __module__ __qualname__rrZ/home/ubuntu/.local/lib/python3.10/site-packages/ml_collections/config_dict/config_dict.pyr 3r c@r )MutabilityErrorNr rrrrr7rrc@r )JSONDecodeErrorNr rrrrr;rrcCstdd|jDS)zATries to ensure: `_is_callable_type(type(obj)) == callable(obj)`.css|]}d|jvVqdS)__call__N)__dict__).0crrr Csz$_is_callable_type..)any__mro__ field_typerrr_is_callable_typeAsrcCs4|dus|tkr dSt||rdSt|ot| S)a]Helper function for type safety exceptions. This function determines whether or not assigning a value to a field violates type safety. Args: value: The value to be assigned. field_type: Type of the field that we would like to assign value to. Returns: True if assigning value to field violates type safety, False otherwise. NF) _NoneType isinstancecallabler)valuerrrr_is_type_safety_violationFs  r#FcCst|}t|tr|turt|St|tr|tur||St|tr+|tur+t|St|tr8|tur8t|St|trC|turC|S|rWt||rWtd |t|t||S)a1Helper function to handle the exceptional type conversions. This function implements the following exceptions for type-checking rules: * An `int` will be converted to a `float` if overriding a `float` field. * Any string value can override a `str` or `unicode` field. The value is converted to `field_type`. * A `tuple` will be converted to a `list` if overriding a `list` field. * A `list` will be converted to a `tuple` if overriding `tuple` field. * Short and long integers are indistinguishable. The final value will always be a `long` if both types are present. Args: value: The value to be assigned. field_type: The type for the field that we would like to assign value to. type_safe: If True, the method will throw an error if the `value` is not of type `field_type` after safe type conversions. Returns: The converted type-safe version of the value if it is one of the cases described. Otherwise, return the value without conversion. Raises: TypeError: if types don't match after safe type conversions. z9{} is of original type {} and cannot be casted to type {}) typer intfloatstrtuplelistr# TypeErrorformat)r"r type_safeoriginal_value_typerrr _safe_cast]s"r.cCst|tr |S|SNr FieldReferenceget)value_or_fieldreferencerrr_get_computed_values r4keyreturncCsJ|dd}td|}|r|d}t|d}||fSd}||fS)zDParse a ConfigDict key into to it's initial part and index (if any)..rz(.*)\[([0-9]+)\]N)splitrematchgroupr%)r5 index_matchindexrrr _parse_keys  r@c@seZdZdZdS)_OpzA named tuple representing a lazily computed op. The _Op named tuple has two fields: fn: The function to be applied. args: a tuple/list of arguments that are used with the op. N)r rr__doc__rrrrrAsrAfnargsc@sBeZdZdZdOddZdPddZdQd d Zd d Zd dZddZ ddZ ddZ dZ ddZ ddZddZddZddZdd Zd!d"Zd#d$Zd%d&Zd'd(Zd)d*Zd+d,Zd-d.Zd/d0Zd1d2Zd3d4Zd5d6Zd7d8Zd9d:Zd;d<Z d=d>Z!d?d@Z"dAdBZ#dCdDZ$dEdFZ%dGdHZ&dIdJZ'dKdLZ(dMdNZ)dS)Rr1aReference to a configuration element. Typed configuration element that can take a None default value. Example:: from ml_collections import config_dict cfg_field = config_dict.FieldReference(0) cfg = config_dict.ConfigDict({ 'optional': configdict.FieldReference(None, field_type=str) 'field': cfg_field, 'nested': {'field': cfg_field} }) with self.assertRaises(TypeError): cfg.optional = 10 # Raises an error because it's defined as an # intfield. cfg.field = 1 # Changes the value of both cfg.field and cfg.nested.field. print(cfg) This class also supports lazy computation. Example:: ref = config_dict.FieldReference(0) # Using ref in a standard operation returns another FieldReference. The new # reference ref_plus_ten will evaluate ref's value only when we call # ref_plus_ten.get() ref_plus_ten = ref + 10 ref.set(3) # change ref's value print(ref_plus_ten.get()) # Prints 13 because ref's value is 3 ref.set(-2) # change ref's value again print(ref_plus_ten.get()) # Prints 8 because ref's value is -2 NFcCs|dur|dur tdt|tr|}nt|}nztd|Wnty2tdt|w||_|||rE|durEtd||_ |durQg|_ dS|g|_ dS)aCreates an instance of FieldReference. Args: default: Default value. field_type: Type for the values contained by the configuration element. If None the type will be inferred from the default value. This value is used as the second argument in calls to isinstance, so it has to follow that function's requirements (class, type or a tuple containing classes, types or tuples). op: An optional operation that is applied to the underlying value when `get()` is called. required: If True, the `get()` method will raise an error if the reference does not contain a value. This argument has no effect when a default value is provided. Setting this to True will raise an error if `op` is not None. Raises: TypeError: If field_type is not None and is different from the type of the default value. ValueError: If both the default value and field_type is None. Nz2Default value cannot be None if field_type is Nonez#field_type should be a type, not {}z-Cannot set required to True if op is not None) ValueErrorr r1get_typer$r*r+ _field_typeset _required_ops)selfdefaultroprequiredrrr__init__s&       zFieldReference.__init__cCs|pt}t||vr dS|t||j}t|tr%||r%dS|jD]}|j D]}t|tr?||r?dSq-q(dS)zFinds cycles in the reference graph. Args: visited: Set containing the ids of all visited nodes in the graph. The default value is the empty set. Returns: True if there is a cycle in the reference graph. TF) rHidadd_valuer r1 has_cyclecopyrJrD)rKvisitedr"rMargrrrrSs   zFieldReference.has_cycleTcCsg|_|dur d|_dSt|tr>|r(t||s(td||t|dd}||_| r<||_t ddSt ||j ||_dS)aOOverwrites the value pointed by a FieldReference. Args: value: New value. type_safe: Check that old and new values are of the same type. Raises: TypeError: If type_safe is true and old and new values are not of the same type. MutabilityError: If a cycle is found in the reference graph. Nz0Reference is of type {} but should be of type {}rRzFound cycle in reference graph.) rJrRr r1 issubclassrFr*r+getattrrSrr.rG)rKr"r, old_valuerrrrH#s    zFieldReference.setcCs |jduS)z5Returns True if the reference points to a None value.N)rRrKrrremptyA zFieldReference.emptycCs~|jr |jdur tdt|j}|jD](}dd|jD}|dus&d|vr/d}td|n |j|g|R}t|}q|S)aEGets the value of the `FieldReference` object. This will dereference `_pointer` and apply all ops to its value. Returns: The result of applying all ops to the dereferenced pointer. Raises: RequiredValueError: if `required` is True and the underlying value for the reference is False. Nz&None value found in required referencecSsg|]}t|qSr)r4rrVrrr Wz&FieldReference.get..z&Cannot apply `%s` to `None`; skipping.) rIrRr r4rJrDrdebugrC)rKr"rMrDrrrr2Es    zFieldReference.getcC|jSr/)rGrZrrrrF`zFieldReference.get_typecCs&t|tr ||kS||kSr/r0rKotherrrr__eq__c  zFieldReference.__eq__cCs&t|tr ||kS||kSr/r0rcrrr__le__irfzFieldReference.__le__cs(fdd|D}tjt||dS)Ncsg|]}t|jqSr)r.rGr]rZrrr^sz,FieldReference._apply_op..rrM)r1rGrA)rKrCrDrrZr _apply_oprs zFieldReference._apply_opcs ttfdd|gdS)aApply a cast op that changes the field_type of this FieldReference. `_apply_op` assumes that the `field_type` does not change after the op is applied whereas `_apply_cast_op` generates a FieldReference with casted field_type. Since the signature is `fn(value, *args)` we need to ignore `value` which now contains a unused default value of field_type. Args: field_type: data type to cast to. Returns: A new FieldReference with of `field_type`. cs|Sr/r)_valrrrsz/FieldReference._apply_cast_op..ri)r1rA)rKrrrr_apply_cast_opys zFieldReference._apply_cast_opcCs|ddS)NcSs|Sr/r)xrrrrmsz)FieldReference.identity..)rjrZrrridentityzFieldReference.identitycCs|t|Sr/)rjoperator attrgetter)rK attr_namerrrattrzFieldReference.attrcC|tj|Sr/)rjrrrQrcrrr__add__rqzFieldReference.__add__cCttj|}||Sr/) functoolspartialrrrQrj)rKrdraddrrr__radd__ zFieldReference.__radd__cCrwr/)rjrrsubrcrrr__sub__rqzFieldReference.__sub__cCryr/)rzr{rrrrj)rKrdrsubrrr__rsub__r~zFieldReference.__rsub__cCrwr/)rjrrmulrcrrr__mul__rqzFieldReference.__mul__cCryr/)rzr{rrrrj)rKrdrmulrrr__rmul__r~zFieldReference.__rmul__cCrwr/rjrrtruedivrcrrr__div__rqzFieldReference.__div__cCryr/rzr{rrrrj)rKrdrdivrrr__rdiv__r~zFieldReference.__rdiv__cCrwr/rrcrrr __truediv__rqzFieldReference.__truediv__cCryr/r)rKrdrtruedivrrr __rtruediv__r~zFieldReference.__rtruediv__cCrwr/)rjrrfloordivrcrrr __floordiv__rqzFieldReference.__floordiv__cCryr/)rzr{rrrrj)rKrd rfloordivrrr __rfloordiv__r~zFieldReference.__rfloordiv__cCrwr/)rjrrpowrcrrr__pow__rqzFieldReference.__pow__cCrwr/)rjrrmodrcrrr__mod__rqzFieldReference.__mod__cCrwr/)rjrrand_rcrrr__and__rqzFieldReference.__and__cCrwr/)rjrror_rcrrr__or__rqzFieldReference.__or__cCrwr/)rjrrxorrcrrr__xor__rqzFieldReference.__xor__cC |tjSr/)rjrrnegrZrrr__neg__ zFieldReference.__neg__cCrr/)rjrrabsrZrrr__abs__rzFieldReference.__abs__cC |tSr/)rnr%rZrrrto_int zFieldReference.to_intcCrr/)rnr&rZrrrto_floatrzFieldReference.to_floatcCrr/)rnr'rZrrrto_strrzFieldReference.to_strcCs0|d|_|d|_|d|_|dd|_dS)NrRrGrJrIF)rRrGrJr2rIrKstaterrr __setstate__s   zFieldReference.__setstate__cCtdNzuFieldReference cannot be used for control flow. For boolean operations use "&" (logical "and") or "|" (logical "or").NotImplementedErrorrZrrr __nonzero__zFieldReference.__nonzero__cCrrrrZrrr__bool__rzFieldReference.__bool__)NNFr/)T)*r rrrBrOrSrHr[r2rFrerg__hash__rjrnrprurxr}rrrrrrrrrrrrrrrrrrrrrrrrrrrr1sP $ 0  r1cCsJt|tsJ|r J|pi}||t|<t|tr!|jdd}n|}|D]{\}}t||vr8|t|}ndt|trw|turw|jrw| rKnQt| |vr`| |t| dnd?Z#d@dAZ$dBdCZ%dqdDdEZ&dqdFdGZ'dHdIZ(dJdKZ)dLdMZ*dNdOZ+dPdQZ,dRdSZ-dZ.dTdUZ/dVdWZ0dpdXdYZ1dZd[Z2dsd\d]Z3dpfd^d_ Z4fd`daZ5e6j7dbdcZ8e6j7fdddeZ9dfdgZ:dhdiZ;djdkZS)traBase class for configuration objects used in DeepMind. This is a container for configurations. It behaves similarly to Lua tables. Specifically: - it has dot-based access as well as dict-style key access, - it is type safe (once a value is set one cannot change its type). Typical usage example:: from ml_collections import config_dict cfg = config_dict.ConfigDict() cfg.float_field = 12.6 cfg.integer_field = 123 cfg.another_integer_field = 234 cfg.nested = config_dict.ConfigDict() cfg.nested.string_field = 'tom' print(cfg) Config dictionaries can also be used to pass named arguments to functions:: from ml_collections import config_dict def print_point(x, y): print "({},{})".format(x, y) point = config_dict.ConfigDict() point.x = 1 point.y = 2 print_point(**point) Note that, depending on your use case, it may be easier to use the `create` function in this package to construct a `ConfigDict`: from ml_collections import config_dict point = config_dict.create(x=1, y=2) Differently from standard `dicts`, `ConfigDicts` also have the nice property that iterating over them is deterministic, in a fashion similar to `collections.OrderedDicts`. T_allow_dotted_keys _sort_keysNF)r sort_keysrr,rrrcst|tr |}tt|ditt|ddtt|d|tt|d|tt|d|tt|d||durHt||t|trctt|d|jtt|d|jdSdS) aCreates an instance of ConfigDict. Warning: In most cases, this faithfully reproduces the reference structure of initial_dictionary, even if initial_dictionary is self-referencing. However, unexpected behavior occurs if self-references are contained within list, tuple, or custom types. For example:: d = {} d['a'] = d d['b'] = [d] cd = ConfigDict(d) cd.a # refers to cd, type ConfigDict. Expected behavior. cd.b # refers to d, type dict. Unexpected behavior. Warning: FieldReference values may be changed. If initial_dictionary contains a FieldReference with a value of type dict or FrozenConfigDict, that value is converted to ConfigDict. Args: initial_dictionary: May be one of the following: 1) dict. In this case, all values of initial_dictionary that are dictionaries are also be converted to ConfigDict. However, dictionaries within values of non-dict type are untouched. 2) ConfigDict. In this case, all attributes are uncopied, and only the top-level object (self) is re-addressed. This is the same behavior as Python dict, list, and tuple. 3) FrozenConfigDict. In this case, initial_dictionary is converted to a ConfigDict version of the initial dictionary for the FrozenConfigDict (reversing any mutability changes FrozenConfigDict made). type_safe: If set to True, once an attribute value is assigned, its type cannot be overridden without .ignore_type() context manager (default: True). convert_dict: If set to True, all dict used as value in the ConfigDict will automatically be converted to ConfigDict (default: True). allow_dotted_keys: If set to True, keys can contain `.`. Integer and float keys are always allowed to have dots. sort_keys: If `True` (default), keys are sorted in alphabetical order. _fields_lockedF _type_safe _convert_dictrrN) r r as_configdictsuperrrr is_lockedr)rKrr,rrr __class__rrrOss$ /    zConfigDict.__init__r6cCra)z)Returns True if config dict is type safe.)rrZrrrrzConfigDict.is_type_safecCra)z%Returns True if keys can contain `.`.)rrZrrrrrzConfigDict.allow_dotted_keyscCra)zCReturns True if it is converting dicts to ConfigDict automatically.)rrZrrrrrzConfigDict.convert_dictcsP|jr|Stt|dd|jD]}|j|}t|}t|tr%|q|S)zSLocks object, preventing user from adding new fields. Returns: self rT)rrrrrr4r lock)rKfieldelementrrrrs   zConfigDict.lockcCra)z!Returns True if object is locked.)rrZrrrrrzConfigDict.is_lockedcsFtt|dd|jD]}t|}t|tr |jr |q|S)zGrants user the ability to add new fields to ConfigDict. In most cases, the unlocked() context manager should be preferred to the direct use of the unlock method. Returns: self rF) rrrrvaluesr4r runlock)rKrrrrrs zConfigDict.unlockr5cCs"z||WSty|YSw)zCReturns value if key is present, or a user defined value otherwise.)KeyError)rKr5rLrrrr2s   zConfigDict.getcCsb|j|}|dur tdt|ts/t|}| |||<Wd|S1s*wY|S)z4Returns a FieldReference initialized on key's value.Nz6Cannot create reference to a field whose value is None)rrEr r1 ignore_typerKr5rrrrget_refs     zConfigDict.get_refcCs||S)aUReturns a one-way FieldReference. Example:: cfg = config_dict.ConfigDict(dict(a=1)) cfg.b = cfg.get_oneway_ref('a') cfg.a = 2 print(cfg.b) # 2 cfg.b = 3 print(cfg.a) # 2 (would have been 3 if using get_ref()) print(cfg.b) # 3 Args: key: Key for field we want to reference. )rrprKr5rrrget_oneway_refszConfigDict.get_oneway_refcs"|rjSfddjDS)aQReturns list of dictionary key, value pairs, sorted by key. Args: preserve_field_references: (bool) Whether to preserve FieldReferences if the ConfigDict has them. By default, False: any FieldReferences will be resolved in the result. Returns: The key, value pairs in the config, sorted by key. csg|]}||fqSrrrkrZrrr^'rhz$ConfigDict.items..)_ordered_fieldsrrKrrrZrrs zConfigDict.itemscCs |jr tt|jS|jS)z4Returns ordered dict shallow cast of _fields member.)r collections OrderedDictsortedrrrZrrrr)szConfigDict._ordered_fieldsccs6|jD]}|r||j|fVq|||fVqdS)aPDeterministically iterates over dictionary key, value pairs. Args: preserve_field_references: (bool) Whether to preserve FieldReferences if the ConfigDict has them. By default, False: any FieldReferences will be resolved in the result. Yields: The key, value pairs in the config, sorted by key. NrrrKrrrrrr1s  zConfigDict.iteritemscs$|ttt|vrtd|dS)Nz{} cannot be overridden.)dirrrrr+rK attributerrr_ensure_mutabilityAszConfigDict._ensure_mutabilityc Cs:z |||||<WdSty}zt|d}~wwr/rrAttributeError)rKrr"errrrEs zConfigDict.__setattr__c Cs8z ||||=WdSty}zt|d}~wwr/rrKrrrrr __delattr__Ls  zConfigDict.__delattr__c Cs,z||WSty}zt|d}~wwr/)rrrrrr __getattr__Ss  zConfigDict.__getattr__c Csx|jst|ttfsd|vrtd|d|jr-||jvr-d}t||| ||j||jvr|j|}z0t|t rG| ||j WdSt ||sft|tr]|jr]t|||j d}t|t||j }Wnty~}z td|d|dd}~ww|jrt|trt||j }n't|t r|rtnt|}|tus|turt||j }| |d||j|<dS) Nr7z]ConfigDict does not accept dots in field names (when `allow_dotted_keys=False`), but the key z contains one.zbKey "{}" does not exist and cannot be added since the config is locked. Other fields present: "{}"r,zCould not override field 'z' (reference). F)rr r%r&rErrr_generate_did_you_mean_messager+r1rHr_should_skip_type_checkrrr$r.r*rr[rr2r)rKr5r" error_msgrrref_typerrrr __setitem__YsZ        zConfigDict.__setitem__cCs`t|ttfr |Stdd|D}t||dd}|r.|r$|d7}|d|d|7}|S)Ncss|]}t|VqdSr/)r'rrrrrsz.r8g? z"Did you mean "{}" instead of "{}"?r)r r%r&r(keysdifflibget_close_matchesr+)rKrequestmessagermatchesrrrrs z)ConfigDict._generate_did_you_mean_messagec Cs|jrtd|js$t|ttfs$d|vr$|dd\}}|||=dSz|j|=WdStyA}z t||t |d}~ww)NzQThis ConfigDict is locked, you have to unlock it before trying to delete a field.r7r8) rrrr r%r&r:rrr')rKr5restrrrr __delitem__s   zConfigDict.__delitem__c Cs|jst|ttfsd|vr|dd\}}|||Sz|j|}t|tr,|WS|WStyD}z t| |t |d}~ww)Nr7r8) rr r%r&r:rr1r2rrr')rKr5rrrrrr __getitem__s      zConfigDict.__getitem__cCs ||jvSr/)rrrrr __contains__rzConfigDict.__contains__cCs8z tj|jddddWStyt|YSw)NTrF)default_flow_style)yamldumpto_dict ExceptionreprrZrrr__repr__s zConfigDict.__repr__cCs0zt|WStyt|YSwr/)rrrrr'rZrrr__str__s  zConfigDict.__str__cCst|jS)z.)r)rrrrrZrrs zConfigDict.valuesccs.|jD]}|r|j|Vq||VqdS)ahDeterministically iterates over values in a config, sorted by their keys. Args: preserve_field_references: (bool) Whether to preserve FieldReferences if the ConfigDict has them. By default, False: any FieldReferences will be resolved in the result. Yields: The values in the config, sorted by their corresponding keys. Nrrrrr itervaluess   zConfigDict.itervaluescCs|ttSr/)rrrrZrrr__dir__rvzConfigDict.__dir__cCr r/)r__len__rZrrrr rzConfigDict.__len__cCr r/)r__iter__rZrrrrrzConfigDict.__iter__cCs@t||jr|j|jk}||j|jkM}||j|jkM}|SdS)z%Override the default Equals behavior.F)r rrrrrr)rKrdsamerrrres  zConfigDict.__eq__cCs || S)zDefine a non-equality test.)rercrrr__ne__ s zConfigDict.__ne__cCst|tr t|t|kSdS)a~Type-invariant equals. This is like `__eq__`, except it does not distinguish `FrozenConfigDict` from `ConfigDict`. For example:: cd = ConfigDict() fcd = FrozenConfigDict() fcd.eq_as_configdict(cd) # Returns True Args: other: Object to compare self to. Returns: same: `True` if `self == other` after conversion to `ConfigDict`. F)r rrcrrreq_as_configdicts zConfigDict.eq_as_configdictcKstj|fi|S)aaReturns a YAML representation of the object. ConfigDict serializes types of fields as well as the values of fields themselves. Deserializing the YAML representation hence requires using YAML's UnsafeLoader: ``` yaml.load(cfg.to_yaml(), Loader=yaml.UnsafeLoader) ``` or equivalently: ``` yaml.unsafe_load(cfg.to_yaml()) ``` Please see the PyYAML documentation and https://msg.pyyaml.org/load for more details on the consequences of this. Args: **kwargs: Keyword arguments for yaml.dump. Returns: YAML representation of the object. )rrrKkwargsrrrto_yaml)szConfigDict.to_yamlc Ks@z tj|fi|WSty}z d|}t|d}~ww)a^Wrapper for json.dumps() method. Produces a more informative error message when there is a problem with string encodings in the ConfigDict. Args: **kwargs: Keyword arguments for json.dumps. Returns: JSON representation of the object. Raises: JSONDecodeError: If there is a problem with string encodings. zDecoding error. Make sure all strings in your ConfigDict use ASCII-compatible encodings. See https://docs.python.org/2.7/howto/unicode.html#the-unicode-type for details. Original error message: {}N)jsondumpsUnicodeDecodeErrorr+r)rKrerror new_messagerrr_json_dumps_wrapperEszConfigDict._json_dumps_wrappercKs|pt}|jdd|i|S)aReturns a JSON representation of the object, fails if there is a cycle. Args: json_encoder_cls: An optional JSON encoder class to customize JSON serialization. **kwargs: Keyword arguments for json.dumps. They cannot contain "cls" as this method specifies it on its own. Returns: JSON representation of the object. Raises: TypeError: If self contains set, frozenset, custom type fields or any other objects that are not JSON serializable. clsNr)CustomJSONEncoderr)rKjson_encoder_clsrrrrto_json_szConfigDict.to_jsoncKs|jddti|S)aReturns a best effort JSON representation of the object. Tries to serialize objects not inherently supported by JSON encoder. This may result in the configdict being partially serialized, skipping the unserializable bits. Ensures that no errors are thrown. Fails if there is a cycle. Args: **kwargs: Keyword arguments for json.dumps. They cannot contain "cls" as this method specifies it on its own. Returns: JSON representation of the object. rNr)r_BestEffortCustomJSONEncoderrrrrto_json_best_effortrszConfigDict.to_json_best_effortcCs|pi}i}||t|<|D]x}t|j|tr$|r$|j|}|}n||}|}t||vr9|t|||<qt|tret|trZ|}tit}||t|<||||n|||}|||<qt|tr|}td| }|j|dd||t|<|||<q|S)aConverts ConfigDict to regular dict recursively with valid references. By default, the output dict will not contain FieldReferences, any present in the ConfigDict will be resolved. However, if `preserve_field_references` is True, the output dict will contain FieldReferences where the original ConfigDict has them. They will not be the same as the ConfigDict's, and their ops will be applied and dropped. Note: As with __eq__() and __init__(), this may not behave as expected on a ConfigDict with self-references contained in lists, tuples, or custom types. Args: visit_map: A mapping from object ids to their dict representation. Method is recursive in nature, and it will call ".to_dict(visit_map)" on each encountered object, unless it is already in visit_map. preserve_field_references: (bool) Whether the output dict should have FieldReferences if the ConfigDict has them. By default, False: any FieldReferences will be resolved and the result will go to the dict. Returns: Dictionary with the same values and references structure as a calling ConfigDict. NFr) rPr rr1r2rrrHrrF)rKrr dict_selfr5 referencer" old_referencerrrrs8             zConfigDict.to_dictcs|pi}|}tt|d|jtt|d|j||t|<|jD]4\}}t |t r4| }t||vrA|t|}n t |trK| |}t |t rW|||q'|||<q'tt|d|jtt|d|j|S)a7Returns a ConfigDict copy with FieldReferences replaced by values. If the object is a FrozenConfigDict, the copy returned is also a FrozenConfigDict. However, note that FrozenConfigDict should already have FieldReferences resolved to values, so this method effectively produces a deep copy. Note: As with __eq__() and __init__(), this may not behave as expected on a ConfigDict with self-references contained in lists, tuples, or custom types. Args: visit_map: A mapping from ConfigDict object ids to their copy. Method is recursive in nature, and it will call ".copy_and_resolve_references(visit_map)" on each encountered object, unless it is already in visit_map. Returns: ConfigDict copy with previous FieldReferences replaced by values. rrrr)rrrrrrrPrrr r1r2copy_and_resolve_referencesr_frozen_setattrrr)rKrconfig_dict_copyr5r"rrrr$s8           z&ConfigDict.copy_and_resolve_referencescsx|tt|d|dtt|d|dd|dD] }|d|||<q |dr:tt|dddSdS)z2Recreates ConfigDict from its dict representation.rrTrrN)rOrrrr2)rKrrrrrrs   zConfigDict.__setstate__ccsB|j}|r |z|VW|r|dSdS|r |ww)z7Context manager which temporarily unlocks a ConfigDict.N)rrr)rK was_lockedrrrunlockeds  $zConfigDict.unlockedc #s8|j}g}t}t|j}|rN|}t||vrq|t|t|t r0| |j nt|t j r>||nt|t jt jfrL|||stt |ddz:t}|D]}||q_|VWdn1suwYWtt |d|dSWtt |d|dStt |d|w)zDContext manager which temporarily turns off type safety recursively.rFN)rrHr)rrpoprPrQr rappendrcollections_abcrextendSequenceSetrr contextlib ExitStack enter_context)rKoriginal_type_safetymanagersrUfieldsrstackmanagerrrrrs6     *zConfigDict.ignore_typecCs0||}t|j|tr|j|St|S)z0Returns type of the field associated with a key.)rr rr1rFr$rrrrrF-s zConfigDict.get_typecOst|dkrtdt|||fD]M}i}t|tr!d|d<|jdi|D]7\}}||vr6|||<q)t|j|trH||||q)t|j|tr\t|tr\td||||<q)qdS)a&Update values based on matching keys in another dict-like object. Mimics the built-in dict's update method: iterates over a given mapping object and adds/overwrites keys with the given mapping's values for those keys. Differs from dict.update in that it operates recursively on existing keys that are already a ConfigDict (i.e. calls their update() on the corresponding value from other), and respects the ConfigDict's type safety status. If keyword arguments are specified, the ConfigDict is updated with those key/value pairs. Args: *other: A (single) dict-like container, e.g. a dict or ConfigDict. **kwargs: Additional keyword arguments to update the ConfigDict. Raises: TypeError: if more than one value for `other` is specified. r8z+update expected at most 1 arguments, got {}Trz value. strip_prefix: A prefix to be stripped from `path`. If specified, only paths matching `strip_prefix` will be processed. Raises: KeyError: if any of the key paths can't be found. cs i|] \}}|r||qSr) startswith)rr5r" strip_prefixrr s z9ConfigDict.update_from_flattened_dict..Nr7rz-Key "{}" cannot be set as "{}" was not found.z3Key "{}" cannot be set as "{}" is not a tuple/list.zinteresting_itemschildren_to_updatefull_keyr"r5 full_childchildr? child_valuechild_strip_prefixrr=rr@sHA      z%ConfigDict.update_from_flattened_dict)NTT)r6rr/F)r)NF)?r rrrB_HAS_DYNAMIC_ATTRIBUTESbool__annotations__rrr'rrOpropertyrrrrrrr2rrrrrrrrrrrrrrrrrr rr r r rrerrrrrrr rr$rr/contextmanagerr(rrFr8r;r@ __classcell__rrrrr=s .B       1       A 2   ? rcCs*t|tsdS|t|tfvrdSdS)z1Returns True if the type check should be skipped.FT)r r1rFr$object)rY new_valuerrrrs rcCs|pg}t|tr Jt||vrtd|t|t|tr.|D]}t||q%n*t|tr;t| |nt|t t frX|D]}t|tttfrRtdt||qD| dS)aoRaises error if obj is NOT a valid input for FrozenConfigDict. Args: obj: Object to check. In the first call (with ancestor_list = None), obj should be of type ConfigDict. During recursion, it may be any type except dict. ancestor_list: List of ancestors of obj in the attribute/element structure, used to detect reference cycles in obj. Raises: ValueError: If obj is an invalid input for FrozenConfigDict, i.e. if it contains a dict within a list/tuple or contains a reference cycle. Also if obj is a dict, which means it wasn't already converted to ConfigDict. zWBad FrozenConfigDict initialization: Cannot contain a cycle in its reference structure.zqBad FrozenConfigDict initialization: Cannot contain a dict, ConfigDict, or FieldReference within a list or tuple.N) r rrPrEr*rr_frozenconfigdict_valid_inputr1r2r)r(r))obj ancestor_listr"rrrrrRs"       rRcCst||vsJg}d}|D]+}t|tttfrJt|tttfr4t||\}}}||M}| |q| |q|rA|d|fSt|d|fS)aConvert tuple to fully immutable tuple. Args: value: Tuple to be made fully immutable (including its elements). visit_map: As used elsewhere. See _frozenconfigdict_fill_seed() documentation. Must not contain id(value) as a key (if it does, an immutable version of value already exists). Returns: immutable_value: Immutable version of value, created with minimal copying (for example, if a value contains no mutable elements, it is returned untouched). same_value: Whether the same value was returned untouched, i.e. with the same memory address. Boolean. visit_map: Updated visit_map Raises: TypeError: If one of the following: 1) value is not a tuple. 2) value contains a dict, ConfigDict, or FieldReference. If it does, value is an invalid attribute of FrozenConfigDict, and this should have been caught in valid_input at initialization. ValueError: id(value) is in visit_map. TF) rPr rrr1r)r(rH_convert_to_immutabler*)r"r value_copy same_valuer new_elementuncopied_elementrrr_tuple_to_immutable6s    rZcCst|}||vr||d|fSd}t|trt|}n t|tr)t||\}}}nt|tr9tt||\}}}nJ|||<|||fS)a]Convert Python built-in type to immutable, copying if necessary. Args: value: To be made immutable type (including its elements). Must have type list, tuple, or set. visit_map: As used elsewhere. See _frozenconfigdict_fill_seed() documentation. Returns: immutable_value: Immutable version of value, created with minimal copying. same_value: Whether the same value was returned untouched, i.e. with the same memory address. Boolean. visit_map: Updated visit_map. Raises: TypeError: If value is an invalid type (not a list, tuple, or set). TF)rPr rH frozensetr(rZr))r"rvalue_idrWimmutable_valuerkrrrrUes"      rUcCst|tsJ|r Jt|d||pi}||t|<|D]9\}}t|tr-tdt||vr:|t|}nt|trRt|tsRt|j d}t ||||}| |||q dS)aFills an empty FrozenConfigDict without copying previously visited nodes. Turns seed (an empty FrozenConfigDict) into a FrozenConfigDict version of initial_configdict. Avoids duplicating nodes of initial_configdict because if a value of initial_configdict has been previously visited, that value is not re-converted to a FrozenConfigDict. If a FieldReference is encountered which contains a dict, its contents will be converted to FrozenConfigDict. Note: As described in the __init__() documentation, this will not replicate the structure of initial_configdict if it contains self-references within lists, tuples, or other types. There is no warning or error in this case. Args: seed: Empty FrozenConfigDict, to be filled in. initial_configdict: The template on which seed is built. Must be of type ConfigDict. visit_map: Dictionary from memory addresses to values, storing the FrozenConfigDict versions of dictionary values. Lists which have been converted to tuples and sets to frozensets are also stored in visit_map to preserve the reference structure of initial_configdict. visit_map need not contain (id(initial_configdict), seed) as a key/value pair. Raises: ValueError: If one of the following, both of which can never happen in practice: 1) seed is not an empty FrozenConfigDict. 2) initial_configdict contains a FieldReference. _configdictzqTrying to initialize a FrozenConfigDict value with a FieldReference. This should never happen, please file a bug.rN) r rrPrrPrr1rErr_frozenconfigdict_fill_seedr%)rinitial_configdictrr5r"value_frozenconfigdictrrrr_s(!      r_cs|eZdZdZdfdd ZdddZdd Zd d Zd d ZddZ ddZ ddZ ddZ ddZ ddZddZZS)raImmutable and hashable type of ConfigDict. See ConfigDict() documentation above for details and usage. FrozenConfigDict is fully immutable. It contains no lists or sets (at initialization, lists and sets are converted to tuples and frozensets). The only potential sources of mutability are attributes with custom types, which are not touched. It is recommended to convert a ConfigDict to FrozenConfigDict after construction if possible. NTcsPtt|t||d}t|t||t|d|jt|d|j dS)aCreates an instance of FrozenConfigDict. Lists and sets are copied into tuples and frozensets. However, copying is kept to a minimum so tuples, frozensets, and other immutable types are not copied unless they contain mutable types. Prohibited initial_dictionary structures: initial_dictionary may not contain any lists or tuples with dictionary, ConfigDict, or FieldReference elements, or else an error is raised at initialization. It also may not contain loops in the reference structure, i.e. the reference structure must be a Directed Acyclic Graph. This includes loops in list-element and tuple-element references. initial_dictionary's reference structure need not be a tree. Warning: Unexpected behavior may occur with types other than Python's built-in types. See ConfigDict() documentation for details. Warning: As with ConfigDict, FieldReference values may be changed. If initial_dictionary contains a FieldReference with a value of type dict or ConfigDict, that value will be converted to FrozenConfigDict. Args: initial_dictionary: May be one of the following: 1) dict. In this case all values of initial_dictionary that are dictionaries are also converted to FrozenConfigDict. If there are dictionaries contained in lists or tuples, an error is raised. 2) ConfigDict. In this case all ConfigDict attributes are also converted to FrozenConfigDict. 3) FrozenConfigDict. In this case all attributes are uncopied, and only the top-level object (self) is re-addressed. type_safe: See ConfigDict documentation. Note that this only matters if the FrozenConfigDict is converted to ConfigDict at some point. )rr,rrN) rrrOrrRr_rPrrr)rKrr,r`rrrrOs& zFrozenConfigDict.__init__cCsl|pi}||d|vsJ||jvrtdt|tttfr/t||\}}}||j|<|S||j|<|S)aSets attribute, analogous to __setattr__(). Args: key: Name of the attribute to set. value: Value of the attribute to set. visit_map: Dictionary from memory addresses to values, storing the FrozenConfigDict versions of value's elements. Lists which have been converted to tuples and sets to frozensets are also stored in visit_map. Returns: visit_map: Updated visit_map. Raises: ValueError: If there is a dot in key, or value contains dicts inside lists or tuples. Also if key is already an attribute, since redefining an attribute is prohibited for FrozenConfigDict. AttributeError: If key is protected (such as '_type_safe' and '_locked'). r7z.Cannot redefine attribute of FrozenConfigDict.)rrrEr r)r(rHrU)rKr5r"rr]rkrrrr% s     z FrozenConfigDict._frozen_setattrcCs||ddS)z8Recreates FrozenConfigDict from its dict representation.r^N)rOrrrrr3szFrozenConfigDict.__setstate__cCr)Nz9FrozenConfigDict is immutable. Cannot call __setattr__().rrKrr"rrrr7zFrozenConfigDict.__setattr__cCr)Nz9FrozenConfigDict is immutable. Cannot call __delattr__().rbrrrrr;rdzFrozenConfigDict.__delattr__cCr)Nz9FrozenConfigDict is immutable. Cannot call __setitem__().rbrcrrrr?rdzFrozenConfigDict.__setitem__cCr)Nz9FrozenConfigDict is immutable. Cannot call __delitem__().rbrrrrrCrdzFrozenConfigDict.__delitem__cCr)Nz2FrozenConfigDict is immutable. Cannot call lock().rbrZrrrrGrdzFrozenConfigDict.lockcCr)Nz4FrozenConfigDict is immutable. Cannot call unlock().rbrZrrrrJrdzFrozenConfigDict.unlockcs~fddd}|jD]&\}}t|tr#|tt|t|f7}q |tt||jj|f7}q t||j|jf}|S)aComputes hash. The hash depends not only on the immutable aspects of the FrozenConfigDict, but on the types of the initial_dictionary at initialization (i.e. on the _configdict attribute). For example, in the following, hash(frozen_1) will not equal hash(frozen_2): d_1 = {'a': (1, )} d_2 = {'a': [1]} frozen_1 = FrozenConfigDict(d_1) frozen_2 = FrozenConfigDict(d_2) Note: This implementation relies on the particulars of the FrozenConfigDict structure. For example, the fact that lists and tuples cannot contain dicts or ConfigDicts is crucial, as is the fact that any loop in the reference structure is prohibited. Note: Due to hash randomization, this hash will likely differ in different Python sessions. For comparisons across sessions, please directly use equality of the serialization. For more, see https://bugs.python.org/issue13703 Returns: frozenconfigdict_hash: The hash value. Raises: TypeError: self contains an unhashable type. cst|trttt|dfSt|ttfr0t|tg}|D] }|}||qtt|St|tr;|Szt|WSt yPt d t |w)zHashes a single value.r8z,FrozenConfigDict contains unhashable type {}) r rHhashr[r)r(r*r1r2r*r+r$)r"value_hash_listr element_hash value_hashrrrijs"         z-FrozenConfigDict.__hash__..value_hashr)rrr rrer^rr)rK fields_hashr5r"frozenconfigdict_hashrrhrrMs    zFrozenConfigDict.__hash__cCst|tr t|t|kSdS)aOverride default Equals behavior. Like __hash__(), this pays attention to the type of initial_dictionary. See __hash__() documentation for details. Warning: This distinguishes FrozenConfigDict from ConfigDict. For example: cd = ConfigDict() fcd = FrozenConfigDict() fcd.__eq__(cd) # Returns False Args: other: Object to compare self to. Returns: same: Boolean self == other. F)r rrrcrrrres zFrozenConfigDict.__eq__cCrar/)r^rZrrrrrbzFrozenConfigDict.as_configdict)NTr/)r rrrBrOr%rrrrrrrrrerrOrrrrrs 2&=rc@seZdZdZddZdS)rznJSON encoder for ConfigDict and FieldReference. The encoder throws an exception for non-supported types. cCsFt|tr |St|tr|jSt|trt|Stdt|)NzI{} is not JSON serializable. Instead use ConfigDict.to_json_best_effort()) r r1r2rrr$r'r*r+rKrSrrrrLs    zCustomJSONEncoder.defaultN)r rrrBrLrrrrrs rcs eZdZdZfddZZS)rz}Best effort JSON encoder for ConfigDict. The encoder tries to serialize non-supported types (doesn't throw exceptions). csz tt||WSty_t|trtt|YSt |r*d |j YSt |r6t |YSt|drJ|jrJt|sJt|jYSt|drVd |YSd t|YSw)Nz function {}rrzunserializable object: {}z!unserializable object of type: {})rrrLr*r rHrr)inspect isfunctionr+r dataclasses is_dataclassasdicthasattrrisclassrr$rlrrrrLs&      z$_BestEffortCustomJSONEncoder.default)r rrrBrLrOrrrrrsrcKs t|dS)aCreates a `ConfigDict` with the given named arguments as key-value pairs. This allows for simple dictionaries whose elements can be accessed directly using field access:: from ml_collections import config_dict point = config_dict.create(x=1, y=2) print(point.x, point.y) This is particularly useful for compactly writing nested configurations:: config = config_dict.create( data=config_dict.create( game='freeway', frame_size=100), model=config_dict.create(num_hidden=1000)) The reason for the existence of this function is that it simplifies the code required for the majority of the use cases of `ConfigDict`, compared to using either `ConfigDict` or `namedtuple's`. Examples of such use cases include training script configuration, and returning multiple named values. Args: **kwargs: key-value pairs to be stored in the `ConfigDict`. Returns: A `ConfigDict` containing the key-value pairs in `kwargs`. )r)r)rrrrcreates rtcCstd||dS)aDefines an entry in a ConfigDict that has no value yet. Example:: config = configdict.create( batch_size = configdict.placeholder(int), frame_shape = configdict.placeholder(tf.TensorShape)) Args: field_type: type of value. required: If True, the placeholder will raise an error on access if the underlying value hasn't been set. Returns: A `FieldReference` with value None and the given type. NrrN)r1rurrr placeholdersrvcCs t|ddS)aDefines an entry in a ConfigDict with unknown but required value. Example:: config = configdict.create( batch_size = configdict.required_placeholder(int)) try: print(config.batch_size) except RequiredValueError: pass config.batch_size = 10 print(config.batch_size) # 10 Args: field_type: type of value. Returns: A `FieldReference` with value None and the given type. T)rN)rvrrrrrequired_placeholders rwcCsZt}|D]#\}}t|trt|||}n|}||kr$t|||qt|||q|S)aReturns copy of conf with old_name recursively replaced by new_name. This is not done in place, no changes are made to conf but a new ConfigDict is returned with the changes made. This is useful if the name of a parameter has been changed in code but you need to load an old config. Example usage: updated_conf = configdict.recursive_rename(conf, "config", "kwargs") Args: conf: a ConfigDict which needs updating old_name: the name used in the ConfigDict which is out of sync with the code new_name: the name used in the code Returns: A ConfigDict which is a copy of conf but with all instances of old_name replaced with new_name. )rrr recursive_renamesetattr)confold_namenew_namenew_confnamernew_crrrrxs rxrIr/):rBrrr+r/rorrzrmrrrr;typingrrrrabslrrr Representeradd_representerABCMetarepresent_namerr rrr$rrr#r.r4r'r%r@ namedtuplerAtotal_orderingr1rrrKrrRrZrUr_r JSONEncoderrrrtrvrwrxrrrrsp    6   DLK (/ 'AX !