o ,wi+@s&dZddlZddlZddlZddlmZmZmZmZm Z m Z m Z m Z m Z ddlmZddlZe dZe dZe e eedeffZGdd d eeejZGd d d eeejZd ed efddZGdddeZejGdddeeZejGdddZeZejZej Z ej!Z!ej"Z"dS)aA registry-based system for constructing models. ## Skeletons and fixtures Skeletons generally express a tree of modules; for example in an EncoderLayer, one might construct Flax modules like so: EncoderLayer attention: normalized_block() layer_norm: layer_norm() body: self_attention() wrapped: MultiHeadDotProductAttention() dropout: dropout() mlp: normalized_block() layer_norm: layer_norm() body: mlp() wrapped: MlpBlock() dropout: dropout() and this becomes even more detailed when libraries are composed. In order to mitigate some verbosity, skeletons allow these tree-like templates to be indexed by the type of the module. However, details of configuration, typically specific to a project like T5, should be filled in later. We call this configuration a "fixture". Generally "skeletons" should only make choices that are quite natural to the types themselves; for example, if an EncoderLayer has an "attention" and "mlp" fields, then those are almost always filled with normalized blocks containing attention and MLP modules. ## Validators Validators are another utility to separate concerns between module authors and projects which may use those modules. A module author might know that their MlpBlock() module must have a parameter `foo` specified whenever another parameter `bar` is True, and it is convenient to express those invariants at the level of the type (MlpBlock, in this case) rather than counting on each project / experimenter to reproduce those checks. ## Global vs. local registries In general, we are going to have a global type-based registry for constructing "skeletons" of models, and then use functions that set configuration in a nested manner. However, the underlying implementation here is rather unopinionated in terms of how "global" its state is, so if specific projects find non-global registries more useful, they can easily work with them. TBD: * Examples with placeholders. * Add a function for validating configs by traversing the config tree. N) AnyCallableDictGenericListOptionalTypeTypeVarUnion)configST.c@$eZdZdejeddfddZdS) SkeletonFncfgreturnNcCdSNselfrrro/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/fiddle/_src/experimental/autobuilders/autobuilders.py__call__UzSkeletonFn.__call____name__ __module__ __qualname__ config_libConfigr rrrrrrSrc@r) ValidatorFnrrNcCrrrrrrrr[rzValidatorFn.__call__rrrrrr!Yr r! fn_or_clsrcCsFz t|du}Wn tyd}Ynw|o"t| o"t| S)avHelper method for determining wheher a function or class can be configured. This will return False for builtin stuff like int/tuple/etc. We generally don't want to configure these classes, partly because they work without requisite arguments, for example int() is 0. Args: fn_or_cls: Function or class. Returns: Whether `fn_or_cls` can be configured. NF)inspect signature ValueError isbuiltinismethod)r" has_signaturerrr_is_configurable_s   r)cs2eZdZdZdeeffdd ZddZZS)DuplicateSkeletonErrorzHIndicates that two skeletons were attempted to be registered for a type.r"cstd|d||_dS)NSkeletonFn for z already exists!)super__init__r")rr" __class__rrr-zs zDuplicateSkeletonError.__init__cCs6t|j}t|j\}}d|jd|d|dS)Nr+z (defined at :z) already exists!)r# getsourcefiler"getsourcelines)rfilename_ line_numberrrr__str__~s  zDuplicateSkeletonError.__str__) rrr__doc__FnOrClsrr-r6 __classcell__rrr.rr*wsr*c@sBeZdZUdZdZeeeed<e j e dZ e eeed<dS) TableEntryaAn entry in the registry for a specific type or function. Attributes: skeleton_fn: A function which can create an initial configuration for this type or function. validators: A list of functions which can validate configuration objects of this type or function. N skeleton_fndefault_factory validators)rrrr7r;rrr __annotations__ dataclassesfieldlistr>rr!rrrrr:s  r:c@seZdZUdZejdddZeee e e fe d< ddee de d eje fd d Zdee d eee gee ffd d Z ddee de d dfddZdee d eee gee ffddZdS)RegistryzAn AutoBuilder registry, which knows how to create a model skeleton. Attributes: table: A mapping from a type or function to configure, to a TableEntry. cCs ttSr) collections defaultdictr:rrrrs zRegistry.r<tableTr"require_skeletonrcCsDt|}|j|}|jdur|rtd|d|S|||S)a Creates a configuration instance of a given function or class. Args: fn_or_cls: Function or class to get a skeleton fdl.Config instance for. require_skeleton: If True, a KeyError will be raised if a skeleton of the desired function/type is not available. If False, an empty fdl.Config of that function/class will be returned. Returns: Configuration instance. Raises: KeyError: If there is no skeleton registered for the given class, and require_skeleton is True (the default). NzNo skeleton registered for .)rrrGr;KeyError)rr"rH base_configentryrrrr s    zRegistry.configcsfdd}|S)aRegisters a function as a skeleton for a given type. Example: @dataclasses.dataclass class MyDense: activation: Callable[..., Any] @ab.skeleton(MyDense): def my_dense_skeleton(config: fdl.Config): config.activation = fdl.Config(nn.gelu, approximate=True) Often the skeleton definition (`my_dense_skeleton`) can be kept separately from the modeling library code, to make the modeling library code agnostic to the choice of configuration library. Args: fn_or_cls: Function that the skeleton is configuring. Returns: Decorator that, when applied to a function definition, will register that function as a skeleton. cs&j}|jdurt||_|Sr)rGr;r*)r;rLr"rrrinners  z Registry.skeleton..innerrrr"rNrrMrskeletonszRegistry.skeletonFattributes_require_skeletonsNc sit|jD]C\}}||vr||dur|||<q |jtjjkr1td|d|dt|jsHt d|d|d|jd|d |j|<q dt j ffd d } ||dS) aRegisters a skeleton based on arg types. Sometimes functions are written to take a more generic class, for example one might have an encoder layer in Flax expressed like, class EncoderLayer(nn.Module): dropout: nn.Module but in practice, `dropout` is almost always `nn.Dropout`, the type signature is just left to be the most general thing in case of experimental overrides. In this case, we can specify that the dropout is `nn.Dropout` as follows: ab.auto_skeleton(EncoderLayer, dropout=nn.Dropout) This is equivalent to writing: @ab.skeleton(EncoderLayer) def encoder_layer_skeleton(config: fdl.Config[EncoderLayer]): config.dropout = ab.config(nn.Dropout, require_skeleton=False) By default, `auto_skeleton` doesn't require the attributes to have defined skeletons, but would use them if available. In this case, the `dropout` class attribute can be filled with a blank `fdl.Config(nn.Dropout)`, if there is not a skeleton for `nn.Dropout`. If you require more advanced skeleton setups, then please use the `skeleton` method. Args: fn_or_cls: Function or class to generate the skeleton for. attributes_require_skeletons: Whether skeletons are required for any class attributes or function arguments to `fn_or_cls`. By default, skeletons are not required, whereupon empty fdl.Config instances are created. **kwargs: Any override types to use for configuration. Raises: ValueError: If there is no annotation for a parameter not mentioned in **kwargs. TypeError: If one of the parameters not mentioned in **kwargs is not a configurable type. Nz Parameter z of zY doesn't have an annotation, and no override type was set in the auto_skeleton decorator.z is of type z*, which cannot be configured. Please pass za=None to your auto-skeleton invocation to avoid automatically adding a Config for this parameter.rcs,D]\}}t||j|dqdS)N)rH)itemssetattrr )rnametype_to_configure arg_to_typerQrrrarg_type_skeletonsz1Registry.auto_skeleton..arg_type_skeleton) r#r$ parametersrR annotation Signatureemptyr%r) TypeErrorrrrP)rr"rQkwargsrT parameterrXrrVr auto_skeletons*.    zRegistry.auto_skeletoncs$dttdttffdd }|S)aDecorator to register a function as a validator for a given type. Args: fn_or_cls: Function that the validator is checking. Returns: Decorator that, when applied to a function definition, will register that function as a validator. validator_fnrcsjj||Sr)rGr>append)rarMrrrN5sz!Registry.validator..inner)r!r rOrrMr validator(s zRegistry.validator)T)F)rrrr7r@rArGrr8rr:r?r boolrrr rrrPr`r!rcrrrrrCs>   % LrC)#r7rDr@r#typingrrrrrrrr r fiddle._srcr rtyping_extensionsr r r8Protocolrr!rdr)r%r* dataclassr:rC_default_registryrPr`rcrrrrs05,  +