o qoi;@sdZddlZddlZddlZddlZddlmZmZmZm Z ddl m Z ddl m Z ddl m Z ddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZe jddddZejddddZe jdgddZ e jdgddZ!e jdgddZ"e j#ddddZ$ddZ%dej&fd d!Z'd"e e(d#ee(fd$d%Z)d"e e(fd&d'Z*dej&fd(d)Z+ d5dej&d*efd+d,Z,d-d.Z-d#e.fd/d0Z/  d6d1eed2eej0d#ej&fd3d4Z1dS)7aLegacy API to use command line flags with Fiddle Buildables. NOTE: Deprecation: These are legacy APIs. Please refer to the new API usage in flags.py and find the documentation in flags_code_lab.md. While it's generally better to check in the full configuration into a source- control system, there are some instances (e.g. hyperparameter sweeps) where it's most effective to set initialization parameters (hyperparameters) from command line flags. This library exposes functions you can call within your program. NOTE: The flags aren't applied in the order they are passed in on the command line. The order followed is: - all fiddlers are applied first, followed by - all tags, followed by - all overrides. ## Quickstart > Note: although different bits of functionality are designed to be usable > piecemeal, this is the fastest way to get all flag-integrated features. First, some definitions: - *A module for base configurations*: So that libraries can be easily reused with or without fiddle, code to define fiddle configurations are often put in a separate package / module. Most commonly, there is one function (called `default` by convention) - *Fiddlers*: A fiddler is a function that takes a `fdl.Buildable` and performs arbitrary mutations. These are commonly also grouped into the same module that defines the base configuration(s). - *Overrides*: Overrides are command line options that set values to parameters within the (nested) fiddle tree. The simplest way to use the fiddle support for absl flags is to pass `flags_parser=absl_flags.flags_parser` when calling `app.run`. Then, simply call `cfg = absl_flags.create_buildable_from_flags(configs)` where `configs` is a module for base configurations and fiddlers. For example: ```py ## ---- config.py ---- import fiddle as fdl def baseline() -> fdl.Config: # A base configuration. my_model = fdl.Config(MyModel) my_optimizer = fdl.Config(MyOptimizer) return fdl.Config(MyTrainer, optimizer=my_optimizer, model=my_model) def set_dtypes_to_bfloat_16(cfg: fdl.Config): # A fiddler. cfg.model.dtype = 'bfloat16' # Arbitrary python. ## ---- main.py ---- from absl import app import fiddle as fdl from fiddle import absl_flags import config def main(argv): cfg = absl_flags.create_buildable_from_flags(configs) trainer = fdl.build(cfg) trainer.train() if __name__ == '__main__': app.run(main, flags_parser=absl_flags.flags_parser) ``` ## Overrides-only If you simply want to use command line flags to override values in a config built through arbitrary python, follow these steps: 1. *main*: Pass `fdl.absl_flags.flags_parser` when calling `absl.app.run`. For example, your `app.run` call should look like the following: `app.run(my_main_function, flags_parser=fiddle.absl_flags.flags_parser)` 2. *Augment a config*: Somewhere in your program, call `fdl.absl_flags.apply_overrides_to(cfg)` to apply settings defined on the command line to `cfg`. (This is often best done right before the call to `fdl.build(cfg)`.) 3. Set values on the command line: `--fdl.my.property=3 --fdl.other.setting=5` ### Configuring multiple fiddle objects If you have multiple fiddle objects that you'd like to configure on the command line, create an empty top-level config and use that. Example: ``` my_program_part_1 = fdl.Config(...) my_program_part_2 = fdl.Config(...) top_level = fiddle.experimental.DictConfig(part1=my_program_part_1) top_level.part2 = my_program_part_2 fdl.absl_flags.apply_overrides_to(top_level) run_part_1(fdl.build(my_program_part_1)) run_part_2(fdl.build(my_program_part_2)) ``` ## Acknowledgements This implementation has drawn inspiration from multiple sources, including [T5X](https://github.com/google-research/t5x), among others. Thank you! N)AnyListOptionalSequence)app)flags)logging)epath)printing) selectors)config)tagging)utils) auto_config) serialization fdl_configz9The name of a function to construct a base Fiddle config.)defaulthelpfdl_config_filezNThe path to a file containing a serialized base Fiddle config in JSON format. fiddlerzCThe name of a fiddler. Fiddlers are functions that modify a config.fdl_setzYPer-parameter configuration settings. Typically accessed via the alias --fdl.foo.bar=123. fdl_tags_setzTFiddle tags setting, by name. Typically accessed via the alias --fdl_tag.foo.bar=123fdl_helpFz\Print out Fiddle-specific help (including information on any loaded configuration) and exit.cOst|i|dtjidS)Nfile)printsysstderr)argskwargsrW/home/ubuntu/.local/lib/python3.10/site-packages/fiddle/_src/absl_flags/legacy_flags.py _print_stderrsr!cfgcCstjD]}t||qdS)z[DEPRECATED] Applies all command line flags to `cfg`. Deprecation: This is a legacy API. Please refer to the new API usage in flags.py and find the documentation in flags_code_lab.md. Args: cfg: The configuration to apply overrides to. N)_FDL_SETvaluer set_value)r"flagrrr apply_overrides_tos r'rreturncCs dtdtfdd}tt||S)aRewrites short-form Fiddle flags. There are two main rewrites: * `--fdl.NAME=VALUE` to `--fdl_set=NAME = VALUE`. * `--fdl_tag.NAME=VALUE` to `--fdl_tags_set=NAME = VALUE`. Args: args: Command-line args. Returns: Rewritten args. argr(cSs|ds |drWd|vr"|jdddd}td|d |d |dr*d }n|dr1d }|jddd\}}|jddd\}}d |d|d|}td|||S|S)Nz--fdl.z --fdl_tag.=.maxsplitrz$Fiddle setting must be of the form `z.NAME=VALUE`; `got: "z".rrz--zRewrote flag "%s" to "%s".) startswithsplit ValueErrorrdebug)r)prefix explicit_name_pathr$ rewrittenrrr _rewrites$  z"rewrite_fdl_args.._rewrite)strlistmap)rr8rrr rewrite_fdl_argssr<cCstt|S)zFlag parser. See absl.app.parse_flags_with_usage and absl.app.main(..., flags_parser). Args: args: All command line arguments. Returns: Whatever `absl.app.parse_flags_with_usage` returns. Sorry! )rparse_flags_with_usager<)rrrr flags_parsers r>cstj|dd}tjD]W}|jddd\}fdd|D}|s=fdd|D}|r1d |d nd }td d |t|dkrKtddtj ||ddj t j |t |dddq dS)a[DEPRECATED] Sets tags based on their name, from CLI flags. Deprecation: This is a legacy API. Please refer to the new API usage in flags.py and find the documentation in flags_code_lab.md. Args: cfg: The configuration to set tags on. Raises: ValueError: If no tags with the given name are found. EnvironmentError: If multiple tags with the given name are found. T)add_superclassesr*r,r-csg|] }|jkr|qSrname.0tagr@rr szset_tags..cs2g|]}dd|jddvr|qS)r5)lowerreplacerArBr@rr rEs "z Did you mean ?rFzNo tags with name z in config.z#There were multiple tags with name z; perhaps some module reloading weirdness is going on? This should be very rare. Please make sure that you are not dynamically creating subclasses of `fdl.Tag` and using 2 instances with the same module and name in the configuration.r)rD)r$r6)r$N)r list_tags _FDL_TAGS_SETr$r0r1lenEnvironmentErrorr selectrHr parse_valuer9)r"all_tagsr&r$ matching_tags loose_matches did_you_meanrr@r set_tagss$    rT source_modulecCsRtjD]#}tj|}|j}t|tjj||d}||g|j Ri|j qdS)a[DEPRECATED] Applies fiddlers to `cfg`. Deprecation: This is a legacy API. Please refer to the new API usage in flags.py and find the documentation in flags_code_lab.md. Args: cfg: The configuration to apply fiddlers to. source_module: source py module where fiddlers are defined. allow_imports: If true, then fully qualified dotted names may be used to specify configs or fiddlers that should be automatically imported. zCould not load fiddlerN) _FIDDLERr$rCallExpressionparse func_nameresolve_function_referenceImportDottedNameDebugContextFIDDLERrr)r"rU allow_imports fiddler_value call_expr fiddler_namerrrr apply_fiddlers_tos  racCs"tjtjt}d|dd}tdtt||durt |dr.d|jnd}ttd|d tt |D]A}t ||}| d sSt |rTqBt |d rx|jrxd|jd dd}td d|}d|}nd}td||qB|rttddSdS)z:Prints flag help, including available symbols in `module`. Nz#Fiddle-specific command line flags:__name__z in rFz,Available symbols (base configs or fiddlers):r5__doc__z z\s+  -  zNBase configs and fiddlers may be specified using fully-qualified dotted names.)rFLAGS module_helprmodulesrdjoin splitlinesr!hasattrdirgetattrr/inspectismodulerfstripr0resub)moduler]flags_help_textsourcerAobjobj_doc docstringrrr _print_help0s4   r}cCsttjptj}|p tjS)z5Returns True if if required Fiddle flags are defined.)bool _FDL_CONFIGr$_FDL_CONFIG_FILE _FDL_HELP)config_definedrrr fdl_flags_suppliedRs rrw pyref_policyc Cstj otj }tjs|rtdtjrtjrtdtjr.t|||r+tt |durL|sLd}tjrAt |j ddt jrLt |j ddtjr~t jtj}|j}t |t jj||d}t|rt|j|ji|j}n0||ji|j}n&tjrtj} tj| |d }Wdn1swYntd t|||d t|t |tjrt d t t!"|} | r| D]} t d | j#d| j$qnt%dt t dt t t&'t()|d t|S)a[DEPRECATED] Returns a fdl.Buildable based on standardized flags. Deprecation: This is a legacy API. Please refer to the new API usage in flags.py and find the documentation in flags_code_lab.md. NOTE: the flags aren't applied in the order they are passed in on the command line. The order followed is: - all fiddlers are applied first, followed by - all tags, followed by - all overrides. Args: module: A common namespace to use as the basis for finding configs and fiddlers. May be `None`; if `None`, only fully qualified Fiddler imports will be used (or alternatively a base configuration can be specified using the `--fdl_config_file` flag.) allow_imports: If true, then fully qualified dotted names may be used to specify configs or fiddlers that should be automatically imported. pyref_policy: An optional `serialization.PyrefPolicy` to use if parsing a serialized Fiddle config (passed via `--fdl_config_file`). Returns: A `fdl.Buildable` based on standardized flags. z>At least one of --fdl_config or --fdl_config_file is required.z:--fdl_config and --fdl_config_file are mutually exclusive.NzsA module must be passed to `create_buildable_from_flags` to use the `{flag}` flag unless `allow_imports` is `True`.z --fdl_config)r&z --fiddlerzCould not init a buildable from)rzThis should be unreachable.)rUr]z,Tags (override as --fdl_tag.=):rirhz No tags present in config.z2Config values (override as --fdl.=): )*rr$rrr UsageErrorr}rexitr!r1formatrVrrWrXrYrZr[ BASE_CONFIGris_auto_config as_buildablerropenr load_jsonreadAssertionErrorrarTr'r rJrA descriptionrtextwrapindentr as_str_flattened) rwr]rmissing_base_configerr_msgr_ base_namebase_fn buildableftagsrDrrr create_buildable_from_flagsXsx       r)F)FN)2rfrrrurrtypingrrrrabslrrretilsr fiddler r fiddle._srcr r fiddle._src.absl_flagsrfiddle._src.experimentalrr DEFINE_stringr DEFINE_pathrDEFINE_multi_stringrVr#rK DEFINE_boolrr! Buildabler'r9r<r>rTrar}r~r PyrefPolicyrrrrr sh            %'  "