o wia@s8dZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddlZddlmZddlmZdde jjfddZGdd d ZGd d d Zd d Zd-ddZddZddZddZddZddZddZddZ dd Z!d!d"Z"d.d#d$Z#d.d%d&Z$d'd(Z%d)d*Z&d.d+d,Z'dS)/z~This library gathers utilities for hyperpyyaml loading Authors * Peter Plantinga 2020 * Aku Rouhe 2020 * Jianchen Li 2022 N)StringIO)RepresenterErrorTcCst|||}|jdtdtd}|jd|dd|dt|dt|dt |d t d t j j j_d tj j j j_t j||d }d t j j j_d tj j j j_d d|D}|D]}||=q^|S)aThis function implements the HyperPyYAML syntax The purpose for this syntax is a compact, structured hyperparameter and function definition. This function implements a few extensions to the yaml syntax, listed below. **PyYAML complex tag shortcuts** Part of our clean structured hyperparameter interface is being able to specify python objects easily and cleanly. This is possible with native YAML using the following syntax: .. code-block:: yaml alignment_saver: !!python/object/new:speechbrain.data_io.TensorSaver kwargs: {save_dir: results/asr/ali} However, due to the extensive use within speechbrain yaml files, we have added a shortcut for this that has the following syntax: .. code-block:: yaml alignment_saver: !new:speechbrain.data_io.TensorSaver save_dir: results/asr/ali In this example, the alignment_saver will be an instance of the ``TensorSaver`` class, with ``'exp/asr/ali'`` passed to the ``__init__()`` method as a keyword argument. This is equivalent to: .. code-block:: python import speechbrain.data_io.data_io alignment_saver = speechbrain.data_io.TensorSaver( save_dir='exp/asr/ali' ) We have also implemented a few more shortcuts::: !!python/name: => !name: !!python/module: => !module: !!python/object/apply: => !apply: **References and copies** Allows internal references to any node in the file. Any node with tag ``!ref`` will create an object reference to the yaml object at the ```` location within the yaml itself, following reference chains. .. code-block:: yaml output_folder: results/asr alignment_saver: !new:speechbrain.data_io.TensorSaver save_dir: !ref Strings values are handled specially: references are substituted but the rest of the string is left in place, allowing filepaths to be easily extended: .. code-block:: yaml output_folder: results/asr alignment_saver: !new:speechbrain.data_io.TensorSaver save_dir: !ref /ali # results/asr/ali A more complex example for demonstration purposes: .. code-block:: yaml key1: {a: !new:object {arg1: 1}} key2: !ref Here, ``key2`` will contain a reference to the ``a`` object, so changing ``a.arg1`` will also change ``key2.arg1``. If you need a deep copy of the object instead of a shallow reference, you can use a similar syntax with the tag ``!copy``. For example: .. code-block:: yaml key1: {a: !new:object {arg1: 1}} key2: !copy These will also implement very basic arithmetic, so: .. code-block:: yaml key1: 1 key2: !ref + 3 # this is 4 **Tuples** One last minor enhancement is an implicit tuple resolver. Passing a string value of ``(3, 4)`` will be given a tag of ``!tuple`` which is then interpreted as a tuple. Arguments --------- yaml_stream : stream A file-like object or string from which to read. overrides : mapping or str A set of overrides for the values read from the stream. As yaml implements a nested structure, so can the overrides. See `speechbrain.utils.data_utils.recursive_update` overrides_must_match : bool Whether an error will be thrown when an override does not match a corresponding key in the yaml_stream. return_dict : bool Whether to return a dictionary rather than the default namespace. loader : Loader Use to parse the yaml_stream, could be `ruamel.yaml.Loader`, `yaml.Loader`, etc. Returns ------- hparams : dict Reflects the structure of ``yaml_stream``. Example ------- >>> yaml_string = """ ... a: 3 ... thing: !new:collections.Counter ... b: !ref ... """ >>> params = load_hyperpyyaml(yaml_string) >>> params["thing"] Counter({'b': 3}) z!tuple)tag constructorz^\(.*\)$()first!new:!name:!module:!apply:)TLoaderFcSsg|] }|dr|qS)__) startswith).0krD/home/ubuntu/.local/lib/python3.10/site-packages/hyperpyyaml/core.py sz$load_hyperpyyaml..)resolve_referencesadd_constructor _make_tuplerecompileadd_implicit_resolveradd_multi_constructor_construct_object_construct_name_construct_module_apply_functionyamlrBaseConstructorconstruct_object __defaults__ruamelloadkeys) yaml_stream overridesoverrides_must_matchloader tuple_patternhparams removal_keyskeyrrrload_hyperpyyamls$        r0c@s(eZdZdZdZddZeddZdS)RefTagzClass for dumping !ref tags to yaml Arguments --------- ref_str : str String including yaml keys in `` notation Example ------- See ``dump_hyperpyyaml`` !refcCs ||_dSN)ref_str)selfr4rrr__init__s zRefTag.__init__cCs||j|jSr3)represent_scalaryaml_tagr4cls representernoderrrto_yamlszRefTag.to_yamlN)__name__ __module__ __qualname____doc__r8r6 classmethodr=rrrrr1s  r1c@s eZdZdZdZeddZdS) PlaceholderzfClass for dumping !PLACEHOLDER tags to yaml Example ------- See ``dump_hyperpyyaml`` !PLACEHOLDERcCs||jdS)N)r7r8r9rrrr=szPlaceholder.to_yamlN)r>r?r@rAr8rBr=rrrrrCs rCcOsHtj}|jttj|jttj|j||g|Ri|dS)aDump yaml including placeholder and reference tags. Arguments --------- yaml_tree : dict An object to dump output_stream : stream A file stream for putting the yaml *args, **kwargs Arguments to forward to ruamel.yaml.YAML().dump() Example ------- >>> to_yaml = {'a': Placeholder(), 'b': RefTag('')} >>> stringio = StringIO() >>> dump_hyperpyyaml(to_yaml, stringio) >>> stringio.getvalue() 'a: !PLACEHOLDER\nb: !ref \n' N) r%r!YAMLr;add_representerr1r=rCdump) yaml_tree output_streamargskwargs ruamel_yamlrrrdump_hyperpyyamls rNFc Csd}t|drtjtj|j}tj}| |}|durFt |dkrFt |t r0| |}z t |||dWn tyEtd|wtd|||t}z||||dW|Styy}z|jddf|_||jd}~ww)aResolves inter-document references, a component of HyperPyYAML. Arguments --------- yaml_stream : stream A file-like object or string with the contents of a yaml file written with the HyperPyYAML syntax. overrides : mapping or str Replacement values, either in a yaml-formatted string or a dict. overrides_must_match : bool Whether an error will be thrown when an override does not match a corresponding key in the yaml_stream. This is the opposite default from ``load_hyperpyyaml`` because ``resolve_references`` doesn't need to be as strict by default. Returns ------- stream A yaml-formatted stream with all references and overrides resolved. Example ------- >>> yaml_string = """ ... constants: ... a: 3 ... b: !ref ... """ >>> overrides = {'constants': {'a': 4}} >>> resolve_references(yaml_string, overrides).getvalue() 'constants:\n a: 4\n b: 4\n' Nnamer) must_matchzLThe structure of the overrides doesn't match the structure of the document: rootz$. Please use the !apply tag instead.)hasattrospathdirnamerealpathrOr%r!rFr&len isinstancestrrecursive_update TypeError ValueError_walk_tree_and_resolverrHseekrrKwith_traceback __traceback__)r(r)r* file_pathrMpreviewerrrrs6!         rcCst|tr&t|D]\}}|dkr|n|d|d}t||||||<q n%t|trK|D]\}}|dkr9|n|d|d}t||||||<q/t|dr|jjpUd}|dkrbt d|d|d vru|d k} t |jg|| d }|S| d r|t d d } |d urt j|| } zt|} Wn tyi} Ynwt|  } t| | } Wd n1swYtj}|| }|S| dr|t dd }t||}|S)aA recursive function for resolving ``!ref``, ``!copy`` and ``!applyref`` tags. Loads additional yaml files if ``!include:`` tags are used. Also throws an error if ``!PLACEHOLDER`` tags are encountered. Arguments --------- key : str The fully-qualified path to current node. current_node : node A node in the yaml tree loaded with ruamel.yaml. tree : node The base node in the yaml tree loaded with ruamel.yaml. file_path : str The location of the directory storing the main yaml file Returns ------- yaml.Node A yaml tree with all references resolved. rQ[]rrErD'z)' is a !PLACEHOLDER and must be replaced.)r2!copyrg) referencereference_list full_tree copy_modez !include:N !applyref:)rXlist enumerater]dictitemsrRrvaluer\recursive_resolverrWrSrTjoinr[openrr%r!rFr&_applyref_function)r/ current_nodetreeraisub_nodesub_keyr tag_valuerkfilenamer)f included_yamlrMfunctionrrrr]TsR    !         r]cCs6||}d|ddd}tj|tjd}t|S)z-Parse scalar node as a list, convert to tuplerdrer )construct_scalarr!r&r tuple)r+r< tuple_string list_string parsed_listrrrrs rcCsht|tjs t|tjjr|j|dd}g|fSt|tjs%t|tjjr0|j|dd}|ifSgifS)NT)deep)rXr! MappingNoder%construct_mapping SequenceNodeconstruct_sequence)r+r<rLrKrrr _load_nodes    rcCsdt|sgifSt|tr'd|vr#d|vr#t|dkr#|d|dfSg|fSt|tr0|ifSt)N_args_kwargs)rYrXrorWrmr\)r<rrr _get_argss  rc Ct|}|durtd|t|std|d|zt||\}}||i|WStyF}z d|}|g|jR|_d}~ww)NzThere is no such class as %srz should be a class, but is zInvalid argument to class %s) pydoclocate ImportErrorinspectisclassr\rr[rKr+callable_stringr< callable_rKrLrcerr_msgrrrr   rc Cst|}|durtd|t|s0t|s0t||\}}|s$|r.td|d||Szt||\}}|s<|rItj |g|Ri|WS|WSt yd}z d|}|g|j R|_ d}~ww)NzThere is no such entity as %sr zK should be class or function, if you specify args or kwargs. Instead it is Invalid argument to callable %s) rrrrr isroutinerr\ functoolspartialr[rK)r+rr<rOrKrLrcrrrrrs.  rcCsft|}|durtd|t||\}}|gks|ikr"tdt|s1td|d||S)NzThere is no such module as %szCannot pass args to moduler z should be module, but is )rrrrr\rismodule)r+ module_namer<modulerKrLrrrrs   rc Cr)NThere is no such callable as %sr  should be a callable, but is r) rrrrrr\rr[rKrrrrr rr c Cst|}|durtd|t|std|d|zt|\}}||i|}|WStyG}z d|}|g|jR|_d}~ww)Nrrlrr) rrrrrr\rr[rK)rr<rrKrLoutrcrrrrru#s"    rucCsd}d|vr|jddd\}}|}|dD]}|d}||vr'td|||}q|r3t|S|durOtjj}|||g7}| tjj dd |S|S) a Find the value referred to by a reference in dot-notation Arguments --------- ref : str The location of the requested value, e.g. 'constants.param' full_tree : dict The dictionary to use for finding values copy_mode : bool Whether to copy the node before dereferencing. Returns ------- node The node in the full_tree dictionary referenced by ``ref``. Example ------- >>> deref('constants[a][b]', {'constants': {'a': {'b': 'c'}}}) 'c' N.r)maxsplitrdrezThe reference "%s" is not validz!apply:getattr)suffix) splitstripr\copydeepcopyr%r!comments CommentedSeq yaml_set_ctagTag)refrjrkattrbranchpartr<rrrderef8s&       rc Cstd}t|tr||s|St|dkr$||ddvr$td|||r>t| d||}||g7}t ||||S| |}|dd|D7}||fdd }| ||}t ||||}t |S) aResolve a reference to a value, following chained references Arguments --------- reference : str a string containing '' in it where x[y] refers to a scalar node in the file. reference_list : list list of prior references in the chain, in order to catch circular references. full_tree : dict the dictionary in which to find all references and their values. copy_mode : bool Whether to perform a deep copy of the referenced node, rather than a shallow reference to the same object. Returns ------- scalar The dereferenced value, with possible string interpolation and arithmetic parsing. Example ------- >>> tree = {'a': 3, 'b': 'x', 'c': '', 'd': '/', 'e': '/'} >>> recursive_resolve('', [], tree) 1.0 >>> recursive_resolve('', [], tree) 'x/x' z<[^>]*>rNzCircular reference detected: <>cSsg|]}|dqS)rr)rmatchrrrrsz%recursive_resolve..cSstt|dd||dS)Nrr)rjrk)rYrr)xrwrkrrr replace_fnsz%recursive_resolve..replace_fn)rrrXrYsearchrWr\ fullmatchrrrrfindallsubparse_arithmetic) rhrirjrkreference_finderrqmatchesrrrrrrrls !     rrc Cs4z ttj|ddjWStttfy|YSw)aParses simple arithmetic operations in references Adapted from https://stackoverflow.com/a/9558001/1761970 Arguments --------- reference_string : str A string with references and possible arithmetic operations. Returns ------- str Result of parsing and applying the arithmetic. Example ------- >>> parse_arithmetic('2 * 6') 12 eval)mode) _ast_evalastparsebodyr[ SyntaxErrorKeyError)reference_stringrrrrs rcCstjtjtjtjtjtjtjtj tj tj tj tj tjtji}tjdkr-t|tjr,|jSn t|tjr6|jSt|tjrL|t|jt|jt|jSt|tjr^|t|jt|jSt|)N))rAddopaddSubrMultmulDivtruedivFloorDivfloordivPowpowModmodsys version_inforXConstantrqNumnBinOptyperleftrightUnaryOpoperandr[)r<opsrrrrs&     rcCst|tjjstd|t|tjjstd||D]4\}}t|tjjr9||vr9t||i|q |rP||vrPtd|ddd| D|||<q dS)aSimilar function to `dict.update`, but for a nested `dict`. From: https://stackoverflow.com/a/3233356 If you have to a nested mapping structure, for example: {"a": 1, "b": {"c": 2}} Say you want to update the above structure with: {"b": {"d": 3}} This function will produce: {"a": 1, "b": {"c": 2, "d": 3}} Instead of: {"a": 1, "b": {"d": 3}} Arguments --------- d : dict mapping to be updated u : dict mapping to update with must_match : bool Whether to throw an error if the key in `u` does not exist in `d`. Example ------- >>> d = {'a': 1, 'b': {'c': 2}} >>> recursive_update(d, {'b': {'d': 3}}) >>> d {'a': 1, 'b': {'c': 2, 'd': 3}} z'Expected to update a mapping, but got: z.Expected a mapping to use for update, but got z Override 'z' not found in: cSsg|]}|qSrr)rr/rrrrsz$recursive_update..N) rX collectionsabcMappingr[rprZgetrr')durPrvrrrrZs' " rZ)NFr)(rArrrr!rros.pathrSrrr ruamel.yamlr%operatorriorruamel.yaml.representerrr r0r1rCrNrr]rrrrrrr rurrrrrrZrrrrsH    4 FR  4?