o 8wi.M@s~dZddlZddlmZddlmZeGdddZGdddZGd d d eZd d Z e Z d dZ e Z GdddZ dS)aqA pipeline for data transformations. Example ------- >>> from hyperpyyaml import load_hyperpyyaml >>> yamlstring = ''' ... pipeline: !new:speechbrain.utils.data_pipeline.DataPipeline ... static_data_keys: [a, b] ... dynamic_items: ... - func: !name:operator.add ... takes: ["a", "b"] ... provides: foo ... - func: !name:operator.sub ... takes: ["foo", "b"] ... provides: bar ... output_keys: ["foo", "bar"] ... ''' >>> hparams = load_hyperpyyaml(yamlstring) >>> hparams["pipeline"]({"a":1, "b":2}) {'foo': 3, 'bar': 1} Author: * Aku Rouhe N) dataclass)DependencyGraphc@seZdZUdZeed<dS) StaticItemzData class that represents a static item. Static items are in-memory items so they don't need to be computed dynamically. keyN)__name__ __module__ __qualname____doc__str__annotations__r r \/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/speechbrain/utils/data_pipeline.pyr s  rc@sHeZdZdZgdgfddZddZddZd d Zd d Zd dZ dS) DynamicItema#Essentially represents a data transformation function. A DynamicItem takes some arguments and computes its value dynamically when called. A straight-forward use-case is to load something from disk dynamically; take the path and provide the loaded data. Instances of this class are often created implicitly via the @takes and @provides decorators or otherwise from specifying the taken and provided arguments and the function. A counterpart is the GeneratorDynamicItem, which should be used for generator functions. Arguments --------- takes : list The keys of the items that this needs to compute its output. func : callable The function that is used to compute the output. provides : list The keys that this provides. NcCs||_||_||_dSN)takesfuncprovides)selfrrrr r r __init__Cs zDynamicItem.__init__cGs |j|Sr)r)rargsr r r __call__H zDynamicItem.__call__cC|jSz1The next argkeys to provide to this, when called.)rrr r r next_takesLzDynamicItem.next_takescCrz.The next keys that this provides, when called.rrr r r next_providesQrzDynamicItem.next_providescCs|jgSzAssuming that this may need to be called multiple times; which keys does it provide at that call. Returns a list, with len equal to the number of times that this may be called. rrr r r provided_in_orderVszDynamicItem.provided_in_ordercCsdS)[Signals that this will not be called any more times on this pipeline call. Nr rr r r reset^szDynamicItem.reset) rrrr rrrrr!r#r r r r r+s rcsHeZdZdZfddZddZddZdd Zd d Zd d Z Z S)GeneratorDynamicItemaEssentially represents a multi-step data transformation. This is the generator function counterpart for DynamicItem (which should be used for regular functions). A GeneratorDynamicItem first takes some arguments and then uses those in multiple steps to incrementally compute some values when called. A typical use-case is a pipeline of transformations on data: e.g. taking in text as a string, and first a tokenized version, and then on the second call providing an integer-encoded version. This can be used even though the integer-encoder needs to be trained on the first outputs. The main benefit is to be able to define the pipeline in a clear function, even if parts of the pipeline depend on others for their initialization. Arguments --------- *args : tuple Forwarded to parent class **kwargs : tuple Forwarded to parent class Example ------- >>> lab2ind = {} >>> def text_pipeline(text): ... text = text.lower().strip() ... text = "".join(c for c in text if c.isalpha() or c == " ") ... words = text.split() ... yield words ... encoded = [lab2ind[word] for word in words] ... yield encoded >>> item = GeneratorDynamicItem( ... func=text_pipeline, ... takes=["text"], ... provides=["words", "words_encoded"]) >>> # First create the integer-encoding: >>> ind = 1 >>> for token in item("Is this it? - This is it."): ... if token not in lab2ind: ... lab2ind[token] = ind ... ind += 1 >>> # Now the integers can be encoded! >>> item() [1, 2, 3, 2, 1, 3] cs"tj|i|d|_d|_dS)Nr)superrcurrent_generatornum_provided_items)rrkwargs __class__r r rs zGeneratorDynamicItem.__init__cGsF|jt|jkr td|js|j||_t|j}|jd7_|S)Nz*DynamicItemPipeline called too many times!)r'lenr RuntimeErrorr&rnext)rroutr r r rs  zGeneratorDynamicItem.__call__cCs|js|jSgSr)r&rrr r r rszGeneratorDynamicItem.next_takescCs |j|j}t|tr|gS|Sr)rr' isinstancer rkeysr r r rs  z"GeneratorDynamicItem.next_providescCs6g}|jD]}t|tr||gq||q|Sr )rr0r append)rin_orderr2r r r r!s    z&GeneratorDynamicItem.provided_in_ordercCs$|jdur |jd|_d|_dS)r"Nr)r&closer'rr r r r#s   zGeneratorDynamicItem.reset) rrrr rrrrr!r# __classcell__r r r)r r$fs 0  r$cfdd}|S)a|Decorator which makes a DynamicItem and specifies its argkeys. If the wrapped object is a generator function (has a yield statement), Creates a GeneratorDynamicItem. If the object is already a DynamicItem, just specifies the argkeys for that. Otherwise creates a new regular DynamicItem, with argkeys specified. The args are always passed to the function at the start. Generators could support sending new arguments, but for such use cases, simply create a new dynamic item. The GeneratorDynamicItem class is meant for pipelines which take in an input and transform it in multiple ways, where the intermediate representations may be needed for e.g. fitting a BPE segmenter. Arguments --------- *argkeys : tuple The data keys expected as input Returns ------- The decorated function, with input argkeys specified Example ------- >>> @takes("text") ... def tokenize(text): ... return text.strip().lower().split() >>> tokenize.provides = ["tokenized"] >>> tokenize(' This Example gets tokenized') ['this', 'example', 'gets', 'tokenized'] csDt|tr|jr td|_|St|rt|dSt|dS)Decorator definition.z!Can't overwrite DynamicItem.takes)rr)r0rr ValueErrorinspectisgeneratorfunctionr$objargkeysr r decorator    ztakes..decoratorr )r?r@r r>r rs ! rcr7)aDecorator which makes a DynamicItem and specifies what keys it provides. If the wrapped object is a generator function (has a yield statement), Creates a GeneratorDynamicItem. If the object is already a DynamicItem, just specifies the provided keys for that. Otherwise creates a new regular DynamicItem, with provided keys specified. Arguments --------- *output_keys : tuple The data keys to be produced by this function Returns ------- The decorated function, with output keys specified NOTE ---- The behavior is slightly different for generators and regular functions, if many output keys are specified, e.g. @provides("signal", "mfcc"). Regular functions should return a tuple with len equal to len(output_keys), while generators should yield the items one by one. >>> @provides("signal", "feat") ... def read_feat(): ... wav = [.1,.2,-.1] ... feat = [s**2 for s in wav] ... return wav, feat >>> @provides("signal", "feat") ... def read_feat(): ... wav = [.1,.2,-.1] ... yield wav ... feat = [s**2 for s in wav] ... yield feat If multiple keys are yielded at once, write e.g., >>> @provides("wav_read", ["left_channel", "right_channel"]) ... def read_multi_channel(): ... wav = [[.1,.2,-.1],[.2,.1,-.1]] ... yield wav ... yield wav[0], wav[1] csDt|tr|jr td|_|St|rt|dSt|dS)r8z*Can't overwrite DynamicItem provides-list.)rr)r0rrr9r:r;r$r< output_keysr r r@3rAzprovides..decoratorr )rCr@r rBr rs . rc@seZdZdZggfddZddZddZdd d Zd d Zd dZ e ddZ ddZ ddZ ddZddZddZddZdS) DataPipelineaOrganises data transformations into a pipeline. Arguments --------- static_data_keys: list The keys which are provided as data dynamic_items: list A list of mappings with "func", "takes", and "provides" output_keys: list The keys to use as outputs Example ------- >>> pipeline = DataPipeline( ... static_data_keys=["text"], ... dynamic_items=[ ... {"func": lambda x: x.lower(), "takes": "text", "provides": "foo"}, ... {"func": lambda x: x[::-1], "takes": "foo", "provides": "bar"}, ... ], ... output_keys=["bar"], ... ) >>> pipeline({"text": "Test"}) {'bar': 'tset'} cCsHt|_d|_i|_i|_g|_i|_||||| |dSr) rdg _exec_order key_to_nodeunaccounted_keys dynamic_itemsoutput_mappingadd_static_keysadd_dynamic_itemsset_output_keys)rstatic_data_keysrIrCr r r r_s  zDataPipeline.__init__cCs,|D]}|jjt|dd}||j|<qdS)zrInforms the pipeline about static items. Static items are the ones provided to __call__ as data. )rdataN)rEadd_noderrG)r static_keysrnode_idr r r rKjs zDataPipeline.add_static_keysc Cs>|D]}z |jdi|Wqty||YqwdS)z#Add multiple dynamic items at once.Nr )add_dynamic_item TypeError)rrIitemr r r rLss zDataPipeline.add_dynamic_itemsNcCsrt|tr|dus |durtd||dSt|tr |g}t|tr(|g}t|t||}||dS)a Adds a dynamic item to the Pipeline. Two calling conventions. For DynamicItem objects, just use: add_dynamic_item(dynamic_item) But otherwise, should use: add_dynamic_item(func, takes, provides) Arguments --------- func : callable, DynamicItem If a DynamicItem is given, adds that directly. Otherwise a DynamicItem is created, and this specifies the callable to use. If a generator function is given, then create a GeneratorDynamicItem. Otherwise creates a normal DynamicItem. takes : list, str List of keys. When func is called, each key is resolved to either an entry in the data or the output of another dynamic_item. The func is then called with these as positional arguments, in the same order as specified here. A single key can be given as a bare string. provides : str, list For regular functions, the key or list of keys that it provides. If you give a generator function, key or list of keys that it yields, in order. Also see the provides decorator. A single key can be given as a bare string. Returns ------- None NzDIf providing a DynamicItem directly, don't specify takes or provides)r0rr9_add_dynamic_item_objectr takes_decoratorprovides_decorator)rrrrdir r r rT{s    zDataPipeline.add_dynamic_itemc Cs|jstdg}|jD]}||jvr"|j|g}||q ||j|q | D]?}|j j |d}|D]$}||j|<||jvr^|j|D]}|j|}|j ||qK|j|=q:|D] } |j || qa|g}q/|j |dS)a Internally adds the object. There is a node in the dependency graph for each call of the DynamicItem. Each call may return multiple keys and depend on multiple keys. An internal dict maps key to the id of the node that produces it. z@Won't add redundant dynamic item which doesn't provide anything.rON)rr9rrGrH setdefaultextendrr3r!rErQadd_edgerI) rr=dependedr dependee_keysprovidedrS dependee_key dependee_nodedep_idr r r rWs0      z%DataPipeline._add_dynamic_item_objectcCs|||_d|_dS)aUse this to change the output keys. Also re-evaluates execution order. So if you request different outputs, some parts of the data pipeline may be skipped. Arguments --------- keys : dict, list, None List of keys (str) to produce in output. If a dict is given; it is used to map internal keys to output keys. From the output_keys dict key:value pairs the key appears outside, and value is the internal key. N)_output_keys_to_mappingrJrFr1r r r rMs  zDataPipeline.set_output_keyscCs4|duri}|St|tr|}|Sdd|D}|S)NcSsi|]}||qSr r .0rr r r sz8DataPipeline._output_keys_to_mapping..)r0dict)r2rJr r r rds z$DataPipeline._output_keys_to_mappingcCs&|jdur |||||j|jS)z Arguments --------- data : dict Dictionary with data entries by key. Returns ------- dict With the keys that were set. N)rF _prepare_run_computerJrrPr r r compute_outputss zDataPipeline.compute_outputscCs,||}|jj||d}||||S)z>Compute output of specific item, without changing output_keys.) selected_keys)rdrEget_evaluation_orderget_selected_node_idsrj)rr2rPrJorderr r r compute_specifics zDataPipeline.compute_specificc s|jrd}|d|j7}t|i|D]G\}}}t|tr7z|jWqty6td|jdwfdd|D}|} ||} t | dkrT| g}  t | | q|j D]} | q`fdd |DS) Nz;These keys are still unaccounted for in the data pipeline: z, z Expected key z in data!cs$g|]}|vr |n|qSr r )rfargkeyrP intermediater r sz)DataPipeline._compute..r+cs*i|]\}}||vr|n|qSr r )rfoutkeyinkeyrsr r rg#sz)DataPipeline._compute..)rHjoinr-r0rrKeyErrorrrr,updateziprIr#items) rrPrprJMSGrSedgesrVr provided_keysvalues dynamic_itemr rsr rjs4        zDataPipeline._computecsfdd|DS)z2Translates selected keys to dependency graph keys.csg|]}j|qSr )rGrerr r ru*sz6DataPipeline.get_selected_node_ids..r )rrmr rr ro(sz"DataPipeline.get_selected_node_idscCs ||Sr)rlrkr r r r,rzDataPipeline.__call__cCs"t|j||j|_dSr)listrErnrorJrrFrkr r r ri/s  zDataPipeline._prepare_run)NN)rrrr rrKrLrTrWrM staticmethodrdrlrqrjrorrir r r r rDEs   /%  # rD)r r: dataclassesrspeechbrain.utils.depgraphrrrr$rrXrrYrDr r r r s   ;l0=