o ½e¦i`ã@sndZddlmZmZmZmZddlZddlZdeeeeffdd„Z deej j fdd „Z Gd d „d ƒZ dS) zaModels and tooling for natural language processing using spaCy Authors * Sylvain de Langen 2024 é)ÚIterableÚIteratorÚListÚUnionNÚsentencecCst|tƒr|Sd |¡S)axEnsures that a sentence is a `str` rather than a list of `str` tokens to be passed to spaCy pipelines correctly. Arguments --------- sentence: str or list of str Sentence to return or list of tokens. Returns ------- str The sentence, returned from the `sentence` argument as-is or joined with spaces from a list of tokens.ú )Ú isinstanceÚstrÚjoin)r©r ú`/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/speechbrain/lobes/models/spacy/nlp.pyÚ _as_sentence s  r ÚdocscCsdd„|DƒS)a>Returns a batch of list of lemmas from a list of Doc (as returned by the pipeline). Arguments --------- docs: iterable of Doc Documents, typically as returned by `nlp.pipe`. Returns ------- list of list of str For each sentence, the sequence of extracted lemmas as `str`s.cSsg|] }dd„|Dƒ‘qS)cSsg|]}|j‘qSr )Úlemma_)Ú.0Útokr r r Ú /sz._extract_lemmas...r )rÚdocr r r r/sz#_extract_lemmas..r )rr r r Ú_extract_lemmas"s rc@sŠeZdZdZdejjfdd„Zedd„ƒZ de e e e e e fde ejjfd d „Zde e e e e e fde e e fd d „Zd S)Ú SpacyPipelineaVWraps a `spaCy pipeline `_ with methods that makes it easier to deal with SB's typical sentence format, and adds some convenience functions if you only care about a specific task. Arguments --------- nlp : spacy.language.Language spaCy text processing pipeline to use.ÚnlpcCs ||_dS)N)r)Úselfrr r r Ú__init__<s zSpacyPipeline.__init__cOsttj|g|¢Ri|¤ŽƒS)aRCreate a pipeline by loading a model using `spacy.load`. Unlike other toolkits, you must explicitly download the model if you want to use a remote model (e.g. `spacy download fr_core_news_md`) rather than just specifying a HF hub name. .. note:: If you only need a subset of modules enabled in the pipeline, e.g. for lemmatization, consider `excluding _` using the `exclude=[...]` argument. Arguments --------- name: str | Path Package name or model path. *args Extra positional arguments passed to `spacy.load`. **kwargs Extra keyword arguments passed to `spacy.load`. Returns ------- New SpacyPipeline )rÚspacyÚload)ÚnameÚargsÚkwargsr r r Ú from_name?szSpacyPipeline.from_nameÚinputsÚreturncCs|j tt|ƒ¡S)aQProcesses a batch of sentences into an iterator of spaCy documents. Arguments --------- inputs: list of sentences (str or list of tokens) Sentences to process, in the form of batches of lists of tokens (list of str) or a str. In the case of token lists, tokens do *not* need to be already tokenized for this specific sequence tagger, and they will be joined with spaces instead. Returns ------- iterator of spacy.tokens.Doc Iterator of documents for the passed sentences.)rÚpipeÚmapr ©rrr r r Ú__call__\szSpacyPipeline.__call__cCs t||ƒƒS)a„Lemmatize a batch of sentences by processing the input sentences, discarding other irrelevant outputs. Arguments --------- inputs: list of sentences (str or list of tokens) Sentences to lemmatize, in the form of batches of lists of tokens (list of str) or a str. In the case of token lists, tokens do *not* need to be already tokenized for this specific sequence tagger, and they will be joined with spaces instead. Returns ------- list of list of str For each sentence, the sequence of extracted lemmas as `str`s.)rr#r r r Ú lemmatizeqs zSpacyPipeline.lemmatizeN)Ú__name__Ú __module__Ú __qualname__Ú__doc__rÚlanguageÚLanguagerÚ staticmethodrrrr rÚtokensÚDocr$r%r r r r r2s  ÿ  þÿ þr)r)ÚtypingrrrrrÚ spacy.tokensr r r-r.rrr r r r Ús