o !wiE@sdZddlZddlZddlZddlZddlZddlmZmZm Z m Z gdZ Gddde Z Gddde ZGd d d e ZGd d d e ZGd dde ZGddde ZGddde ZGddde ZGdddeZGdddeZedddZGdddeZGddde ZGd d!d!e ZGd"d#d#e ZGd$d%d%e ZGd&d'd'e ZGd(d)d)e ZGd*d+d+e Z dS),a@ This file implements the building blocks for transforming a collection of input strings to the desired format in order to calculate the WER of CER. In principle, for word error rate calculations, every string of a sentence needs to be collapsed into a list of strings, where each string is a *single* word. This is done with [transforms.ReduceToListOfListOfWords][]. A composition of multiple transformations must therefore *always* end with [transforms.ReduceToListOfListOfWords][]. For the character error rate, every string of a sentence also needs to be collapsed into a list of strings, but here each string is a *single* character. This is done with [transforms.ReduceToListOfListOfChars][]. Similarly, a composition of multiple transformations must therefore also always end with [transforms.ReduceToListOfListOfChars][]. N)IterableUnionListMapping)AbstractTransformComposeExpandCommonEnglishContractionsRemoveEmptyStringsReduceToListOfListOfWordsReduceToListOfListOfCharsReduceToSingleSentenceRemoveKaldiNonWordsRemoveMultipleSpacesRemovePunctuationRemoveSpecificWordsRemoveWhiteSpaceStripSubstituteRegexesSubstituteWords ToLowerCase ToUpperCasec@sJeZdZdZdeeeeffddZdefddZdeefd d Z d S) rz( The base class of a Transform. sentencescCs6t|tr ||St|tr||Std|)z Transforms one or more strings. Args: sentences: The strings to transform. Returns: (Union[str, List[str]]): The transformed strings. z7input {} was expected to be a string or list of strings) isinstancestrprocess_stringlist process_list ValueErrorformat)selfrr M/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/jiwer/transforms.py__call__Gs   zAbstractTransform.__call__scCstN)NotImplementedErrorrr#r r r!r]sz AbstractTransform.process_stringinpcfdd|DS)Ncg|]}|qSr r.0r#rr r! az2AbstractTransform.process_list..r rr'r r-r!r`zAbstractTransform.process_listN) __name__ __module__ __qualname____doc__rrrr"rrr r r r!rBs rc@s*eZdZdZdeefddZddZdS)ra Chain multiple transformations back-to-back to create a pipeline combining multiple transformations. Note that each transformation needs to end with either `ReduceToListOfListOfWords` or `ReduceToListOfListOfChars`, depending on whether word error rate, or character error rate is desired. Example: ```python3 import jiwer jiwer.Compose([ jiwer.RemoveMultipleSpaces(), jiwer.ReduceToListOfListOfWords() ]) ``` transformscC ||_dS)zV Args: transforms: The list of transformations to chain. Nr6)rr6r r r!__init__x zCompose.__init__cCs|jD]}||}q|Sr$r8)rtexttrr r r!r"s  zCompose.__call__N)r2r3r4r5rrr9r"r r r r!rds rc@s@eZdZd deefddZdefddZdeefd d Zd S) BaseRemoveTransformtokens_to_removecCs||_||_dSr$)r? replace_token)rr?r@r r r!r9s zBaseRemoveTransform.__init__r#cCs|jD] }|||j}q|Sr$)r?replacer@)rr#wr r r!rs z"BaseRemoveTransform.process_stringr'cr()Ncr)r r*r+r-r r!r.r/z4BaseRemoveTransform.process_list..r r0r r-r!rr1z BaseRemoveTransform.process_listNr>) r2r3r4rrr9rrrr r r r!r=sr=c@@eZdZdZd defddZdefddZd eefd d Zd S)r a Transforms a single input sentence, or a list of input sentences, into a list with lists of words, which is the expected format for calculating the edit operations between two input sentences on a word-level. A sentence is assumed to be a string, where words are delimited by a token (such as ` `, space). Each string is expected to contain only a single sentence. Empty strings (no output) are removed for the list. Example: ```python import jiwer sentences = ["hi", "this is an example"] print(jiwer.ReduceToListOfListOfWords()(sentences)) # prints: [['hi'], ['this', 'is', 'an, 'example']] ```  word_delimitercCr7)zo Args: word_delimiter: the character which delimits words. Default is ` ` (space). NrFrrFr r r!r9 z"ReduceToListOfListOfWords.__init__r#cCsdd||jDgS)NcSg|] }t|dkr|qSlenr,rBr r r!r.z.)splitrFr&r r r!rsz(ReduceToListOfListOfWords.process_stringr'cC<g}|D]}||d}||qt|dkrggS|SNrrappendrNrr'sentence_collectionsentence list_of_wordsr r r!r  z&ReduceToListOfListOfWords.process_listNrE r2r3r4r5rr9rrrr r r r!r s r c@0eZdZdZdefddZdeefddZdS) r a Transforms a single input sentence, or a list of input sentences, into a list with lists of characters, which is the expected format for calculating the edit operations between two input sentences on a character-level. A sentence is assumed to be a string. Each string is expected to contain only a single sentence. Example: ```python import jiwer sentences = ["hi", "this is an example"] print(jiwer.ReduceToListOfListOfChars()(sentences)) # prints: [['h', 'i'], ['t', 'h', 'i', 's', ' ', 'i', 's', ' ', 'a', 'n', ' ', 'e', 'x', 'a', 'm', 'p', 'l', 'e']] ``` r#cCsdd|DgS)NcSg|]}|qSr r rOr r r!r.z.r r&r r r!rsz(ReduceToListOfListOfChars.process_stringr'cCrRrSrTrVr r r!rrZz&ReduceToListOfListOfChars.process_listNr2r3r4r5rrrrr r r r!r sr c@rD)r a& Transforms multiple sentences into a single sentence. This operation can be useful when the number of reference and hypothesis sentences differ, and you want to do a minimal alignment over these lists. Note that this creates an invariance: `wer([a, b], [a, b])` might not be equal to `wer([b, a], [b, a])`. Example: ```python3 import jiwer sentences = ["hi", "this is an example"] print(jiwer.ReduceToSingleSentence()(sentences)) # prints: ['hi this is an example'] ``` rErFcCr7)zd :param word_delimiter: the character which delimits words. Default is ` ` (space). NrGrHr r r!r9s zReduceToSingleSentence.__init__r#cCs|Sr$r r&r r r!rsz%ReduceToSingleSentence.process_stringr'cCs2dd|D}t|dkrgSd|j|gS)NcSrJrKrMr,ir r r!r.rPz7ReduceToSingleSentence.process_list..rz{})rNrrFjoin)rr' filtered_inpr r r!rs z#ReduceToSingleSentence.process_listNr[r\r r r r!r s r c@4eZdZdZdeeeffddZdefddZdS) ram Transform strings by substituting substrings matching regex expressions into another substring. Example: ```python import jiwer sentences = ["is the world doomed or loved?", "edibles are allegedly cultivated"] # note: the regex string "\b(\w+)ed\b", matches every word ending in 'ed', # and "\1" stands for the first group ('\w+). It therefore removes 'ed' in every match. print(jiwer.SubstituteRegexes({r"doom": r"sacr", r"\b(\w+)ed\b": r"\1"})(sentences)) # prints: ["is the world sacr or lov?", "edibles are allegedly cultivat"] ``` substitutionscCr7)zj Args: substitutions: a mapping of regex expressions to replacement strings. Nrfrrfr r r!r9 r:zSubstituteRegexes.__init__r#cCs&|jD] \}}t|||}q|Sr$)rfitemsresubrr#keyvaluer r r!r(sz SubstituteRegexes.process_stringNr2r3r4r5rrr9rr r r r!r src@re) ra| This transform can be used to replace a word into another word. Note that the whole word is matched. If the word you're attempting to substitute is a substring of another word it will not be affected. For example, if you're substituting `foo` into `bar`, the word `foobar` will NOT be substituted into `barbar`. Example: ```python import jiwer sentences = ["you're pretty", "your book", "foobar"] print(jiwer.SubstituteWords({"pretty": "awesome", "you": "i", "'re": " am", 'foo': 'bar'})(sentences)) # prints: ["i am awesome", "your book", "foobar"] ``` rfcCr7)z[ Args: substitutions: A mapping of words to replacement words. Nrgrhr r r!r9DrIzSubstituteWords.__init__r#cCs2|jD]\}}tdt|||}q|S)Nz\b{}\b)rfrirjrkrescaperlr r r!rKszSubstituteWords.process_stringNror r r r!r/srcs*eZdZdZdeeffdd ZZS)ra% Can be used to filter out certain words. As words are replaced with a ` ` character, make sure to that `RemoveMultipleSpaces`, `Strip()` and `RemoveEmptyStrings` are present in the composition _after_ `RemoveSpecificWords`. Example: ```python import jiwer sentences = ["yhe awesome", "the apple is not a pear", "yhe"] print(jiwer.RemoveSpecificWords(["yhe", "the", "a"])(sentences)) # prints: [' awesome', ' apple is not pear', ' '] # note the extra spaces ``` words_to_removecsdd|D}t|dS)zM Args: words_to_remove: List of words to remove. cSsi|]}|dqSr[r )r,wordr r r! jsz0RemoveSpecificWords.__init__..N)superr9)rrqmapping __class__r r!r9eszRemoveSpecificWords.__init__)r2r3r4r5rrr9 __classcell__r r rvr!rRsrcs(eZdZdZddeffdd ZZS)ra| This transform filters out white space characters. Note that by default space (` `) is also removed, which will make it impossible to split a sentence into a list of words by using `ReduceToListOfListOfWords` or `ReduceToSingleSentence`. This can be prevented by replacing all whitespace with the space character. If so, make sure that `jiwer.RemoveMultipleSpaces`, `Strip()` and `RemoveEmptyStrings` are present in the composition _after_ `RemoveWhiteSpace`. Example: ```python import jiwer sentences = ["this is an example", "hello world "] print(jiwer.RemoveWhiteSpace()(sentences)) # prints: ["thisisanexample", "helloworld"] print(jiwer.RemoveWhiteSpace(replace_by_space=True)(sentences)) # prints: ["this is an example", "hello world "] # note the trailing spaces ``` Freplace_by_spacecs2ddtjD}|r d}nd}tj||ddS)zq Args: replace_by_space: every white space character is replaced with a space (` `) cSr^r r )r,cr r r!r.r_z-RemoveWhiteSpace.__init__..rEr>)r@N)string whitespacertr9)rry charactersr@rvr r!r9s zRemoveWhiteSpace.__init__)F)r2r3r4r5boolr9rxr r rvr!rosrrLcCs$ttjd}tdd|D}|S)z9Compute the punctuation characters only once and memoize.rLcss,|]}tt|drt|VqdS)PN) unicodedatacategorychr startswithrar r r! s  z._get_punctuation_characters..)rangesys maxunicodeset) codepoints punctuationr r r!_get_punctuation_characterss rcs eZdZdZfddZZS)ra This transform filters out punctuation. The punctuation characters are defined as all unicode characters whose category name starts with `P`. See [here](https://www.unicode.org/reports/tr44/#General_Category_Values) for more information. Example: ```python import jiwer sentences = ["this is an example!", "hello. goodbye"] print(jiwer.RemovePunctuation()(sentences)) # prints: ['this is an example', "hello goodbye"] ``` cst}t|dSr$)rrtr9)rpunctuation_charactersrvr r!r9szRemovePunctuation.__init__)r2r3r4r5r9rxr r rvr!rsrc@r]) rao Filter out multiple spaces between words. Example: ```python import jiwer sentences = ["this is an example ", " hello goodbye ", " "] print(jiwer.RemoveMultipleSpaces()(sentences)) # prints: ['this is an example ', " hello goodbye ", " "] # note that there are still trailing spaces ``` r#cCtdd|S)Nz\s\s+rErjrkr&r r r!rz#RemoveMultipleSpaces.process_stringr'cr()Ncr)r r*r+r-r r!r.r/z5RemoveMultipleSpaces.process_list..r r0r r-r!rr1z!RemoveMultipleSpaces.process_listNr`r r r r!rsrc@eZdZdZdefddZdS)ra~ Removes all leading and trailing spaces. Example: ```python import jiwer sentences = [" this is an example ", " hello goodbye ", " "] print(jiwer.Strip()(sentences)) # prints: ['this is an example', "hello goodbye", ""] # note that there is an empty string left behind which might need to be cleaned up ``` r#cC|Sr$stripr&r r r!rzStrip.process_stringNr2r3r4r5rrr r r r!rrc@r]) r a  Remove empty strings from a list of strings. Example: ```python import jiwer sentences = ["", "this is an example", " ", " "] print(jiwer.RemoveEmptyStrings()(sentences)) # prints: ['this is an example'] ``` r#cCrr$rr&r r r!rrz!RemoveEmptyStrings.process_stringr'cr()Ncsg|] }|dkr|qSrCr*r+r-r r!r.sz3RemoveEmptyStrings.process_list..r r0r r-r!rr1zRemoveEmptyStrings.process_listNr`r r r r!r sr c@r)ru6 Replace common contractions such as `let's` to `let us`. Currently, this method will perform the following replacements. Note that `␣` is used to indicate a space (` `) to get around markdown rendering constrains. | Contraction | transformed into | | ------------- |:----------------:| | `won't` | `␣will not` | | `can't` | `␣can not` | | `let's` | `␣let us` | | `n't` | `␣not` | | `'re` | `␣are` | | `'s` | `␣is` | | `'d` | `␣would` | | `'ll` | `␣will` | | `'t` | `␣not` | | `'ve` | `␣have` | | `'m` | `␣am` | Example: ```python import jiwer sentences = ["she'll make sure you can't make it", "let's party!"] print(jiwer.ExpandCommonEnglishContractions()(sentences)) # prints: ["she will make sure you can not make it", "let us party!"] ``` r#cCstdd|}tdd|}tdd|}tdd|}td d |}td d |}td d|}tdd|}tdd|}tdd|}tdd|}|S)Nzwon'tzwill notzcan\'tzcan notzlet\'szlet uszn\'tz notz\'rez arez\'sz isz\'dz wouldz\'llz willz\'tz\'vez havez\'mz amrr&r r r!rsz.ExpandCommonEnglishContractions.process_stringNrr r r r!rs rc@r)rz Convert every character into lowercase. Example: ```python import jiwer sentences = ["You're PRETTY"] print(jiwer.ToLowerCase()(sentences)) # prints: ["you're pretty"] ``` r#cCrr$)lowerr&r r r!r@rzToLowerCase.process_stringNrr r r r!r1src@r)rz Convert every character to uppercase. Example: ```python import jiwer sentences = ["You're amazing"] print(jiwer.ToUpperCase()(sentences)) # prints: ["YOU'RE AMAZING"] ``` r#cCrr$)upperr&r r r!rTrzToUpperCase.process_stringNrr r r r!rDrrc@r)r a Remove any word between `[]` and `<>`. This can be useful when working with hypotheses from the Kaldi project, which can output non-words such as `[laugh]` and ``. Example: ```python import jiwer sentences = ["you like [laugh]"] print(jiwer.RemoveKaldiNonWords()(sentences)) # prints: ["you like "] # note the extra spaces ``` r#cCr)Nz[<\[][^>\]]*[>\]]r>rr&r r r!rkrz"RemoveKaldiNonWords.process_stringNrr r r r!r Xsr )!r5r functoolsrjr{rtypingrrrr__all__objectrrr=r r r rrrr lru_cacherrrrr rrrr r r r r!s8"#-%%"#*  6