o 5t¾i# ã @sždZddlmZddlmZmZdedededeeeffdd „Zd eed edefd d „Z dded ede defdd„Z ddedede deefdd„Z dS)zCVarious utility functions for word and character n-gram extraction.é)ÚCounter)ÚListÚTupleÚlineÚ min_orderÚ max_orderÚreturnc Csfg}| ¡}t||dƒD]}tdt|ƒ|dƒD]}| t||||…ƒ¡qq t|ƒt|ƒfS)aExtracts all ngrams (min_order <= n <= max_order) from a sentence. :param line: A string sentence. :param min_order: Minimum n-gram order. :param max_order: Maximum n-gram order. :return: a Counter object with n-grams counts and the sequence length. ér)ÚsplitÚrangeÚlenÚappendÚtupler)rrrÚngramsÚtokensÚnÚi©rúM/home/ubuntu/.local/lib/python3.10/site-packages/sacrebleu/metrics/helpers.pyÚextract_all_word_ngramss ÿrrrcs(t‡‡fdd„ttˆƒˆdƒDƒƒS)z¸Extracts n-grams with order `n` from a list of tokens. :param tokens: A list of tokens. :param n: The order of n-grams. :return: a Counter object with n-grams counts. cs"g|] }d ˆ||ˆ…¡‘qS)ú )Újoin©Ú.0r©rrrrÚ !s"z'extract_word_ngrams..r )rr r )rrrrrÚextract_word_ngramss(rFÚinclude_whitespacecs:|s d ˆ ¡¡‰t‡‡fdd„ttˆƒˆdƒDƒƒS)a&Yields counts of character n-grams from a sentence. :param line: A segment containing a sequence of words. :param n: The order of the n-grams. :param include_whitespace: If given, will not strip whitespaces from the line. :return: a dictionary containing ngrams and counts Úcóg|] }ˆ||ˆ…‘qSrrr©rrrrr/óz'extract_char_ngrams..r )rr rr r )rrrrr rÚextract_char_ngrams$s(r"cs`g}|s d ˆ ¡¡‰td|dƒD]‰t‡‡fdd„ttˆƒˆdƒDƒƒ}| |¡q|S)aHExtracts all character n-grams at once for convenience. :param line: A segment containing a sequence of words. :param max_order: The maximum order of the n-grams. :param include_whitespace: If given, will not strip whitespaces from the line. :return: a list of Counter objects containing ngrams and counts. rr crrrrr rrrBr!z+extract_all_char_ngrams..)rr r rr r )rrrÚcountersrrr rÚextract_all_char_ngrams2s ( r$N)F) Ú__doc__Ú collectionsrÚtypingrrÚstrÚintrrÚboolr"r$rrrrÚs  " ÿÿÿÿÿ