o
ei @ s. d Z ddlZ d
ddZdd Zdd d ZdS )zU
N-gram counting, discounting, interpolation, and backoff
Authors
* Aku Rouhe 2020
NT and ) at the ends of
each sentence. The token should not be predicted, so some special
care needs to be taken for unigrams.
Arguments
---------
sequence : iterator
The sequence (any iterable type) to pad.
pad_left : bool
Whether to pad on the left side as well. True by default.
left_pad_symbol : any
The token to use for left side padding. "" by default.
right_pad_symbol : any
The token to use for right side padding. "" by default.
Returns
-------
generator
A generator that yields the padded sequence.
Example
-------
>>> for token in pad_ends(["Speech", "Brain"]):
... print(token)
Speech
Brain
) itertoolschaintuple)sequencepad_leftleft_pad_symbolright_pad_symbol r U/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/speechbrain/lm/counting.pypad_ends s
&r
c c s |dkr t d|dkr| D ]}|fV qdS t| }g }t|ddD ]\}}|| ||d kr5 nq$dS |D ]}t||f V || |d= q:dS )a
Produce all Nth order N-grams from the sequence.
This will generally be used in an N-gram counting pipeline.
Arguments
---------
sequence : iterator
The sequence from which to produce N-grams.
n : int
The order of N-grams to produce
Yields
------
tuple
Yields each ngram as a tuple.
Returns
-------
None
Example
-------
>>> for ngram in ngrams("Brain", 3):
... print(ngram)
('B', 'r', 'a')
('r', 'a', 'i')
('a', 'i', 'n')
r z
N must be >=1 N)start)
ValueErroriter enumerateappendr )r ntokeniteratorhistory hist_lengthr r r ngrams? s(
r Fc c sh |dkr t dt| }g }|s|t| |D ]}t||kr%|d= |t|fV || qdS )ao
Produce each token with the appropriate context.
The function produces as large N-grams as possible, so growing from
unigrams/bigrams to max_n.
E.G. when your model is a trigram model, you'll still only have one token
of context (the start of sentence) for the first token.
In general this is useful when evaluating an N-gram model.
Arguments
---------
sequence : iterator
The sequence to produce tokens and context from.
max_n : int
The maximum N-gram length to produce.
predict_first : bool
To produce the first token in the sequence to predict (without
context) or not. Essentially this should be False when the start of
sentence symbol is the first in the sequence.
Yields
------
Any
The token to predict
tuple
The context to predict conditional on.
Example
-------
>>> for token, context in ngrams_for_evaluation("Brain", 3, True):
... print(f"p( {token} |{' ' if context else ''}{' '.join(context)} )")
p( B | )
p( r | B )
p( a | B r )
p( i | r a )
p( n | a i )
r zMax N must be >=1N)r r r nextlenr )r max_n
predict_firstr r r r r r ngrams_for_evaluationt s ( r )Tr r )F)__doc__r r
r r r r r r s
.5