o i#@sddlZddlmZddlmZmZmZmZmZm Z m Z m Z m Z ddl mZmZe e eefZe dZeeegeeefZddded ed ed e eegefd ef d dZdddeded ed e eegefd ef ddZ dded e eegefd efddZddefdeeded ed ed ed eeef ddZddefdeeded eeefddZefde eded eeefddZdS) N)partial) AnyCallableIterableIteratorListOptionalSequenceTypeVarUnion) minibatchregistryItemT get_lengthsizebufferdiscard_oversizerreturncC,|durd|ini}ttf|||d|S)aCreate a batcher that uses the `batch_by_padded_size` strategy. The padded size is defined as the maximum length of sequences within the batch multiplied by the number of sequences in the batch. size (int or Sequence[int]): The largest padded size to batch sequences into. Can be a single integer, or a sequence, allowing for variable batch sizes. buffer (int): The number of sequences to accumulate before sorting by length. A larger buffer will result in more even sizing, but if the buffer is very large, the iteration order will be less random, which can result in suboptimal training. discard_oversize (bool): Whether to discard sequences that are by themselves longer than the largest padded batch size. get_length (Callable or None): Function to get the length of a sequence item. The `len` function is used by default. Nr)rrr)rminibatch_by_padded_size)rrrr optionalsrK/home/ubuntu/.local/lib/python3.10/site-packages/spacy/training/batchers.py"configure_minibatch_by_padded_sizesr tolerancecCr)aCreate a batcher that uses the "minibatch by words" strategy. size (int or Sequence[int]): The target number of words per batch. Can be a single integer, or a sequence, allowing for variable batch sizes. tolerance (float): What percentage of the size to allow batches to exceed. discard_oversize (bool): Whether to discard sequences that by themselves exceed the tolerated size. get_length (Callable or None): Function to get the length of a sequence item. The `len` function is used by default. Nr)rrr)rminibatch_by_words)rrrrrrrrconfigure_minibatch_by_words8srcCs(|durd|ini}ttfd|i|S)zCreate a batcher that creates batches of the specified size. size (int or Sequence[int]): The target number of items per batch. Can be a single integer, or a sequence, allowing for variable batch sizes. Nrr)rr )rrrrrrconfigure_minibatchSsrFseqsc #st|tr t|}nt|}t||dD]3tt|}t||D]"}fdd|D}t dd|Dt |} |rE| |krEq&|Vq&qdS)aqMinibatch a sequence by the size of padded batches that would result, with sequences binned by length within a window. The padded size is defined as the maximum length of sequences within the batch multiplied by the number of sequences in the batch. size (int or Sequence[int]): The largest padded size to batch sequences into. buffer (int): The number of sequences to accumulate before sorting by length. A larger buffer will result in more even sizing, but if the buffer is very large, the iteration order will be less random, which can result in suboptimal training. discard_oversize (bool): Whether to discard sequences that are by themselves longer than the largest padded batch size. get_length (Callable or None): Function to get the length of a sequence item. The `len` function is used by default. )rcsg|]}|qSrr).0i outer_batchrr ~sz,minibatch_by_padded_size..cs|]}t|VqdSNlen)r"seqrrr z+minibatch_by_padded_size..N) isinstanceint itertoolsrepeatiterr listnext_batch_by_lengthmaxr*) r!rrrrsize_ target_sizeindicessubbatch padded_sizerr$rr_s   rg?ccsjt|tr t|}nt|}t|}||}g}g} d} d} |D]} || } | ||kr5|s4| gVq"| dkrI| | |krI|| | | 7} q"| | | ||kr]| | | | 7} q"|rb|Vt|}||}| }| } g} d} | | |kr|| | | 7} q"| | ||kr| | | | 7} q"|r|Vt|}||}| g}| } q"|| |r|VdSdS)aCreate minibatches of roughly a given number of words. If any examples are longer than the specified batch length, they will appear in a batch by themselves, or be discarded if discard_oversize=True. seqs (Iterable[Sequence]): The sequences to minibatch. size (int or Sequence[int]): The target number of words per batch. Can be a single integer, or a sequence, allowing for variable batch sizes. tolerance (float): What percentage of the size to allow batches to exceed. discard_oversize (bool): Whether to discard sequences that by themselves exceed the tolerated size. get_length (Callable or None): Function to get the length of a sequence item. The `len` function is used by default. rN)r.r/r0r1r2r4appendextend)r!rrrrr7r8tol_sizebatchoverflow batch_size overflow_sizer+n_wordsrrrrs\              r max_wordscsfddt|D}|g}g}|D]$\}}|s!||q|t|d|kr1||q|||g}q|rA||tdd|Dt|ksPJdd|D}||S)zGiven a list of sequences, return a batched list of indices into the list, where the batches are grouped by length, in descending order. Batches may be at most max_words in size, defined as max sequence length * size. csg|] \}}||fqSrr)r"r#r+rrrr&sz$_batch_by_length..csr'r(r))r"brrrr,r-z#_batch_by_length..cSsg|]}tt|qSr)r3sorted)r"r?rrrr&s) enumeratesortr<r*sumreverse)r!rDrlengths_indicesbatchesr?lengthr#rrrr5s"      r5r()r0 functoolsrtypingrrrrrrr r r utilr rr/SizingrBatcherTboolrfloatrrr*rrr5rrrrs ,  '    *  M