o TigS)@sddlZddlZddlZddlZddlZddlmZmZmZm Z m Z m Z ddl m Z mZmZmZmZmZddZ   dDded ee ed e ee ed ed ed eef ddZ  dEded ee edee ed ed ed eef ddZ  dFdedee ed ed ed ee f ddZ  dEdedee ed ed ed eef ddZ dGdddddddddddddd ded e ed e e ed ed!ed"ed#ed$ed%ed&ed'ed(ed)e e e ed*e e ee ee efd+ed,ed eef"d-d.Z dHd/dddddddddddd0dd1ded2e e ee e efd ed ed3ed4ed"ed#ed$ed%ed&ed'ed(ed)e e e ed*e e ee ee efd5e e ed6ed7eegefd eef&d8d9Z dHd/dddddddddddd0dd1ded2e e ee e efd ed ed3ed4ed"ed#ed$ed%ed&ed'ed(ed)e e e ed*e e ee ee efd5e e ed6ed7eegefd eef&d:d;ZGdd?Z d@dAZ!dBdCZ"dS)IN) AsyncIterableCallableIterableListOptionalUnion)GenerationResultGenerationStepResult Generator ScoringResultTranslationResult TranslatorcCsXttdtttdtttdtttdtttdtttdtttdt dS)z4Registers additional attributes to compiled modules.translate_iterablescore_iterablegenerate_tokensgenerate_iterableasync_generate_tokensN) setattrr translator_translate_iterabletranslator_score_iterabletranslator_generate_tokensr generator_generate_iterablegenerator_score_iterablegenerator_generate_tokensgenerator_async_generate_tokensrrJ/home/ubuntu/.local/lib/python3.10/site-packages/ctranslate2/extensions.pyregister_extensionss      r examples translatorsource target_prefixmax_batch_size batch_typereturncks<|g}|dur ||t|j|||fi|EdHdS)a6Translates an iterable of tokenized examples. This method is built on top of :meth:`ctranslate2.Translator.translate_batch` to efficiently translate an arbitrarily large stream of data. It enables the following optimizations: * stream processing (the iterable is not fully materialized in memory) * parallel translations (if the translator has multiple workers) * asynchronous batch prefetching * local sorting by length Arguments: source: An iterable of tokenized source examples. target_prefix: An optional iterable of tokenized target prefixes. max_batch_size: The maximum batch size. batch_type: Whether :obj:`max_batch_size` is the number of "examples" or "tokens". **kwargs: Any translation options accepted by :meth:`ctranslate2.Translator.translate_batch`. Returns: A generator iterator over :class:`ctranslate2.TranslationResult` instances. Example: This method can be used to efficiently translate text files: .. code-block:: python # Replace by your own tokenization and detokenization functions. tokenize_fn = lambda line: line.strip().split() detokenize_fn = lambda tokens: " ".join(tokens) with open("input.txt") as input_file: source = map(tokenize_fn, input_file) results = translator.translate_iterable(source, max_batch_size=64) for result in results: tokens = result.hypotheses[0] target = detokenize_fn(tokens) print(target) N)append_process_iterabletranslate_batch)r r!r"r#r$kwargs iterablesrrrrs0 r@targetcks(t|j||g||fi|EdHdS)a}Scores an iterable of tokenized examples. This method is built on top of :meth:`ctranslate2.Translator.score_batch` to efficiently score an arbitrarily large stream of data. It enables the following optimizations: * stream processing (the iterable is not fully materialized in memory) * parallel scoring (if the translator has multiple workers) * asynchronous batch prefetching * local sorting by length Arguments: source: An iterable of tokenized source examples. target: An iterable of tokenized target examples. max_batch_size: The maximum batch size. batch_type: Whether :obj:`max_batch_size` is the number of "examples" or "tokens". **kwargs: Any scoring options accepted by :meth:`ctranslate2.Translator.score_batch`. Returns: A generator iterator over :class:`ctranslate2.ScoringResult` instances. Nr' score_batch)r r!r,r#r$r)rrrr[sr generator start_tokensck&t|j|g||fi|EdHdS)acGenerates from an iterable of tokenized prompts. This method is built on top of :meth:`ctranslate2.Generator.generate_batch` to efficiently run generation on an arbitrarily large stream of data. It enables the following optimizations: * stream processing (the iterable is not fully materialized in memory) * parallel generations (if the generator has multiple workers) * asynchronous batch prefetching * local sorting by length Arguments: start_tokens: An iterable of tokenized prompts. max_batch_size: The maximum batch size. batch_type: Whether :obj:`max_batch_size` is the number of "examples" or "tokens". **kwargs: Any generation options accepted by :meth:`ctranslate2.Generator.generate_batch`. Returns: A generator iterator over :class:`ctranslate2.GenerationResult` instances. N)r'generate_batch)r/r0r#r$r)rrrrrtokensckr1)a9Scores an iterable of tokenized examples. This method is built on top of :meth:`ctranslate2.Generator.score_batch` to efficiently score an arbitrarily large stream of data. It enables the following optimizations: * stream processing (the iterable is not fully materialized in memory) * parallel scoring (if the generator has multiple workers) * asynchronous batch prefetching * local sorting by length Arguments: tokens: An iterable of tokenized examples. max_batch_size: The maximum batch size. batch_type: Whether :obj:`max_batch_size` is the number of "examples" or "tokens". **kwargs: Any score options accepted by :meth:`ctranslate2.Generator.score_batch`. Returns: A generator iterator over :class:`ctranslate2.ScoringResult` instances. Nr-)r/r4r#r$r)rrrrr3rFi) max_decoding_lengthmin_decoding_length sampling_topk sampling_toppsampling_temperaturereturn_log_probrepetition_penaltyno_repeat_ngram_size disable_unksuppress_sequences end_tokenmax_input_lengthuse_vmapr7r8r9r:r;r<r=r>r?r@rArBrCc csFt|j|g|dur |gnd| | | | | ||||||||dEdHdS)a{Yields tokens as they are generated by the model. Arguments: source: Source tokens. target_prefix: Optional target prefix tokens. max_decoding_length: Maximum prediction length. min_decoding_length: Minimum prediction length. sampling_topk: Randomly sample predictions from the top K candidates. sampling_topp: Keep the most probable tokens whose cumulative probability exceeds this value. sampling_temperature: Sampling temperature to generate more random samples. return_log_prob: Include the token log probability in the result. repetition_penalty: Penalty applied to the score of previously generated tokens (set > 1 to penalize). no_repeat_ngram_size: Prevent repetitions of ngrams with this size (set 0 to disable). disable_unk: Disable the generation of the unknown token. suppress_sequences: Disable the generation of some sequences of tokens. end_token: Stop the decoding on one of these tokens (defaults to the model EOS token). max_input_length: Truncate inputs after this many tokens (set 0 to disable). use_vmap: Use the vocabulary mapping file saved in this model Returns: A generator iterator over :class:`ctranslate2.GenerationStepResult` instances. Note: This generation method is not compatible with beam search which requires a complete decoding. N) r=r>r?r@rAr7r8r9r:r; return_scoresrBrC)_generate_tokensr()r r!r"r7r8r9r:r;r<r=r>r?r@rArBrCrrrrs&.riT) max_length min_lengthr9r:r;r<r=r>r?r@rA static_promptcache_static_promptcallbackpromptrFrGrHrIrJccst|dkrt|dtr|g}t|j|fid|d|d| d| d| d| d|d |d |d |d |d |d| d|d|ddd|EdHdS)aYields tokens as they are generated by the model. Arguments: prompt: Batch of start tokens. If the decoder starts from a special start token like , this token should be added to this input. max_batch_size: The maximum batch size. batch_type: Whether :obj:`max_batch_size` is the number of "examples" or "tokens". max_length: Maximum generation length. min_length: Minimum generation length. sampling_topk: Randomly sample predictions from the top K candidates. sampling_topp: Keep the most probable tokens whose cumulative probability exceeds this value. sampling_temperature: Sampling temperature to generate more random samples. return_log_prob: Include the token log probability in the result. repetition_penalty: Penalty applied to the score of previously generated tokens (set > 1 to penalize). no_repeat_ngram_size: Prevent repetitions of ngrams with this size (set 0 to disable). disable_unk: Disable the generation of the unknown token. suppress_sequences: Disable the generation of some sequences of tokens. end_token: Stop the decoding on one these tokens (defaults to the model EOS token). static_prompt: If the model expects a static prompt (a.k.a. system prompt) it can be set here to simplify the inputs and optionally cache the model state for this prompt to accelerate future generations. cache_static_prompt: Cache the model state after the static prompt and reuse it for future generations using the same static prompt. callback: Optional function that is called for each generated token when obj:`beam_size` is 1. If the callback function returns ``True``, the decoding will stop for this batch index. Returns: A generator iterator over :class:`ctranslate2.GenerationStepResult` instances. Note: This generation method is not compatible with beam search which requires a complete decoding. rr#r$r=r>r?r@rArFrGr9r:r;rDrHrIinclude_prompt_in_resultFrJN)len isinstancestrrEr2)r/rKr#r$rFrGr9r:r;r<r=r>r?r@rArHrIrJrrrrsR8     rcCst|dkrt|dtr|g}t|j|fid|d|d| d| d| d| d|d |d |d |d |d |d| d|d|ddd|2z 3dHW}|VqL6dS)aYields tokens asynchronously as they are generated by the model. Arguments: prompt: Batch of start tokens. If the decoder starts from a special start token like , this token should be added to this input. max_batch_size: The maximum batch size. batch_type: Whether :obj:`max_batch_size` is the number of "examples" or "tokens". max_length: Maximum generation length. min_length: Minimum generation length. sampling_topk: Randomly sample predictions from the top K candidates. sampling_topp: Keep the most probable tokens whose cumulative probability exceeds this value. sampling_temperature: Sampling temperature to generate more random samples. return_log_prob: Include the token log probability in the result. repetition_penalty: Penalty applied to the score of previously generated tokens (set > 1 to penalize). no_repeat_ngram_size: Prevent repetitions of ngrams with this size (set 0 to disable). disable_unk: Disable the generation of the unknown token. suppress_sequences: Disable the generation of some sequences of tokens. end_token: Stop the decoding on one of these tokens (defaults to the model EOS token). static_prompt: If the model expects a static prompt (a.k.a. system prompt) it can be set here to simplify the inputs and optionally cache the model state for this prompt to accelerate future generations. cache_static_prompt: Cache the model state after the static prompt and reuse it for future generations using the same static prompt. callback: Optional function that is called for each generated token when obj:`beam_size` is 1. If the callback function returns ``True``, the decoding will stop for this batch index. Returns: An async generator iterator over :class:`ctranslate2.GenerationStepResult` instances. Note: This generation method is not compatible with beam search which requires a complete decoding. rr#r$r=r>r?r@rArFrGr9r:r;rDrHrIrLFrJN)rMrNrOAsyncGeneratorr2)r/rKr#r$rFrGr9r:r;r<r=r>r?r@rArHrIrJ step_resultrrrr_sV8     rc@s,eZdZddZddZddZddZd S) rPcOs0t|_t|_d|_||_||_||_ dSN) asyncioQueuequeue threadingEventshutdown_event iterator_task process_funcargsr))selfrZr[r)rrr__init__s   zAsyncGenerator.__init__csjt|jg|jRi|jD]}|j|IdHtdIdH|j r)nq|jdIdHdS)Ng-C6?) rErZr[r)rUputrSsleeprXis_set)r\rQrrrproducers  zAsyncGenerator.producercCst||_|SrR)rS create_taskrarY)r\rrr __aiter__szAsyncGenerator.__aiter__cs\|jrtz|jIdH}|dur|jt|WStjy-|jtwrR)rXr`StopAsyncIterationrUgetsetrSCancelledError)r\itemrrr __anext__s   zAsyncGenerator.__anext__N)__name__ __module__ __qualname__r]rarcrirrrrrPs  rPc/stt|dddurddfdd}|dd|d||i|fd d }tj|dd }| }|durKnt|t rR|z|VWn t yd YnwqB| dS) NrJcSsdS)NFr)rQrrrsz"_generate_tokens..cs|}|p|SrR)r^r`)rQuser_callback_result)generator_closed step_results user_callbackrr _callbacks  z#_generate_tokens.._callbackTr6) asynchronous beam_sizerJc sTz D]}|qWnty"}z |WYd}~nd}~wwddSrR)result Exceptionr^)rue) async_resultsrprr_catch_exceptions z*_generate_tokens.._catch_exception)r,daemon) rUrTrVrWreupdateThreadstartrNrv GeneratorExitrfjoin)rZr[r)rrrythreadrQr)rxrorprqrrEs>     rEc ks|dkr tdt|dkr|d}ntj|}|||dd|dkr*|dn|}t}t|||D]#}|||i||rY|d rY| V|rY|d sJq6|rg| V|s\dSdS)Nr6zmax_batch_size must be >= 1rT)r#r$rs) ValueErrorrM itertools zip_longestr{ collectionsdeque_batch_iteratorextenddonepopleftru) rZr*r#r$r)iterableread_batch_sizerUstreamsrrrr's.    r'ccsd}d}|D]u}t|ts|f}|dkr%|r$t|d|kr$|Vd}n-|dkrLt|t|d}|rKt|dd||krK|Vd}t|d}ntd||dur_tdd|D}t||D]\}}|durvt|dkrvtd||qdq|dur|VdSdS) Nrrr4r6zInvalid batch type %scss|]}gVqdSrRr).0_rrr Fsz"_batch_iterator..z+Input iterables do not have the same length)rNtuplerMmaxrzipr&)r batch_sizer$rrFexamplebatchelementrrrr-s:     r)Nrr)r+r)rrrR)rr)#rSrrrUrVtypingrrrrrrctranslate2._extrr r r r r rrOintrrrrfloatboolrrrrPrEr'rrrrrs     A   *  (  (       E        T        R(6