o پiP@sddlZddlmZddlmZddlmZmZmZmZm Z m Z ddl m Z ddl mZmZmZer6ddlZe eeeeejejejfZGdddZed d Gd d d Zed d Gd ddZGdddZGdddeZdS)N)copy) dataclass) TYPE_CHECKINGAnyIteratorListOptionalUnion)sequence_generator)BeamSearchSampler GreedySamplerMultinomialSamplerc@s@eZdZddZdddddedfddZd eed eedefd d Zd ed e eedefddZ d ede fddZ   dde eeefde ede e eeefde dde e ee eee ff ddZ   dde eeefde ede e eeefde ddee eeeeeeff ddZdS)SequenceGeneratorcCs,||_||_||_|j|_||_|j|_dSN)fsmmodelsampler tokenizerdevicesamples num_samples)selfrrrrrI/home/ubuntu/.local/lib/python3.10/site-packages/outlines/generate/api.py__init__s  zSequenceGenerator.__init__prompt_token_idsz torch.Tensor token_idsreturncCs&dd|D}ddt||D}|S)aZGet the tokens generated so far. Parameters ---------- prompt_token_ids Tensor that contains the token ids of the sequences' prompts. token_ids The generated token ids. Returns ------- A tensor that contains the token ids that have been generated so far. cSsg|]}t|qSrlen).0promptrrr 3z=SequenceGenerator.get_generated_token_ids..cSsg|] \}}||dqSrr)r cur_token_idslengthrrrr"4s )zip)rrrprompt_lengthsrrrget_generated_token_ids s z)SequenceGenerator.get_generated_token_idsgenerated_sequencesstop_sequencescstfdd|DS)aDetermine whether one of the stop sequences has been generated. Parameters ---------- generated_sequences The list of sequences generated so far. stop_sequences The list that contains the sequence which stop the generation when found. Returns ------- True if at least one of the stop sequences has been found in each generated sequence. cs"g|] tfddDqS)csg|]}|vqSrrr seq generatedrrr"Pr#zGSequenceGenerator.is_stop_sequence_found...)any)r r*r-rr"Osz.)all)rr)r*rr0ris_stop_sequence_found;s  z(SequenceGenerator.is_stop_sequence_foundsequencecsd|r0fdd|D}tdd|Dr0tdd|D}||}d||t||S)aRemove the stop sequences from the generated sequences. Parameters ---------- sequence One of the generated sequences. stop_sequences The list that contains the sequence which stop the generation when found. cg|]}|qSr)findr+r3rrr"dz:SequenceGenerator.strip_stop_sequences..cSsg|]}|dkqSr)r indexrrrr"er#cSsg|]}|dkr|qSr8r)r irrrr"gsN)r/minr:r)rr3r* match_indexesmin_match_index_valuemin_match_index_posrr6rstrip_stop_sequencesUs  z&SequenceGenerator.strip_stop_sequencescC|SaXTranslate the generated sequence to another type. This method is for instance overridden when generating JSON to either return a dictionnary or a Pydantic model. Parameters ---------- sequence A generated sequences. Returns ------- The formatted sequence. rrr3rrrformat_sequencepz!SequenceGenerator.format_sequenceNprompts max_tokensstop_atrngztorch.Generatorc s@ddl}t|tr |g}t|tr|g}|j}|dur(|jjd}|j|\}}| j}| j}j}t |} |j ||dd}|j ||dd}ddt | |D} fddt | |D} |j | ||jjd} tjj| || || |d } z-t| }|sr|j}||}|rt |d|krWnrj|rWn Wn tyYnwq|j}||}j|}fd d|D}fd d|D}t}t d| ||D] }|||||q| d kr |d kr |ddS| d kr|dS|d krdd|DS|S)aGenerate the full text sequence. Since `SequenceGenerator.stream` calls the tokenizer at every step this method loops over the generator returned by `sequence_generator` itself so the tokenizer is called only once after all token ids have been generated. Parameters ---------- prompts A string or list of strings that are passed to the model before generating the first token. max_tokens An integer representing maximum number of tokens that will be generated (per prompt) stop_at A string or list of strings at which the text generated will stop rng The random number generator. Defaults to a non-seeded `torch.Generator` instance. Returns ------- The generation(s), potentially cast to another type. rNrdimcSg|]}dqSrrr _rrrr"z.SequenceGenerator.__call__..cg|]}jqSrrrrOrrrr"r7dtyperrITcsg|]}|qSr)r@r r3rr*rrr"s cr4r)rDrXrTrrr"r7cSg|]}|dqSrNrr rrrrr"r#)torch isinstancestrr Generatorrseedrencodetorrepeat_interleaverangezerosfloatr rrnextrr(r2decode StopIterationlistappend)rrFrGrHrIr]rrattention_masks batch_size fsm_statesfsmsweightsstates last_statergenerated_token_idsr.stripped formattedoutputr;rrYr__call__s             zSequenceGenerator.__call__c sFddl}t|tr |g}t|tr|g}|jj|\}}|j}||j}jt||j |dd}|j |dd}ddt D}fddt D} |j |j |jd} |durw|j |jd}|tjj| || |||d d ttttttttfffd d } | S) aGenerate the text sequence one token at a time. Since `Tokenizer.decode` strips the whitespaces from the tokens we have no choice but to decode the generated token ids at each step and compare the current decoded strings to the previously decoded strings. Parameters ---------- prompts A string or list of strings that are passed to the model before generating the first token. max_tokens An integer representing maximum number of tokens that will be generated (per prompt) stop_at A string or list of strings at which the text generated will stop rng The random number generator. Defaults to a non-seeded `torch.Generator` instance. Returns ------- A string or list of strings that contain the generated text. rNrKcSrMrNrrOrrrr"/rQz,SequenceGenerator.stream..crRrrSrOrTrrr"0r7rUrJrWrc 3stddtD}d}ddtD} r |ks$t|r&dSz t}|d7}Wn ty:YdSw|jdd| df}j|}rifddt||D}fddt||D}d dt|||D}|}t}tdD] }| |||qdkrdkr|ddVndkr|dVndkrd d|DVn|Vq) NcSrM)rrOrrrr"GszESequenceGenerator.stream..token_generator..rcSrM)FrrOrrrr"KrQTrZcs"g|] \}}|p|gqSr)r2)r generated_sequencestoprYrrr"Yscs*g|]\}}|r|n|qSr)rDr@)r r3r{rYrrr"cs cSs"g|] \}}}|t|dqSrr)r tokenr3r{rrrr"mscSr[rNrr\rrrr"r#) rer1rhrjrrrir&rkrl) previously_generated_sequences num_generatedis_stop_at_reachedr3rtr) next_tokensrwr;rnrGrrrrr*rrtoken_generatorFsf       z1SequenceGenerator.stream..token_generator)r]r^r_rrrbrcrrrdrerfrgr`rar rrrr r) rrFrGrHrIr]rrmrorprqrrrrstreamsH    6 ?zSequenceGenerator.streamNNN)__name__ __module__ __qualname__rrr(r_boolr2rr@FormattedOutputrDr intrxrrrrrrrsh     yrT)frozenc@sBeZdZUdZeeed<eeee efed<eeed<dS)GenerationParametersz3Generation parameters used in Outlines' public API.rGrHraN) rrr__doc__rr__annotations__r r_rrrrrrs  rc@sVeZdZUdZeed<dZeed<dZe e ed<dZ e eed<dZ e e ed<dS) SamplingParametersz*Sampling parameters available in Outlines.rrZrNtop_ptop_k temperature) rrrrr_rrrrrrgrrrrrrrs  rc @seZdZdZddZdeedeeee efdeefddZ d ed e fd d Z d dZ   ddeee efdeedeeee efdeefddZ   ddeee efdeedeeee efdeefddZdS)SequenceGeneratorAdaptera!Class used to unify the interface to the model providers' generation functions. Attributes ---------- model The wrapped model. logits_processor The logits processor to use to generate text. sampler The sampler to use to generate text. cCs|||_||_t|trtd|j|j|j|j|_ dSt|t r+td|jddd|_ dSt|t r.)r^rkrD)r sequencesrrTrrs  z SequenceGeneratorAdapter._formatNrFcKs:||||}|jj||t|j|jfi|}||S)z/Generate text from a prompt of list of prompts.)rrgeneraterrrr)rrFrGrHramodel_specific_paramsr completionsrrrrxs  z!SequenceGeneratorAdapter.__call__cKs0||||}|jj||t|j|jfi|Sz;Return a text generator from a prompt or a list of prompts.)rrrrrr)rrFrGrHrarrrrrrs zSequenceGeneratorAdapter.streamr)rrrrrrrr r_rrrrDrrxrrrrrrsH   rc @s eZdZ   ddeeeefdeeefdeedeeeeefdeef ddZ   ddeeeefdeeeeeeeeffdeedeeeeefdeef d d Z e deeeefdeeeeeeeffd eeeeffd d Z dS)VisionSequenceGeneratorAdapterNrFmediarGrHrac KsL|||\}}||||}|jj|||t|j|jfi|}||S)z Generate text from a prompt of list of prompts. Media: A URI to construct media or media object itself. Used as AutoProcessor argument. )_validate_prompt_media_typesrrrrrrr) rrFrrGrHrarrrrrrrxs z'VisionSequenceGeneratorAdapter.__call__cKsB|||\}}||||}|jj|||t|j|jfi|Sr)rrrrrrr)rrFrrGrHrarrrrrr7s z%VisionSequenceGeneratorAdapter.streamrcCs.dd}|||std|d|||fS)zi Prepare media as PIL.Image and ensure for every prompt str there is one List[PIL.Image] csddlmt|tr9t|trt|t|krdSt||D]\}}t|tr3tfdd|Ds6dSqdSt|trKtfdd|DsKdSdS)NrImageFc3|] }t|jVqdSrr^rr mrrr _s  zcVisionSequenceGeneratorAdapter._validate_prompt_media_types..valid_types..c3rrrrrrrrdsT)PILrr^rkrr&r_r1)rFr subpromptsubmediarrr valid_typesXs   zPVisionSequenceGeneratorAdapter._validate_prompt_media_types..valid_typeszsExpected (prompts, media) to be of type (str, List[Image])), or (List[str], List[List[Image]]) instead got prompts=z, media=) TypeError)clsrFrrrrrrNs z;VisionSequenceGeneratorAdapter._validate_prompt_media_typesr) rrrr r_rrrrrxr classmethodrrrrrrsJ  # r)datetimer dataclassesrtypingrrrrrr outlines.generate.generatorr outlines.samplersr r r r]r_rrgrdatetimerrrrrrrrrrs(    y {