o پiY@sddlZddlmZmZmZmZddlmZmZer,ddl m Z ddl m Z ddl mZGdddZd efd d Z dddZdS)N) TYPE_CHECKINGListOptionalUnion)GenerationParametersSamplingParameters)PreTrainedTokenizerBaseLLMSamplingParamsc @sreZdZdZdddZddZdd d d eeeefd e d e de dde f ddZ ddZde efddZdS)VLLMa Represents a vLLM model. We wrap models from model providing libraries in order to give all of them the same interface in Outlines and allow users to easily switch between providers. This class wraps the `vllm.LLM` class from the `vllm` library. modelr cCs||_d|_||_dS)N)r lora_request_get_tokenizer tokenizer)selfrrH/home/ubuntu/.local/lib/python3.10/site-packages/outlines/models/vllm.py__init__sz VLLM.__init__cCsZt|jdr |j}nt|jdr$t|jjdr|jjj}n |jj}ntdt|dS)N get_tokenizerrzZThe provided LLM instance neither has a `tokenizer` attribute or a `get_tokenizer` method.)r)hasattrrrr ValueErroradapt_tokenizer)rrrrrrs      zVLLM._get_tokenizerNT)sampling_paramsuse_tqdmpromptsgeneration_parameterssampling_parametersrr rcCsddlm}|dur |}t|\}} } |dur||_| dur+t| tr(| g} | |_| dur2| |_|dur9|gng|_ t|\} } } }}|j dkrQ| |_ | |_ | dur]|j dkr]| |_ |durp|j dkrp||_ |dkrpd|_|dur||jdkr|||_| dkrd|_|jj|||j|d }d d |D}t|}t|d}|dkr|dkr|ddS|dkr|dS|dkrd d |DS|S) aGenerate text using vLLM. Arguments --------- prompts A prompt or list of prompts. generation_parameters An instance of `GenerationParameters` that contains the prompt, the maximum number of tokens, stop sequences and seed. All the arguments to `SequenceGeneratorAdapter`'s `__cal__` method. logits_processor The logits processor to use when generating text. sampling_parameters An instance of `SamplingParameters`, a dataclass that contains the name of the sampler to use and related parameters as available in Outlines. sampling_params An instance of `vllm.sampling_params.SamplingParams`. The values passed via this dataclass supersede the values of the parameters in `generation_parameters` and `sampling_parameters`. See the vLLM documentation for more details: https://docs.vllm.ai/en/latest/dev/sampling_params.html. use_tqdm A boolean in order to display progress bar while inferencing Returns ------- The generated text, of shape `(n_batch, n_samples)`. If there are only one batch and several samples, the list is of shape `(n_samples)`. If this is a batch with several sequences but only one sample the list is of shape `(n_batch)`. If there is only one sequence and one sample, a string is returned. rr Ng? beam_searchT)rrrcSsg|] }dd|jDqS)cSsg|]}|jqSr)text).0samplerrr sz,VLLM.generate...)outputsr#batchrrrr%sz!VLLM.generate..cSsg|]}|dqS)rrr'rrrr%s)vllm.sampling_paramsr dataclassesastuple max_tokens isinstancestrstopseedlogits_processorsnbest_oftop_ptop_krepetition_penalty temperatureuse_beam_searchrgeneraterlen)rrrlogits_processorrrrr r,stop_atr0sampler num_samplesr4r5r7results batch_size sample_sizerrrr9+s\ +    z VLLM.generatecOstd)zReturn a text generator. Streaming is not yet available for `vllm.LLM`. TODO: Implement the streaming functionality ourselves. z4Streaming is not available for the vLLM integration.)NotImplementedError)rargskwargsrrrstreamsz VLLM.stream adapter_pathcCs0ddlm}|durd|_dS||d||_dS)Nr) LoRARequestr)vllm.lora.requestrGr)rrFrGrrr load_loras  zVLLM.load_lora)rr )__name__ __module__ __qualname____doc__rrrr.rrrrboolr9rErIrrrrr s&  k r model_namecKs$ddlm}||fi|}t|S)a6Load a vLLM model. Arguments --------- model_name The name of the model to load from the HuggingFace hub. vllm_model_params vLLM-specific model parameters. See the vLLM code for the full list: https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/llm.py rr )vllmr r )rOvllm_model_paramsr rrrrrPs rPrrreturncsLddlm_tj_dttt fdtffdd }|_ S)aAdapt a tokenizer to use to compile the FSM. The API of Outlines tokenizers is slightly different to that of `transformers`. In addition we need to handle the missing spaces to Llama's tokenizer to be able to compile FSMs for this model. Parameters ---------- tokenizer The tokenizer of the model. Returns ------- PreTrainedTokenizerBase The adapted tokenizer. r)SPIECE_UNDERLINEtokenrRcs6|g}t|tur|s|dkrd|S|S)Nz<0x20> )convert_tokens_to_stringtyper. startswith)rTstringrSrrrconvert_token_to_strings  z0adapt_tokenizer..convert_token_to_string) transformersrS get_vocab vocabularysetall_special_tokensspecial_tokensrr.bytesr[)rr[rrZrrs     r)rrrRr)r*typingrrrroutlines.generate.apirrr\rrPr r)r r r.rrrrrs