o پiF9@sddlZddlZddlZddlmZmZmZmZmZm Z m Z m Z m Z ddl mZddlmZmZddlmZer@ddlmZmZGdddeZGd d d e d d ZGd ddZ ddedeedefddZdS)N) TYPE_CHECKINGDictIteratorListOptionalSetTuple TypedDictUnion)Unpack)GenerationParametersSamplingParameters) Tokenizer)LlamaLogitsProcessorListc @seZdZdddZdeedeefddZ dd eeeefd e d e de eeeeffd dZ dedefddZ ddZ ddZddZddZdS)LlamaCppTokenizermodelrcCs||_||jg|_|j|_t|_t|_ ||_d|_ z|j j |_ |j j |_ WntyPt|D]}||g}||j |<q>Ynwddt|j dddD|_ d|_dS)NcSsi|]\}}||qSr).0toktok_idrrL/home/ubuntu/.local/lib/python3.10/site-packages/outlines/models/llamacpp.py 0sz.LlamaCppTokenizer.__init__..cSs|dS)Nr)xrrr2sz,LlamaCppTokenizer.__init__..)key) token_eos eos_token_id tokenizerdecode eos_token pad_token_idsetspecial_tokensdict vocabulary _hf_tokenizer tokenizer_ hf_tokenizer get_vocabAttributeErrorrangen_vocabsorteditems_hash)selfrt token_piecerrr__init__s(     zLlamaCppTokenizer.__init__ token_idsreturncCs|j|}|jdddgS)Nutf-8ignoreerrors)r detokenizer )r1r5 decoded_bytesrrrr 7s zLlamaCppTokenizer.decodeTpromptadd_bosspecialcsHt|tr tdjj|jddd||d}fdd|D}||fS)Nz=llama-cpp-python tokenizer doesn't support batch tokenizationr7r8r9)r>r?csg|] }|jkr dndqS)rr)r")rtoken_idr1rr Fsz,LlamaCppTokenizer.encode..) isinstancelistNotImplementedErrorrtokenizeencode)r1r=r>r?r5attention_maskrrArrG;s  zLlamaCppTokenizer.encodetokencCsF|jdur!ddlm}|j|g}||s|dkrd|}|S|S)Nr)SPIECE_UNDERLINEz<0x20> )r'transformers.file_utilsrJconvert_tokens_to_string startswith)r1rIrJ token_strrrrconvert_token_to_stringKs  z)LlamaCppTokenizer.convert_token_to_stringcCst|tsdS||kS)NF)rCr __getstate__)r1otherrrr__eq__Vs zLlamaCppTokenizer.__eq__cCs |jdur tt||_|jSN)r0hashpickledumpsrArrr__hash__[s zLlamaCppTokenizer.__hash__cCs|j|j|j|jt|jfS)z3Create a stable representation for outlines.caching)r&rr!r"r.r$rArrrrQ`s zLlamaCppTokenizer.__getstate__cCstd)Nz(Cannot load a pickled llamacpp tokenizer)rE)r1staterrr __setstate__jszLlamaCppTokenizer.__setstate__Nrr)TT)__name__ __module__ __qualname__r4rintstrr r boolrrGrPrSrXrQrZrrrrrs$    rc@seZdZUeeed<eed<eed<eed<eed<eed<eed<ded <eeee efed <eed <eed <eed <eed<eed<eed<eed<eed<e ed<dS)LlamaCppParamssuffix temperaturetop_pmin_p typical_pseed max_tokensrlogits_processorstopfrequence_penaltypresence_penaltyrepeat_penaltytop_ktfs_z mirostat_mode mirostat_tau mirostat_etastreamN) r\r]r^rr`__annotations__floatr_r rrarrrrrbns&  rbF)totalc @seZdZdZdddZeddZded ed e e fd d Z d e e ee fded ed e e de f ddZd e e ee fded ed e e dee f ddZde fddZdS)LlamaCppaERepresents a model provided by the `llama-cpp-python` library. We wrap models from model providing libraries in order to give all of them the same interface in Outlines and allow users to easily switch between providers. This class wraps the `llama_cpp.Llama` class from the `llama-cpp-python` library. rrcCs ||_dSrT)r)r1rrrrr4s zLlamaCpp.__init__cCs t|jSrT)rrrArrrrs zLlamaCpp.tokenizergeneration_parameterssampling_parametersllama_cpp_paramscKsjddlm}t|\}}}d|vr||d<d|vr||d<d|vr2|dur+d|d<n|d|d<n|dd|d<t|\} } } } } | d krLtd | dkrTtd d |vre| dura| |d <nd |d <d|vrmd|d<d|vr~| durz| |d<nd|d<d|vr| dur| |d<nd |d<d|vrd |d<d|d<|durd|vr|d||S||g|d<|S)zfPrepare the generation parameters. `llama-cpp-python` uses different default values r)rrkrhriNr beam_searchz.token_generator)rCr`rErrr)r1rryrrzr{rrrrrt's % zLlamaCpp.stream adapter_pathcCs"|jj|drtd|dS)Nr~z%Failed to apply LoRA from lora path: )r_modelapply_lora_from_file RuntimeError)r1rrrr load_loraes zLlamaCpp.load_loraNr[)r\r]r^__doc__r4propertyrr r r rbrr r`rrrrtrrrrrrxsF   ] 6 >rxrepo_idfilenamer6cKsZddlm}d|vrd|d<d|vrd|d<d|vrtd|j||fi|}t|S)aLoad a model from the `llama-cpp-python` library. We use the `Llama.from_pretrained` classmethod that downloads models directly from the HuggingFace hub, instead of asking users to specify a path to the downloaded model. One can still load a local model by initializing `llama_cpp.Llama` directly. Arguments --------- repo_id The name of the model repository. filename: A filename of glob pattern to match the model file in the repo. llama_cpp_model_params Llama-specific model parameters. See the `llama-cpp-python` documentation for the full list: https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.__init__ r)rn_ctxverboseFraXThe pre-tokenizer in `llama.cpp` handles unicode improperly (https://github.com/ggerganov/llama.cpp/pull/5613) Outlines may raise a `RuntimeError` when building the regex index. To circumvent this error when using `models.llamacpp()` you may pass the argument`tokenizer=llama_cpp.llama_tokenizer.LlamaHFTokenizer.from_pretrained()` )rrwarningswarnfrom_pretrainedrx)rrllamacpp_model_paramsrrrrrllamacppms rrT)rrVrtypingrrrrrrrr r typing_extensionsr outlines.generate.apir r outlines.models.tokenizerrrrrrrbrxr`rrrrrs*,  Ul