o پi; @sddlZddlZddlmZmZmZmZmZmZddl m Z m Z ddl m Z er:ddlZddlmZmZddlmZdgZeedd fZd d ZGd d d e ZGdddZdiiddfdedeededefddZdiifdedeededefddZdS)N) TYPE_CHECKINGIteratorListOptionalTupleUnion)GenerationParametersSamplingParameters) Tokenizer)PreTrainedModelPreTrainedTokenizer)OutlinesLogitsProcessor transformers)torch.DoubleTensorr.cCszddlm}WntyGddd}Ynwzddlm}Wnty1Gddd}Ynwzddlm}WntyJGd d d }Ynwzdd lm}WntycGd d d }Ynw||||fS)zGet all the Llama tokenizer types/classes that need work-arounds. When they can't be imported, a dummy class is created. r)LlamaTokenizerc@ eZdZdS)z1get_llama_tokenizer_types..LlamaTokenizerN__name__ __module__ __qualname__rrP/home/ubuntu/.local/lib/python3.10/site-packages/outlines/models/transformers.pyrr)LlamaTokenizerFastc@r)z5get_llama_tokenizer_types..LlamaTokenizerFastNrrrrrr%rr)CodeLlamaTokenizerc@r)z5get_llama_tokenizer_types..CodeLlamaTokenizerNrrrrrr,rr)CodeLlamaTokenizerFastc@r)z9get_llama_tokenizer_types..CodeLlamaTokenizerFastNrrrrrr3rr)transformers.models.llamar ImportErrorrtransformers.models.code_llamarr)rrrrrrrget_llama_tokenizer_typess2    rc@seZdZdZdddZdeeeefdedfd d Z d d deefd dZ dedefddZ ddZ ddZ ddZddZdS)TransformerTokenizerz@Represents a tokenizer for models in the `transformers` library. tokenizerr cKs~||_|jj|_|jj|_|jjdur|jj|j_|j|_n |jj|_|jj|_t|jj|_|j|_ t |jt |_ dSN) r! eos_token_id eos_token pad_token_id pad_tokensetall_special_tokensspecial_tokens get_vocab vocabulary isinstanceris_llama)selfr!kwargsrrr__init__As        zTransformerTokenizer.__init__promptreturn)torch.LongTensorr3cKs2d|d<d|d<|j|fi|}|d|dfS)NTpaddingptreturn_tensors input_idsattention_maskr!)r.r1r/outputrrrencodeRszTransformerTokenizer.encode token_idsr3cCs|jj|dd}|S)NT)skip_special_tokens)r! batch_decode)r.r<textrrrdecodeZszTransformerTokenizer.decodetokencCs>ddlm}|j|g}|jr||s|dkrd|S|S)Nr)SPIECE_UNDERLINEz<0x20> )transformers.file_utilsrBr!convert_tokens_to_stringr- startswith)r.rArBstringrrrconvert_token_to_string^s z,TransformerTokenizer.convert_token_to_stringcCsJt|t|r#t|drt|dr|j|jko|j|jkS|j|jkStS)N model_namer/)r,typehasattrrIr/r!NotImplemented)r.otherrrr__eq__js  zTransformerTokenizer.__eq__cCsddlm}t||jS)Nr)Hasher)datasets.fingerprintrOhashr!)r.rOrrr__hash__ts zTransformerTokenizer.__hash__cCsd|ji}|SNr!r9r.staterrr __getstate__ys z!TransformerTokenizer.__getstate__cCs||ddSrS)r0rTrrr __setstate__}sz!TransformerTokenizer.__setstate__N)r!r )rrr__doc__r0rstrrrr;r@rHrNrRrVrWrrrrr >s     r c@s4eZdZdZ    d$ddZ d%d d d d d eed edeeffddZ d%d d d d d eed dfddZ de e e e fde dedded e e e e e e e ff ddZde e e e fde dedded ee e e e ff ddZde e e e fde dedded ef ddZddZd&d"d#ZdS)' Transformersz"Represents a `transformers` model.modelr r!r cCs||_t||_dSr")r[r r!)r.r[r!rrrr0szTransformers.__init__Nr7r3r8past_key_valuesr2ztorch.FloatTensorc Cszddl}Wn tytdYnwd|jkr dks#JJ|r,|dd}||j||ddd|d }Wdn1sFwY|j|jfS) aCompute a forward pass through the transformer model. Parameters ---------- input_ids The input token ids. Must be one or two dimensional. attention_mask The attention mask. Must be one or two dimensional. past_key_values A tuple of tuples containing the cached key and value tensors for each attention head. Returns ------- The computed logits and the new cached key and value tensors. rNzGThe `torch` library needs to be installed to use `transformers` models.).r^TF)r8 return_dictoutput_attentionsoutput_hidden_statesr\)torchrndim unsqueezeinference_moder[logitsr\)r.r7r8r\rbr:rrrforwards*    zTransformers.forwardcCs,||||\}}|ddddf}||fS)N.r^)rg)r.r7r8r\rfkv_cachenext_token_logitsrrr__call__szTransformers.__call__promptsgeneration_parameterslogits_processorr sampling_parametersc Cst|tr|j|g\}}n|j|\}}||jj||jjd}dt|jj j vr5|d=| ||||}|j ||fi|} t|trQ| d} || S)aGenerate text using `transformers`. Arguments --------- prompts A prompt or list of prompts. generation_parameters An instance of `GenerationParameters` that contains the prompt, the maximum number of tokens, stop sequences and seed. All the arguments to `SequenceGeneratorAdapter`'s `__cal__` method. logits_processor The logits processor to use when generating text. sampling_parameters An instance of `SamplingParameters`, a dataclass that contains the name of the sampler to use and related parameters as available in Outlines. Returns ------- The generated text r7r8r8r)r,rYr!r;tor[deviceinspect signaturerg parameterskeys_get_generation_kwargs_generate_output_seqsqueeze_decode_generation) r.rkrlrmrnr7r8inputsgeneration_kwargs generated_idsrrrgenerates(      zTransformers.generatec cst|tr|j|g\}}n|j|\}}||jj||jjd}dt|jj j vr6|d=| ||||}|j ||fi|} t|trR| d} t| dD]} | d| d} || VqYdS)a Temporary stream stand-in which implements stream() signature and equivalent behaviour but isn't yielded until generation completes. TODO: implement following completion of https://github.com/huggingface/transformers/issues/30810 ror8rr^N)r,rYr!r;rpr[rqrrrsrgrtrurvrwrxrangesizeselectrdry) r.rkrlrmrnr7r8rzr{r|ioutput_group_idsrrrstreams0     zTransformers.streamc Csddlm}m}m}t|\}} } t|\} } } }}|dur$td}| dur,|| |dur6||g}nd}||| | p>d| ||| dk| dkrJ| nd|jj|jj d }t |||jjd S) zR Conert outlines generation parameters into model.generate kwargs r)GenerationConfigLogitsProcessorListset_seedNi@ multinomial beam_search) max_new_tokens stop_stringsnum_return_sequencestop_ptop_k temperature do_sample num_beamsr#r%)rmgeneration_configr!) rrrr dataclassesastupleintr!r#r%dict)r.rkrlrmrnrrrrstop_atseedsampler num_samplesrrrlogits_processor_listrrrrrv+s:   z#Transformers._get_generation_kwargsc Ks|d}|jjdi|d|i|}|jjjr|}n |dd|jddf}|jp,d}|dkrGt|trG|d} |jp?d} | | | d}|S)Nr7rrrr^r) r[r}configis_encoder_decodershaperr,listrview) r.rkrzrr{r7 output_idsr|r batch_sizerrrrrwZs"     z!Transformers._generate_output_seqr| torch.TensorcsttjdkrjgdStjdkrjStjdkr2fddttDStdj)Nrrr]csg|] }j|qSr)r!r@).0rr|r.rr xsz3Transformers._decode_generation..z7Generated outputs aren't 1D, 2D or 3D, but instead are )lenrr!r@r~ TypeError)r.r|rrrryrs    zTransformers._decode_generation)r[r r!r r")r|r)rrrrXr0rr KVCacheTypergrjrrYrrr r}rrrrvrwryrrrrrZs|   2   : , /rZrIrq model_kwargstokenizer_kwargsc Cs|dus|durz ddlm}m}Wn tytdw|dur$|}|dur*|}|dur2||d<|j|fi|}|dd|j|fi|} t|| S)aInstantiate a model from the `transformers` library and its tokenizer. Parameters ---------- model_name The name of the model as listed on Hugging Face's model page. device The device(s) on which the model should be loaded. This overrides the `device_map` entry in `model_kwargs` when provided. model_kwargs A dictionary that contains the keyword arguments to pass to the `from_pretrained` method when loading the model. tokenizer_kwargs A dictionary that contains the keyword arguments to pass to the `from_pretrained` method when loading the tokenizer. Returns ------- A `TransformersModel` model instance. Nr)AutoModelForCausalLM AutoTokenizerzWThe `transformers` library needs to be installed in order to use `transformers` models. device_map padding_sideleft)rrrrfrom_pretrained setdefaultrZ) rIrqrr model_classtokenizer_classrrr[r!rrrrs$   cCs:zddlm}Wn tytdwt|||||dS)Nr)MambaForCausalLMzaThe `mamba_ssm`, `torch` and `transformer` libraries needs to be installed in order to use Mamba.)rIrqrrr)rrr)rIrqrrrrrrmambas r)rrrtypingrrrrrroutlines.generate.apirr outlines.models.tokenizerr rbrr r outlines.processorsr __all__rrr rZrYrrrrrrsT   *C 6