o
Ni
@ s d dl mZmZmZmZmZmZmZmZm Z d dl
Z d dl Z d dl
Z
d dlZd dlmZmZmZ d dlZd dlmZmZmZmZ d dlmZ d dlmZ d dl Z dd Z G dd d eZ G d
d d eZ d d
Z!e"dkrd dl m Z d dl m#Z#m$Z$ e j%e _%e j&e _&e$j'dej(dZ)e#'dZ*e)+dZ)e), Z)dZ-e*e-dddj.Z.e.+dZ.e/ E e)j%e.ddddd d dd d Z0e1e*j2e0dd e)j%e.ddddd d dd dd
Z3d!Z4e3D ]
Z5e*j2e5dd Z6e4e67 Z4qe1e4 W d dS 1 sw Y dS dS )" ) GenerationConfigGenerationMixinLogitsProcessorListStoppingCriteriaListDisjunctiveConstraintBeamSearchScorerPhrasalConstraintConstrainedBeamSearchScorerPreTrainedModelN)GenerateOutput SampleOutputlogger)CallableListOptionalUnion)nnc C sP | dkrd S t | t j rt j| tj| t| dt jj _
d S )NT) torch manual_seedcuda is_availablemanual_seed_allnprandomseedbackendscudnn
deterministic)r r V/home/ubuntu/.local/lib/python3.10/site-packages/transformers_stream_generator/main.py
setup_seed s
r! c s e Zd Z fddZ ZS )StreamGenerationConfigc s$ t jdi | |dd| _d S )N do_streamFr )super__init__popr# )selfkwargs __class__r r r% $ s z StreamGenerationConfig.__init__)__name__
__module__ __qualname__r%
__classcell__r r r) r r" # s r" c @ s" e Zd Ze ddeej dee dee dee dee
e ejge e f d ee
d
eeejf fd d Ze dd
ejdee dee dee dee dee deee e e f dee
dee
dee
dee
d ee
d
eeejf fddZdS )NewGenerationMixinNFr inputsgeneration_configlogits_processorstopping_criteriaprefix_allowed_tokens_fn synced_gpusreturnc % s
t | | du r&| jjr#t| j} | | jkr#td | | _| j t
j dCi |}
|dur9|nt }|durB|nt
} jdu rt jdurt|
dddu r\td j} t| trh| d } td| d | _| | j|
\} }
}
| jd } j|
d< j|
d < j|
d
< dtt | j j v }d |
v}|
dddu r|r|r| !| j j|
d< | jj"sو jdurt#$| ddd f jkdkrtd
| jj"rd |
vr| %| |
|
}
| jj"r| j&| j' j|
| j(d}n| }|jd }|ddu o
j)du}|r# j*du r#td j) dt+ n |r3 j*dur3 j*| _)n
|s@ j*dur@t,d j-durZ j- j)krZt,d j- d j) d| j)kry| jj"rgdnd}td| d| d j) d t. d s|d d _/ j0dup j1duo j/d u } j2duo j2d ko j3d u o j4duo j4dko j/d u } j5d ko҈ j6d ko҈ j3d u o| o| o؈ j/d u } j5d ko j6d ko j3d u o j/d u o| o| o j/d u } j5d ko j6d ko j/d u o| o| } j5d ko/ j6d ko/ j3d u o/| o/| o5 j/d u } j5d koN j6d koN j3d u oN| oN| oT j/d u } j5d kog j6d kog| og| om j/d u } j6 j5kryt,d | r j3d u rt,d!| j(j7|j(j7krtd"|j(j7 d#| j(j7 d$| j(j7 d%t+ | j8 || ||d&}| j9 |d'}|r݈ j:d krt,d( j: d)| j;|f|| j j j< j=|d*|
S |r j:d krt,d( j: d+| j>|f j2 j4|| j j j< j=|d, |
S |r7| ? } | j@dC| j:| jj"d-|
\}}
| jA|f|| | j j j< j=|d.|
S |rf| ? } | j@dC| j:| jj"d-|
\}}
| jB|f|| | j j j< j=|d.|
S |r j: j5krtt,d/|j)du r~t,d0tC| j5| j( jD jE j:d1} | j@dC| j5| jj"d-|
\}}
| jF|| f|| j j j< j=|d*|
S |r| ? } |j)du rt,d0tC| j: j5| j( jD jEd2} | j@dC| j5 j: | jj"d-|
\}}
| jG|| f|| | j j j< j=|d.|
S | rt j: j5krt,d/ j5 j6 dkr t,d3|j)du r&t,d0|d4du o2 jHd5k} | s:t,d6tC| j5|j)| j( jD jE j: j6d7} | j@dC| j5| jj"d-|
\}}
| jI|| f|| j j j< j=|d*|
S |rr j: j5krt,d/|j)du rt,d0 j5d krt,d8 j3rt,d9 j6dur j6d krt,d:g } j0dur j0} j1dur; fd;d<}!t j1trtJ j1dkr|! j1D ]`}"t|"d trt|"trtJ|"dkr|! tKd=d> |"D r|! tKd?d> |"D r|! tL|"}#n!t|"tr tJ|"dkr#|! tKd@d> |"D r0|! tM|"}#| N|# qtO| | j5| j( jD jE j:dA}$| j@dC| j5| jj"d-|
\}}
| jP|f|$|| j j j< j=|dB|
S dS )Da
Generates sequences of token ids for models with a language modeling head.
Most generation-controlling parameters are set in `generation_config` which, if not passed, will be set to the
model's default generation configuration. You can override any `generation_config` by passing the corresponding
parameters to generate(), e.g. `.generate(inputs, num_beams=4, do_sample=True)`.
For an overview of generation strategies and code examples, check out the [following
guide](./generation_strategies).
Parameters:
inputs (`torch.Tensor` of varying shape depending on the modality, *optional*):
The sequence used as a prompt for the generation or as model inputs to the encoder. If `None` the
method initializes it with `bos_token_id` and a batch size of 1. For decoder-only models `inputs`
should of in the format of `input_ids`. For encoder-decoder models *inputs* can represent any of
`input_ids`, `input_values`, `input_features`, or `pixel_values`.
generation_config (`~generation.GenerationConfig`, *optional*):
The generation configuration to be used as base parametrization for the generation call. `**kwargs`
passed to generate matching the attributes of `generation_config` will override them. If
`generation_config` is not provided, the default will be used, which had the following loading
priority: 1) from the `generation_config.json` model file, if it exists; 2) from the model
configuration. Please note that unspecified parameters will inherit [`~generation.GenerationConfig`]'s
default values, whose documentation should be checked to parameterize generation.
logits_processor (`LogitsProcessorList`, *optional*):
Custom logits processors that complement the default logits processors built from arguments and
generation config. If a logit processor is passed that is already created with the arguments or a
generation config an error is thrown. This feature is intended for advanced users.
stopping_criteria (`StoppingCriteriaList`, *optional*):
Custom stopping criteria that complement the default stopping criteria built from arguments and a
generation config. If a stopping criteria is passed that is already created with the arguments or a
generation config an error is thrown. This feature is intended for advanced users.
prefix_allowed_tokens_fn (`Callable[[int, torch.Tensor], List[int]]`, *optional*):
If provided, this function constraints the beam search to allowed tokens only at each step. If not
provided no constraint is applied. This function takes 2 arguments: the batch ID `batch_id` and
`input_ids`. It has to return a list with the allowed tokens for the next generation step conditioned
on the batch ID `batch_id` and the previously generated tokens `inputs_ids`. This argument is useful
for constrained generation conditioned on the prefix, as described in [Autoregressive Entity
Retrieval](https://arxiv.org/abs/2010.00904).
synced_gpus (`bool`, *optional*, defaults to `False`):
Whether to continue running the while loop until max_length (needed for ZeRO stage 3)
kwargs:
Ad hoc parametrization of `generate_config` and/or additional model-specific kwargs that will be
forwarded to the `forward` function of the model. If the model is an encoder-decoder model, encoder
specific kwargs should not be prefixed and decoder specific kwargs should be prefixed with *decoder_*.
Return:
[`~utils.ModelOutput`] or `torch.LongTensor`: A [`~utils.ModelOutput`] (if `return_dict_in_generate=True`
or when `config.return_dict_in_generate=True`) or a `torch.FloatTensor`.
If the model is *not* an encoder-decoder model (`model.config.is_encoder_decoder=False`), the possible
[`~utils.ModelOutput`] types are:
- [`~generation.GreedySearchDecoderOnlyOutput`],
- [`~generation.SampleDecoderOnlyOutput`],
- [`~generation.BeamSearchDecoderOnlyOutput`],
- [`~generation.BeamSampleDecoderOnlyOutput`]
If the model is an encoder-decoder model (`model.config.is_encoder_decoder=True`), the possible
[`~utils.ModelOutput`] types are:
- [`~generation.GreedySearchEncoderDecoderOutput`],
- [`~generation.SampleEncoderDecoderOutput`],
- [`~generation.BeamSearchEncoderDecoderOutput`],
- [`~generation.BeamSampleEncoderDecoderOutput`]
Na% You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation)attention_maskzThe attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.r z)Setting `pad_token_id` to `eos_token_id`:z for open-end generation.output_attentionsoutput_hidden_states use_cacheencoder_outputsr zA decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.)decoder_start_token_id bos_token_id model_kwargsdevice
max_lengthzUNeither `max_length` nor `max_new_tokens` has been set, `max_length` will default to z (`generation_config.max_length`). Controlling `max_length` via the config is deprecated and `max_length` will be removed from the config in v5 of Transformers -- we recommend using `max_new_tokens` to control the maximum length of the generation.a. Both `max_new_tokens` and `max_length` have been set but they serve the same purpose -- setting a limit to the generated output length. Remove one of those arguments. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)z3Unfeasible length constraints: the minimum length (z%) is larger than the maximum length ()decoder_input_ids input_idszInput length of z is z , but `max_length` is set to zX. This can lead to unexpected behavior. You should consider increasing `max_new_tokens`.r# F Tz;`num_beam_groups` has to be smaller or equal to `num_beams`zbDiverse beam search cannot be used in sampling mode. Make sure that `do_sample` is set to `False`.z~You are calling .generate() with the `input_ids` being on a device type different than your model's device. `input_ids` is on z, whereas the model is on z. You may experience unexpected behaviors or slower generation. Please make sure that you have put `input_ids` to the correct device by calling for example input_ids = input_ids.to('z ') before running `.generate()`.)r1 input_ids_seq_lengthencoder_input_idsr4 r2 )r1 r3 z)num_return_sequences has to be 1, but is z when doing greedy search.)r2 r3 pad_token_id eos_token_id
output_scoresreturn_dict_in_generater5 z when doing contrastive search.) top_k
penalty_alphar2 r3 rG rH rI rJ r5 )rC expand_sizeis_encoder_decoder)r2
logits_warperr3 rG rH rI rJ r5 zA`num_return_sequences` has to be smaller or equal to `num_beams`.z5`max_length` needs to be a stopping_criteria for now.)
batch_size num_beamsr? length_penaltydo_early_stoppingnum_beam_hyps_to_keep)rP rQ r? rR rS zK`num_beams` should be divisible by `num_beam_groups` for group beam search. typical_p ?z?Decoder argument `typical_p` is not supported with beam groups.)rP rQ r@ r? rR rS rT num_beam_groupszB`num_beams` needs to be greater than 1 for constrained generation.z9`do_sample` needs to be false for constrained generation.z?`num_beam_groups` not supported yet for constrained generation.c s t d j d)Nzn`force_words_ids` has to either be a `List[List[List[int]]]` or `List[List[int]]`of positive integers, but is .)
ValueErrorforce_words_idsr r1 r r typeerror s
z.NewGenerationMixin.generate..typeerrorc s s | ] }t |t V qd S N)
isinstancelist.0 token_idsr r r s
z.NewGenerationMixin.generate..c s s" | ] }t d d |D V qdS )c s $ | ]
}t |t p
|d k V qdS r Nr^ intra token_idr r r rc
z8NewGenerationMixin.generate...N)anyr` r r r rc s
c s rd re rf rh r r r rc rj ) constraintsrP rQ r? rR rS rT )constrained_beam_scorerr2 r3 rG rH rI rJ r5 r )Qr! _validate_model_classr1 _from_model_configr" from_model_configconfigwarningswarncopydeepcopyupdater r rG rH getr
warningr^ r_ _prepare_model_inputsr= shaper8 r9 r: setinspect signatureforward
parameterskeys&_prepare_attention_mask_for_generationrN r sum._prepare_encoder_decoder_kwargs_for_generation)_prepare_decoder_input_ids_for_generationr< r? r@ max_new_tokens UserWarningrY
min_lengthhasattrr# rl rZ rK do_samplerL rQ rW type_get_logits_processor_get_stopping_criterianum_return_sequences
greedy_searchrI rJ contrastive_search_get_logits_warper _expand_inputs_for_generationsample
sample_streamr rR early_stopping beam_search beam_samplerU group_beam_searchlenrk r r appendr constrained_beam_search)%r' r0 r1 r2 r3 r4 r5 r r( new_generation_configr> rH
inputs_tensormodel_input_namerP accepts_attention_maskrequires_attention_maskrC rE has_default_max_lengthinput_ids_stringis_constraint_gen_mode is_contrastive_search_gen_modeis_greedy_gen_modeis_sample_gen_modeis_sample_gen_stream_modeis_beam_gen_modeis_beam_sample_gen_modeis_group_beam_gen_moderO beam_scorerhas_default_typical_pfinal_constraintsr\ word_ids
constraintrm r r[ r generate* s T
zNewGenerationMixin.generaterC rO r@ rG rH r8 r9 rI rJ c
+ s. |dur|nt }|dur|nt }|dur"tdt t||}|dur(|nt }|dur1|n| jj}|dur;|n| jj}t |t
rG|g}|
durM|
n| jj }
|durW|n| jj }| dura| n| jj
} | durk| n| jj} | ru|
rudnd}| r}|r}dnd}| r|rdnd}| r| rdnd}||jd d}d} | rt|rdnd |j}tj|tjjd
| dkrdS | j|fi |
}| di |d|| d }| r|rq|j ddd ddf }|||}|||}| r'|
r||f7 }|r|| j j r|j fn|j f7 }| j j r||j!f7 }| r'|| j j r"|j"fn|j#f7 }t$j%j&|d d
}tj'|dd(d |durQ|du rGt)d | |d| V tj*| dddf gd d
}| j+||
| j j d}
|dur|,t- fdd|D . }|/ dks|||r| sdS d}q)a
Generates sequences of token ids for models with a language modeling head using **multinomial sampling** and
can be used for text-decoder, text-to-text, speech-to-text, and vision-to-text models.
In most cases, you do not need to call [`~generation.GenerationMixin.sample`] directly. Use generate() instead.
For an overview of generation strategies and code examples, check the [following
guide](./generation_strategies).
Parameters:
input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
The sequence used as a prompt for the generation.
logits_processor (`LogitsProcessorList`, *optional*):
An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`]
used to modify the prediction scores of the language modeling head applied at each generation step.
stopping_criteria (`StoppingCriteriaList`, *optional*):
An instance of [`StoppingCriteriaList`]. List of instances of class derived from [`StoppingCriteria`]
used to tell if the generation loop should stop.
logits_warper (`LogitsProcessorList`, *optional*):
An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsWarper`] used
to warp the prediction score distribution of the language modeling head applied before multinomial
sampling at each generation step.
max_length (`int`, *optional*, defaults to 20):
**DEPRECATED**. Use `logits_processor` or `stopping_criteria` directly to cap the number of generated
tokens. The maximum length of the sequence to be generated.
pad_token_id (`int`, *optional*):
The id of the *padding* token.
eos_token_id (`int`, *optional*):
The id of the *end-of-sequence* token.
output_attentions (`bool`, *optional*, defaults to `False`):
Whether or not to return the attentions tensors of all attention layers. See `attentions` under
returned tensors for more details.
output_hidden_states (`bool`, *optional*, defaults to `False`):
Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors
for more details.
output_scores (`bool`, *optional*, defaults to `False`):
Whether or not to return the prediction scores. See `scores` under returned tensors for more details.
return_dict_in_generate (`bool`, *optional*, defaults to `False`):
Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
synced_gpus (`bool`, *optional*, defaults to `False`):
Whether to continue running the while loop until max_length (needed for ZeRO stage 3)
model_kwargs:
Additional model specific kwargs will be forwarded to the `forward` function of the model. If model is
an encoder-decoder model the kwargs should include `encoder_outputs`.
Return:
[`~generation.SampleDecoderOnlyOutput`], [`~generation.SampleEncoderDecoderOutput`] or `torch.LongTensor`:
A `torch.LongTensor` containing the generated tokens (default behaviour) or a
[`~generation.SampleDecoderOnlyOutput`] if `model.config.is_encoder_decoder=False` and
`return_dict_in_generate=True` or a [`~generation.SampleEncoderDecoderOutput`] if
`model.config.is_encoder_decoder=True`.
Examples:
```python
>>> from transformers import (
... AutoTokenizer,
... AutoModelForCausalLM,
... LogitsProcessorList,
... MinLengthLogitsProcessor,
... TopKLogitsWarper,
... TemperatureLogitsWarper,
... StoppingCriteriaList,
... MaxLengthCriteria,
... )
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("gpt2")
>>> # set pad_token_id to eos_token_id because GPT2 does not have a EOS token
>>> model.config.pad_token_id = model.config.eos_token_id
>>> model.generation_config.pad_token_id = model.config.eos_token_id
>>> input_prompt = "Today is a beautiful day, and"
>>> input_ids = tokenizer(input_prompt, return_tensors="pt").input_ids
>>> # instantiate logits processors
>>> logits_processor = LogitsProcessorList(
... [
... MinLengthLogitsProcessor(15, eos_token_id=model.generation_config.eos_token_id),
... ]
... )
>>> # instantiate logits processors
>>> logits_warper = LogitsProcessorList(
... [
... TopKLogitsWarper(50),
... TemperatureLogitsWarper(0.7),
... ]
... )
>>> stopping_criteria = StoppingCriteriaList([MaxLengthCriteria(max_length=20)])
>>> torch.manual_seed(0) # doctest: +IGNORE_RESULT
>>> outputs = model.sample(
... input_ids,
... logits_processor=logits_processor,
... logits_warper=logits_warper,
... stopping_criteria=stopping_criteria,
... )
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Today is a beautiful day, and a wonderful day.\n\nI was lucky enough to meet the']
```Nz`max_length` is deprecated in this function, use `stopping_criteria=StoppingCriteriaList(MaxLengthCriteria(max_length=max_length))` instead.r r rD FTg rV )op) return_dictr8 r9 r )dim) num_sampleszGIf `eos_token_id` is defined, make sure that `pad_token_id` is defined.)rN c 3 s | ]} |kV qd S r] r )ra i next_tokensr r rc s z3NewGenerationMixin.sample_stream..)0r r rr rs r validate_stopping_criteriar1 rG rH r^ rg rI r8 r9 rJ newrz fill_r tensortor? dist
all_reduceReduceOpSUMitem prepare_inputs_for_generationlogitsrq rN decoder_attentions
attentionscross_attentionsdecoder_hidden_states
hidden_statesr
functionalsoftmax multinomialsqueezerY cat#_update_model_kwargs_for_generationmulr longmax)r' rC r2 r3 rO r@ rG rH r8 r9 rI rJ r5 r> scoresr r r unfinished_sequencesthis_peer_finishedthis_peer_finished_flag model_inputsoutputsnext_token_logitsnext_token_scoresprobsr r r r s ~
z NewGenerationMixin.sample_stream)NNNNNFr ) NNNNNNNNNNF)r+ r, r- r no_gradr Tensorr" r r r rg r boolr r
LongTensorr r r r r r r r/ ) s ,
r/ c C s t jt_t jt_d S r] )r/ r r
r r r r r init_stream_support s r __main__)r
)
AutoTokenizerAutoModelForCausalLMzbigscience/bloom-560m) torch_dtypezcuda:0zhello?
ptF)return_tensorsadd_special_tokens T g333333?gffffff?g333333?)r r rK top_p temperaturerepetition_penaltyr r )skip_special_tokens) r r rK r r r r r r# )7 transformersr r r r r r r r r
numpyr r rr r| transformers.generation.utilsr r r
r typingr r r r r torch.distributed distributedr rt r! r" r/ r r+ r r r r from_pretrainedfloat16model tokenizerr eval prompt_textrC r resultprintdecode generator
stream_resultxchunkr r r r s , J
"