o ۷iY@s8ddlZddlZddlmZddlmZddlmZddl m Z ddl m Z e e ZdZGdddZGd d d ZGd d d eZGd ddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdd d eZGd!d"d"eZdS)#N)sparse)add_start_docstrings) get_loggerad Args: input_ids (`jnp.ndarray` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`PreTrainedTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) scores (`jnp.ndarray` of shape `(batch_size, config.vocab_size)`): Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam search or log softmax for each vocabulary token when using beam search kwargs (`dict[str, Any]`, *optional*): Additional logits processor specific kwargs. Return: `jnp.ndarray` of shape `(batch_size, config.vocab_size)`: The processed prediction scores. c@4eZdZdZeedejdejdejfddZdS)FlaxLogitsProcessorzSAbstract base class for all logit processors that can be applied during generation. input_idsscoresreturncCt|jd)z"Flax method for processing logits.H is an abstract class. Only classes inheriting this class can be called.NotImplementedError __class__selfrr ra/home/ubuntu/vllm_env/lib/python3.10/site-packages/transformers/generation/flax_logits_process.py__call__6 zFlaxLogitsProcessor.__call__N __name__ __module__ __qualname____doc__r!LOGITS_PROCESSOR_INPUTS_DOCSTRINGjnpndarrayrrrrrr3"rc@r)FlaxLogitsWarperzjAbstract base class for all logit warpers that can be applied during generation with multinomial sampling.rr r cCr )zFlax method for warping logits.r r rrrrrArzFlaxLogitsWarper.__call__Nrrrrrr>rrc @s8eZdZdZeedejdejdedejfddZ dS) FlaxLogitsProcessorLista. This class can be used to create a list of [`FlaxLogitsProcessor`] or [`FlaxLogitsWarper`] to subsequently process a `scores` input tensor. This class inherits from list and adds a specific *__call__* method to apply each [`FlaxLogitsProcessor`] or [`FlaxLogitsWarper`] to the inputs. rr cur_lenr c s|D]C}t|jj}t|dkr?tfddt|ddDs4tdt|d|j d||||fi}q||||}q|S)Nc3s|]}|vVqdSNr).0argkwargsrr Usz3FlaxLogitsProcessorList.__call__..rz,Make sure that all the required parameters: z for z$ are passed to the logits processor.) inspect signaturer parameterslenalllistkeys ValueErrorr)rrr r!r' processor function_argsrr&rrPs &z FlaxLogitsProcessorList.__call__N) rrrrrrrrintrrrrrr Is&r c@>eZdZdZdefddZdejdejdedejfd d Z d S) FlaxTemperatureLogitsWarperz [`FlaxLogitsWarper`] for temperature (exponential scaling output probability distribution). Args: temperature (`float`): The value used to module the logits distribution. temperaturecCs*t|tr |dkstd|||_dS)Nrz:`temperature` has to be a strictly positive float, but is ) isinstancefloatr0r6)rr6rrr__init__is z$FlaxTemperatureLogitsWarper.__init__rr r!r cCs||j}|Sr#)r6rrr r!rrrros z$FlaxTemperatureLogitsWarper.__call__N) rrrrr8r9rrr3rrrrrr5`s$r5c@sReZdZdZed dfdededefddZd ejd ejd ed ejfd dZ dS)FlaxTopPLogitsWarpera= [`FlaxLogitsWarper`] that performs top-p, i.e. restricting to top tokens summing to prob_cut_off <= prob_cut_off. Args: top_p (`float`): If set to < 1, only the smallest set of most probable tokens with probabilities that add up to `top_p` or higher are kept for generation. filter_value (`float`, *optional*, defaults to -inf): All filtered values will be set to this float value. min_tokens_to_keep (`int`, *optional*, defaults to 1): Minimum number of tokens that cannot be filtered. Inftop_p filter_valuemin_tokens_to_keepcCs^t|tr |dks |dkrtd|t|tr|dkr$td|||_||_||_dS)Nrg?z.`top_p` has to be a float > 0 and < 1, but is r=z:`min_tokens_to_keep` has to be a positive integer, but is )r7r8r0r3r>r?r@)rr>r?r@rrrr9s zFlaxTopPLogitsWarper.__init__rr r!r c Cst||jd\}}t||j}tjj|ddj dd}||j k}t |d}||j dddf dO}|j ddd|jf d}t|||} tj|| d} | S)Naxisr=rT)laxtop_kshaper full_liker?jaxnnsoftmaxcumsumr>rollatsetr@where sort_key_val) rrr r! topk_scores topk_indices mask_scorescumulative_probs score_masktopk_next_scores next_scoresrrrrs  zFlaxTopPLogitsWarper.__call__N rrrrr8r3r9rrrrrrrr;ts" $ r;c@sReZdZdZed dfdededefddZd ejd ejd ed ejfd dZ dS)FlaxTopKLogitsWarpera [`FlaxLogitsWarper`] that performs top-k, i.e. restricting to the k highest probability elements. Args: top_k (`int`): The number of highest probability vocabulary tokens to keep for top-k-filtering. filter_value (`float`, *optional*, defaults to -inf): All filtered values will be set to this float value. min_tokens_to_keep (`int`, *optional*, defaults to 1): Minimum number of tokens that cannot be filtered. r<r=rEr?r@cCs6t|tr |dkrtd|t|||_||_dS)Nrz6`top_k` has to be a strictly positive integer, but is )r7r3r0maxrEr?)rrEr?r@rrrr9s  zFlaxTopKLogitsWarper.__init__rr r!r cCs|j\}}t|||j}t|j|jd}t||\}} tt||dddf||f } | } | | } |j |  | }| ||} | S)NrA) rFrfullr?minrErD broadcast_toarangeflattenrMrNreshape)rrr r! batch_size vocab_sizenext_scores_flattopkrQrRshifttopk_scores_flattopk_indices_flatrWrrrrs *  zFlaxTopKLogitsWarper.__call__NrXrrrrrYs" $rYc@s>eZdZdZdefddZdejdejdedejfd d Zd S) !FlaxForcedBOSTokenLogitsProcessorz [`FlaxLogitsProcessor`] that enforces the specified token as the first generated token. Args: bos_token_id (`int`): The id of the token to force as the first generated token. bos_token_idcCs ||_dSr#)ri)rrirrrr9s z*FlaxForcedBOSTokenLogitsProcessor.__init__rr r!r cCsNt|jtd }dt|d}t||jdd|jfd|}|SNinfr=r) rr[rFr8bool_rOrMrirNrrr r! new_scores apply_penaltyrrrrs$z*FlaxForcedBOSTokenLogitsProcessor.__call__N rrrrr3r9rrrrrrrrhs$rhc@BeZdZdZdedefddZdejdejded ejfd d Zd S) !FlaxForcedEOSTokenLogitsProcessorae [`FlaxLogitsProcessor`] that enforces the specified token as the last generated token when `max_length` is reached. Args: max_length (`int`): The maximum length of the sequence to be generated. eos_token_id (`int`): The id of the token to force as the last generated token when `max_length` is reached. max_length eos_token_idcCs||_||_dSr#)rsrt)rrsrtrrrr9s z*FlaxForcedEOSTokenLogitsProcessor.__init__rr r!r cCsTt|jtd }dt||jd}t||jdd|jf d|}|Srj) rr[rFr8rlrsrOrMrtrNrmrrrrs$z*FlaxForcedEOSTokenLogitsProcessor.__call__Nrprrrrrrs $rrc@rq) FlaxMinLengthLogitsProcessora3 [`FlaxLogitsProcessor`] enforcing a min-length by setting EOS probability to 0. Args: min_length (`int`): The minimum length below which the score of `eos_token_id` is set to `-float("Inf")`. eos_token_id (`int`): The id of the *end-of-sequence* token. min_lengthrtcCsPt|tr |dkrtd|t|tr|dkr td|||_||_dS)Nrz2`min_length` has to be a positive integer, but is z4`eos_token_id` has to be a positive integer, but is )r7r3r0rvrt)rrvrtrrrr9s  z%FlaxMinLengthLogitsProcessor.__init__rr r!r cCsFdt||jdd}t||jdd|jftd |}|S)Nr=rrk)rcliprvrOrMrtrNr8rrr r!rorrrrs*z%FlaxMinLengthLogitsProcessor.__call__Nrprrrrrus $ ruc@s&eZdZdZddZdefddZdS)(FlaxSuppressTokensAtBeginLogitsProcessora [`FlaxLogitsProcessor`] suppressing a list of tokens as soon as the `generate` function starts generating using `begin_index` tokens. This should ensure that the tokens defined by `begin_suppress_tokens` are not sampled at the beginning of the generation. Args: begin_suppress_tokens (`list[int]`): Tokens to not sample. begin_index (`int`): Index where the tokens are suppressed. cCst||_||_dSr#)r.begin_suppress_tokens begin_index)rrzr{rrrr9s  z1FlaxSuppressTokensAtBeginLogitsProcessor.__init__r!cCsBdt||j}t||jdd|jftd |}|S)Nr=rk)rrlr{rOrMrzrNr8rxrrrr!s*z1FlaxSuppressTokensAtBeginLogitsProcessor.__call__N)rrrrr9r3rrrrrrys ryc@r4) !FlaxSuppressTokensLogitsProcessorz [`FlaxLogitsProcessor`] suppressing a list of tokens at each decoding step. The processor will set their log probs to be `-inf` so they are not sampled. Args: suppress_tokens (`list`): Tokens to not sample. suppress_tokenscCst||_dSr#)r.r})rr}rrrr93sz*FlaxSuppressTokensLogitsProcessor.__init__rr r!r cCs |jd|jftd }|S)N.rk)rMr}rNr8r:rrrr6sz*FlaxSuppressTokensLogitsProcessor.__call__N) rrrrr.r9rrr3rrrrrr|)s $r|c@s8eZdZdZddZdejdejdedejfdd Zd S) FlaxForceTokensLogitsProcessora [`FlaxLogitsProcessor`] that takes a list of pairs of integers which indicates a mapping from generation indices to token indices that will be forced before sampling. The processor will set their log probs to 0 and all other tokens to `-inf` so that they are sampled at their corresponding index. Args: force_token_map (`list`): Map giving token ids and indices where they will be forced to be sampled. cCsbt|}tjt|dtjdd}|D]\}}|dur(|j||}qt||_ dS)Nr=dtyperA) dictronesrZr/int32itemsrMrNforce_token_array)rforce_token_maprindextokenrrrr9Gs z'FlaxForceTokensLogitsProcessor.__init__rr r!r csBfddtjjdkfddfddS)NcsXjd}j|}tjjdtd }tj|dfjd}t||d|f}|S)Nrrrkr=) rFrr ones_likerr8zerosrDdynamic_update_slice)generation_idxra current_tokenrnupdates)r rrr _force_tokenSs  z=FlaxForceTokensLogitsProcessor.__call__.._force_tokenrcSr#rrr rr_z9FlaxForceTokensLogitsProcessor.__call__..cs*tjdkfddfddS)NrcsSr#rr)rr!rrrdszKFlaxForceTokensLogitsProcessor.__call__....crr#rrrrrrfr)rDcondrrrr!r rrrras    )rDrrrFr:rrrrRs   z'FlaxForceTokensLogitsProcessor.__call__N) rrrrr9rrr3rrrrrr~<s $ r~c@s eZdZdZddZddZdS)#FlaxWhisperTimeStampLogitsProcessora{ Whisper specific Processor. This processor can be used to force a list of tokens. The processor will set their log probs to `inf` so that they are sampled at their corresponding index. Args: generate_config (`GenerateConfig`): The generate config used to generate the output. The following parameters are required: eos_token_id (`int`, *optional*, defaults to 50257): The id of the *end-of-sequence* token. no_timestamps_token_id (`int`, *optional*, defaults to 50363): The id of the `"<|notimestamps|>"` token. max_initial_timestamp_index (`int`, *optional*, defaults to 1): Used to set the maximum value of the initial timestamp. This is used to prevent the model from predicting timestamps that are too far in the future. cCsp|j|_|j|_|jd|_|d|_|jr|jd7_t|dr'|j|_n|j|_|jdur6|j|_dSdS)Nr=rmax_initial_timestamp_index)rtno_timestamps_token_idtimestamp_beginr{is_multilingualhasattrrrb)rgenerate_config model_configdecoder_input_lengthrrrr9}s      z,FlaxWhisperTimeStampLogitsProcessor.__init__c s|jddjftd }fdd}t|||}tjkdd}tj du|d}j j }t||jdd|ddftd |}tj j |dd}fd d }t|||}|S) Nrkc stjdkdd}t|djk|d}tjdkdd}t|djkd|}t|t|dk|jjdtd |jdjtd |S)Nr=TFrrrk)rrOr{rrMrNr8rt) input_ids_kscores_klast_was_timestamppenultimate_was_timestampr!rrr handle_pairss*zBFlaxWhisperTimeStampLogitsProcessor.__call__..handle_pairsTFr=rArBcsXtjj|jddd}t|dj}t||k|jdjt d |S)NrArBrk) rHrI logsumexprrrZrOrMrNr8) logprobs_krtimestamp_logprobmax_text_token_logprob)rrrhandle_cumulative_probsszMFlaxWhisperTimeStampLogitsProcessor.__call__..handle_cumulative_probs) rMrrNr8rHvmaprrOr{rrrI log_softmax) rrr r!rapply_max_initial_timestamp last_allowedlogprobsrrrrrs&  $  z,FlaxWhisperTimeStampLogitsProcessor.__call__N)rrrrr9rrrrrrls rc@sleZdZdZdefddZdejdedefdd Zd ejd ejfd d Z dejdejded ejfddZ dS) FlaxNoRepeatNGramLogitsProcessora9 [`FlaxLogitsProcessor`] that enforces no repetition of n-grams. See [Fairseq](https://github.com/pytorch/fairseq/blob/a07cb6f40480928c9e0548b737aadd36ee66ac76/fairseq/sequence_generator.py#L345). Args: ngram_size (`int`): All ngrams of size `ngram_size` can only occur once. ngram_sizecCs*t|tr |dkrtd|||_dS)Nrz;`ngram_size` has to be a strictly positive integer, but is )r7r3r0r)rrrrrr9s z)FlaxNoRepeatNGramLogitsProcessor.__init__rrbr!c sj\}|jd}|jd}fdd}|jdf}tjd||tj|jd} t||k d} t j | | ff|fjdS)a get a matrix of size (batch_size,) + (vocab_size,)*n (for n-grams) that represent the n-grams that occurred previously. The BCOO representation allow to store only the few non-zero entries, instead of the full (huge) matrix r=csD|||j|tgfddtjDS)Ncs"g|] }t|fqSr)rarray)r$j)brposrr s"zZFlaxNoRepeatNGramLogitsProcessor.get_previous_ngrams..body_fun..)rMrNrrranger)ivalrarr)rrrbody_funs zFFlaxNoRepeatNGramLogitsProcessor.get_previous_ngrams..body_funrrfloat32)rF) rFrrHrD fori_looprrrr^astyperBCOO) rrrbr!seq_len seq_ngrams cur_ngramsrrFall_update_indicesdatarrrget_previous_ngramss   z4FlaxNoRepeatNGramLogitsProcessor.get_previous_ngrams latest_tokensr cCs$tjtjdd}t|||S)zt Determines which tokens must be banned given latest tokens and the previously seen ngrams. cSs |t|Sr#)tuple)rprevious_ngramsrrrinner_fns zIFlaxNoRepeatNGramLogitsProcessor.get_banned_tokens_mask..inner_fn)rsparsifyrHr bcoo_todense)rrrrrrrget_banned_tokens_masks z7FlaxNoRepeatNGramLogitsProcessor.get_banned_tokens_maskr cs8fdd}tjjdk|fdd}|S)Nc sj\}}|}tjjdjdfjd}tj|tj djdfjdjdfd} || d}t |t d S)Nrr=r)rrboolrk)rFrrrrrrHrDr dynamic_slicerrrOr8)_rbrrbanned_tokens_indices_maskr!rr rrrtrue_fn s  $ z:FlaxNoRepeatNGramLogitsProcessor.__call__..true_fnr=crr#rrrrrrrz;FlaxNoRepeatNGramLogitsProcessor.__call__..)rHrDrr)rrr r!routputrrrr s"z)FlaxNoRepeatNGramLogitsProcessor.__call__N) rrrrr3r9rrrrrrrrrrs  "$ r)r)rHjax.laxrD jax.numpynumpyrjax.experimentalrutilsr utils.loggingrrloggerrrrr.r r5r;rYrhrrruryr|r~rrrrrrs.       ,#0a