o پi)@sddlZddlmZmZmZmZmZerddlZGdddeZGdddZ e Z GdddZ e Z d e d ed gd ffd d Zded ed gd ffddZded ed gd ffddZGdddZeZdS)N) TYPE_CHECKINGCallableOptionalProtocolTuplec@s0eZdZUeed<        d dd Zd S) Samplersamplesnext_token_logitstorch.DoubleTensorsequence_weightsrngtorch.GeneratorreturncCsdSN)selfr r r rrE/home/ubuntu/.local/lib/python3.10/site-packages/outlines/samplers.py__call__ szSampler.__call__N)r r r r r r rr )__name__ __module__ __qualname__int__annotations__rrrrrrs rc@s.eZdZdZddZ      d dd Zd S) GreedySampleraGreedy Sampling algorithm. Greedy sampling consists in choosing the token with the largest likelihood at every step. We don't allow more than one sample. We could attribute this a meaning, for instance the k-th sample represents the k-th most likely token. In which case it would be equivalent to beam search without the sequence weights. Attributes ---------- samples The number of samples taken for each input sequence. cCs d|_dS)Nr)rrrr__init__% zGreedySampler.__init__r r r rc Cs`ddl}|jjj|dd}|j|ddd}|j|jd|jd}|||d| }|||fS) aCall the greedy sampler. Parameters ---------- next_token_logits A tensor of shape ``(n_seqs, vocab_size,)`` that represents the probability distribution of the next token over the vocabulary. sequence_weights A tensor of shape ``(n_seqs,)`` that represents the cumulative weight of each sequence. rng A random number generator. Returns ------- A tuple with an array that contains the ids of the sampled tokens of shape ``(n_seqs, 1)``, an array that contains the ancestors of each sampled id of shape ``(n_seqs,)`` and an array that contains the updated cumulative weights of each sequence of shape ``(n_seqs,)``. rNdimT)r keepdimdevicer) torchnn functional log_softmaxargmaxarangeshaper#gathersqueeze) rr r _r$logprobsnext_token_ids ancestorsweightsrrrr(s  zGreedySampler.__call__N)r r r r rr )rrr__doc__rrrrrrrsrc @sbeZdZdZ ddddddedeedeedeefd d Zd d d d dddedfddZ dS)MultinomialSamplera<Multinomial sampling algorithm. Multinomial sampling consists in randomly sampling the next token assuming its distribution is a Categorical distribution parametrized by the next-token logits. Attributes ---------- samples The number of samples taken for each input sequence. rN)top_ktop_p temperaturerr4r5r6cCsp||_||_||_||_g|_|dur|jt|n |dur(|jt||dur6|jt|dSdSr) rr4r5r6logits_processorsappendkeep_top_k_logitskeep_top_p_logitsrescale_logits)rrr4r5r6rrrrbszMultinomialSampler.__init__r r r r r rr r r c Csddl}|}|jD]}||}q |jjj|dd}|j|d|d}|jjj|dd} |j|jd|j d} || | d| } || | fS)a Call the multinomial sampler. Parameters ---------- next_token_logits A tensor of shape ``(n_seqs, vocab_size,)`` that represents the probability distribution of the next token over the vocabulary. sequence_weights A tensor of shape ``(n_seqs,)`` that represents the cumulative weight of each sequence. rng A random number generator. Returns ------- A tuple with an array that contains the ids of the sampled tokens of shape ``(n_seqs, 1)``, an array that contains the ancestors of each sampled id of shape ``(n_seqs,)`` and an array that contains the updated cumulative weights of each sequence of shape ``(n_seqs,)``. rNrrr) num_samples generatorr") r$r7r%r&softmax multinomialr'r)r*r#r+r,) rr r r r$altered_next_token_logitslogit_processorprobsr/r.r0r1rrrrxs    zMultinomialSampler.__call__r) rrrr2rrfloatrrrrrrrr3Ss2 r3kr torch.TensorcsJddlttr dkrtdddjdjffdd }|S) zBuild a function that masks logits values smaller than the top `k` ones. Parameters ---------- k The ranking below which logit values are replaced by `-math.inf`. rNrz.`k` must be a strictly positive integers, got instead.logitsrcs8t|d}|||ddk}||tj S)Nrr).rN)minsizetopk masked_fillmathinf)rI num_to_keepmask_idxrFr$rrlogits_processorsz+keep_top_k_logits..logits_processor)r$ isinstancer ValueErrorTensor)rFrSrrRrr9s  r9pcsHddldks dkrtdddjdjffd d }|S) aBuild a function that masks the lowest probability tokens whose cumulative probability is below a certain threshold. Parameters ---------- p The value of the threshold. We keep the highest probability tokens whose cumulative distribution is greater than or equal to `p` and mask the others. Its value must be between 0 (excluded) and 1 (included). rNg?zO`p` must be a floating point number between 0 (excluded) and 1 (included), got rHrIrcsXj|dd\}}jjj|ddjdd}|dk}|d||}||tj S)NF) descendingrrr) sortr%r&r?cumsumscatterrMrNrO)rI sorted_logits sorted_idxcumulative_probabiltiessorted_masked_idxrQrWr$rrrSs z+keep_top_p_logits..logits_processor)r$rUrV)rWrSrrarr:s   r:r6csDttr dkrtdddkrtdd fdd }|S) zBuild a function that rescales the token probabilities exponentially. Parameters ---------- temperature The value by which we rescale the logits. rXzE`temperature` must be a strictly positive floating point number, got rHzFPlease use the greedy sampler instead of setting the temperature to 0.rIrGrcs|Srr)rIr6rrrSsz(rescale_logits..logits_processorN)rIrGrrG)rTrErU)r6rSrrbrr;s  r;c@s:eZdZdZddefddZddddd ed fd d Zd S)BeamSearchSamplerzBeam Search sampling algorithm. Attributes ---------- samples The number of samples taken for each input sequence. Equivalent to the number of beams. rbeamscCs ||_dSrr)rrdrrrrrzBeamSearchSampler.__init__r r r rr<c Csddl}|jjj|dd}||d|}|jd|j}|jd}|||j|}| |dkr>|ddd|f}|j ||jdddd\}} |j | |dd } | |} |j d||j|j|j d d} | | } | |j|} ||j|}| |j|d} | | |fS) a Call the beam search sampler. Parameters ---------- next_token_logits A tensor of shape ``(n_seqs, vocab_size,)`` that represents the probability distribution of the next token over the vocabulary. sequence_weights A tensor of shape ``(n_seqs,)`` that represents the cumulative weight of each sequence. rng A random number generator. Returns ------- A tuple with an array that contains the ids of the sampled tokens of shape ``(n_seqs, 1)``, an array that contains the ancestors of each sampled id of shape ``(n_seqs,)`` and an array that contains the updated cumulative weights of each sequence of shape ``(n_seqs,)``. rNrrrT)r largestsortedfloor) rounding_moder")r$r%r&r' unsqueeze expand_asr*rviewallrLdivr)r#) rr r r-r$r.r1 batch_size vocab_sizeindicesr0r/first_batch_idxrrrrs.    zBeamSearchSampler.__call__NrD)rrrr2rrrrrrrrrcs rc)rNtypingrrrrrr$rrgreedyr3r@rr9rEr:r;rc beam_searchrrrrs <R M