o پik"@sddlZddlZddlmZmZmZmZmZmZm Z er&ddl Z ddl m Z Gddde ZejddGdd d Zd ed ed ed dddddddeedddeefddZd ed deedddeefddZd ed deedeeeefddZd ed deedefddZ        d/d d!Z      d0d"d#Zd ed ddded fd$d%Zdeedddeefd&d'Zd(ee dddee fd)d*Zd+dd,eddfd-d.ZdS)1N) TYPE_CHECKINGCallableIterableIteratorListOptionalTuple)Guidec@s eZdZdS)ContextLengthExceededErrorN)__name__ __module__ __qualname__rrO/home/ubuntu/.local/lib/python3.10/site-packages/outlines/generate/generator.pyr sr T)frozenc@s:eZdZUded<ded<ded<ded<eeed<dS)GenerationState torch.Tensor token_idskv_cachelogitsweights fsm_statesN)r r r __annotations__rintrrrrrs rmodelsamplerfsmsr rrsequence_weightsattention_masksrrngztorch.Generatorreturnccsddl}|dur |}d} z |||| \} } Wn ty%tdwt||} t| | } || ||\} }}t|| |}t||}t| |} t |dkrYt ||}t ||}t ||| }t ||}|rqt|| | ||VdSt|| | ||Vq)a2Generates sequences of tokens. Parameters ---------- model A callable that generates a probability distribution over the vocabulary when passed a tensor of token ids. sampler A callable that returns the next token ids, their ancestor sequence and the updated sequence weights when passed a distribution over the vocabulary. token_ids A tensor of token ids on which the sequence distribution is conditioned, of shape ``(n_seqs, n_prompt_tokens)`` sequence_weights A tensor that contains the initial weights of the sequences, of shape ``(n_seqs,)`` attention_masks A tensor of tensors that represent the tokens considered at the attention layer, of shape ``(n_seqs, n_prompt_tokens)``. fsms List of finite-state machines that drive the text generation, one for each sequence in the batch. fsm_states The initial states of the finite-state machine for each sequence in the batch. Yields ------ A new sequence. rNTz9The input length exceeds the context length of the model.)torch Generator IndexErrorr get_allowed_tokens bias_logitsupdate_token_idsupdate_attention_masksreorder_kv_cachelen reorder_fsmsreorder_fsm_statesget_next_fsm_statesis_generation_finishedr)rrrrrrrrr"rrallowed_tokens biased_logitsnext_token_ids ancestors is_finishedrrrsequence_generatorsV)           r4r1cCsddt|||DS)a Parameters ---------- fsm The finite-state machine used to monitor this batch. next_token_ids The tokens that were just generated. Returns ------- A `torch.Tensor` object that represents the next logit mask. cSs&g|]\}}}||t|dqS)r)get_next_stater).0fsm fsm_statetoken_idrrr sz'get_next_fsm_states..zip)rrr1rrrr-ss r-cCsddt||DS)a^Get the new instructions for each sequence from the finite-state machine. Parameters ---------- fsm The finite-state machine used to monitor this batch. fsm_states The FSM states corresponding to each sequence in the batch. Returns ------- A nested list that contains the ids of the logits to keep. cSsg|] \}}||jqSr)get_next_instructiontokensr6r7staterrrr:sz&get_allowed_tokens..r;rrrrrr%sr%cCstddt||DS)aDetermine if the generation is finished. A generation is considered finished if the FSM of every sequence in the batch is in a final state. A better solution is to return finished sequences as soon as their FSM is in a final state. Parameters ---------- fsm The finite-state machine used to monitor this batch. fsm_states The FSM states corresponding to each sequence in the batch. Returns ------- Whether all sequences are finished sampling. cSsg|] \}}||qSr)is_final_stater?rrrr:sz*is_generation_finished..)allr<rArrrr.sr.r2cCs(ddl}||d|}|j||gddS)aAppend the sampled tokens to the running sequence of tokens. Parameters ---------- token_ids The current token sequences next_token_ids The tokens that were just generated and that we need to append to the existing sequences. ancestors The sequences to which the token ids need to be added. Returns ------- A new sequence of token ids that contains the tokens that were just generated. rN)dim)r" index_select concatenate)rr1r2r"rrrr'sr'cCsBddl}||d|}|j||j|jddd|jdgddS)aExpand the attention masks. Parameters ---------- attention_masks The attention masks for each sequence in the batch. ancestors The sequences to which the token ids need to be added. Returns ------- The attention masks padded with 1s. rNrD)r!device)axis)r"rFrGonesshaperI)rr2r"rrrr(sr(cCs$g}|D] }|||q|SN)appendcopy)rr2reordered_fsmsancestorrrrr+sr+cCs g}|D] }|||q|SrM)rN)rr2reordered_statesrQrrrr,sr,rc Csbddl}|dur dSt}|D]}t}|D]}||d||j}||f7}q||f7}q|S)aMRe-order the KV-cache based on the ancestors. In transformers, the object that stores the KV-cache is a tuple who elements are the key cache and the value cache. Each of these caches are tuples where each element correpond to a layer. To each layer corresponds a tensor whose first dimension is the batch size. rN)r"tuplerFtorI)rr2r" new_kv_cache cache_itemnew_cache_itemlayerrrrr)s   r)rallowed_token_idscCs^ddl}|j|tj |jd}t|D]\}}|dur&|||f|||f<q||||<q|S)aMask the logits. The function iterates over a nested list where each list corresponds to the indices that need to be masked for each row in the array. Parameters ---------- logits Two dimensional tensor that contains the next-token probability distribution. allowed_token_ids A list that contains the tokens that can be generated by the model. Returns ------- A view of the original logits tensor where some values are masked. rNrH)r" full_likemathinfrI enumerate)rrYr"r0iidsrrrr&sr&)rrr1rr2rr r)rrr2rr r) dataclassesr[typingrrrrrrrr"outlines.fsm.guider Exceptionr dataclassrrr4r-r%boolr.r'r(r+r,r)r&rrrrs$   [