o ̳i0?@sddlmZmZmZmZddlZddlmZdejdejdejfddZ d ddd d ejd e d ee deejdejf ddZ d*dd dddedejdejdeejdeejd e d ee deejejffddZ dejdejdejdejfddZ d*dejdee dejfddZdejfd d!Zedd ddddd"ded#ejd$e d%e d e d ee deee d&eejd'eedeejejffd(d)ZdS)+)CallableListOptionalTupleN)TransformerDecoderprobsqreturncCstj||dddjtjdS)z(Samples from a multinomial distribution.T)dimkeepdim)dtype)torchargmaxtoint)rrrT/home/ubuntu/.local/lib/python3.10/site-packages/torchtune/generation/_generation.pymultinomial_sample_one srg? temperaturetop_krlogitsrrcCs|t|d}|dur.t|t||d\}}|ddd}t||ktd |}tj j j |dd}|durCt | d}t||S)a:Generic sample from a probability distribution. Includes support for Top-K sampling and Temperature. Args: logits (torch.Tensor): logits from which to sample temperature (float): value to scale the predicted logits by, default 1.0. top_k (Optional[int]): If specified, we prune the sampling to only token ids within the top_k probabilities q (Optional[torch.Tensor]): randomly sampled tensor for softmax sampling trick. If None, we use the default softmax sampling trick. Default None. Example: >>> from torchtune.generation import sample >>> logits = torch.empty(3, 3).uniform_(0, 1) >>> sample(logits) tensor([[1], [2], [0]], dtype=torch.int32) Returns: torch.Tensor: sampled token id gh㈵>Nr Infr )maxrtopkminsizeselect unsqueezewherefloatnn functionalsoftmax empty_like exponential_r)rrrrv_pivotrrrrsamples r,)maskrrmodel input_posxr-cCs8||||ddddf}t||||d|dfS)a Generates the next tokens given a prompt, and also returns the corresponding logits. Args: model (TransformerDecoder): model used for generation input_pos (torch.Tensor): tensor with the positional encodings associated with the given prompt, with shape [bsz x seq_length]. x (torch.Tensor): tensor with the token IDs associated with the given prompt, with shape [bsz x seq_length]. q (Optional[torch.Tensor]): randomly sampled tensor for softmax sampling trick. See https://github.com/pytorch-labs/gpt-fast/blob/32971d3129541c5bfb4f715abc33d1c5f408d204/generate.py#L40 mask (Optional[torch.Tensor]): attention mask with shape [bsz x seq_length x seq_length], default None. temperature (float): value to scale the predicted logits by, default 1.0. top_k (Optional[int]): Top-k value to use for sampling, default None. Returns: Tuple[torch.Tensor, torch.Tensor]: tuple of two tensors: - tokens (torch.Tensor): tensor with the generated tokens, with shape [bsz x 1]. - logits (torch.Tensor): tensor with the logits associated with the generated tokens, with shape [bsz x 1 x vocab_size]. )r/r-Nr rr)r,cloner!)r.r/r0rr-rrrrrrgenerate_next_tokenBs$r2tokens stop_tokensstop_token_reachedcCst||}||O}|S)z2Updates which sequences have reached a stop token.)risinflatten)r3r4r5stop_token_reached_currrrrupdate_stop_tokens_trackermsr9 padding_masktarget_seq_lencCs|j\}}|dur |n|}||krtdtjtj|||jtddd|dd}|dd| |dddddf d|d|j ddd  t d g|S) a Converts a padding mask of shape ``[bsz, seq_len]`` to a ``[bsz, seq_len, seq_len]`` causal attention mask suitable for consumption by :func:`~torch.nn.functional.scaled_dot_product_attention`. If ``target_seq_len`` is provided, this will return a mask of shape ``[bsz, seq_len, target_seq_len]``. This is useful when generating masks for static KV caches where the maximum length the caches have been setup with are longer than the current sequence. Args: padding_mask (torch.Tensor): Boolean tensor where False indicates the corresponding token in the sequence is a padding token and should be masked out in attention, with shape [bsz x seq_length] target_seq_len (Optional[int]): target sequence length to create attention mask with. Default None. Returns: torch.Tensor: Boolean causal mask with shape - [bsz, seq_length, seq_length] or - [bsz, seq_length, target_seq_len] if ``target_seq_len`` was specified. Raises: AssertionError: if ``target_seq_len < seq_len``, the sequence length of the padding mask. Example: >>> padding_mask = torch.tensor([[False, True, True, True]]) >>> get_causal_mask_from_padding_mask(padding_mask, target_seq_len=5) tensor([[[ True, False, False, False, False], [False, True, False, False, False], [False, True, True, False, False], [False, True, True, True, False]]]) ]) NzNtarget_seq_len cannot be shorter than the sequence length of the padding mask.devicer r)diagonalrr )dim1dim2T)shapeAssertionErrorrtrilonesr=boolrepeatnarrowmul_expandr>copy_Tensor)r:r;bszseq_lenr-rrr!get_causal_mask_from_padding_maskys  0rOcCs|dd|tjS)a Calculates position ids given a padding mask which right-shifts position ids to start from the first valid token. Args: padding_mask (torch.Tensor): Boolean tensor where False indicates the corresponding token in the sequence is a padding token and should be masked out in attention. Shape [bsz, seq_len] Returns: torch.Tensor: position ids which are appropriately shifted according to any padding values. Example: >>> padding_mask = torch.tensor([False, False, False, True, True, True, True, True]) >>> get_position_ids_from_padding_mask(padding_mask) torch.Tensor([0, 0, 0, 0, 1, 2, 3, 4]) r r)cumsumrrr)r:rrr"get_position_ids_from_padding_masksrQ)pad_idrrr4rngcustom_generate_next_tokenpromptmax_generated_tokensrRrSrTc Csv|jdkr |ddn|}|durt}|\} } | |} |} |} | s)| n|j}| |k}|sKtj j j |d|fdd}t ||d}t |}nttj| |tj|jdd}tjd| | jd d}| ru|ddd| f}n |ddd| d| f}d}|durtj| |jjf|jd jd|d }t||ddd| f|||||d \}}tj| |gdd } | }tj| tj|jd}|rtj||j|jd nd}tj| | dftj|jd}|durt|||}|r| |fSt |dD]}|durtj||!| dgdd }| r,|dd|f"}|dd|dddf"}n!| }|ddd|df}|ddd|dd|df}d}|duretj| |jjf|jd jd|d }||||||||d\}}tj| |gdd } tj||gdd }|d7}|durt|||}|rnq|dur| |9} ||dd|j#d ddf9}| |fS)a1 Generates tokens from a model conditioned on a prompt, and also returns logits for the generations. Args: model (TransformerDecoder): model used for generation prompt (torch.Tensor): tensor with the token IDs associated with the given prompt, with shape either [seq_length] or [bsz x seq_length]. max_generated_tokens (int): number of tokens to be generated pad_id (int): token ID to use for padding, default 0. temperature (float): value to scale the predicted logits by, default 1.0. top_k (Optional[int]): If specified, we prune the sampling to only token ids within the top_k probabilities, default None. stop_tokens (Optional[List[int]]): If specified, generation is stopped when any of these tokens are generated, default None. rng (Optional[torch.Generator]): random number generator, default None. custom_generate_next_token (Optional[Callable]): If specified, we'll use the ``custom_generate_next_token function``. This is generally only useful if you want to specify a ``torch.compile`` version of the generate next token for performance reasons. If None, we use the default :func:`generate_next_token`. Default is None. Note: This function has only been tested with decoder-only models. Examples: >>> model = torchtune.models.llama3.llama3_8b() >>> tokenizer = torchtune.models.llama3.llama3_tokenizer() >>> prompt = tokenizer.encode("Hi my name is") >>> rng.manual_seed(42) >>> output, logits = generate(model, torch.tensor(prompt), max_generated_tokens=100, pad_id=0) >>> print(tokenizer.decode(output[0].tolist())) Hi my name is Jeremy and I'm a friendly language model assistant! Returns: Tuple[torch.Tensor, torch.Tensor]: tuple of two tensors: - tokens (torch.Tensor): tensor with the generated tokens, with shape ``[bsz x seq_len + num_generated_tokens]`` where ``num_generated_tokens`` may be less than ``max_generated_tokens`` if ``stop_tokens`` are provided. - logits (torch.Tensor): tensor with the logits associated with the generated tokens, with shape ``[bsz x num_generated_tokens x vocab_size]``. rr NrT)value)r;)r r=)r=) generator)r/r-r0rrrrr<)r/r0r-rrr)$ndimviewr2rr1caches_are_enableddecoder_max_cache_seq_lenallrr$r%padrOrQrDrErFr=r!arangeemptytok_embeddingsnum_embeddingsr(squeezecatzerostensorr int32r9itemrangereshape contiguousrB)r.rUrVrRrrr4rSrTrM prompt_lengthtotal_response_lengthgenerated_tokensincremental_decoding max_seq_len padding_masksmasksr/ curr_masksrr3generated_logitscurr_posr5stop_token_maskr*curr_input_posrrrrgenerates6       "      "rx)N)typingrrrrrtorchtune.modules.transformerrrLrr#rr,r2r9rOrQno_grad Generatorrxrrrrs  4 +   1