o eiZ@shdZddlmZddlmZddlZddlmZddlmZm Z ddl m Z dd l m Z mZmZmZmZdd lmZdd lmZdd lmZdd lmZmZddlmZddlmZee Z!eeddGdddeZ"eeGdddeZ#eddeGdddeZ$eGddde$Z%eddGddde$Z&eddGd d!d!e$eZ'gd"Z(dS)#zRAG model implementation.)Callable) dataclassN)nn)CacheEncoderDecoderCache)PreTrainedConfig)GenerationConfigGenerationMixinGenerationModeLogitsProcessorListStoppingCriteriaList)GENERATION_MODES_MAPPING) ModelOutput)PreTrainedModel)auto_docstringlogging) RagConfig) RagRetrieverzI Base class for retriever augmented marginalized models outputs. ) custom_introc@szeZdZUdZdZejdBed<dZejdBed<dZ ejdBed<dZ e dBed<dZ ejdBed<dZ ejdBed<dZejdBed <dZejdBed <dZejdBed <dZeejd fdBed <dZeejd fdBed<dZejdBed<dZeejd fdBed<dZeejd fdBed<dZeejd fdBed<dZeejd fdBed<dZeejd fdBed<dS)RetrievAugLMMarginOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head. The score is possibly marginalized over all documents for each vocabulary token. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains precomputed hidden-states (key and values in the attention blocks) of the decoder that can be used (see `past_key_values` input) to speed up sequential decoding. retrieved_doc_embeds (`torch.FloatTensor` of shape `(batch_size, config.n_docs, hidden_size)`, *optional*, returned when *output_retrieved=True*): Embedded documents retrieved by the retriever. Is used with `question_encoder_last_hidden_state` to compute the `doc_scores`. retrieved_doc_ids (`torch.LongTensor` of shape `(batch_size, config.n_docs)`, *optional*, returned when *output_retrieved=True*): The indexes of the embedded documents retrieved by the retriever. context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input ids post-processed from the retrieved documents and the question encoder input_ids by the retriever. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. question_encoder_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden states at the output of the last layer of the question encoder pooled output of the model. question_enc_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the question encoder at the output of each layer plus the initial embedding outputs. question_enc_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the question encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_enc_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the generator encoder of the model. generator_enc_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the generator encoder at the output of each layer plus the initial embedding outputs. generator_enc_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the generator encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_dec_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the generator decoder at the output of each layer plus the initial embedding outputs. generator_dec_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the generator decoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_cross_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Cross-attentions weights of the generator decoder, after the attention softmax, used to compute the weighted average in the cross-attention heads. Nlosslogits doc_scorespast_key_valuesretrieved_doc_embedsretrieved_doc_idscontext_input_idscontext_attention_mask"question_encoder_last_hidden_state.question_enc_hidden_statesquestion_enc_attentionsgenerator_enc_last_hidden_stategenerator_enc_hidden_statesgenerator_enc_attentionsgenerator_dec_hidden_statesgenerator_dec_attentionsgenerator_cross_attentions)__name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rrrrrr LongTensorrrr r!tupler"r#r$r%r&r'r(r2r2b/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/rag/modeling_rag.pyr$s& Frc@sheZdZUdZdZejdBed<dZejdBed<dZ e dBed<dZ ejdBed<dZ ej dBed<dZej dBed<dZej dBed <dZejdBed <dZeejd fdBed <dZeejd fdBed <dZejdBed<dZeejd fdBed<dZeejd fdBed<dZeejd fdBed<dZeejd fdBed<dZeejd fdBed<dS)RetrievAugLMOutputa" logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head. The score is possibly marginalized over all documents for each vocabulary token. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains precomputed hidden-states (key and values in the attention blocks) of the decoder that can be used (see `past_key_values` input) to speed up sequential decoding. retrieved_doc_embeds (`torch.FloatTensor` of shape `(batch_size, config.n_docs, hidden_size)`, *optional*, returned when *output_retrieved=True*): Embedded documents retrieved by the retriever. Is used with `question_encoder_last_hidden_state` to compute the `doc_scores`. retrieved_doc_ids (`torch.LongTensor` of shape `(batch_size, config.n_docs)`, *optional*, returned when *output_retrieved=True*): The indexes of the embedded documents retrieved by the retriever. context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input ids post-processed from the retrieved documents and the question encoder input_ids by the retriever. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. question_encoder_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden states at the output of the last layer of the question encoder pooled output of the model. question_enc_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the question encoder at the output of each layer plus the initial embedding outputs. question_enc_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the question encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_enc_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the generator encoder of the model. generator_enc_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the generator encoder at the output of each layer plus the initial embedding outputs. generator_enc_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the generator encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_dec_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings and one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden states of the generator decoder at the output of each layer plus the initial embedding outputs. generator_dec_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the generator decoder, after the attention softmax, used to compute the weighted average in the self-attention heads. generator_cross_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Cross-attentions weights of the generator decoder, after the attention softmax, used to compute the weighted average in the cross-attention heads. Nrrrrrrrr .r!r"r#r$r%r&r'r()r)r*r+r,rr-r.r/rrrrrr0rrr r!r1r"r#r$r%r&r'r(r2r2r2r3r4s$ Dr4a RAG models were released with the paper [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://huggingface.co/papers/2005.11401) by Patrick Lewis, Ethan Perez, Aleksandra Piktus et al. RAG is a retriever augmented model and encapsulate three components: a question encoder, a dataset retriever and a generator, the encoder and generator are trainable while the retriever is just an indexed dataset. c @sTeZdZUeed<dZdZdZe   d de dBde dBde dBde fd d Z dS) RagPreTrainedModelconfigragTN.question_encoder_pretrained_model_name_or_path'generator_pretrained_model_name_or_path retrieverreturncKs~dd|D}dd|D}|D]}|d|=q|D]}|d|=q|dd}|durd|dus8Jdd d lm} d |vr[d d lm} | j|fi|d di\} }| |d <| j|fi|}|dd} | dur|dusvJdd dlm} d |vrd d lm} | j|fi|d di\}}||d <| j|fi|} |d }|durt j |j | j fi|}||| ||dS)a Instantiates an question encoder and a generator from one or two base classes of the library from pretrained model checkpoints. The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated). To train the model, you need to first set it back in training mode with `model.train()`. Params: question_encoder_pretrained_model_name_or_path (`str`, *optional*, defaults to `None`): Information necessary to initiate the question encoder. Can be either: - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co. - A path to a *directory* containing model weights saved using [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. generator_pretrained_model_name_or_path (`str`, *optional*, defaults to `None`): Information necessary to initiate the generator. Can be either: - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co. - A path to a *directory* containing model weights saved using [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. model_args (remaining positional arguments, *optional*): All remaining positional arguments will be passed to the underlying model's `__init__` method. retriever ([`RagRetriever`], *optional*): The retriever to use. kwwargs (remaining dictionary of keyword arguments, *optional*): Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., `output_attentions=True`). - To update the question_encoder configuration, use the prefix *question_encoder_* for each configuration parameter. - To update the generator configuration, use the prefix *generator_* for each configuration parameter. - To update the parent model configuration, do not use a prefix for each configuration parameter. Behaves differently depending on whether a `config` is provided or automatically loaded. Example: ```python >>> from transformers import RagModel >>> # initialize a RAG from two pretrained models. >>> model = RagModel.from_pretrained_question_encoder_generator( ... "facebook/dpr-question_encoder-single-nq-base", "google-t5/t5-small" ... ) >>> # saving model after fine-tuning >>> model.save_pretrained("./rag") >>> # load fine-tuned model >>> model = RagModel.from_pretrained("./rag") ```cS,i|]\}}|dr|tdd|qS)question_encoder_N startswithlen.0argumentvaluer2r2r3 ) zQRagPreTrainedModel.from_pretrained_question_encoder_generator..cSr<) generator_Nr>rAr2r2r3rE/rFr=rGmodelNznIf `model` is not defined as an argument, a `question_encoder_pretrained_model_name_or_path` has to be defined AutoModelr6) AutoConfigreturn_unused_kwargsTzqIf `generator_model` is not defined as an argument, a `generator_pretrained_model_name_or_path` has to be definedAutoModelForSeq2SeqLM)question_encoder generatorr6r:) itemspopauto.modeling_autorKauto.configuration_autorLfrom_pretrainedrOgetr'from_question_encoder_generator_configsr6)clsr8r9r:kwargskwargs_question_encoderkwargs_generatorkeyrPrKrLquestion_encoder_configrQrOgenerator_configr6r2r2r3*from_pretrained_question_encoder_generatorsx<             z=RagPreTrainedModel.from_pretrained_question_encoder_generator)NNN) r)r*r+rr/base_model_prefix_supports_flash_attn_supports_sdpa classmethodstrrrr`r2r2r2r3r5s$  r5c!s eZdZ    ddedBdedBdedBdedBffdd Ze              ddej dBd ej dBd e e ej dBd ej dBd ej dBd edBdej dBdej dBdej dBdedBdedBdedBdedBdedBde ej eBfddZZS)RagModelNr6rPrQr:c s |dus|dur |dusJd|dur!tj|j|jfi|}nt||js2Jd|d|jt||durHddlm}| |j }|durXddlm }| |j }||_ |j durst|tspJdt|j d ||_ ||_ ||_ d|_d |_|dS)  question_encoder (`PreTrainedModel`, *optional*): The model responsible for encoding the question into hidden states for retrieval. generator (`PreTrainedModel`, *optional*): The model responsible for generating text based on retrieved documents. retriever (`RagRetriever`, *optional*): The component responsible for retrieving documents from a knowledge base given the encoded question. NzQEither a configuration or an question_encoder and a generator has to be provided.zconfig: z has to be of type rIrJrNz`self.retriever` is of type z&, but should be of type `RagRetriever`F)rrXr6 isinstance config_classsuper__init__rTrK from_configrPrOrQr:rtype ctx_encodercontext_encoder_training post_init)selfr6rPrQr:rZrKrO __class__r2r3rkus8"        zRagModel.__init__ input_idsattention_maskencoder_outputsdecoder_input_idsdecoder_attention_maskrrrr use_cacheoutput_attentionsoutput_hidden_statesoutput_retrievedn_docsr;c Ksx|dur|n|jj}| dur| n|jj} | dur| n|jj} | dur$| n|jj} | dur.| n|jj} |jduoF|dupB| dupB|duoF|du}|dur|r|j||dd}|d}|j||j dt j d t |jjdd|dd }|jr|d |d |d |d |d|df\}} }}}}| |}| |} | |}| |}|j||ddj}|d||jd}t |d|ddd}nM|d |d |d |df\}} }}| |}| |}| |} t |d|ddd}n|dusJd| dus Jd|dusJd|dusJd|jd|dks7Jd|d|jdd|durC|j|dd}|durO|j|dd}|j|| ||||| | dd }|skd}d}d}d}d}n|j}|j}|rw| sd}d} d}d}td*id|jd|d|jd |d | d |d |d!|d"|d#|d$|jd%|jd&|j d'|j!d(|j"d)|j#S)+ay input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [`RagConfig`], used to initialize the model, specifies which generator to use, it also specifies a compatible generator tokenizer. Use that tokenizer class to obtain the indices. [What are input IDs?](../glossary#input-ids) encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*) Tuple consists of (`generator_enc_last_hidden_state`, *optional*: `generator_enc_hidden_states`, *optional*: `generator_enc_attentions`). `generator_enc_last_hidden_state` of shape `(batch_size, n_docs * sequence_length, hidden_size)` is a sequence of hidden-states at the output of the last layer of the generator's encoder. Used by the ([`RagModel`]) model during decoding. decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Provide for generation tasks. `None` by default, construct as per instructions for the generator model you're using with your RAG instance. decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. If the model has is not initialized with a `retriever` `doc_scores` has to be provided to the forward pass. `doc_scores` can be computed via `question_encoder_last_hidden_state` and `retrieved_doc_embeds`, see examples for more information. context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input IDs post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model was not initialized with a `retriever` ``context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`,*optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model has is not initialized with a `retriever` `context_attention_mask` has to be provided to the forward pass. `context_attention_mask` are returned by [`~RagRetriever.__call__`]. output_retrieved (`bool`, *optional*): Whether or not to return the `retrieved_doc_embeds`, `retrieved_doc_ids`, `context_input_ids` and `context_attention_mask`. See returned tensors for more detail. n_docs (`int`, *optional*): The number of documents to retrieve. Example: ```python >>> from transformers import AutoTokenizer, RagRetriever, RagModel >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("facebook/rag-token-base") >>> retriever = RagRetriever.from_pretrained( ... "facebook/rag-token-base", index_name="exact", use_dummy_dataset=True ... ) >>> # initialize with RagRetriever to do everything in one forward call >>> model = RagModel.from_pretrained("facebook/rag-token-base", retriever=retriever) >>> inputs = tokenizer("How many people live in Paris?", return_tensors="pt") >>> outputs = model(input_ids=inputs["input_ids"]) ```NT)ru return_dictrcpudevicedtypeprefixptrr}return_tensorsrrrtokenized_doc_idstokenized_doc_attention_maskdoc_idsrrIzMake sure that `context_input_ids` are passed, if no `retriever` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.zMake sure that `context_attention_mask` are passed, if no `retriever` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.zMake sure that `doc_scores` are passed, if no `retriever` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.z^Make sure that `doc_scores` are passed when passing `encoder_outputs` to the forward function.M The first dimension of `context_input_ids` should be a multiple of `n_docs`= , but is .dim) rtrurvrwrxrryrzr~Nrrrrr r!r"r#r$r%r&r'r(r2)$r6r}ryrzr{r|r:rPdetachtor-float32numpygetattrrQrorn pooler_outputviewshapebmm unsqueeze transposesqueezerepeat_interleave hidden_states attentionsr4rrencoder_last_hidden_stateencoder_hidden_statesencoder_attentionsdecoder_hidden_statesdecoder_attentionscross_attentions)rqrtrurvrwrxrrrrryrzr{r|r}rZhas_to_retrievequestion_enc_outputsr retriever_outputsrretrieved_doc_input_idsretrieved_doc_attention_maskr gen_outputsr!r"r2r2r3forwards&J                       zRagModel.forwardNNNN)NNNNNNNNNNNNNN)r)r*r+rrrrkrr-r0Tensorr1r. BoolTensorrboolintr4r __classcell__r2r2rrr3rfssx4      rfzu A RAG-sequence model implementation. It performs RAG-sequence specific marginalization in the forward pass. c&seZdZ    d2dedBdedBdedBdedBffdd Zdefdd Zd efd d Ze                 d3d e j dBde j dBde e e j dBde j dBde jdBdedBde j dBde j dBde jdBdedBdedBdedBdedBdedBdedBde j dBdedBdef$dd Zed!d"Zed#d$Zed%d&Ze          d4d e j dBde j dBde j dBde j dBde jdBd'edBd(edBd)edBdedBde j fd*d+Z d5d.d/Zed0d1ZZS)6RagSequenceForGenerationNr6rPrQr:c j|dus|dur |dusJd|dur tj|j|jfi|}t|t||||d|_|dSrgNzHEither a configuration or an encoder and a generator has to be provided.)r6rPrQr:rrXr6rjrkrfr7rprqr6rPrQr:rZrrr2r3rks  z!RagSequenceForGeneration.__init__cC ||j_dSrr7r:rqr:r2r2r3 set_retriever z&RagSequenceForGeneration.set_retrieverrncCd|j_||j_dSNTr7rornrqrnr2r2r3 set_context_encoder_for_training z9RagSequenceForGeneration.set_context_encoder_for_trainingrtrurvrwrxrrrrryrzr{r|exclude_bos_score reduce_losslabelsr}r;cKs6|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur*|dur(|}d} |j|||||||| || | | | |d}d}|durS|j|j|j|||jj||d}t did|d|jd|jd|j d |j d |j d |j d |jd |jd|jd|jd|jd|jd|jd|jd|jd|jS)a3 input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [`RagConfig`], used to initialize the model, specifies which generator to use, it also specifies a compatible generator tokenizer. Use that tokenizer class to obtain the indices. [What are input IDs?](../glossary#input-ids) encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*) Tuple consists of (`generator_enc_last_hidden_state`, *optional*: `generator_enc_hidden_states`, *optional*: `generator_enc_attentions`). `generator_enc_last_hidden_state` of shape `(batch_size, n_docs * sequence_length, hidden_size)` is a sequence of hidden-states at the output of the last layer of the generator's encoder. Used by the ([`RagModel`]) model during decoding. decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Provide for generation tasks. `None` by default, construct as per instructions for the generator model you're using with your RAG instance. decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input IDs post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model was not initialized with a `retriever` ``context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`,*optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model has is not initialized with a `retriever` `context_attention_mask` has to be provided to the forward pass. `context_attention_mask` are returned by [`~RagRetriever.__call__`]. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. If the model has is not initialized with a `retriever` `doc_scores` has to be provided to the forward pass. `doc_scores` can be computed via `question_encoder_last_hidden_state` and `retrieved_doc_embeds`, see examples for more information. output_retrieved (`bool`, *optional*): Whether or not to return the `retrieved_doc_embeds`, `retrieved_doc_ids`, `context_input_ids` and `context_attention_mask`. See returned tensors for more detail. exclude_bos_score (`bool`, *optional*): Only relevant if `labels` is passed. If `True`, the score of the BOS token is disregarded when computing the loss. reduce_loss (`bool`, *optional*): Only relevant if `labels` is passed. If `True`, the NLL loss is reduced using the `torch.Tensor.sum` operation. n_docs (`int`, *optional*): The number of documents to retrieve. Example: ```python >>> from transformers import AutoTokenizer, RagRetriever, RagSequenceForGeneration >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("facebook/rag-sequence-nq") >>> retriever = RagRetriever.from_pretrained( ... "facebook/rag-sequence-nq", index_name="exact", use_dummy_dataset=True ... ) >>> # initialize with RagRetriever to do everything in one forward call >>> model = RagSequenceForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever) >>> inputs = tokenizer("How many people live in Paris?", return_tensors="pt") >>> targets = tokenizer(text_target="In Paris, there are 10 million people.", return_tensors="pt") >>> input_ids = inputs["input_ids"] >>> labels = targets["input_ids"] >>> outputs = model(input_ids=input_ids, labels=labels) >>> # or use retriever separately >>> model = RagSequenceForGeneration.from_pretrained("facebook/rag-sequence-nq", use_dummy_dataset=True) >>> # 1. Encode >>> question_hidden_states = model.question_encoder(input_ids)[0] >>> # 2. Retrieve >>> docs_dict = retriever(input_ids.numpy(), question_hidden_states.detach().numpy(), return_tensors="pt") >>> doc_scores = torch.bmm( ... question_hidden_states.unsqueeze(1), docs_dict["retrieved_doc_embeds"].float().transpose(1, 2) ... ).squeeze(1) >>> # 3. Forward to generator >>> outputs = model( ... context_input_ids=docs_dict["context_input_ids"], ... context_attention_mask=docs_dict["context_attention_mask"], ... doc_scores=doc_scores, ... decoder_input_ids=labels, ... ) ```NFrtrurvrwrxrrrrryrzr{r|r})repsilonrr}rrrrrrrrr r!r"r#r$r%r&r'r(r2)r6r}rrr7get_nllrrlabel_smoothingrrrrrrr r!r"r#r$r%r&r'r()rqrtrurvrwrxrrrrryrzr{r|rrrr}rZoutputsrr2r2r3rsg      z RagSequenceForGeneration.forwardcC|jjSrrrqr2r2r3r:_z"RagSequenceForGeneration.retrievercCrrr7rQrr2r2r3rQcrz"RagSequenceForGeneration.generatorcCrrr7rPrr2r2r3rPgrz)RagSequenceForGeneration.question_encoderdo_deduplicationnum_return_sequences num_beamsc KsZ| dur| n|jj} |dur|n|jj}|dur|n|jj} |dur$|n|jj}|dus4|dus4Jd|jdurg|durg|j||dd} |j|| jdt j d t |j jdd| dd d }||}g} || d <|| d <d| d <|dur~|jdn|jd| }t|D]}||| |d| }|j j|fi| }|rt tdd|D}|jd}|dur|||d|d}|||dd}nC|dusJd|dusJd||d}||| |d| }||d}|||dddf}||d}|||||dd}|d | d}| ||q|j| |jj jdS)a Implements RAG sequence "thorough" decoding. Read the [`~generation.GenerationMixin.generate`]` documentation for more information on how to set other generate input parameters. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): The sequence used as a prompt for the generation. If `input_ids` is not passed, then `context_input_ids` has to be provided. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input IDs post-processed from the retrieved documents and the question encoder input_ids by the retriever. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model is not initialized with a `retriever` or `input_ids` is not given, `context_input_ids` and `context_attention_mask` have to be provided to the forward pass. They are returned by [`~RagRetriever.__call__`]. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. If the model is not initialized with a `retriever` or `input_ids` is not given, `doc_scores` has to be provided to the forward pass. `doc_scores` are returned by [`~RagRetriever.__call__`]. do_deduplication (`bool`, *optional*): Whether or not to deduplicate the generations from different context documents for a given input. Has to be set to `False` if used while training with distributed backend. num_return_sequences(`int`, *optional*, defaults to 1): The number of independently computed returned sequences for each element in the batch. Note that this is not the value we pass to the `generator`'s `[`~generation.GenerationMixin.generate`]` function, where we set `num_return_sequences` to `num_beams`. num_beams (`int`, *optional*, defaults to 1): Number of beams for beam search. 1 means no beam search. n_docs (`int`, *optional*, defaults to `config.n_docs`) Number of documents to retrieve and/or number of documents for which to generate an answer. kwargs (`dict[str, Any]`, *optional*): Additional kwargs will be passed to [`~generation.GenerationMixin.generate`]. Return: `torch.LongTensor` of shape `(batch_size * num_return_sequences, sequence_length)`: The generated sequences. The second dimension (sequence length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. Nz= At least one of input_ids or context_input_ids must be givenrurrrrrrrrrrurcSsi|] }t||qSr2)retolist)rBkr2r2r3rEsz5RagSequenceForGeneration.generate..T)rrzMake sure that `context_attention_mask` are passed, if no `input_ids` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.zMake sure that `doc_scores` are passed, if no `input_ids` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.)rrrrrr) pad_token_id)r6r}rrrr:rPrrr-rrrrQrrangegeneratestacklistvaluesrepeattopkappend _cat_and_padr)rqrtrurrrrrrr} model_kwargsnum_doc_return_sequencesquestion_hidden_stateshypos batch_sizeindexgenerator_input_idsoutput_sequencesnum_candidates new_input_idsrindividual_input_idsindividual_attention_maskindividual_doc_scores top_cand_indsr2r2r3rks~A       z!RagSequenceForGeneration.generateFcsBtddddfjddjjjgd|dur#|njj}jj p/jjj }|duo@dddf | } fdd} t j j|dd|jd||d|d} t j j|dddd} | ddddddddf} | ddddddddf}| ddddddddf}tj| || |gdd}ddd|dd|ksJ|jdd}|jdd d }| ||\}}|r| r|ddddddfdn|d}|d}|d}|d}| }| }|r|}|}||d}d ||||}|S) NrrcDjjj}|r||d||d|d|dfSNrreqr6rQrany masked_fill_rll smooth_objpad_maskrqtargetr2r3 _mask_pads   z4RagSequenceForGeneration.get_nll.._mask_padsrrrIrrTrkeepdim?)r-catnewrfill_r6rQrr} bos_token_idrallr functional log_softmaxrsizerrrgathersum logsumexp)rq seq_logitsrrrrrr}ruse_bosr seq_logprobs doc_logprobsfirst_token_scoressecond_token_scores remainder rag_logprobsrrnll_loss smooth_losseps_irr2rr3rs@2"   2   z RagSequenceForGeneration.get_nllcCsv|dtdd|Dtdd|D|}d}|D]}|||||jdd|jdf<||jd7}q|S)Nrcs|]}|jdVqdS)rNrrBtr2r2r3 ?z8RagSequenceForGeneration._cat_and_pad..csr)rNrrr2r2r3r?rr)rr maxrr)tensorsroutputindrr2r2r3r=s 2$z%RagSequenceForGeneration._cat_and_padrNNNNNNNNNNNNNNNNN) NNNNNNNNN)FrFN)r)r*r+rrrrkrrrr-r0rr1rrr.rrrrpropertyr:rQrPno_gradrr staticmethodrrr2r2rrr3rs      !       ;rzo A RAG-token model implementation. It performs RAG-token specific marginalization in the forward pass. c&s^eZdZ    dAdedBdedBdedBdedBffdd Zdefdd Zd efd d Z      dBd dZ e ddZ e ddZ e ddZ eddZdCddZe                 dDdejdBdejdBdeeejdBdejdBdejdBdedBdejdBd ejdBd!ejdBd"edBd#edBd$edBd%edBd&edBd'edBd(ejdBd)edBd*ef$d+d,Zeddddddddeef dejdBdejdBdejdBd ejdBd!ejdBd)edBd-e dBd.e!eejge"efdBd/edBd0edBd*ejfd1d2Z#d3d4Z$d5d6Z%d7d8Z&d9d:Z'dCd;d<Z(dEd?d@Z)Z*S)FRagTokenForGenerationNr6rPrQr:c rrrrrrr2r3rkMs  zRagTokenForGeneration.__init__cCrrrrr2r2r3rmrz#RagTokenForGeneration.set_retrieverrncCrrrrr2r2r3rprz6RagTokenForGeneration.set_context_encoder_for_trainingc Ks4|dur|ddddf}d||||||d|d S)NrT) rtrvrrrwrrydo_marginalizer}r2) rqrwrruryrvrr}rZr2r2r3prepare_inputs_for_generationts z3RagTokenForGeneration.prepare_inputs_for_generationcCrrrrr2r2r3r:rzRagTokenForGeneration.retrievercCrrrrr2r2r3rQrzRagTokenForGeneration.generatorcCrrrrr2r2r3rPrz&RagTokenForGeneration.question_encoderc sddd}tt|D]S}t|tr@fdd|jj|j|jj|j|jj|j|jj|jfD\}}}}||||f}nfdd|j|j|j|jfD\}}||f}||f7}q t ||S)zeReorders cache for generation. BART-inspired but we need to take care of the extra dimension for docscSs^|jd|jd}|jd|g|jddR}|d|}|jdg|jddR}|S)NrrrrI)rr index_select)r new_orderr}resultr2r2r3_reorder_stackeds  z>RagTokenForGeneration._reorder_cache.._reorder_stackedr2c3"|] }||jVqdSrrrrBxr+beam_idxr2r3r  z7RagTokenForGeneration._reorder_cache..c3r,rr-r.r0r2r3rr2) rr@rhrself_attention_cachelayerskeysrcross_attention_cacherm) rr1reordered_pastidxself_attention_kself_attention_vcross_attention_kcross_attention_v new_tupler2r0r3_reorder_caches$          z$RagTokenForGeneration._reorder_cachecCsp|dur|n|jj}tjj|dd|jd||d|d}tj|dd}|| d d}tj |ddS)Nrrrr) r6r}rrrrrrr-rr )rqr rr}rr log_prob_sumr2r2r3 marginalizesz!RagTokenForGeneration.marginalizertrurvrwrxrrrrryrzr{r|r&rrr}r;cKsX|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur*|dur(|}d} |j|||||||| || | | | |d}d}|j}|dur[|dusLJ|j|j|j|||jj|d}|re| ||j|}t did|d|d|jd|j d |j d |j d |jd |jd |jd|jd|jd|jd|jd|jd|jd|jd|jS)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [`RagConfig`], used to initialize the model, specifies which generator to use, it also specifies a compatible generator tokenizer. Use that tokenizer class to obtain the indices. [What are input IDs?](../glossary#input-ids) encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*) Tuple consists of (`generator_enc_last_hidden_state`, *optional*: `generator_enc_hidden_states`, *optional*: `generator_enc_attentions`). `generator_enc_last_hidden_state` of shape `(batch_size, n_docs * sequence_length, hidden_size)` is a sequence of hidden-states at the output of the last layer of the generator's encoder. Used by the ([`RagModel`]) model during decoding. decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Provide for generation tasks. `None` by default, construct as per instructions for the generator model you're using with your RAG instance. decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input IDs post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model was not initialized with a `retriever` ``context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`,*optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model has is not initialized with a `retriever` `context_attention_mask` has to be provided to the forward pass. `context_attention_mask` are returned by [`~RagRetriever.__call__`]. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. If the model has is not initialized with a `retriever` `doc_scores` has to be provided to the forward pass. `doc_scores` can be computed via `question_encoder_last_hidden_state` and `retrieved_doc_embeds`, see examples for more information. output_retrieved (`bool`, *optional*): Whether or not to return the `retrieved_doc_embeds`, `retrieved_doc_ids`, `context_input_ids` and `context_attention_mask`. See returned tensors for more detail. do_marginalize (`bool`, *optional*): If `True`, the logits are marginalized over all documents by making use of `torch.nn.functional.log_softmax`. reduce_loss (`bool`, *optional*): Only relevant if `labels` is passed. If `True`, the NLL loss is reduced using the `torch.Tensor.sum` operation. n_docs (`int`, *optional*): The number of documents to retrieve. Example: ```python >>> from transformers import AutoTokenizer, RagRetriever, RagTokenForGeneration >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("facebook/rag-token-nq") >>> retriever = RagRetriever.from_pretrained( ... "facebook/rag-token-nq", index_name="exact", use_dummy_dataset=True ... ) >>> # initialize with RagRetriever to do everything in one forward call >>> model = RagTokenForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever) >>> inputs = tokenizer("How many people live in Paris?", return_tensors="pt") >>> targets = tokenizer(text_target="In Paris, there are 10 million people.", return_tensors="pt") >>> input_ids = inputs["input_ids"] >>> labels = targets["input_ids"] >>> outputs = model(input_ids=input_ids, labels=labels) >>> # or use retriever separately >>> model = RagTokenForGeneration.from_pretrained("facebook/rag-token-nq", use_dummy_dataset=True) >>> # 1. Encode >>> question_hidden_states = model.question_encoder(input_ids)[0] >>> # 2. Retrieve >>> docs_dict = retriever(input_ids.numpy(), question_hidden_states.detach().numpy(), return_tensors="pt") >>> doc_scores = torch.bmm( ... question_hidden_states.unsqueeze(1), docs_dict["retrieved_doc_embeds"].float().transpose(1, 2) ... ).squeeze(1) >>> # 3. Forward to generator >>> outputs = model( ... context_input_ids=docs_dict["context_input_ids"], ... context_attention_mask=docs_dict["context_attention_mask"], ... doc_scores=doc_scores, ... decoder_input_ids=labels, ... ) >>> # or directly generate >>> generated = model.generate( ... context_input_ids=docs_dict["context_input_ids"], ... context_attention_mask=docs_dict["context_attention_mask"], ... doc_scores=doc_scores, ... ) >>> generated_string = tokenizer.batch_decode(generated, skip_special_tokens=True) ```NFr)rrr}rrrrrrrrr r!r"r#r$r%r&r'r(r2)r6r}r&rr7rrrrr@rrrrrrr r!r"r#r$r%r&r'r()rqrtrurvrwrxrrrrryrzr{r|r&rrr}rZrrrr2r2r3rso       zRagTokenForGeneration.forwardgeneration_configprefix_allowed_tokens_fnlogits_processorstopping_criteriac  s|d| ddd} |j|fi| \}} |}|tjtjtjtjfvr,td|dt t |t |}| | |||| | dddu}|||durWn|jj|jdur|dur|j||dd}|j||jdtjd t |jjd dd d }|d |d|d}}}||}||}||}t|d|ddd}|jddksJdd|jdd|jd|j j!}|||dd}tj"|j#df|j$tj%t&|'j(d}|jd}|d}d&fdd }|||j#d}|||j#d|d<|j)|j#dd}|| d<|| d<|| d<| d <|j*||||| |j(d!}|j+|| d"}|j,|| d|jd|j-dd#|.||| | d$<|||f|||d%| | S)'a Implements RAG token decoding. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): The sequence used as a prompt for the generation. If `input_ids` is not passed, then `context_input_ids` has to be provided. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) context_input_ids (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Input IDs post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model has is not initialized with a `retriever`, `context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. context_attention_mask (`torch.LongTensor` of shape `(batch_size * config.n_docs, config.max_combined_length)`, *optional*, returned when *output_retrieved=True*): Attention mask post-processed from the retrieved documents and the question encoder `input_ids` by the retriever. If the model has is not initialized with a `retriever`, `context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. doc_scores (`torch.FloatTensor` of shape `(batch_size, config.n_docs)`): Score between each retrieved document embeddings (see `retrieved_doc_embeds`) and `question_encoder_last_hidden_state`. If the model has is not initialized with a `retriever`, `context_input_ids` has to be provided to the forward pass. `context_input_ids` are returned by [`~RagRetriever.__call__`]. n_docs (`int`, *optional*, defaults to `config.n_docs`) Number of documents to retrieve and/or number of documents for which to generate an answer. generation_config (`~generation.GenerationConfig`, *optional*): The generation configuration to be used as base parametrization for the generation call. `**kwargs` passed to generate matching the attributes of `generation_config` will override them. If `generation_config` is not provided, the default will be used, which has the following loading priority: 1) from the `generation_config.json` model file, if it exists; 2) from the model configuration. Please note that unspecified parameters will inherit [`~generation.GenerationConfig`]'s default values, whose documentation should be checked to parameterize generation. prefix_allowed_tokens_fn (`Callable[[int, torch.Tensor], list[int]]`, *optional*): If provided, this function constraints the beam search to allowed tokens only at each step. If not provided no constraint is applied. This function takes 2 arguments `inputs_ids` and the batch ID `batch_id`. It has to return a list with the allowed tokens for the next generation step conditioned on the previously generated tokens `inputs_ids` and the batch ID `batch_id`. This argument is useful for constrained generation conditioned on the prefix, as described in [Autoregressive Entity Retrieval](https://huggingface.co/papers/2010.00904). logits_processor (`LogitsProcessorList`, *optional*): Custom logits processors that complement the default logits processors built from arguments and a model's config. If a logit processor is passed that is already created with the arguments or a model's config an error is thrown. stopping_criteria (`StoppingCriteriaList`, *optional*): Custom stopping criteria that complement the default stopping criteria built from arguments and a model's config. If a stopping criteria is passed that is already created with the arguments or a model's config an error is thrown. kwargs (`dict[str, Any]`, *optional*): Ad hoc parametrization of `generate_config` and/or additional model-specific kwargs that will be forwarded to the `forward` function of the model. Return: `torch.LongTensor` of shape `(batch_size * num_return_sequences, sequence_length)`: The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. NFz!RAG model is not compatible with z5 generation. Please check your generation parameters.rurrrrrrrrrrrrIrrrT)rtrur~)rrrlast_hidden_statecsl|ddddfdf|jdd}||f|jdd}||f|jddS)Nrr)reshaperexpand)tensorrrr}r2r3extend_enc_output s,"z9RagTokenForGeneration.generate..extend_enc_output)rrrrvr})rAinput_ids_seq_lengthencoder_input_idsrBrCr)rArD)generation_modermax_cache_lengthprefill_outputs)rCrDrAr)/_extract_generation_mode_kwargs_prepare_generation_configget_generation_moder SAMPLE GREEDY_SEARCH BEAM_SEARCH BEAM_SAMPLE ValueErrorrrmr_validate_model_kwargscopy_validate_generation_moderW_prepare_special_tokensr6r}r:rPrrr-rrrQrrrrrr7 get_encoderfullrdecoder_start_token_idlongnext parametersrr_get_logits_processor_get_stopping_criteria_prepare_cache_for_generation max_length_prefill)rqrtrurrrr}rArBrCrDrZgeneration_mode_kwargsrrMdecoding_methodkwargs_has_attention_maskroutrencoderrvrKrErJprepared_logits_processorprepared_stopping_criteriar2rIr3rvsQ             zRagTokenForGeneration.generatecCs|||}|Sr)r>)rqrr1r2r2r3_temporary_reorder_cacheEs z.RagTokenForGeneration._temporary_reorder_cachecC |jjSr)r7rQget_input_embeddingsrr2r2r3rpLrz*RagTokenForGeneration.get_input_embeddingscCror)r7rQget_output_embeddingsrr2r2r3rqOrz+RagTokenForGeneration.get_output_embeddingscCs|jj|Sr)r7rQset_output_embeddings)rqnew_embeddingsr2r2r3rrRsz+RagTokenForGeneration.set_output_embeddingscCsX|dur|jj}||j}|ddddf|ddddf<||dddf<|S)zCShift input ids one token to the right, and pad with start_token_idNrrr)r6r^ new_zerosrclone)rqrtstart_token_idshifted_input_idsr2r2r3shift_tokens_rightUs  (z(RagTokenForGeneration.shift_tokens_rightFrcs |dur|njj}tddddfjddjjjgdfdd} |||} d | ksDJ|j dd} |j ddd} || | \} } | d} | d} | } | } |rs| } | } ||d} d || | | }|S) Nrrcrrrrrr2r3rerz1RagTokenForGeneration.get_nll.._mask_padsrrTrr)r6r}r-rrrrrQrr@rrr r r)rqr rrrrr}rrrrrrrrr2rr3r^s*2   zRagTokenForGeneration.get_nllr)NNNNNNrr!)FrN)+r)r*r+rrrrkrrr'r"r:rQrPr$r>r@rr-r0r.r1rrrrrrrr#r r r rrrrnrprqrrrxrrr2r2rrr3r%Gs              -    O  r%)rfr5rr%))r,collections.abcr dataclassesrr-r cache_utilsrrconfiguration_utilsr generationr r r r r generation.utilsrmodeling_outputsrmodeling_utilsrutilsrrconfiguration_ragr retrieval_ragr get_loggerr)loggerrr4r5rfrr%__all__r2r2r2r3sb          ZW   3 ;