o i*Y@sdZddlmZddlmZmZddlZddlmZmZddl m Z ddl m Z dd l mZmZmZd d lmZd d lmZeeZeeddGdddeZeeddGdddeZeeddGdddeZeGddde ZGdddeZGdddeZGdddeZGdddeZGd d!d!eZ ed"dGd#d$d$eZ!ed%dGd&d'd'eZ"ed(dGd)d*d*e Z#gd+Z$dS),z5PyTorch DPR model for Open Domain Question Answering.) dataclass)OptionalUnionN)Tensornn)BaseModelOutputWithPooling)PreTrainedModel) ModelOutputauto_docstringlogging) BertModel) DPRConfigz6 Class for outputs of [`DPRQuestionEncoder`]. ) custom_introc@PeZdZUdZejed<dZee ejdfed<dZ ee ejdfed<dS)DPRContextEncoderOutputa pooler_output (`torch.FloatTensor` of shape `(batch_size, embeddings_size)`): The DPR encoder outputs the *pooler_output* that corresponds to the context representation. Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer. This output is to be used to embed contexts for nearest neighbors queries with questions embeddings. pooler_outputN. hidden_states attentions __name__ __module__ __qualname____doc__torch FloatTensor__annotations__rrtuplerr r a/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/transformers/models/dpr/modeling_dpr.pyr*  rc@r)DPRQuestionEncoderOutputa pooler_output (`torch.FloatTensor` of shape `(batch_size, embeddings_size)`): The DPR encoder outputs the *pooler_output* that corresponds to the question representation. Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer. This output is to be used to embed questions for nearest neighbors queries with context embeddings. rN.rrrr r r r!r#=r"r#c@steZdZUdZejed<dZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dS) DPRReaderOutputa start_logits (`torch.FloatTensor` of shape `(n_passages, sequence_length)`): Logits of the start index of the span for each passage. end_logits (`torch.FloatTensor` of shape `(n_passages, sequence_length)`): Logits of the end index of the span for each passage. relevance_logits (`torch.FloatTensor` of shape `(n_passages, )`): Outputs of the QA classifier of the DPRReader that corresponds to the scores of each passage to answer the question, compared to all the other passages. start_logitsN end_logitsrelevance_logits.rr) rrrrrrrr&rr'rrrr r r r!r$Ps  r$c@seZdZdZddZdS)DPRPreTrainedModelTcCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|tj rX|jj |jj ddSdS)zInitialize the weightsg)meanstdNg?) isinstancerLinearweightdatanormal_configinitializer_rangebiaszero_ Embedding padding_idx LayerNormfill_)selfmoduler r r! _init_weightsls      z DPRPreTrainedModel._init_weightsN)rrr_supports_sdpar:r r r r!r(hs r(cseZdZdZdeffdd Z      ddedeed eed eed ed ed ede e e edfffddZ e defddZZS) DPREncoder bert_modelr0csdt|t|dd|_|jjjdkrtd|j|_|jdkr,t |jjj|j|_ | dS)NF)add_pooling_layerrz!Encoder hidden_size can't be zero) super__init__rr=r0 hidden_size ValueErrorprojection_dimrr, encode_proj post_initr8r0 __class__r r!r@s   zDPREncoder.__init__NF input_idsattention_masktoken_type_ids inputs_embedsoutput_attentionsoutput_hidden_states return_dictreturn.c Csv|j|||||||d}|d} | dddddf} |jdkr%|| } |s1| | f|ddSt| | |j|jdS)NrIrJrKrLrMrNrOrr )last_hidden_staterrr)r=rCrDrrr) r8rIrJrKrLrMrNrOoutputssequence_output pooled_outputr r r!forwards*    zDPREncoder.forwardcCs|jdkr |jjS|jjjS)Nr)rCrD out_featuresr=r0rA)r8r r r!embeddings_sizes  zDPREncoder.embeddings_size)NNNFFF)rrrbase_model_prefixrr@rrboolrrrrVpropertyintrX __classcell__r r rGr!r<}s8 #r<csjeZdZdZdeffdd Z    ddeded eed ed ed ed e e e edfffddZ Z S)DPRSpanPredictorencoderr0csFt|t||_t|jjd|_t|jjd|_| dS)Nr r) r?r@r<r_rr,rX qa_outputs qa_classifierrErFrGr r!r@s   zDPRSpanPredictor.__init__NFrIrJrLrMrNrOrP.cCs|dur|n|dd\}}|j||||||d} | d} || } | jddd\} } | d} | d} || dddddf}| ||} | ||} ||}|si| | |f| ddSt| | || j | j dS)Nr )rJrLrMrNrOrr)dim)r%r&r'rr) sizer_r`splitsqueeze contiguousraviewr$rr)r8rIrJrLrMrNrO n_passagessequence_lengthrSrTlogitsr%r&r'r r r!rVs6$     zDPRSpanPredictor.forward)NFFF)rrrrYrr@rrrZrr$rrVr]r r rGr!r^s, r^c@"eZdZUdZeed<dZdZdS)DPRPretrainedContextEncoder An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. r0N ctx_encoderrrrrrrload_tf_weightsrYr r r r!rm rmc@rl)DPRPretrainedQuestionEncoderrnr0Nquestion_encoderrpr r r r!rsrrrsc@rl)DPRPretrainedReaderrnr0Nspan_predictorrpr r r r!ru rrruzf The bare DPRContextEncoder transformer outputting pooler outputs as context representations. ceZdZdeffdd Ze       ddeedeedeedeed eed eed eed e e e ed fffddZ Z S)DPRContextEncoderr0c(t|||_t||_|dSN)r?r@r0r<rorErFrGr r!r@   zDPRContextEncoder.__init__NrIrJrKrLrMrNrOrP.c Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur*|dur*td|dur3|}n|dur@|dd}ntd|durK|jn|j} |durc|dur]tj|| dn||jj k}|durptj |tj | d}|j |||||||d} |s| ddSt | j| j| jd S) aS input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. To match pretraining, DPR input sequence should be formatted with [CLS] and [SEP] tokens as follows: (a) For sequence pairs (for a pair title+text for example): ``` tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP] token_type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1 ``` (b) For single sequences (for a question for example): ``` tokens: [CLS] the dog is hairy . [SEP] token_type_ids: 0 0 0 0 0 0 0 ``` DPR is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Examples: ```python >>> from transformers import DPRContextEncoder, DPRContextEncoderTokenizer >>> tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base") >>> model = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base") >>> input_ids = tokenizer("Hello, is my dog cute ?", return_tensors="pt")["input_ids"] >>> embeddings = model(input_ids).pooler_output ```NDYou cannot specify both input_ids and inputs_embeds at the same timerb5You have to specify either input_ids or inputs_embedsdevicedtyperrQrrrr)r0rMrNuse_return_dictrBrdrrones pad_token_idzeroslongrorrrr r8rIrJrKrLrMrNrO input_shaperrSr r r!rV&sB1    zDPRContextEncoder.forwardNNNNNNN)rrrrr@r rrrZrrrrVr]r r rGr!rx6 rxzh The bare DPRQuestionEncoder transformer outputting pooler outputs as question representations. crw)DPRQuestionEncoderr0cryrz)r?r@r0r<rtrErFrGr r!r@r{zDPRQuestionEncoder.__init__NrIrJrKrLrMrNrOrP.c Cs(|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur*|dur*td|dur9||||}n|durF|dd}ntd|durQ|jn|j} |duri|durctj || dn||jj k}|durvtj |tj | d}|j |||||||d} |s| ddSt| j| j| jd S) aj input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. To match pretraining, DPR input sequence should be formatted with [CLS] and [SEP] tokens as follows: (a) For sequence pairs (for a pair title+text for example): ``` tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP] token_type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1 ``` (b) For single sequences (for a question for example): ``` tokens: [CLS] the dog is hairy . [SEP] token_type_ids: 0 0 0 0 0 0 0 ``` DPR is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Examples: ```python >>> from transformers import DPRQuestionEncoder, DPRQuestionEncoderTokenizer >>> tokenizer = DPRQuestionEncoderTokenizer.from_pretrained("facebook/dpr-question_encoder-single-nq-base") >>> model = DPRQuestionEncoder.from_pretrained("facebook/dpr-question_encoder-single-nq-base") >>> input_ids = tokenizer("Hello, is my dog cute ?", return_tensors="pt")["input_ids"] >>> embeddings = model(input_ids).pooler_output ``` Nr|rbr}r~rrQrr)r0rMrNrrB%warn_if_padding_and_no_attention_maskrdrrrrrrrtr#rrrrr r r!rVsD1     zDPRQuestionEncoder.forwardr)rrrrr@r rrrZrr#rrVr]r r rGr!rrrzE The bare DPRReader transformer outputting span predictions. cseZdZdeffdd Ze      ddeedeedeedeed eed eed e e e ed fffd dZ Z S) DPRReaderr0cryrz)r?r@r0r^rvrErFrGr r!r@r{zDPRReader.__init__NrIrJrLrMrNrOrP.c Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur*|dur*td|dur9||||}n|durF|dd}ntd|durQ|jn|j}|dur_tj ||d}|j ||||||dS)a input_ids (`tuple[torch.LongTensor]` of shapes `(n_passages, sequence_length)`): Indices of input sequence tokens in the vocabulary. It has to be a sequence triplet with 1) the question and 2) the passages titles and 3) the passages texts To match pretraining, DPR `input_ids` sequence should be formatted with [CLS] and [SEP] with the format: `[CLS] [SEP] [SEP] ` DPR is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left. Indices can be obtained using [`DPRReaderTokenizer`]. See this class documentation for more details. [What are input IDs?](../glossary#input-ids) inputs_embeds (`torch.FloatTensor` of shape `(n_passages, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Examples: ```python >>> from transformers import DPRReader, DPRReaderTokenizer >>> tokenizer = DPRReaderTokenizer.from_pretrained("facebook/dpr-reader-single-nq-base") >>> model = DPRReader.from_pretrained("facebook/dpr-reader-single-nq-base") >>> encoded_inputs = tokenizer( ... questions=["What is love ?"], ... titles=["Haddaway"], ... texts=["'What Is Love' is a song recorded by the artist Haddaway"], ... return_tensors="pt", ... ) >>> outputs = model(**encoded_inputs) >>> start_logits = outputs.start_logits >>> end_logits = outputs.end_logits >>> relevance_logits = outputs.relevance_logits ``` Nr|rbr}r~)rLrMrNrO) r0rMrNrrBrrdrrrrv) r8rIrJrLrMrNrOrrr r r!rVs.0  zDPRReader.forward)NNNNNN)rrrrr@r rrrZrr$rrVr]r r rGr!rs0r)rxrmr(rsrurr)%r dataclassesrtypingrrrrrmodeling_outputsrmodeling_utilsr utilsr r r bert.modeling_bertrconfiguration_dprr get_loggerrloggerrr#r$r(r<r^rmrsrurxrr__all__r r r r!s\        8>  de W