o eiXU@sdZddlmZddlZddlmZmZddlmZddlm Z ddl m Z m Z m Z d d lmZd d lmZe eZee d dGddde Zee d dGddde Zee d dGddde Ze Gddde ZGdddeZGdddeZGdddeZGdddeZGdd d eZe d!dGd"d#d#eZe d$dGd%d&d&eZe d'dGd(d)d)eZ gd*Z!dS)+z5PyTorch DPR model for Open Domain Question Answering.) dataclassN)Tensornn)BaseModelOutputWithPooling)PreTrainedModel) ModelOutputauto_docstringlogging) BertModel) DPRConfigz6 Class for outputs of [`DPRQuestionEncoder`]. ) custom_introc@PeZdZUdZejed<dZeejdfdBed<dZ eejdfdBed<dS)DPRContextEncoderOutputa pooler_output (`torch.FloatTensor` of shape `(batch_size, embeddings_size)`): The DPR encoder outputs the *pooler_output* that corresponds to the context representation. Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer. This output is to be used to embed contexts for nearest neighbors queries with questions embeddings. pooler_outputN. hidden_states attentions __name__ __module__ __qualname____doc__torch FloatTensor__annotations__rtuplerrrb/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/dpr/modeling_dpr.pyr(  rc@r)DPRQuestionEncoderOutputa pooler_output (`torch.FloatTensor` of shape `(batch_size, embeddings_size)`): The DPR encoder outputs the *pooler_output* that corresponds to the question representation. Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer. This output is to be used to embed questions for nearest neighbors queries with context embeddings. rN.rrrrrrrr!;r r!c@steZdZUdZejed<dZejdBed<dZejdBed<dZ e ejdfdBed<dZ e ejdfdBed<dS) DPRReaderOutputa start_logits (`torch.FloatTensor` of shape `(n_passages, sequence_length)`): Logits of the start index of the span for each passage. end_logits (`torch.FloatTensor` of shape `(n_passages, sequence_length)`): Logits of the end index of the span for each passage. relevance_logits (`torch.FloatTensor` of shape `(n_passages, )`): Outputs of the QA classifier of the DPRReader that corresponds to the scores of each passage to answer the question, compared to all the other passages. start_logitsN end_logitsrelevance_logits.rr) rrrrrrrr$r%rrrrrrrr"Ns  r"c@seZdZdZdS)DPRPreTrainedModelTN)rrr_supports_sdparrrrr&fsr&cseZdZdZdeffdd Z      ddededBd edBd edBd ed ed edee edfBfddZ e de fddZ ZS) DPREncoder bert_modelconfigcsdt|t|dd|_|jjjdkrtd|j|_|jdkr,t |jjj|j|_ | dS)NF)add_pooling_layerrz!Encoder hidden_size can't be zero) super__init__r r)r* hidden_size ValueErrorprojection_dimrLinear encode_proj post_initselfr* __class__rrr-ns   zDPREncoder.__init__NF input_idsattention_masktoken_type_ids inputs_embedsoutput_attentionsoutput_hidden_states return_dictreturn.c Ksv|j|||||||d} | d} | dddddf} |jdkr%|| } |s1| | f| ddSt| | | j| jdS)Nr8r9r:r;r<r=r>rr )last_hidden_staterrr)r)r0r2rrr) r5r8r9r:r;r<r=r>kwargsoutputssequence_output pooled_outputrrrforwardys*    zDPREncoder.forwardcCs|jdkr |jjS|jjjS)Nr)r0r2 out_featuresr)r*r.)r5rrrembeddings_sizes  zDPREncoder.embeddings_size)NNNFFF)rrrbase_model_prefixrr-rboolrrrFpropertyintrH __classcell__rrr6rr(ks8 $r(csfeZdZdZdeffdd Z    ddeded edBd ed ed ed ee edfBfddZ Z S)DPRSpanPredictorencoderr*csFt|t||_t|jjd|_t|jjd|_| dS)Nr r ) r,r-r(rOrr1rH qa_outputs qa_classifierr3r4r6rrr-s   zDPRSpanPredictor.__init__NFr8r9r;r<r=r>r?.cKs|dur|n|dd\}} |j||||||d} | d} || } | jddd\} }| d} |d}|| dddddf}| || } ||| }||}|si| ||f| ddSt| ||| j | j dS)Nr )r9r;r<r=r>rr )dim)r#r$r%rr) sizerOrPsplitsqueeze contiguousrQviewr"rr)r5r8r9r;r<r=r>rB n_passagessequence_lengthrCrDlogitsr#r$r%rrrrFs6$     zDPRSpanPredictor.forward)NFFF) rrrrIrr-rrJr"rrFrMrrr6rrNs,  rNc@eZdZUdZeed<dZdS)DPRPretrainedContextEncoder An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. r* ctx_encoderNrrrrrrrIrrrrr] r]c@r\)DPRPretrainedQuestionEncoderr^r*question_encoderNr`rrrrrbrarbc@r\)DPRPretrainedReaderr^r*span_predictorNr`rrrrrdrardzf The bare DPRContextEncoder transformer outputting pooler outputs as context representations. ceZdZdeffdd Ze       ddedBdedBdedBdedBd edBd edBd edBd ee ed fBfddZ Z S)DPRContextEncoderr*c(t|||_t||_|dSN)r,r-r*r(r_r3r4r6rrr-    zDPRContextEncoder.__init__Nr8r9r:r;r<r=r>r?.c Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur*|dur*td|dur3|} n|dur@|dd} ntd|durK|jn|j} |durc|dur]tj| | dn||jj k}|durptj | tj | d}|j |||||||d} |s| ddSt | j| j| jd S) aS input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. To match pretraining, DPR input sequence should be formatted with [CLS] and [SEP] tokens as follows: (a) For sequence pairs (for a pair title+text for example): ``` tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP] token_type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1 ``` (b) For single sequences (for a question for example): ``` tokens: [CLS] the dog is hairy . [SEP] token_type_ids: 0 0 0 0 0 0 0 ``` DPR is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Examples: ```python >>> from transformers import DPRContextEncoder, DPRContextEncoderTokenizer >>> tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base") >>> model = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base") >>> input_ids = tokenizer("Hello, is my dog cute ?", return_tensors="pt")["input_ids"] >>> embeddings = model(input_ids).pooler_output ```NDYou cannot specify both input_ids and inputs_embeds at the same timerR5You have to specify either input_ids or inputs_embedsdevicedtypernr@r rrr)r*r<r=use_return_dictr/rTrnrones pad_token_idzeroslongr_rrrr r5r8r9r:r;r<r=r>rB input_shapernrCrrrrFsB2    zDPRContextEncoder.forwardNNNNNNN) rrrrr-r rrJrrrFrMrrr6rrg6 rgzh The bare DPRQuestionEncoder transformer outputting pooler outputs as question representations. crf)DPRQuestionEncoderr*crhri)r,r-r*r(rcr3r4r6rrr-vrjzDPRQuestionEncoder.__init__Nr8r9r:r;r<r=r>r?.c Ks(|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur*|dur*td|dur9||||} n|durF|dd} ntd|durQ|jn|j} |duri|durctj | | dn||jj k}|durvtj | tj | d}|j |||||||d} |s| ddSt| j| j| jd S) aj input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. To match pretraining, DPR input sequence should be formatted with [CLS] and [SEP] tokens as follows: (a) For sequence pairs (for a pair title+text for example): ``` tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP] token_type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1 ``` (b) For single sequences (for a question for example): ``` tokens: [CLS] the dog is hairy . [SEP] token_type_ids: 0 0 0 0 0 0 0 ``` DPR is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Examples: ```python >>> from transformers import DPRQuestionEncoder, DPRQuestionEncoderTokenizer >>> tokenizer = DPRQuestionEncoderTokenizer.from_pretrained("facebook/dpr-question_encoder-single-nq-base") >>> model = DPRQuestionEncoder.from_pretrained("facebook/dpr-question_encoder-single-nq-base") >>> input_ids = tokenizer("Hello, is my dog cute ?", return_tensors="pt")["input_ids"] >>> embeddings = model(input_ids).pooler_output ``` NrkrRrlrmror@r rq)r*r<r=rrr/%warn_if_padding_and_no_attention_maskrTrnrrsrtrurvrcr!rrrrwrrrrF}sD2     zDPRQuestionEncoder.forwardry) rrrrr-r rrJr!rrFrMrrr6rr{przr{zE The bare DPRReader transformer outputting span predictions. cs~eZdZdeffdd Ze      ddedBdedBdedBdedBd edBd edBd ee ed fBfd dZ Z S) DPRReaderr*crhri)r,r-r*rNrer3r4r6rrr-rjzDPRReader.__init__Nr8r9r;r<r=r>r?.c Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur*|dur*td|dur9||||}n|durF|dd}ntd|durQ|jn|j} |dur_tj || d}|j ||||||dS)a input_ids (`tuple[torch.LongTensor]` of shapes `(n_passages, sequence_length)`): Indices of input sequence tokens in the vocabulary. It has to be a sequence triplet with 1) the question and 2) the passages titles and 3) the passages texts To match pretraining, DPR `input_ids` sequence should be formatted with [CLS] and [SEP] with the format: `[CLS] [SEP] [SEP] ` DPR is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left. Indices can be obtained using [`DPRReaderTokenizer`]. See this class documentation for more details. [What are input IDs?](../glossary#input-ids) inputs_embeds (`torch.FloatTensor` of shape `(n_passages, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Examples: ```python >>> from transformers import DPRReader, DPRReaderTokenizer >>> tokenizer = DPRReaderTokenizer.from_pretrained("facebook/dpr-reader-single-nq-base") >>> model = DPRReader.from_pretrained("facebook/dpr-reader-single-nq-base") >>> encoded_inputs = tokenizer( ... questions=["What is love ?"], ... titles=["Haddaway"], ... texts=["'What Is Love' is a song recorded by the artist Haddaway"], ... return_tensors="pt", ... ) >>> outputs = model(**encoded_inputs) >>> start_logits = outputs.start_logits >>> end_logits = outputs.end_logits >>> relevance_logits = outputs.relevance_logits ``` NrkrRrlrm)r;r<r=r>) r*r<r=rrr/r|rTrnrrsre) r5r8r9r;r<r=r>rBrxrnrrrrFs.1  zDPRReader.forward)NNNNNN) rrrrr-r rrJr"rrFrMrrr6rr}s0 r})rgr]r&rbrdr{r})"r dataclassesrrrrmodeling_outputsrmodeling_utilsrutilsrr r bert.modeling_bertr configuration_dprr get_loggerrloggerrr!r"r&r(rNr]rbrdrgr{r}__all__rrrrsZ        9?  ef X