o izF@s dZddlZddlZddlmZddlmZmZddlZddlm Z ddl m Z ddl m Z dd lmZdd lmZdd lmZmZmZmZdd lmZdd lmZmZmZddlmZmZm Z m!Z!ddl"m#Z#ddl$m%Z%e &e'Z(dZ)dZ*dZ+dZ,ddZ-Gddde j.Z/Gddde j.Z0Gddde j.Z1de0iZ2Gdd d e j.Z3Gd!d"d"e j.Z4Gd#d$d$e j.Z5Gd%d&d&eZ6Gd'd(d(e j.Z7Gd)d*d*e j.Z8eGd+d,d,eZ9eGd-d.d.eZ:eGd/d0d0eZ;eGd1d2d2eZGd7d8d8e j.Z?Gd9d:d:e j.Z@Gd;d<dZCGd?d@d@eZDGdAdBdBeDZEedCeBGdDdEdEeDZFedFeBGdGdHdHeDZGedIeBGdJdKdKeDZHedLeBGdMdNdNeDZIdOZJedPeBGdQdRdReDZKgdSZLdS)TzPyTorch REALM model.N) dataclass)OptionalUnion)nn)CrossEntropyLoss)ACT2FN)Cache)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentionsMaskedLMOutput ModelOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)add_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings)deprecate_kwarg) RealmConfigz(google/realm-cc-news-pretrained-embedderz'google/realm-cc-news-pretrained-encoderz&google/realm-cc-news-pretrained-scorerrc Csfzddl}ddl}ddl}Wn tytdwtj|}t d||j |}g}g} |D] \} } t d| d| |j || } | | | | q6t|| D]\} } t|tryd| vryt d| d |jjd q\| d s| d rt|tr| d d} | dd} | d s| d rt|tr| d d} | drt|trdnd} | d| d} | d| d} | d| d} | d| d} | d| d} | dr*t|trdnd}| d|d} | d|d } | d!|d"} | d#|d$} | d%|d} | d&|d$} n"| d'rLt|tr8dnd}| d(|d } | d)|d"} | d*} td+d,| Drgt dd*| q\|}| D]m}|d-|r{|d.|}n|g}|dd/ks|dd0krt|d1}n4|dd2ks|dd3krt|d4}n z t||d}Wntyt dd*| Yqkwt|d5krt|d6}||}qk|d7dd8krt|d1}n |d/kr| | } z|j!| j!ksJd9|j!d:| j!d;Wnt"y!}z |j#|j!| j!f7_#d}~wwt d<| t$%| |_&q\|S)=z'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape readerz Skipping z as it is not z 's parameterbertclszbert/z reader/realm/zcls/z reader/cls/zrealm/zreader/zreader/module/bert/zreader/module/cls/z reader/dense/zqa_outputs/dense_intermediate/zreader/dense_1/zqa_outputs/dense_output/zreader/layer_normalizationzqa_outputs/layer_normalizationzmodule/module/module/z embedder/z!module/module/module/module/bert/zmodule/module/module/LayerNorm/zcls/LayerNorm/zmodule/module/module/dense/z cls/dense/z,module/module/module/module/cls/predictions/zcls/predictions/zmodule/module/module/bert/z%module/module/module/cls/predictions/zmodule/module/zmodule/module/LayerNorm/zmodule/module/dense//css|]}|dvVqdS))adam_vadam_mAdamWeightDecayOptimizerAdamWeightDecayOptimizer_1 global_stepN).0nr$r$g/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/deprecated/realm/modeling_realm.py ss  z+load_tf_weights_in_realm..z [A-Za-z]+_\d+z_(\d+)kernelgammaweight output_biasbetabiasri _embeddingszPointer shape z and array shape z mismatchedzInitialize PyTorch weight )'renumpy tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variables load_variableappendzip isinstance RealmReader __class____name__ startswithRealmForOpenQAreplaceRealmKnowledgeAugEncoder RealmEmbeddersplitanyjoin fullmatchgetattrAttributeErrorlenint transposeshapeAssertionErrorargstorch from_numpydata)modelconfigtf_checkpoint_pathr1nptftf_path init_varsnamesarraysnamerRarray reader_prefixembedder_prefixpointerm_name scope_namesnumer$r$r'load_tf_weights_in_realm1s                     rjc sheZdZdZfddZ     ddeejdeejdeejd eejd e d ej f d d Z Z S)RealmEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_t|dd|_|jdt|jddd|jd tj|jtjd dddS) N) padding_idxepsposition_embedding_typeabsolute position_ids)rF) persistenttoken_type_idsdtype)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutrMroregister_bufferrUarangeexpandzerosrqsizelongselfrYrBr$r'rxs  zRealmEmbeddings.__init__Nr input_idsrtrq inputs_embedspast_key_values_lengthreturnc Cs|dur |}n|dd}|d}|dur&|jdd|||f}|durPt|drE|jddd|f}||d|} | }n tj|tj|jjd}|durY| |}| |} || } |j dkrp| |} | | 7} | | } || } | S)Nrrrrtrrvdevicerp)rrqhasattrrtrrUrrrr}rrorrr) rrrtrqrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr embeddingsrr$r$r'forwards,        zRealmEmbeddings.forward)NNNNr) rC __module__ __qualname____doc__rxrrU LongTensor FloatTensorrPTensorr __classcell__r$r$rr'rks* rkcseZdZdfdd ZdejdejfddZedd d d       dd ejdeej deej deej deej d ee dee de ejfddZ ZS)RealmSelfAttentionNcst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_|p\t|dd|_|jdksh|jd kry|j|_t d |jd |j|_|j|_dS) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()rorp relative_keyrelative_key_queryr/r)rwrxr{num_attention_headsr ValueErrorrPattention_head_size all_head_sizerLinearquerykeyvaluerattention_probs_dropout_probrrMror~rydistance_embedding is_decoderrrYrorr$r'rxs*   zRealmSelfAttention.__init__xrcCs6|dd|j|jf}||}|ddddS)Nrrrr/r)rrrviewpermute)rr new_x_shaper$r$r'transpose_for_scoress z'RealmSelfAttention.transpose_for_scorespast_key_valuepast_key_values4.58new_nameversionF hidden_statesattention_mask head_maskencoder_hidden_statesencoder_attention_maskoutput_attentionscCs||}|du} | r|dur|d} |d} |}nP| r/|||} |||} |}n;|durZ|||} |||} tj|d| gdd} tj|d| gdd} n|||} |||} ||} |du} |jrz| | f}t| | dd}|j dks|j dkr | j d| j d}}| rtj |dtj |j d dd}ntj|tj |j d dd}tj|tj |j d dd}||}|||jd}|j| jd }|j dkrtd | |}||}n|j dkr td | |}td | |}|||}|t|j}|dur||}tjj|dd}||}|dur0||}t|| }|dddd }|dd|jf}||}|rX||fn|f}|jrd||f}|S)Nrrr/dimrrrrrruzbhld,lrd->bhlrzbhrd,lrd->bhlrr) rrrrrUcatrmatmulrQrorRtensorrrrrrr~torveinsummathsqrtrr functionalsoftmaxrr contiguousrr)rrrrrrrrmixed_query_layeris_cross_attention key_layer value_layer query_layer use_cacheattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapeoutputsr$r$r'rsn             zRealmSelfAttention.forwardNNNNNNF)rCrrrxrUrrrrrr booltuplerrr$r$rr'rs6  rc8eZdZfddZdejdejdejfddZZS)RealmSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNrm) rwrxrrr{denserrrrrrrr$r'rxd zRealmSelfOutput.__init__r input_tensorrcC&||}||}|||}|Srrrrrrrr$r$r'rj  zRealmSelfOutput.forwardrCrrrxrUrrrr$r$rr'rc $reagercseZdZdfdd ZddZedddd       dd ejd eej d eej deej deej dee dee de ejfddZ ZS)RealmAttentionNcs4tt|j||d|_t||_t|_dS)Nro) rwrxREALM_SELF_ATTENTION_CLASSES_attn_implementationrroutputset pruned_headsrrr$r'rxws   zRealmAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr)rOrrrrrrrrrrrrunion)rheadsindexr$r$r' prune_headss zRealmAttention.prune_headsrrrrFrrrrrrrc Cs<||||||||}||d|} | f|dd} | S)Nrr)rr) rrrrrrrr self_outputsattention_outputrr$r$r'rs  zRealmAttention.forwardrr)rCrrrxrrrUrrrr rrrrr$r$rr'rvs6  rc2eZdZfddZdejdejfddZZS)RealmIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) rwrxrrr{intermediate_sizerr@ hidden_actstrrintermediate_act_fnrrr$r'rxs   zRealmIntermediate.__init__rrcC||}||}|Sr)rr rrr$r$r'r  zRealmIntermediate.forwardrr$r$rr'rs rcr) RealmOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) rwrxrrrr{rrrrrrrrr$r'rxrzRealmOutput.__init__rrrcCrrrrr$r$r'rrzRealmOutput.forwardrr$r$rr'rrrcseZdZfddZedddd      dd ejd eejd eejd eejd eejdee dee de ejfddZ ddZ ZS) RealmLayercsrt|j|_d|_t||_|j|_|j|_|jr-|js&t|dt|dd|_ t ||_ t ||_ dS)Nrz> should be used as a decoder model if cross attention is addedrpr)rwrxchunk_size_feed_forward seq_len_dimr attentionradd_cross_attentionrcrossattentionr intermediaterrrrr$r'rxs   zRealmLayer.__init__rrrrNFrrrrrrrc Cs|dur |ddnd}|j|||||d} | d} |jr(| dd} | d} n| dd} d} |jro|durot|dsDtd|d|durN|d dnd}|| ||||||}|d} | |dd} |d} | | } t|j|j|j| }|f| } |jr| | f} | S) Nr/)rrrrrrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) rrrrrrfeed_forward_chunkrr)rrrrrrrrself_attn_past_key_valueself_attention_outputsrrpresent_key_valuecross_attn_present_key_valuecross_attn_past_key_valuecross_attention_outputs layer_outputr$r$r'rsP         zRealmLayer.forwardcCs||}|||}|Sr)rr)rrintermediate_outputrr$r$r'rs  zRealmLayer.feed_forward_chunkr)rCrrrxrrUrrrr rrrrrr$r$rr'rs6   ArcseZdZfddZ         ddejdeejdeejd eejd eejd eed ee d ee dee dee de e eje ffddZ ZS) RealmEncodercs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr$)r)r%_rYr$r' "sz)RealmEncoder.__init__..F) rwrxrYr ModuleListrangenum_hidden_layerslayergradient_checkpointingrrr"r'rxs   zRealmEncoder.__init__NFTrrrrrrrroutput_hidden_states return_dictrc  Cs2| rdnd} |r dnd} |r|jjrdnd} |jr%|jr%|r%tdd}|r)dnd}t|jD]H\}}| r;| |f} |durC||nd}|||||||durS||nd|}|d}|rd||df7}|rx| |df} |jjrx| |df} q0| r| |f} | stdd ||| | | fDSt ||| | | d S) Nr$zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frrrrr/css|] }|dur|VqdSrr$)r%vr$r$r'r(Zsz'RealmEncoder.forward..)last_hidden_staterr attentionscross_attentions) rYrr(trainingr5 warning_once enumerater'rr )rrrrrrrrrr)r*all_hidden_statesall_self_attentionsall_cross_attentionsnext_decoder_cachei layer_modulelayer_head_mask layer_outputsr$r$r'r%sb       zRealmEncoder.forward) NNNNNNFFT)rCrrrxrUrrrr rrrr rrr$r$rr'r sD      r cr) RealmPoolercs*tt|j|j|_t|_dSr)rwrxrrr{rTanh activationrrr$r'rxos zRealmPooler.__init__rrcCs(|dddf}||}||}|S)Nr)rr<)rrfirst_token_tensor pooled_outputr$r$r'rts  zRealmPooler.forwardrr$r$rr'r:ns r:c@sPeZdZUdZdZeejed<dZ ee ejed<dZ ee ejed<dS)RealmEmbedderOutputa* Outputs of [`RealmEmbedder`] models. Args: projected_score (`torch.FloatTensor` of shape `(batch_size, config.retriever_proj_size)`): Projected score. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nprojected_scorerr-) rCrrrr@rrUr__annotations__rrr-r$r$r$r'r?}s r?c@sHeZdZUdZdZeejed<dZ eejed<dZ eejed<dS)RealmScorerOutputa' Outputs of [`RealmScorer`] models. Args: relevance_score (`torch.FloatTensor` of shape `(batch_size, config.num_candidates)`): The relevance score of document candidates (before softmax). query_score (`torch.FloatTensor` of shape `(batch_size, config.retriever_proj_size)`): Query score derived from the query embedder. candidate_score (`torch.FloatTensor` of shape `(batch_size, config.num_candidates, config.retriever_proj_size)`): Candidate score derived from the embedder. Nrelevance_score query_scorecandidate_score) rCrrrrCrrUrrArDrEr$r$r$r'rBs  rBc@seZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eej ed<dZ eej ed<dZeejed<dZeejed <dZeejed <dZeejed <dZeeejed <dZeeejed <dS)RealmReaderOutputa+ Outputs of [`RealmReader`] models. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `start_positions`, `end_positions`, `has_answers` are provided): Total loss. retriever_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `start_positions`, `end_positions`, `has_answers` are provided): Retriever loss. reader_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `start_positions`, `end_positions`, `has_answers` are provided): Reader loss. retriever_correct (`torch.BoolTensor` of shape `(config.searcher_beam_size,)`, *optional*): Whether or not an evidence block contains answer. reader_correct (`torch.BoolTensor` of shape `(config.reader_beam_size, num_candidates)`, *optional*): Whether or not a span candidate contains answer. block_idx (`torch.LongTensor` of shape `()`): The index of the retrieved evidence block in which the predicted answer is most likely. candidate (`torch.LongTensor` of shape `()`): The index of the retrieved span candidates in which the predicted answer is most likely. start_pos (`torch.IntTensor` of shape `()`): Predicted answer starting position in *RealmReader*'s inputs. end_pos (`torch.IntTensor` of shape `()`): Predicted answer ending position in *RealmReader*'s inputs. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlossretriever_loss reader_lossretriever_correctreader_correct block_idx candidate start_posend_posrr-)rCrrrrGrrUrrArHrIrJ BoolTensorrKrLrrMrN IntTensorrOrrr-r$r$r$r'rFs #rFc@s4eZdZUdZdZeeed<dZee j ed<dS)RealmForOpenQAOutputz Outputs of [`RealmForOpenQA`] models. Args: reader_output (`dict`): Reader output. predicted_answer_ids (`torch.LongTensor` of shape `(answer_sequence_length)`): Predicted answer ids. N reader_outputpredicted_answer_ids) rCrrrrSrdictrArTrUrr$r$r$r'rRs  rRc$eZdZfddZddZZS)RealmPredictionHeadTransformcsVtt|j|j|_t|jtrt |j|_ n|j|_ tj |j|j d|_ dSr) rwrxrrr{rr@r r rtransform_act_fnrrrrr$r'rxs  z%RealmPredictionHeadTransform.__init__cCs"||}||}||}|Sr)rrXrr r$r$r'rs   z$RealmPredictionHeadTransform.forwardrCrrrxrrr$r$rr'rWs  rWcs,eZdZfddZddZddZZS)RealmLMPredictionHeadcsLtt||_tj|j|jdd|_t t |j|_ |j |j_ dS)NF)r.) rwrxrW transformrrr{rzdecoder ParameterrUrr.rrr$r'rxs  zRealmLMPredictionHead.__init__cCs|j|j_dSr)r.r\rr$r$r' _tie_weights z"RealmLMPredictionHead._tie_weightscCr r)r[r\r r$r$r'rrzRealmLMPredictionHead.forward)rCrrrxr_rrr$r$rr'rZs  rZcrV)RealmOnlyMLMHeadcstt||_dSr)rwrxrZ predictionsrrr$r'rxs zRealmOnlyMLMHead.__init__cCs||}|Sr)rb)rsequence_outputprediction_scoresr$r$r'rs zRealmOnlyMLMHead.forwardrYr$r$rr'ras racrV)RealmScorerProjectioncs>tt||_t|j|j|_tj |j|j d|_ dSr) rwrxrZrbrrr{retriever_proj_sizerrrrrr$r'rx!s  zRealmScorerProjection.__init__cCr r)rrr r$r$r'r'rzRealmScorerProjection.forwardrYr$r$rr're s recrV)RealmReaderProjectioncsXt||_t|j|jd|_t|jd|_tj |j|j d|_ t |_ dS)Nr/rrm)rwrxrYrrr{span_hidden_sizedense_intermediate dense_outputrreader_layer_norm_epslayer_normalizationReLUrelurrr$r'rx.s zRealmReaderProjection.__init__csfdd}tjfdd}|}|jddd\}}||\}}} tj|d|d } tj|d|d } | | } | } | } | d} | || | j d 7} | ||fS) Ncsj\}fddtfddtjjD\}}t|d}t|d}tjd|d}tjd|d}||}|||fS)aK Generate span candidates. Args: masks: [num_retrievals, max_sequence_len] Returns: starts: [num_spans] ends: [num_spans] span_masks: [num_retrievals, num_spans] whether spans locate in evidence block. cs6tj|djd}tj|djd}||fS)Nrr)rUrr)widthcurrent_starts current_ends)masksmax_sequence_lenr$r'_spans_given_widthDszRRealmReaderProjection.forward..span_candidates.._spans_given_widthc3s|] }|dVqdS)rNr$)r%w)rur$r'r(IszIRealmReaderProjection.forward..span_candidates..rrrrr)rRr?r%rYmax_span_widthrUr index_select)rsr!startsends start_masks end_masks span_masksr^)rursrtr'span_candidates7s "   z6RealmReaderProjection.forward..span_candidatescSd||t|jSN?typerUfinfominmaskrvr$r$r' mask_to_scoreVz4RealmReaderProjection.forward..mask_to_scorer/rrrrrwru) rUfloat32richunkryrnrlrjsqueezerv)rr block_maskrrstart_projectionend_projectioncandidate_startscandidate_endscandidate_maskcandidate_start_projectionscandidate_end_projectionscandidate_hidden reader_logitsr$r^r'r6s     zRealmReaderProjection.forwardrYr$r$rr'rg-s rgaH This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`RealmConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a5 Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. c@s2eZdZUdZeed<eZdZddZ ddZ dS) RealmPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. rYrealmcCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|tj rX|jj |jj ddSdS)zInitialize the weightsg)meanstdNr)r@rrr+rWnormal_rYinitializer_ranger.zero_ryrlrfill_)rmoduler$r$r' _init_weightss      z"RealmPreTrainedModel._init_weightscGsTg}|D]#}|dur|dq|j}t|dkr"|d|df}||q|S)z.Flatten inputs' shape to (-1, input_shape[-1])Nr/rr)r>rRrOr)rinputsflattened_inputsrrr$r$r'_flatten_inputss   z$RealmPreTrainedModel._flatten_inputsN) rCrrrrrArjload_tf_weightsbase_model_prefixrrr$r$r$r'rs  rcs^eZdZdZdfdd ZddZddZd d Z dd d ZZ S)RealmBertModelz? Same as the original BertModel but remove docstrings. TcsDt|||_t||_t||_|rt|nd|_| dSr) rwrxrYrkrr encoderr:pooler post_init)rrYadd_pooling_layerrr$r'rxs    zRealmBertModel.__init__cCs|jjSrrr}r^r$r$r'get_input_embeddingssz#RealmBertModel.get_input_embeddingscCs ||j_dSrrrrr$r$r'set_input_embeddingss z#RealmBertModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrr'rr)rheads_to_pruner'rr$r$r' _prune_headsszRealmBertModel._prune_headsNc CsF| dur| n|jj} | dur| n|jj} | dur| n|jj} |jjr-| dur(| n|jj} nd} |dur;|dur;td|durJ||||}n|durW|dd}ntd|\}}|durf|j n|j }| durq| nd}|durt j |||f|d}|durt |jdr|jjddd|f}|||}|}n t j|t j|d}|||}|jjr|dur|\}}}||f}|durt j ||d}||}nd}|||jj}|j|||||d }|j|||||| | | | | d }|d}|jdur||nd}| s||f|d dSt|||j|j|j|jd S) NFzDYou cannot specify both input_ids and inputs_embeds at the same timerrz5You have to specify either input_ids or inputs_embedsrrortr)rrqrtrr) rrrrrrrr)r*r)r, pooler_outputrrr-r.)rYrr)use_return_dictrrr%warn_if_padding_and_no_attention_maskrrget_seq_lengthrUonesrrrtrrrget_extended_attention_maskinvert_attention_mask get_head_maskr&rrr rrr-r.)rrrrtrqrrrrrrrr)r*r batch_sizerrrrrextended_attention_maskencoder_batch_sizeencoder_sequence_lengthr!encoder_hidden_shapeencoder_extended_attention_maskembedding_outputencoder_outputsrcr>r$r$r'rs       zRealmBertModel.forward)T NNNNNNNNNNNNN) rCrrrrxrrrrrr$r$rr'rs(  rz`The embedder of REALM outputting projected score that will be used to calculate relevance score.cseZdZdgZfddZddZddZee de e e d  dd e ejd e ejd e ejde ejde ejde ejde ede ede edeee ffddZZS)rHzcls.predictions.decoder.biasc0t|t|j|_t|j|_|dSr)rwrxrrYrrerrrrr$r'rxks    zRealmEmbedder.__init__cC |jjjSrrrr}r^r$r$r'rr z"RealmEmbedder.get_input_embeddingscC||jj_dSrrrr$r$r'rur`z"RealmEmbedder.set_input_embeddingsbatch_size, sequence_length output_type config_classNrrrtrqrrrr)r*rc Csj| dur| n|jj} |j||||||||| d } | d} || } | s,| f| ddSt| | j| jdS)a Returns: Example: ```python >>> from transformers import AutoTokenizer, RealmEmbedder >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/realm-cc-news-pretrained-embedder") >>> model = RealmEmbedder.from_pretrained("google/realm-cc-news-pretrained-embedder") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> projected_score = outputs.projected_score ``` Nrrtrqrrrr)r*rr/r)r@rr-)rYrrrr?rr-) rrrrtrqrrrr)r* realm_outputsrr@r$r$r'rxs*! zRealmEmbedder.forward) NNNNNNNNN)rCrr_tied_weights_keysrxrrrREALM_INPUTS_DOCSTRINGformatrr?_CONFIG_FOR_DOCrrUrrrrrrrr$r$rr'rHdsJ      rHzoThe scorer of REALM outputting relevance scores representing the score of document candidates (before softmax).c!seZdZdZdfdd Zeedee e d             dde e j de e jd e e j d e e j d e e j d e e jd e e j de e jde e jde e jde ede ede edeee ffddZZS) RealmScorerz Args: query_embedder ([`RealmEmbedder`]): Embedder for input sequences. If not specified, it will use the same embedder as candidate sequences. Ncs8t|t|j|_|dur|n|j|_|dSr)rwrxrHrYembedderquery_embedderr)rrYrrr$r'rxs   zRealmScorer.__init__rrrrrtrqcandidate_input_idscandidate_attention_maskcandidate_token_type_idscandidate_inputs_embedsrrrr)r*rc Cs| dur| n|jj} |dur| durtd|dur"|dur"td|j||||| | | | | d }||||\}}}|j||||| || | | d }|d}|d}|d|jj|jj}t d||}| si|||fSt |||dS) a candidate_input_ids (`torch.LongTensor` of shape `(batch_size, num_candidates, sequence_length)`): Indices of candidate input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) candidate_attention_mask (`torch.FloatTensor` of shape `(batch_size, num_candidates, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) candidate_token_type_ids (`torch.LongTensor` of shape `(batch_size, num_candidates, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) candidate_inputs_embeds (`torch.FloatTensor` of shape `(batch_size * num_candidates, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `candidate_input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *candidate_input_ids* indices into associated vectors than the model's internal embedding lookup matrix. Returns: Example: ```python >>> import torch >>> from transformers import AutoTokenizer, RealmScorer >>> tokenizer = AutoTokenizer.from_pretrained("google/realm-cc-news-pretrained-scorer") >>> model = RealmScorer.from_pretrained("google/realm-cc-news-pretrained-scorer", num_candidates=2) >>> # batch_size = 2, num_candidates = 2 >>> input_texts = ["How are you?", "What is the item in the picture?"] >>> candidates_texts = [["Hello world!", "Nice to meet you!"], ["A cute cat.", "An adorable dog."]] >>> inputs = tokenizer(input_texts, return_tensors="pt") >>> candidates_inputs = tokenizer.batch_encode_candidates(candidates_texts, max_length=10, return_tensors="pt") >>> outputs = model( ... **inputs, ... candidate_input_ids=candidates_inputs.input_ids, ... candidate_attention_mask=candidates_inputs.attention_mask, ... candidate_token_type_ids=candidates_inputs.token_type_ids, ... ) >>> relevance_score = outputs.relevance_score ```Nz5You have to specify either input_ids or input_embeds.zJYou have to specify either candidate_input_ids or candidate_inputs_embeds.rrrrz bd,bnd->bn)rCrDrE) rYrrrrrrnum_candidatesrfrUrrB)rrrrtrqrrrrrrrr)r* query_outputsflattened_input_idsflattened_attention_maskflattened_token_type_idscandidate_outputsrDrErCr$r$r'rsNI    zRealmScorer.forwardrr)rCrrrrxrrrrrBrrrUrrrrrrrr$r$rr'rs^       rzrThe knowledge-augmented encoder of REALM outputting masked language model logits and marginal log-likelihood loss.cseZdZdgZfddZddZddZdd Zd d Ze e d e e ed             ddeejdeejdeejdeejdeejdeejdeejdeejdeejdeedeedeedeee ffddZZS)rGzcls.predictions.decodercrr)rwrxrrYrrarrrrr$r'rxQs    z!RealmKnowledgeAugEncoder.__init__cCrrrr^r$r$r'rWrz-RealmKnowledgeAugEncoder.get_input_embeddingscCrrrrr$r$r'rZr`z-RealmKnowledgeAugEncoder.set_input_embeddingscCrr)rrbr\r^r$r$r'get_output_embeddings]rz.RealmKnowledgeAugEncoder.get_output_embeddingscCs||jj_|j|jj_dSr)rrbr\r.)rnew_embeddingsr$r$r'set_output_embeddings`s z.RealmKnowledgeAugEncoder.set_output_embeddingsz+batch_size, num_candidates, sequence_lengthrNrrrtrqrrrClabelsmlm_maskrr)r*rc  Cs|| dur| n|jj} |dur|durtd||||\} }}|j| |||||| | | d }|d}||}|}d}|dur|\}}| durRtj|tj d} n| tj } t dd}| d|jj }|d |jj d}||| ||jj| }|dd}||}|d }tt|| t|  }| s|f|d d }|dur|f|S|St|||j|jd S) a relevance_score (`torch.FloatTensor` of shape `(batch_size, num_candidates)`, *optional*): Relevance score derived from RealmScorer, must be specified if you want to compute the masked language modeling loss. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` mlm_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid calculating joint loss on certain positions. If not specified, the loss will not be masked. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. Returns: Example: ```python >>> import torch >>> from transformers import AutoTokenizer, RealmKnowledgeAugEncoder >>> tokenizer = AutoTokenizer.from_pretrained("google/realm-cc-news-pretrained-encoder") >>> model = RealmKnowledgeAugEncoder.from_pretrained( ... "google/realm-cc-news-pretrained-encoder", num_candidates=2 ... ) >>> # batch_size = 2, num_candidates = 2 >>> text = [["Hello world!", "Nice to meet you!"], ["The cute cat.", "The adorable dog."]] >>> inputs = tokenizer.batch_encode_candidates(text, max_length=10, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits ```NzZYou have to specify `relevance_score` when `labels` is specified in order to compute loss.rrrunone) reductionrrrr/r)rGlogitsrr-)rYrrrrrrrU ones_likerrrrrztiler log_softmax unsqueeze logsumexpnansumsumr rr-)rrrrtrqrrrCrrrr)r*rrr joint_outputs joint_outputrdrEmasked_lm_lossrrloss_fct mlm_logits mlm_targetsmasked_lm_log_probcandidate_log_probjoint_gold_log_probmarginal_gold_log_probsrr$r$r'rds^8         z RealmKnowledgeAugEncoder.forward) NNNNNNNNNNNN)rCrrrrxrrrrrrrrr rrrUrrrrrrrr$r$rr'rGIsd        rGzThe reader of REALM.c#seZdZfddZeedeee d              dde e j de e j de e j d e e j d e e j d e e j d e e j d e e jde e j de e j de e jde ede ede edeeeffddZZS)rAcs>t||j|_t||_t||_t||_| dSr) rwrx num_labelsrrrarrg qa_outputsrrrr$r'rxs     zRealmReader.__init__z!reader_beam_size, sequence_lengthrNrrrtrqrrrCrstart_positions end_positions has_answersrr)r*rc$ CsF|dur|n|jj}|durtd|durtd|d|jjkr'td|j||||||| | |d }|d}|||d|jj\}}}t |d|jjd}||7}t tj |dd j }t tj |dd j }tj |d|d }tj |d|d }d}d}d}d}d}| dur| dur| durd d }d d}|d} | d| } | d| } | }t|}!|||| d|jj| d|jjd}t|}"|||}||d|d}||!tj9}||"tj9}||}|s||||f|dd}#|dur|||||f|#S|#St||||||||||j|jd S)ar relevance_score (`torch.FloatTensor` of shape `(searcher_beam_size,)`, *optional*): Relevance score, which must be specified if you want to compute the logits and marginal log loss. block_mask (`torch.BoolTensor` of shape `(searcher_beam_size, sequence_length)`, *optional*): The mask of the evidence block, which must be specified if you want to compute the logits and marginal log loss. start_positions (`torch.LongTensor` of shape `(searcher_beam_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(searcher_beam_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. has_answers (`torch.BoolTensor` of shape `(searcher_beam_size,)`, *optional*): Whether or not the evidence block has answer(s). Returns: NzCYou have to specify `relevance_score` to calculate logits and loss.zOYou have to specify `block_mask` to separate question block and evidence block.rzQThe input sequence length must be greater than or equal to config.max_span_width.rrrrrrwcSs\ttt|ddt|d}ttt|ddt|d}tt||dS)zCompute correct span.rrrr)rUeqrrJ logical_and)rr gold_starts gold_ends is_gold_start is_gold_endr$r$r'compute_correct_candidatesDsz7RealmReader.forward..compute_correct_candidatescSs@tjfdd}tj||||jddd}tj|dd}||S)z3Loss based on the negative marginal log-likelihood.cSrrrrr$r$r'rTrzERealmReader.forward..marginal_log_loss..mask_to_scorerurrr)rUrrrv)r is_correctr log_numeratorlog_denominatorr$r$r'marginal_log_lossQsz.RealmReader.forward..marginal_log_loss)rrrrr/) rGrHrIrJrKrLrMrNrOrr-)rYrrrrxrrreader_beam_sizerUrargmaxmaxvaluesryclamprJrrrrrFrr-)$rrrrtrqrrrCrrrrrr)r*rrcrrrretriever_logitspredicted_block_indexpredicted_candidatepredicted_start predicted_end total_lossrHrIrJrKrr ignored_indexany_retriever_correctany_reader_correctrr$r$r'rs&         zRealmReader.forward)NNNNNNNNNNNNNN)rCrrrxrrrrrFrrrUrrrPrrrrrr$r$rr'rAsb        rAay Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token (should not be used in this model by design). [What are token type IDs?](../glossary#token-type-ids) answer_ids (`list` of shape `(num_answers, answer_length)`, *optional*): Answer ids for computing the marginal log-likelihood loss. Indices should be in `[-1, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-1` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z?`RealmForOpenQA` for end-to-end open domain question answering.cseZdZdfdd ZeddZddZee de e e d     dd e ejd e ejd e ejd e ejde edeee ff ddZZS)rENc s`t|t||_t||_|dtdj |j |j ftj t dd||_|dS)N block_embr$cpu)rrvr)rwrxrHrrArrrUr new_emptynum_block_recordsrfrr retrieverr)rrYrrr$r'rxs      zRealmForOpenQA.__init__cCs|jr|jjS|jjSr)r/rYsearcher_beam_sizer r^r$r$r'rsz!RealmForOpenQA.searcher_beam_sizecCs|j||_dS)zSend `self.block_emb` to a specific device. Args: device (`str` or `torch.device`): The device to which `self.block_emb` will be sent. N)rr)rrr$r$r'block_embedding_tosz!RealmForOpenQA.block_embedding_toz1, sequence_lengthrrrrt answer_idsr*rc Cs|dur|n|jj}|dur|jddkrtd|j|||dd}|d}td|j||jj }tj ||j dd \} } | } tj |jd| d } |j| |||jjd \} } }}||jj }|jtjj|jj d }||jtj| durtj| tj|jj d } tj| tj|jj d } tj|tj|jj d }td| | |jj }|j|jd|jj|jd|jj|jd|jj||| | |dd }|j|j}||j|jd}|s||fSt ||dS)a Returns: Example: ```python >>> import torch >>> from transformers import RealmForOpenQA, RealmRetriever, AutoTokenizer >>> retriever = RealmRetriever.from_pretrained("google/realm-orqa-nq-openqa") >>> tokenizer = AutoTokenizer.from_pretrained("google/realm-orqa-nq-openqa") >>> model = RealmForOpenQA.from_pretrained("google/realm-orqa-nq-openqa", retriever=retriever) >>> question = "Who is the pioneer in modern computer science?" >>> question_ids = tokenizer([question], return_tensors="pt") >>> answer_ids = tokenizer( ... ["alan mathison turing"], ... add_special_tokens=False, ... return_token_type_ids=False, ... return_attention_mask=False, ... ).input_ids >>> reader_output, predicted_answer_ids = model(**question_ids, answer_ids=answer_ids, return_dict=False) >>> predicted_answer = tokenizer.decode(predicted_answer_ids) >>> loss = reader_output.loss ```Nrrz'The batch_size of the inputs must be 1.T)rrtrr*z BD,QD->QBrr)krrw) max_lengthrorzD,BD->B) rrrtrCrrrrr*)rSrT)!rYrrRrrrUrrrrtopkrrryrrreader_seq_lenrspecial_tokens_maskrr logical_not_ logical_and_rtrrrr rrLrNrOrR)rrrrtrr*question_outputsquestion_projection batch_scoresr!retrieved_block_idsretrieved_block_embrrNrO concat_inputsrretrieved_logitsrSpredicted_blockrTr$r$r'rsV% zRealmForOpenQA.forwardr)NNNN)rCrrrxpropertyrrrREALM_FOR_OPEN_QA_DOCSTRINGrrrRrrrUrrrrrrrr$r$rr'rEs0   rE)rHrErGrrArrj)Mrrr7 dataclassesrtypingrrrUrtorch.nnr activationsr cache_utilsr modeling_layersr modeling_outputsr r r rmodeling_utilsr pytorch_utilsrrrutilsrrrrutils.deprecationrconfiguration_realmr get_loggerrCr5_EMBEDDER_CHECKPOINT_FOR_DOC_ENCODER_CHECKPOINT_FOR_DOC_SCORER_CHECKPOINT_FOR_DOCrrjModulerkrrrrrrrr r:r?rBrFrRrWrZrarergREALM_START_DOCSTRINGrrrrHrrGrAr/rE__all__r$r$r$r's          k@4WP1  C 2(N(!