o eiv@sdZddlZddlmZddlZddlmZddlmZmZm Z ddl m Z ddl mZmZdd lmZdd lmZmZdd lmZdd lmZdd lmZmZmZddlmZee Z!eeddGdddeZ"eeddGdddeZ#eeddGdddeZ$eeddGdddeZ%eeddGdddeZ&eeddGd d!d!eZ'eed"dGd#d$d$eZ(eed%dGd&d'd'eZ)eed(dGd)d*d*eZ*eed+dGd,d-d-eZ+Gd.d/d/ej,Z-Gd0d1d1ej,Z.Gd2d3d3ej,Z/Gd4d5d5ej,Z0Gd6d7d7ej,Z1Gd8d9d9ej,Z2Gd:d;d;ej,Z3Gdd?d?ej,Z5Gd@dAdAej,Z6GdBdCdCej,Z7GdDdEdEej,Z8eGdFdGdGeZ9edHdGdIdJdJe9Z:dKdLZ;GdMdNdNej,ZedUdGdVdWdWe9Z?edXdGdYdZdZe9Z@ed[dGd\d]d]e9ZAed^dGd_d`d`e9ZBeGdadbdbe9ZCeGdcdddde9ZDgdeZEdS)fzPyTorch LUKE model.N) dataclass)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FNgelu)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel)apply_chunking_to_forward) ModelOutputauto_docstringlogging) LukeConfigz3 Base class for outputs of the LUKE model. ) custom_introc@>eZdZUdZdZejdBed<dZe ejdfdBed<dS)BaseLukeModelOutputWithPoolingax pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. entity_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, entity_length, hidden_size)`): Sequence of entity hidden-states at the output of the last layer of the model. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. Nentity_last_hidden_state.entity_hidden_states __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rtupler#r#d/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/luke/modeling_luke.pyr$s  rzV Base class for model's outputs, with potential hidden states and attentions. c@r)BaseLukeModelOutputa entity_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, entity_length, hidden_size)`): Sequence of entity hidden-states at the output of the last layer of the model. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. Nr.rrr#r#r#r$r%;s  r%c@seZdZUdZdZejdBed<dZejdBed<dZ ejdBed<dZ ejdBed<dZ ejdBed<dZ e ejdBed<dZe ejd fdBed <dZe ejd fdBed <dS) LukeMaskedLMOutputa: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): The sum of masked language modeling (MLM) loss and entity prediction loss. mlm_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Masked language modeling (MLM) loss. mep_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Masked entity prediction (MEP) loss. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). entity_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the entity prediction head (scores for each entity vocabulary token before SoftMax). entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. Nlossmlm_lossmep_losslogits entity_logits hidden_states.r attentions)rrrrr'rr r!r(r)r*r+r,r"rr-r#r#r#r$r&Os r&z2 Outputs of entity classification models. c@eZdZUdZdZejdBed<dZejdBed<dZ e ejdfdBed<dZ e ejdfdBed<dZ e ejdfdBed<dS) EntityClassificationOutput loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification scores (before SoftMax). entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. Nr'r*.r,rr- rrrrr'rr r!r*r,r"rr-r#r#r#r$r/q  r/z7 Outputs of entity pair classification models. c@r.) EntityPairClassificationOutputr0Nr'r*.r,rr-r1r#r#r#r$r3r2r3z7 Outputs of entity span classification models. c@r.) EntitySpanClassificationOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification loss. logits (`torch.FloatTensor` of shape `(batch_size, entity_length, config.num_labels)`): Classification scores (before SoftMax). entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. Nr'r*.r,rr-r1r#r#r#r$r4r2r4z4 Outputs of sentence classification models. c@r.) LukeSequenceClassifierOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification (or regression if config.num_labels==1) loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. Nr'r*.r,rr-r1r#r#r#r$r5r2r5z@ Base class for outputs of token classification models. c@r.) LukeTokenClassifierOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification loss. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_labels)`): Classification scores (before SoftMax). entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. Nr'r*.r,rr-r1r#r#r#r$r6r2r6z/ Outputs of question answering models. c@seZdZUdZdZejdBed<dZejdBed<dZ ejdBed<dZ e ejdfdBed<dZ e ejdfdBed<dZ e ejdfdBed <dS) LukeQuestionAnsweringModelOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Total span extraction loss is the sum of a Cross-Entropy for the start and end positions. entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. Nr' start_logits end_logits.r,rr-)rrrrr'rr r!r8r9r,r"rr-r#r#r#r$r7s  r7z, Outputs of multiple choice models. c@r.) LukeMultipleChoiceModelOutputa loss (`torch.FloatTensor` of shape *(1,)*, *optional*, returned when `labels` is provided): Classification loss. logits (`torch.FloatTensor` of shape `(batch_size, num_choices)`): *num_choices* is the second dimension of the input tensors. (see *input_ids* above). Classification scores (before SoftMax). entity_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, entity_length, hidden_size)`. Entity hidden-states of the model at the output of each layer plus the initial entity embedding outputs. Nr'r*.r,rr-r1r#r#r#r$r:s  r:cs:eZdZdZfddZ    d ddZddZZS) LukeEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_|j|_tj|j|j|jd|_ dS)N padding_idxeps)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutr=selfconfig __class__r#r$rA&s   zLukeEmbeddings.__init__Nc Cs|dur|durt||j|j}n||}|dur!|}n|dd}|dur8tj|tj|j jd}|durA| |}| |}| |}|||}| |}||}|S)Ndtypedevice)"create_position_ids_from_input_idsr=torX&create_position_ids_from_inputs_embedssizerzeroslong position_idsrFrHrJrKrO) rQ input_idstoken_type_idsr_ inputs_embeds input_shaperHrJ embeddingsr#r#r$forward5s"        zLukeEmbeddings.forwardcCsN|dd}|d}tj|jd||jdtj|jd}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor NrUrrVr)r\raranger=r^rX unsqueezeexpand)rQrbrcsequence_lengthr_r#r#r$r[Vs  z5LukeEmbeddings.create_position_ids_from_inputs_embeds)NNNN)rrrrrArer[ __classcell__r#r#rSr$r;!s  !r;csFeZdZdeffdd Z d dejdejdejdBfdd ZZS) LukeEntityEmbeddingsrRcst||_tj|j|jdd|_|j|jkr$tj |j|jdd|_ t|j |j|_ t|j |j|_tj|j|jd|_t|j|_dS)Nrr<Fbiasr>)r@rArRrrBentity_vocab_sizeentity_emb_sizeentity_embeddingsrDLinearentity_embedding_denserGrHrIrJrKrLrMrNrOrPrSr#r$rAis  zLukeEntityEmbeddings.__init__N entity_idsr_rac Cs|dur t|}||}|jj|jjkr||}||jdd}|dk | d}||}tj |dd}||j ddjdd}| |}|||}| |}||}|S)Nr)minrUdimgHz>)r zeros_likerprRrorDrrrHclamptype_asrgsumrJrKrO) rQrsr_rarprHposition_embedding_maskrJrdr#r#r$rews       zLukeEntityEmbeddings.forwardN) rrrrrAr LongTensorrerjr#r#rSr$rkhsrkcs2eZdZfddZddZ  d ddZZS) LukeSelfAttentioncst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_|j |_ t |j|j|_ t |j|j|_ t |j|j|_|j rpt |j|j|_t |j|j|_t |j|j|_t |j|_dS)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .)r@rArDnum_attention_headshasattr ValueErrorintattention_head_size all_head_sizeuse_entity_aware_attentionrrqquerykeyvalue w2e_query e2w_query e2e_queryrMattention_probs_dropout_probrOrPrSr#r$rAs&  zLukeSelfAttention.__init__cCs6|dd|j|jf}|j|}|ddddS)NrUrrr)r\rrviewpermute)rQx new_x_shaper#r#r$transpose_for_scoress z&LukeSelfAttention.transpose_for_scoresNFcCs|d}|dur |}n tj||gdd}|||}|||}|jr|dur|||} |||} || |} || |} |ddddd|ddf} |ddddd|ddf}|dddd|dddf}|dddd|dddf}t | | dd}t | | dd}t | | dd}t | | dd}tj||gdd}tj||gdd}tj||gdd}n|||}t || dd}|t |j}|dur||}tjj|dd}||}t ||}|dddd}|dd|jf}|j|}|ddd|ddf}|dur5d}n |dd|dddf}|rL|||f}|S||f}|S)NrrvrUrurrr)r\rcatrrrrrrrrmatmul transposemathsqrtrr functionalsoftmaxrOr contiguousrr)rQword_hidden_statesrattention_maskoutput_attentions word_sizeconcat_hidden_states key_layer value_layerw2w_query_layerw2e_query_layere2w_query_layere2e_query_layer w2w_key_layer e2w_key_layer w2e_key_layer e2e_key_layerw2w_attention_scoresw2e_attention_scorese2w_attention_scorese2e_attention_scoresword_attention_scoresentity_attention_scoresattention_scores query_layerattention_probs context_layernew_context_layer_shapeoutput_word_hidden_statesoutput_entity_hidden_statesoutputsr#r#r$resT          zLukeSelfAttention.forwardNF)rrrrArrerjr#r#rSr$rs   rc8eZdZfddZdejdejdejfddZZS)LukeSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNr>) r@rArrqrDdenserKrLrMrNrOrPrSr#r$rA zLukeSelfOutput.__init__r, input_tensorreturncC&||}||}|||}|Sr}rrOrKrQr,rr#r#r$re  zLukeSelfOutput.forwardrrrrArTensorrerjr#r#rSr$r $rcs*eZdZfddZ  dddZZS) LukeAttentioncs"tt||_t||_dSr})r@rArrQroutputrPrSr#r$rAs  zLukeAttention.__init__NFc Cs|d}|||||}|dur|d}|}ntj|dddd}tj||gdd}|||} | ddd|ddf} |durFd} n | dd|dddf} | | f|dd} | S)Nrrrrv)r\rQrrr) rQrrrrr self_outputsconcat_self_outputsrattention_outputword_attention_outputentity_attention_outputrr#r#r$res&  zLukeAttention.forwardrrrrrArerjr#r#rSr$rs   rc2eZdZfddZdejdejfddZZS)LukeIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr}) r@rArrqrDintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrPrSr#r$rA9s   zLukeIntermediate.__init__r,rcCs||}||}|Sr})rrrQr,r#r#r$reAs  zLukeIntermediate.forwardrr#r#rSr$r8s rcr) LukeOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r@rArrqrrDrrKrLrMrNrOrPrSr#r$rAIrzLukeOutput.__init__r,rrcCrr}rrr#r#r$reOrzLukeOutput.forwardrr#r#rSr$rHrrcs2eZdZfddZ  d ddZddZZS) LukeLayercs:t|j|_d|_t||_t||_t||_ dSNr) r@rAchunk_size_feed_forward seq_len_dimr attentionr intermediaterrrPrSr#r$rAWs   zLukeLayer.__init__NFc Cs|d}|j||||d}|dur|d}n tj|dddd}|dd}t|j|j|j|} | ddd|ddf} |durFd} n | dd|dddf} | | f|}|S)Nr)rrrrv)r\rrrrfeed_forward_chunkrr) rQrrrrrself_attention_outputsconcat_attention_outputr layer_outputword_layer_outputentity_layer_outputr#r#r$re_s(    zLukeLayer.forwardcCs||}|||}|Sr})rr)rQrintermediate_outputrr#r#r$rs  zLukeLayer.feed_forward_chunkr)rrrrArerrjr#r#rSr$rVs    #rcs.eZdZfddZ    dddZZS) LukeEncodercs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr#)r).0_rRr#r$ sz(LukeEncoder.__init__..F) r@rArRr ModuleListrangenum_hidden_layerslayergradient_checkpointingrPrSrr$rAs   zLukeEncoder.__init__NFTc Cs|rdnd}|r dnd}|rdnd} t|jD],\} } |r'||f}||f}| ||||} | d}|dur:| d}|rC| | df} q|rP||f}||f}|s`tdd||| ||fDSt||| ||dS)Nr#rrrcs|] }|dur|VqdSr}r#rvr#r#r$ z&LukeEncoder.forward..)last_hidden_stater,r-rr) enumeraterr"r%) rQrrrroutput_hidden_states return_dictall_word_hidden_statesall_entity_hidden_statesall_self_attentionsi layer_module layer_outputsr#r#r$resL       zLukeEncoder.forward)NFFTrr#r#rSr$rs  rcr) LukePoolercs*tt|j|j|_t|_dSr})r@rArrqrDrTanh activationrPrSr#r$rAs zLukePooler.__init__r,rcCs(|dddf}||}||}|S)Nr)rr)rQr,first_token_tensor pooled_outputr#r#r$res  zLukePooler.forwardrr#r#rSr$rs rc$eZdZfddZddZZS)EntityPredictionHeadTransformcsVtt|j|j|_t|jt rt |j|_ n|j|_ tj |j|j d|_ dSr)r@rArrqrDrorrrrr transform_act_fnrKrLrPrSr#r$rAs  z&EntityPredictionHeadTransform.__init__cCs"||}||}||}|Sr})rrrKrr#r#r$res   z%EntityPredictionHeadTransform.forwardrr#r#rSr$rs  rcr)EntityPredictionHeadcsHt||_t||_tj|j|jdd|_ t t |j|_ dS)NFrl)r@rArRr transformrrqrorndecoder Parameterrr]rmrPrSr#r$rAs  zEntityPredictionHead.__init__cCs||}|||j}|Sr})r r rmrr#r#r$res zEntityPredictionHead.forwardrr#r#rSr$r s r c@s>eZdZUeed<dZdZddgZe de j fddZ d S) LukePreTrainedModelrRlukeTrrkmodulecCst|tjr tj|jd|jjd|jdurt |jdSdSt|tj rX|j dkr2t |jn tj|jd|jjd|j durTt |jddsVt |j|j dSdSdSt|tjrlt |jt|jdSdS)zInitialize the weightsg)meanstdNr_is_hf_initializedF)rrrqinitnormal_weightrRinitializer_rangermzeros_rB embedding_dimr=getattrrKones_)rQrr#r#r$ _init_weightss      z!LukePreTrainedModel._init_weightsN) rrrrr!base_model_prefixsupports_gradient_checkpointing_no_split_modulesrno_gradrModulerr#r#r#r$r s r zt The bare LUKE model transformer outputting raw hidden-states for both word tokens and entities without any cseZdZd!dedeffdd ZddZdd Zd d Zd d Z e            d"de j dBde j dBde j dBde j dBde j dBde j dBde j dBde j dBde j dBdedBdedBdedBdeeBfddZde j de j dBfdd ZZS)# LukeModelTrRadd_pooling_layercsNt|||_t||_t||_t||_|rt |nd|_ | dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r@rArRr;rdrkrprencoderrpooler post_init)rQrRr"rSr#r$rAs     zLukeModel.__init__cC|jjSr}rdrFrQr#r#r$get_input_embeddings+zLukeModel.get_input_embeddingscC ||j_dSr}r'rQrr#r#r$set_input_embeddings. zLukeModel.set_input_embeddingscCs|jjSr}rpr(r#r#r$get_entity_embeddings1r*zLukeModel.get_entity_embeddingscCs ||j_dSr}r/r,r#r#r$set_entity_embeddings4r.zLukeModel.set_entity_embeddingsNr`rrar_rsentity_attention_maskentity_token_type_idsentity_position_idsrbrrrrc Ks| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur*| dur*td|dur9||||}n| durF| dd}ntd|\}}|durU|jn| j}|duretj ||f|d}|durrtj |tj |d}|dur|d}|durtj ||f|d}|durtj ||ftj |d}|j |||| d}| ||}|durd}n||||}|j|||| | | d }|d }|jdur||nd}| s||f|ddSt|||j|j|j|jd S) uz entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. Examples: ```python >>> from transformers import AutoTokenizer, LukeModel >>> tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-base") >>> model = LukeModel.from_pretrained("studio-ousia/luke-base") # Compute the contextualized entity representation corresponding to the entity mention "Beyoncé" >>> text = "Beyoncé lives in Los Angeles." >>> entity_spans = [(0, 7)] # character-based entity span corresponding to "Beyoncé" >>> encoding = tokenizer(text, entity_spans=entity_spans, add_prefix_space=True, return_tensors="pt") >>> outputs = model(**encoding) >>> word_last_hidden_state = outputs.last_hidden_state >>> entity_last_hidden_state = outputs.entity_last_hidden_state # Input Wikipedia entities to obtain enriched contextualized representations of word tokens >>> text = "Beyoncé lives in Los Angeles." >>> entities = [ ... "Beyoncé", ... "Los Angeles", ... ] # Wikipedia entity titles corresponding to the entity mentions "Beyoncé" and "Los Angeles" >>> entity_spans = [ ... (0, 7), ... (17, 28), ... ] # character-based entity spans corresponding to "Beyoncé" and "Los Angeles" >>> encoding = tokenizer( ... text, entities=entities, entity_spans=entity_spans, add_prefix_space=True, return_tensors="pt" ... ) >>> outputs = model(**encoding) >>> word_last_hidden_state = outputs.last_hidden_state >>> entity_last_hidden_state = outputs.entity_last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerUz5You have to specify either input_ids or inputs_embeds)rXrVr)r`r_rarb)rrrrr)r pooler_outputr,r-rr)rRrruse_return_dictr%warn_if_padding_and_no_attention_maskr\rXronesr]r^rdget_extended_attention_maskrpr#r$rr,r-rr)rQr`rrar_rsr2r3r4rbrrrkwargsrc batch_size seq_lengthrXentity_seq_lengthword_embedding_outputextended_attention_maskentity_embedding_outputencoder_outputssequence_outputrr#r#r$re7slI     zLukeModel.forwardword_attention_maskcCs|}|durtj||gdd}|dkr$|dddddddf}n|dkr7|ddddddf}n td|jd|j|jd}d |t|jj}|S) ac Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: word_attention_mask (`torch.LongTensor`): Attention mask for word tokens with ones indicating tokens to attend to, zeros for tokens to ignore. entity_attention_mask (`torch.LongTensor`, *optional*): Attention mask for entity tokens with ones indicating tokens to attend to, zeros for tokens to ignore. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. NrUrvrrz&Wrong shape for attention_mask (shape ))rWg?) rrrwrshaperZrWfinfort)rQrCr2rr?r#r#r$r9s  z%LukeModel.get_extended_attention_mask)T) NNNNNNNNNNNN)rrrrboolrAr)r-r0r1rrr~r r"rrer9rjr#r#rSr$r!sh      r!cCs2||}tj|dd||}||S)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rrv)nerrcumsumrzr^)r`r=maskincremental_indicesr#r#r$rYs  rYcs(eZdZdZfddZddZZS) LukeLMHeadz*Roberta Head for masked language modeling.csZtt|j|j|_tj|j|jd|_t|j|j |_ t t |j |_dSr)r@rArrqrDrrKrL layer_normrCr r rr]rmrPrSr#r$rAs zLukeLMHead.__init__cKs*||}t|}||}||}|Sr})rr rMr )rQfeaturesr:rr#r#r$res   zLukeLMHead.forward)rrrrrArerjr#r#rSr$rLs rLz The LUKE model with a language modeling head and entity prediction head on top for masked language modeling and masked entity prediction. c!seZdZdddZfddZddZdd Ze dd ej d Bd ej d Bd ej d Bdej d Bdej d Bdej d Bdej d Bdej d Bdej d Bdej d Bdej d Bde d Bde d Bde d Bde e BfddZZS)LukeForMaskedLMz/luke.entity_embeddings.entity_embeddings.weightzlm_head.decoder.bias)z!entity_predictions.decoder.weightz lm_head.biascs@t|t||_t||_t||_t |_ | dSr}) r@rAr!rrLlm_headr entity_predictionsrrloss_fnr%rPrSr#r$rAs      zLukeForMaskedLM.__init__cCr&r}rPr r(r#r#r$get_output_embeddings*r*z%LukeForMaskedLM.get_output_embeddingscCr+r}rS)rQnew_embeddingsr#r#r$set_output_embeddings-r.z%LukeForMaskedLM.set_output_embeddingsNr`rrar_rsr2r3r4labels entity_labelsrbrrrrcKs,|dur|n|jj}|j||||||||| | | dd }d}d}||j}| durD| |j} ||d|jj | d}|durD|}d}d}|j durq| |j }| durq||d|jj | d}|durm|}n||}|st dd||||||j|j|jfDSt||||||j|j|jdS)aC entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` entity_labels (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` NT r`rrar_rsr2r3r4rbrrrrUcsrr}r#rr#r#r$rs  z*LukeForMaskedLM.forward..)r'r(r)r*r+r,rr-)rRr6rrPrrZrXrRrrCrrQrnr"r,rr-r&)rQr`rrar_rsr2r3r4rWrXrbrrrr:rr'r(r*r)r+r#r#r$re0sl1    zLukeForMaskedLM.forwardNNNNNNNNNNNNNN)rrr_tied_weights_keysrArTrVrrr~r rGr"r&rerjr#r#rSr$rOsj       rOz The LUKE model with a classification head on top (a linear layer on top of the hidden state of the first entity token) for entity classification tasks, such as Open Entity. ceZdZfddZe             ddejdBdejdBdejdBdejdBdejdBd ejdBd ejdBd ejdBd ejdBd ejdBdedBdedBdedBde e BfddZ Z S)LukeForEntityClassificationcsJt|t||_|j|_t|j|_t |j |j|_ | dSr} r@rAr!r num_labelsrrMrNrOrqrD classifierr%rPrSr#r$rAs   z$LukeForEntityClassification.__init__Nr`rrar_rsr2r3r4rbrWrrrrcKs| dur| n|jj} |j||||||||| | | dd }|jdddddf}||}||}d}| durZ| |j} | jdkrJt j || }nt j | d| d|}| smtdd|||j|j|jfDSt|||j|j|jd S) u entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. labels (`torch.LongTensor` of shape `(batch_size,)` or `(batch_size, num_labels)`, *optional*): Labels for computing the classification loss. If the shape is `(batch_size,)`, the cross entropy loss is used for the single-label classification. In this case, labels should contain the indices that should be in `[0, ..., config.num_labels - 1]`. If the shape is `(batch_size, num_labels)`, the binary cross entropy loss is used for the multi-label classification. In this case, labels should only contain `[0, 1]`, where 0 and 1 indicate false and true, respectively. Examples: ```python >>> from transformers import AutoTokenizer, LukeForEntityClassification >>> tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-large-finetuned-open-entity") >>> model = LukeForEntityClassification.from_pretrained("studio-ousia/luke-large-finetuned-open-entity") >>> text = "Beyoncé lives in Los Angeles." >>> entity_spans = [(0, 7)] # character-based entity span corresponding to "Beyoncé" >>> inputs = tokenizer(text, entity_spans=entity_spans, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: person ```NTrYrrrUcsrr}r#rr#r#r$rz6LukeForEntityClassification.forward..r'r*r,rr-)rRr6rrrOr`rZrXndimrr cross_entropy binary_cross_entropy_with_logitsrrzr"r,rr-r/rQr`rrar_rsr2r3r4rbrWrrrr:rfeature_vectorr*r'r#r#r$resF>     z#LukeForEntityClassification.forward NNNNNNNNNNNNN) rrrrArrr~r rGr"r/rerjr#r#rSr$r]Z       r]z The LUKE model with a classification head on top (a linear layer on top of the hidden states of the two entity tokens) for entity pair classification tasks, such as TACRED. cseZdZfddZe             ddejdBdejdBdejdBdejdBdejdBd ejdBd ejdBd ejdBd ejdBd ejdBdedBdedBdedBde e BfddZ Z S)LukeForEntityPairClassificationcsPt|t||_|j|_t|j|_t |j d|jd|_ | dS)NrFr^rPrSr#r$rA,s   z(LukeForEntityPairClassification.__init__Nr`rrar_rsr2r3r4rbrWrrrrcKs| dur| n|jj} |j||||||||| | | dd }tj|jdddddf|jdddddfgdd}||}||}d}| durk| |j } | j dkr[t j || }nt j |d| d|}| s~tdd |||j|j|jfDSt|||j|j|jd S) u entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. labels (`torch.LongTensor` of shape `(batch_size,)` or `(batch_size, num_labels)`, *optional*): Labels for computing the classification loss. If the shape is `(batch_size,)`, the cross entropy loss is used for the single-label classification. In this case, labels should contain the indices that should be in `[0, ..., config.num_labels - 1]`. If the shape is `(batch_size, num_labels)`, the binary cross entropy loss is used for the multi-label classification. In this case, labels should only contain `[0, 1]`, where 0 and 1 indicate false and true, respectively. Examples: ```python >>> from transformers import AutoTokenizer, LukeForEntityPairClassification >>> tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-large-finetuned-tacred") >>> model = LukeForEntityPairClassification.from_pretrained("studio-ousia/luke-large-finetuned-tacred") >>> text = "Beyoncé lives in Los Angeles." >>> entity_spans = [ ... (0, 7), ... (17, 28), ... ] # character-based entity spans corresponding to "Beyoncé" and "Los Angeles" >>> inputs = tokenizer(text, entity_spans=entity_spans, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: per:cities_of_residence ```NTrYrrrvrUcsrr}r#rr#r#r$rraz:LukeForEntityPairClassification.forward..rb)rRr6rrrrrOr`rZrXrcrrrdrerrzr"r,rr-r3rfr#r#r$re8sJA0     z'LukeForEntityPairClassification.forwardrh) rrrrArrr~r rGr"r3rerjr#r#rSr$rj%rirjz The LUKE model with a span classification head on top (a linear layer on top of the hidden states output) for tasks such as named entity recognition. c#seZdZfddZe               ddejdBdejdBdejdBdejdBdejdBd ejdBd ejdBd ejdBd ejdBd ejdBdejdBdejdBdedBdedBdedBde e Bf ddZ Z S)LukeForEntitySpanClassificationcsNt|t||_|j|_t|j|_t |j d|j|_ | dS)Nrr^rPrSr#r$rAs   z(LukeForEntitySpanClassification.__init__Nr`rrar_rsr2r3r4entity_start_positionsentity_end_positionsrbrWrrrrcKs|dur|n|jj}|j||||||||| | |dd }|jd}| ddd|} | j|jjkr9| |jj} t |jd| }| ddd|} | j|jjkrY| |jj} t |jd| }t j |||j gdd}| |}||}d}| dur| |j} | jdkrtj|d|j| d}ntj|d| d|}|stdd |||j|j|jfDSt|||j|j|jd S) u entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. entity_start_positions (`torch.LongTensor`): The start positions of entities in the word token sequence. entity_end_positions (`torch.LongTensor`): The end positions of entities in the word token sequence. labels (`torch.LongTensor` of shape `(batch_size, entity_length)` or `(batch_size, entity_length, num_labels)`, *optional*): Labels for computing the classification loss. If the shape is `(batch_size, entity_length)`, the cross entropy loss is used for the single-label classification. In this case, labels should contain the indices that should be in `[0, ..., config.num_labels - 1]`. If the shape is `(batch_size, entity_length, num_labels)`, the binary cross entropy loss is used for the multi-label classification. In this case, labels should only contain `[0, 1]`, where 0 and 1 indicate false and true, respectively. Examples: ```python >>> from transformers import AutoTokenizer, LukeForEntitySpanClassification >>> tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-large-finetuned-conll-2003") >>> model = LukeForEntitySpanClassification.from_pretrained("studio-ousia/luke-large-finetuned-conll-2003") >>> text = "Beyoncé lives in Los Angeles" # List all possible entity spans in the text >>> word_start_positions = [0, 8, 14, 17, 21] # character-based start positions of word tokens >>> word_end_positions = [7, 13, 16, 20, 28] # character-based end positions of word tokens >>> entity_spans = [] >>> for i, start_pos in enumerate(word_start_positions): ... for end_pos in word_end_positions[i:]: ... entity_spans.append((start_pos, end_pos)) >>> inputs = tokenizer(text, entity_spans=entity_spans, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> predicted_class_indices = logits.argmax(-1).squeeze().tolist() >>> for span, predicted_class_idx in zip(entity_spans, predicted_class_indices): ... if predicted_class_idx != 0: ... print(text[span[0] : span[1]], model.config.id2label[predicted_class_idx]) Beyoncé PER Los Angeles LOC ```NTrYrUrurrvcsrr}r#rr#r#r$r:raz:LukeForEntitySpanClassification.forward..rb)rRr6rrr\rgrhrXrZrgatherrrrOr`rcrrrdrr_rerzr"r,rr-r4)rQr`rrar_rsr2r3r4rlrmrbrWrrrr:rrD start_states end_statesrgr*r'r#r#r$resXO       z'LukeForEntitySpanClassification.forward)NNNNNNNNNNNNNNN) rrrrArrr~r rGr"r4rerjr#r#rSr$rksf       rkz The LUKE Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cr\)LukeForSequenceClassificationcsZt||j|_t||_t|jdur|jn|j|_ t |j |j|_ | dSr}r@rAr_r!rrrMclassifier_dropoutrNrOrqrDr`r%rPrSr#r$rAPs   z&LukeForSequenceClassification.__init__Nr`rrar_rsr2r3r4rbrWrrrrcKs| dur| n|jj} |j||||||||| | | dd }|j}||}||}d}| dur| |j} |jjdur^|j dkrDd|j_n|j dkrZ| j t j ksU| j t j krZd|j_nd|j_|jjdkr|t}|j dkrv||| }n+||| }n%|jjdkrt}||d|j | d}n|jjdkrt}||| }| std d |||j|j|jfDSt|||j|j|jd S) a entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTrYr regressionsingle_label_classificationmulti_label_classificationrUcsrr}r#rr#r#r$rraz8LukeForSequenceClassification.forward..rb)rRr6rr5rOr`rZrX problem_typer_rWrr^rrsqueezerrrr"r,rr-r5)rQr`rrar_rsr2r3r4rbrWrrrr:rrr*r'loss_fctr#r#r$re\sb+      "       z%LukeForSequenceClassification.forwardrh) rrrrArrr~r rGr"r5rerjr#r#rSr$rqIrirqz The LUKE Model with a token classification head on top (a linear layer on top of the hidden-states output). To solve Named-Entity Recognition (NER) task using LUKE, `LukeForEntitySpanClassification` is more suitable than this class. cr\)LukeForTokenClassificationcs^t||j|_t|dd|_t|jdur|jn|j|_ t |j |j|_ | dSNF)r"rrrPrSr#r$rAs  z#LukeForTokenClassification.__init__Nr`rrar_rsr2r3r4rbrWrrrrcKs| dur| n|jj} |j||||||||| | | dd }|j}||}||}d}| durD| |j} t}|| d|j | d}| sWt dd|||j |j |jfDSt|||j |j |jdS)aM entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) NTrYrUcsrr}r#rr#r#r$r$raz5LukeForTokenClassification.forward..rb)rRr6rrrOr`rZrXrrr_r"r,rr-r6)rQr`rrar_rsr2r3r4rbrWrrrr:rrBr*r'ryr#r#r$resD+   z"LukeForTokenClassification.forwardrh) rrrrArrr~r rGr"r6rerjr#r#rSr$rzsZ       rzc!seZdZfddZe              ddejdBdejdBdejdBdejdBdejdBd ejdBd ejdBd ejdBd ejdBd ejdBdejdBdedBdedBdedBde e BfddZ Z S)LukeForQuestionAnsweringcs@t||j|_t|dd|_t|j|j|_| dSr{) r@rAr_r!rrrqrD qa_outputsr%rPrSr#r$rA5s  z!LukeForQuestionAnswering.__init__Nr`rrar_rsr2r3r4rbstart_positions end_positionsrrrrcKsB|dur|n|jj}|j||||||||| | | dd }|j}||}|jddd\}}|d}|d}d}| dur| durt| dkrM| d} t| dkrZ| d} |d}| d|| d|t |d}||| }||| }||d }|st d d ||||j |j |jfDSt||||j |j |jd S) a  entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. NTrYrrUrvr) ignore_indexrcsrr}r#rr#r#r$rs z3LukeForQuestionAnswering.forward..)r'r8r9r,rr-)rRr6rrr}splitrxlenr\clamp_rr"r,rr-r7)rQr`rrar_rsr2r3r4rbr~rrrrr:rrBr*r8r9 total_loss ignored_indexry start_lossend_lossr#r#r$re@sf(             z LukeForQuestionAnswering.forwardrZ) rrrrArrr~r rGr"r7rerjr#r#rSr$r|3s`       r|cr\)LukeForMultipleChoicecsPt|t||_t|jdur|jn|j|_t |j d|_ | dSr) r@rAr!rrrMrsrNrOrqrDr`r%rPrSr#r$rAs   zLukeForMultipleChoice.__init__Nr`rrar_rsr2r3r4rbrWrrrrcKs| dur| n|jj} |dur|jdn| jd}|dur%|d|dnd}|dur4|d|dnd}|durC|d|dnd}|durR|d|dnd}| dure| d| d| dnd} |durt|d|dnd}|dur|d|dnd}|dur|d|dnd}|dur|d|d|dnd}|j||||||||| | | dd }|j}||}||}|d|}d}| dur| |j } t }||| }| st dd|||j |j|jfDSt|||j |j|jd S) a^ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) entity_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`): Indices of entity tokens in the entity vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. entity_attention_mask (`torch.FloatTensor` of shape `(batch_size, entity_length)`, *optional*): Mask to avoid performing attention on padding entity token indices. Mask values selected in `[0, 1]`: - 1 for entity tokens that are **not masked**, - 0 for entity tokens that are **masked**. entity_token_type_ids (`torch.LongTensor` of shape `(batch_size, entity_length)`, *optional*): Segment token indices to indicate first and second portions of the entity token inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *portion A* entity token, - 1 corresponds to a *portion B* entity token. entity_position_ids (`torch.LongTensor` of shape `(batch_size, entity_length, max_mention_length)`, *optional*): Indices of positions of each input entity in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) NrrUruTrYcsrr}r#rr#r#r$r5rz0LukeForMultipleChoice.forward..rb)rRr6rErr\rr5rOr`rZrXrr"r,rr-r:)rQr`rrar_rsr2r3r4rbrWrrrr: num_choicesrrr*reshaped_logitsr'ryr#r#r$res|C      zLukeForMultipleChoice.forwardrh) rrrrArrr~r rGr"r:rerjr#r#rSr$rsZ       r) r]rjrkrr|rqrzrOr!r )Frr dataclassesrrrtorch.nnrrrrr activationsr r modeling_layersr modeling_outputsr r modeling_utilsr pytorch_utilsrutilsrrrconfiguration_luker get_loggerrloggerrr%r&r/r3r4r5r6r7r:r r;rkrrrrrrrrrr r r!rYrLrOr]rjrkrqrzr|r__all__r#r#r#r$s         G+m*2BR{wfu !