o eie@sndZddlmZddlZddlmZddlmZmZmZddl m Z ddl m Z dd lmZdd lmZmZmZmZmZmZdd lmZmZdd lmZdd lmZddlmZm Z m!Z!m"Z"ddl#m$Z$e"%e&Z'Gdddej(Z)Gdddej(Z*Gdddej(Z+Gdddej(Z,Gdddej(Z-Gdddej(Z.Gdddej(Z/Gdd d ej(Z0Gd!d"d"ej(Z1 #dCd$ej(d%ej2d&ej2d'ej2d(ej2dBd)e3d*e3fd+d,Z4Gd-d.d.ej(Z5Gd/d0d0ej(Z6Gd1d2d2eZ7Gd3d4d4ej(Z8e Gd5d6d6eZ9e Gd7d8d8e9Z:e Gd9d:d:e9Z;e d;d<Gd=d>d>e9ZdS)DzPyTorch MarkupLM model.)CallableN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward)TransformersKwargsauto_docstringcan_return_tuplelogging)MarkupLMConfigcs*eZdZdZfddZdddZZS)XPathEmbeddingszConstruct the embeddings from xpath tags and subscripts. We drop tree-id in this version, as its info can be covered by xpath. cstj|_tj|jj|_tj |_ t |_ tj|jdj|_ tdjj|_tfddt|jD|_tfddt|jD|_dS)Ncg|] }tjjqS)r Embeddingmax_xpath_tag_unit_embeddingsxpath_unit_hidden_size.0_configrl/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/markuplm/modeling_markuplm.py >z,XPathEmbeddings.__init__..crr)rrmax_xpath_subs_unit_embeddingsr!r"r%rr'r(Er))super__init__ max_depthrLinearr! hidden_sizexpath_unitseq2_embeddingsDropouthidden_dropout_probdropoutReLU activationxpath_unitseq2_inner inner2emb ModuleListrangexpath_tag_sub_embeddingsxpath_subs_sub_embeddingsselfr& __class__r%r'r,1s"     zXPathEmbeddings.__init__Nc Csg}g}t|jD](}||j||dddd|f||j||dddd|fq tj|dd}tj|dd}||}||| | |}|S)Ndim) r9r-appendr:r;torchcatr7r3r5r6)r=xpath_tags_seqxpath_subs_seqxpath_tags_embeddingsxpath_subs_embeddingsixpath_embeddingsrrr'forwardKs&(zXPathEmbeddings.forward)NN)__name__ __module__ __qualname____doc__r,rL __classcell__rrr>r'r+s rcsPeZdZdZfddZeddZed ddZ d d d ZZ S)MarkupLMEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cst||_tj|j|j|jd|_t|j |j|_ |j |_ t ||_ t|j|j|_tj|j|jd|_t|j|_|jdt|j ddd|j|_tj|j |j|jd|_ dS)N) padding_idxeps position_idsrr@F) persistent)r+r,r&rr vocab_sizer/ pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingsr-rrKtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsr1r2r3register_bufferrDarangeexpandrSr<r>rr'r,`s    zMarkupLMEmbeddings.__init__cCsJ|dd}|d}tj|d||dtj|jd}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr@rdtypedevicer)sizerDrclongrg unsqueezerd) inputs_embedsrS input_shapesequence_lengthrVrrr'&create_position_ids_from_inputs_embedsxs  z9MarkupLMEmbeddings.create_position_ids_from_inputs_embedsrcCs6||}tj|dd|||}||S)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rrA)neintrDcumsumtype_asri) input_idsrSpast_key_values_lengthmaskincremental_indicesrrr'"create_position_ids_from_input_idss  z5MarkupLMEmbeddings.create_position_ids_from_input_idsNcCs@|dur |}n|dd}|dur|jn|j}|dur2|dur+|||j}n|||j}|dur?tj|tj|d}|durH||}|dura|j j tj t t ||jgtj|d}|durz|j jtj t t ||jgtj|d}|} ||} ||} |||} | | | | } || } || } | S)Nr@re)rhrgrwrSrnrDzerosrir[r& tag_pad_idonestuplelistr- subs_pad_idr]r_rKr`r3)r=rsrFrGtoken_type_idsrVrkrlrgwords_embeddingsr]r_rK embeddingsrrr'rLs8         zMarkupLMEmbeddings.forward)r)NNNNNN) rMrNrOrPr, staticmethodrnrwrLrQrrr>r'rR]s   rRc8eZdZfddZdejdejdejfddZZS)MarkupLMSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNrT) r+r,rr.r/denser`rar1r2r3r<r>rr'r, zMarkupLMSelfOutput.__init__ hidden_states input_tensorreturncC&||}||}|||}|SNrr3r`r=rrrrr'rL  zMarkupLMSelfOutput.forwardrMrNrOr,rDTensorrLrQrrr>r'r $rc2eZdZfddZdejdejfddZZS)MarkupLMIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) r+r,rr.r/intermediate_sizer isinstance hidden_actstrr intermediate_act_fnr<r>rr'r,s   zMarkupLMIntermediate.__init__rrcC||}||}|Sr)rrr=rrrr'rL  zMarkupLMIntermediate.forwardrrrr>r'rs rcr)MarkupLMOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r+r,rr.rr/rr`rar1r2r3r<r>rr'r,rzMarkupLMOutput.__init__rrrcCrrrrrrr'rLrzMarkupLMOutput.forwardrrrr>r'rrrcr)MarkupLMPoolercs*tt|j|j|_t|_dSr)r+r,rr.r/rTanhr5r<r>rr'r,s zMarkupLMPooler.__init__rrcCs(|dddf}||}||}|S)Nr)rr5)r=rfirst_token_tensor pooled_outputrrr'rLs  zMarkupLMPooler.forwardrrrr>r'rs rcr)MarkupLMPredictionHeadTransformcsVtt|j|j|_t|jtrt |j|_ n|j|_ tj |j|j d|_ dSr) r+r,rr.r/rrrrr transform_act_fnr`rar<r>rr'r,s  z(MarkupLMPredictionHeadTransform.__init__rrcCs"||}||}||}|Sr)rrr`rrrr'rLs   z'MarkupLMPredictionHeadTransform.forwardrrrr>r'rs  rcs$eZdZfddZddZZS)MarkupLMLMPredictionHeadcsBtt||_tj|j|jdd|_t t |j|_ dS)NT)bias) r+r,r transformrr.r/rYdecoder ParameterrDrxrr<r>rr'r,"s  z!MarkupLMLMPredictionHead.__init__cCrr)rrrrrr'rL+rz MarkupLMLMPredictionHead.forward)rMrNrOr,rLrQrrr>r'r!s  rcr)MarkupLMOnlyMLMHeadcstt||_dSr)r+r,r predictionsr<r>rr'r,3s zMarkupLMOnlyMLMHead.__init__sequence_outputrcCs||}|Sr)r)r=rprediction_scoresrrr'rL7s zMarkupLMOnlyMLMHead.forwardrrrr>r'r2s rmodulequerykeyvalueattention_maskscalingr3c Ks|t||dd|}|dur||}tjj|dtjd|j}tjj |||j d}t||} | dd } | |fS)Nrr@)rBrf)ptrainingr) rDmatmul transposer functionalsoftmaxfloat32torfr3r contiguous) rrrrrrr3kwargs attn_weights attn_outputrrr'eager_attention_forward=s  rc VeZdZfddZ  d dejdejdBdedBdee d e ejf d d Z Z S) MarkupLMSelfAttentioncst|j|jdkrt|dstd|jd|jd||_|j|_t|j|j|_|j|j|_ t |j|j |_ t |j|j |_ t |j|j |_t |j|_|j|_|jd|_dS)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()g)r+r,r/num_attention_headshasattr ValueErrorr&rpattention_head_size all_head_sizerr.rrrr1attention_probs_dropout_probr3attention_dropoutrr<r>rr'r,Us"  zMarkupLMSelfAttention.__init__NFrroutput_attentionsrrcKs|jdd}g|d|jR}|||dd}|||dd}|||dd} t|j j t } | |||| |f|j sIdn|j |jd|\} } | jg|dR} |ri| | f} | S| f} | S)Nr@rrr)r3r)shaperrviewrrrr get_interfacer&_attn_implementationrrrrreshaper)r=rrrrrl hidden_shape query_states key_states value_statesattention_interfacerroutputsrrr'rLjs2  zMarkupLMSelfAttention.forwardNF rMrNrOr,rDr FloatTensorboolrrr{rLrQrrr>r'rTs rc r) MarkupLMAttentioncs"tt||_t||_dSr)r+r,rr=routputr<r>rr'r,s  zMarkupLMAttention.__init__NFrrrrrcKs>|j|f||d|}||d|}|f|dd}|SN)rrrr)r=r)r=rrrr self_outputsattention_outputrrrr'rLszMarkupLMAttention.forwardrrrrr>r'rs rc s^eZdZfddZ  ddejdejdBdedBdee d e ejf d d Z d d Z Z S) MarkupLMLayercs:t|j|_d|_t||_t||_t||_ dS)Nr) r+r,chunk_size_feed_forward seq_len_dimr attentionr intermediaterrr<r>rr'r,s   zMarkupLMLayer.__init__NFrrrrrc KsN|j|f||d|}|d}|dd}t|j|j|j|}|f|}|Sr)rrfeed_forward_chunkrr) r=rrrrself_attention_outputsrr layer_outputrrr'rLs  zMarkupLMLayer.forwardcCs||}|||}|Sr)rr)r=rintermediate_outputrrrr'rs  z MarkupLMLayer.feed_forward_chunkr)rMrNrOr,rDrrrrrr{rLrrQrrr>r'rs    rcsreZdZfddZe    ddejdejdBdedBd edBd edBd e e d e eje Bfd dZ ZS)MarkupLMEncodercs:t|_tfddtjD|_d|_dS)Ncsg|]}tqSr)r)r#rJr%rr'r(sz,MarkupLMEncoder.__init__..F) r+r,r&rr8r9num_hidden_layerslayergradient_checkpointingr<r>r%r'r,s   zMarkupLMEncoder.__init__NFTrrroutput_hidden_states return_dictrrc Ks|rdnd}|r dnd}t|jD]"\} } |r||f}| |||fi|} | d}|r3|| df}q|r;||f}t|||dS)Nrrr)last_hidden_stater attentions) enumeraterr ) r=rrrrrrall_hidden_statesall_self_attentionsrJ layer_module layer_outputsrrr'rLs.   zMarkupLMEncoder.forward)NFFT)rMrNrOr,rrDrrrrrr{r rLrQrrr>r'rs,  rcs2eZdZUeed<dZefddZZ S)MarkupLMPreTrainedModelr&markuplmcsZt|t|trt|jdSt|tr+t|j t |j j d ddSdS)zInitialize the weightsr@rWN)r+ _init_weightsrrinitzeros_rrRcopy_rVrDrcrrd)r=rr>rr'rs   &z%MarkupLMPreTrainedModel._init_weights) rMrNrOr__annotations__base_model_prefixrDno_gradrrQrrr>r'rs rcseZdZdfdd ZddZddZee          dd ej dBd ej dBd ej dBd ej dBd ej dBdej dBdej dBde dBde dBde dBde e BfddZZS) MarkupLMModelTcsDt|||_t||_t||_|rt|nd|_| dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r+r,r&rRrrencoderrpooler post_init)r=r&add_pooling_layerr>rr'r, s    zMarkupLMModel.__init__cCs|jjSrrr[)r=rrr'get_input_embeddingssz"MarkupLMModel.get_input_embeddingscCs ||j_dSrr)r=rrrr'set_input_embeddingss z"MarkupLMModel.set_input_embeddingsNrsrFrGrr~rVrkrrrrc Ks`|dur|n|jj}| dur| n|jj} | dur| n|jj} |dur*|dur*td|dur9||||} n|durF|dd} ntd|durQ|jn|j} |dur_tj | | d}|durltj | tj | d}| d d}|j |jd }d |d }|j||||||d }|j|||| d d}|d}|jdur||nd}t|||j|jdS)a xpath_tags_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Tag IDs for each token in the input sequence, padded up to config.max_depth. xpath_subs_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Subscript IDs for each token in the input sequence, padded up to config.max_depth. Examples: ```python >>> from transformers import AutoProcessor, MarkupLMModel >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> model = MarkupLMModel.from_pretrained("microsoft/markuplm-base") >>> html_string = " Page Title " >>> encoding = processor(html_string, return_tensors="pt") >>> outputs = model(**encoding) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 4, 768] ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer@z5You have to specify either input_ids or inputs_embeds)rgrerr)rfg?g)rsrFrGrVr~rkT)rrrr)r pooler_outputrr)r&rruse_return_dictr%warn_if_padding_and_no_attention_maskrhrgrDrzrxrirjrrfrrrr rr)r=rsrFrGrr~rVrkrrrrrlrgextended_attention_maskembedding_outputencoder_outputsrrrrr'rL"sV'   zMarkupLMModel.forward)T) NNNNNNNNNN)rMrNrOr,rrrrrD LongTensorrrr{r rLrQrrr>r'r sN    rcseZdZfddZee            ddejdBdejdBdejdBdejdBdejdBd ejdBd ejdBd ejdBd ejdBd edBdedBdedBde eje BfddZ Z S)MarkupLMForQuestionAnsweringcs@t||j|_t|dd|_t|j|j|_| dSNF)r) r+r, num_labelsrrrr.r/ qa_outputsrr<r>rr'r,s  z%MarkupLMForQuestionAnswering.__init__NrsrFrGrr~rVrkstart_positions end_positionsrrrrc  Ks| dur| n|jj} |j|||||||| | dd }|d}||}|jddd\}}|d}|d}d}|dur| durt|dkrP|d}t| dkr]| d} |d}| d|| d|t |d}|||}||| }||d }t ||||j |j d S) ae xpath_tags_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Tag IDs for each token in the input sequence, padded up to config.max_depth. xpath_subs_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Subscript IDs for each token in the input sequence, padded up to config.max_depth. Examples: ```python >>> from transformers import AutoProcessor, MarkupLMForQuestionAnswering >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base-finetuned-websrc") >>> model = MarkupLMForQuestionAnswering.from_pretrained("microsoft/markuplm-base-finetuned-websrc") >>> html_string = " My name is Niels " >>> question = "What's his name?" >>> encoding = processor(html_string, questions=question, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = encoding.input_ids[0, answer_start_index : answer_end_index + 1] >>> processor.decode(predict_answer_tokens).strip() 'Niels' ```NT rFrGrr~rVrkrrrrrr@rA) ignore_indexr)loss start_logits end_logitsrr)r&r rrsplitsqueezerlenrhclamp_rrrr)r=rsrFrGrr~rVrkrrrrrrrrlogitsrr total_loss ignored_indexloss_fct start_lossend_lossrrr'rLsL0          z$MarkupLMForQuestionAnswering.forward) NNNNNNNNNNNN) rMrNrOr,rrrDrrr{rrLrQrrr>r'rsV        rzC MarkupLM Model with a `token_classification` head on top. ) custom_introceZdZfddZee           ddejdBdejdBdejdBdejdBdejdBd ejdBd ejdBd ejdBd edBd edBdedBde eje BfddZ Z S)MarkupLMForTokenClassificationcsbt||j|_t|dd|_|jdur|jn|j}t||_ t |j |j|_ | dSr)r+r,rrrclassifier_dropoutr2rr1r3r.r/ classifierrr=r&r'r>rr'r,s   z'MarkupLMForTokenClassification.__init__NrsrFrGrr~rVrklabelsrrrrc  Ks| dur| n|jj} |j|||||||| | dd } | d}||}d}|dur9t}||d|jj|d}t||| j| j dS)a xpath_tags_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Tag IDs for each token in the input sequence, padded up to config.max_depth. xpath_subs_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Subscript IDs for each token in the input sequence, padded up to config.max_depth. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Examples: ```python >>> from transformers import AutoProcessor, AutoModelForTokenClassification >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> processor.parse_html = False >>> model = AutoModelForTokenClassification.from_pretrained("microsoft/markuplm-base", num_labels=7) >>> nodes = ["hello", "world"] >>> xpaths = ["/html/body/div/li[1]/div/span", "/html/body/div/li[1]/div/span"] >>> node_labels = [1, 2] >>> encoding = processor(nodes=nodes, xpaths=xpaths, node_labels=node_labels, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> loss = outputs.loss >>> logits = outputs.logits ```NTrrr@rrrr) r&r rr(rrrrrr)r=rsrFrGrr~rVrkr*rrrrrrrrr!rrr'rLs8. z&MarkupLMForTokenClassification.forward NNNNNNNNNNN) rMrNrOr,rrrDrrr{r rLrQrrr>r'r&sP      r&z MarkupLM Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cr%)!MarkupLMForSequenceClassificationcsdt||j|_||_t||_|jdur|jn|j}t ||_ t |j |j|_ |dSr)r+r,rr&rrr'r2rr1r3r.r/r(rr)r>rr'r,Ys    z*MarkupLMForSequenceClassification.__init__NrsrFrGrr~rVrkr*rrrrc  KsH| dur| n|jj} |j|||||||| | dd } | d}||}||}d}|dur|jjdurW|jdkr=d|j_n|jdkrS|jtj ksN|jtj krSd|j_nd|j_|jjdkrut }|jdkro|| | }n+|||}n%|jjdkrt }||d|j|d}n|jjdkrt}|||}t||| j| jd S) a xpath_tags_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Tag IDs for each token in the input sequence, padded up to config.max_depth. xpath_subs_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Subscript IDs for each token in the input sequence, padded up to config.max_depth. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoProcessor, AutoModelForSequenceClassification >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> model = AutoModelForSequenceClassification.from_pretrained("microsoft/markuplm-base", num_labels=7) >>> html_string = " Page Title " >>> encoding = processor(html_string, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> loss = outputs.loss >>> logits = outputs.logits ```NTrr regressionsingle_label_classificationmulti_label_classificationr@r+)r&r rr3r( problem_typerrfrDrirprrrrrrrr)r=rsrFrGrr~rVrkr*rrrrrrrrr!rrr'rLhsR-     "       z)MarkupLMForSequenceClassification.forwardr,) rMrNrOr,rrrDrrr{rrLrQrrr>r'r-QsP      r-)rr-r&rr)r)?rPcollections.abcrrDrtorch.nnrrrrr activationsr modeling_layersr modeling_outputsr r r rrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrrrconfiguration_markuplmr get_loggerrMloggerModulerrRrrrrrrrrfloatrrrrrrrrr&r-__all__rrrr'sv          2t 9'-ul` q