o eiC@sdZddlZddlZddlmZddlmZmZmZddlm Z ddl m Z ddl mZdd lmZmZmZmZmZdd lmZdd lmZdd lmZmZd dlmZeeZ Gdddej!Z"Gdddej!Z#Gdddej!Z$Gdddej!Z%Gdddej!Z&Gdddej!Z'Gdddej!Z(GdddeZ)Gdd d ej!Z*Gd!d"d"ej!Z+eGd#d$d$eZ,eGd%d&d&e,Z-ed'd(Gd)d*d*e,Z.eGd+d,d,e,Z/Gd-d.d.ej!Z0eGd/d0d0e,Z1gd1Z2dS)2zPyTorch LiLT model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward)auto_docstringlogging) LiltConfigcs>eZdZfddZ    d ddZddZdd ZZS) LiltTextEmbeddingscsttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_|jdt|jddd|j|_tj|j|j|jd|_ dS)N padding_idxeps position_idsrF) persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandrselfconfig __class__d/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/lilt/modeling_lilt.pyr*s   zLiltTextEmbeddings.__init__Nc Cs|dur|dur|||j|j}n||}|dur"|}n|dd}|dur9tj|tj|j jd}|durB| |}| |}||}| |}||7}| |}||}||fS)Nrdtypedevice)"create_position_ids_from_input_idsrtor;&create_position_ids_from_inputs_embedssizer/zeroslongrr$r(r&r)r-) r3 input_idstoken_type_idsr inputs_embeds input_shaper( embeddingsr&r7r7r8forward>s(       zLiltTextEmbeddings.forwardcCs2||}tj|dd||}||S)a Args: Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. x: torch.Tensor x: Returns: torch.Tensor rdim)neintr/cumsumtype_asrA)r3rBrmaskincremental_indicesr7r7r8r<bs  z5LiltTextEmbeddings.create_position_ids_from_input_idscCsN|dd}|d}tj|jd||jdtj|jd}|d|S)z Args: We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids.: inputs_embeds: torch.Tensor Returns: torch.Tensor Nrrr9r)r?r/r0rrAr; unsqueezer1)r3rDrEsequence_lengthrr7r7r8r>os z9LiltTextEmbeddings.create_position_ids_from_inputs_embeds)NNNN)__name__ __module__ __qualname__rrGr<r> __classcell__r7r7r5r8r)s  $ rcs&eZdZfddZdddZZS)LiltLayoutEmbeddingscstt|j|jd|_t|j|jd|_t|j|jd|_t|j|jd|_ |j |_ tj|j |j|j |j d|_tj|j|j|j d|_tj|j|j |jd|_t|j|_dS)Nr) in_features out_featuresr)rrrr max_2d_position_embeddingsr"x_position_embeddingsy_position_embeddingsh_position_embeddingsw_position_embeddingsr#rr%channel_shrink_ratiobox_position_embeddingsLinearbox_linear_embeddingsr)r*r+r,r-r2r5r7r8rs  zLiltLayoutEmbeddings.__init__Nc CsJz:||dddddf}||dddddf}||dddddf}||dddddf}WntyK}ztd|d}~ww||dddddf|dddddf}||dddddf|dddddf} tj|||||| gdd} || } ||} | | } | | } | | } | S)Nrrrz;The `bbox` coordinate values should be within 0-1000 range.rrH) r[r\ IndexErrorr]r^r/catrbr`r)r-) r3bboxrleft_position_embeddingsupper_position_embeddingsright_position_embeddingslower_position_embeddingser]r^spatial_position_embeddingsr`r7r7r8rGs6  22   zLiltLayoutEmbeddings.forward)NN)rRrSrTrrGrUr7r7r5r8rVs rVcs6eZdZd fdd Zd ddZ  d dd ZZS) LiltSelfAttentionNcst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|j|j|j|_t |j|j|j|j|_t |j|j|j|j|_t |j|_|j|_||_dS)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ())rrr"num_attention_headshasattr ValueErrorrKattention_head_size all_head_sizerraquerykeyvaluer_ layout_query layout_key layout_valuer+attention_probs_dropout_probr- layer_idx)r3r4r|r5r7r8rs2   zLiltSelfAttention.__init__rcCs:|dd|j|j|f}|j|}|ddddS)Nrrrcrr)r?rprsviewpermute)r3xr new_x_shaper7r7r8transpose_for_scoress  z&LiltSelfAttention.transpose_for_scoresFcCs|j|||jd}|j|||jd}|j|||jd}||}|||} |||} ||} t | | dd} t || dd} | t |j }| t |j |j}||} ||} |durr| |} tjdd| }||}t ||}|dddd}|dd|j|jf}|j|}|dur| |} tjdd| }||}t || }|dddd}|dd|jf}|j|}||f}|r||f}|S) N)rrrHrrcrr)rrzr_ryrxrurvrwr/matmul transposemathsqrtrsrSoftmaxr-r~ contiguousr?rtr})r3 hidden_states layout_inputsattention_maskoutput_attentionslayout_value_layerlayout_key_layerlayout_query_layermixed_query_layer key_layer value_layer query_layerattention_scoreslayout_attention_scorestmp_attention_scorestmp_layout_attention_scoreslayout_attention_probslayout_context_layernew_context_layer_shapeattention_probs context_layeroutputsr7r7r8rGsF          zLiltSelfAttention.forwardN)rNF)rRrSrTrrrGrUr7r7r5r8rms   rmc8eZdZfddZdejdejdejfddZZS)LiltSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNr) rrrrar"denser)r*r+r,r-r2r5r7r8r" zLiltSelfOutput.__init__r input_tensorreturncC&||}||}|||}|Srrr-r)r3rrr7r7r8rG(  zLiltSelfOutput.forwardrRrSrTrr/TensorrGrUr7r7r5r8r! $rc sVeZdZd fdd Z  d dejdejdejdBdedBd eejf d d Z Z S) LiltAttentionNcsJtt||d|_t||_|j}|j|j|_t||_||_dS)Nr|) rrrmr3routputr"r_ layout_output)r3r4r|ori_hidden_sizer5r7r8r0s    zLiltAttention.__init__Frrrrrc CsH|||||}||d|}||d|}||f|dd}|S)Nrrrc)r3rr) r3rrrr self_outputsattention_outputlayout_attention_outputrr7r7r8rG:szLiltAttention.forwardrr) rRrSrTrr/r FloatTensorbooltuplerGrUr7r7r5r8r/src2eZdZfddZdejdejfddZZS)LiltIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) rrrrar"intermediate_sizer isinstance hidden_actstrrintermediate_act_fnr2r5r7r8rOs   zLiltIntermediate.__init__rrcCs||}||}|Sr)rr)r3rr7r7r8rGWs  zLiltIntermediate.forwardrr7r7r5r8rNs rcr) LiltOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) rrrrarr"rr)r*r+r,r-r2r5r7r8r_rzLiltOutput.__init__rrrcCrrrrr7r7r8rGerzLiltOutput.forwardrr7r7r5r8r^rrc sfeZdZdfdd Z  ddejdejdejdBdedBd eejf d d Z d d Z ddZ Z S) LiltLayerNcst|j|_d|_t||d|_t||_t||_ |j }|j }|j |j |_ |j |j |_ t||_ t||_||_ ||_ dS)Nrr)rrchunk_size_feed_forward seq_len_dimr attentionr intermediaterrr"rr_layout_intermediater)r3r4r|rori_intermediate_sizer5r7r8rms      zLiltLayer.__init__Frrrrrc Csf|j||||d}|d}|d}|dd}t|j|j|j|} t|j|j|j|} | | f|}|S)N)rrrrc)rrfeed_forward_chunkrrlayout_feed_forward_chunk) r3rrrrself_attention_outputsrrr layer_outputlayout_layer_outputr7r7r8rG~s"  zLiltLayer.forwardcC||}|||}|Sr)rrr3rintermediate_outputrr7r7r8r  zLiltLayer.feed_forward_chunkcCrr)rrrr7r7r8rrz#LiltLayer.layout_feed_forward_chunkrr) rRrSrTrr/rrrrrGrrrUr7r7r5r8rls" rcsleZdZfddZ    ddejdejdejdBd edBd edBd edBd eeje Bfd dZ Z S) LiltEncodercs4t|_tfddtjD|_dS)Ncsg|]}tqSr7)r).0_r4r7r8 sz(LiltEncoder.__init__..)rrr4r ModuleListrangenum_hidden_layerslayerr2r5rr8rs $zLiltEncoder.__init__NFTrrrroutput_hidden_states return_dictrc Cs|rdnd}|r dnd}t|jD]#\} } |r||f}| ||||} | d}| d}|r4|| df}q|r<||f}|sJtdd|||fDSt|||dS)Nr7rrrccss|] }|dur|VqdSrr7)rvr7r7r8 sz&LiltEncoder.forward..)last_hidden_stater attentions) enumeraterrr ) r3rrrrrrall_hidden_statesall_self_attentionsi layer_module layer_outputsr7r7r8rGs<    zLiltEncoder.forward)NFFT) rRrSrTrr/rrrrr rGrUr7r7r5r8rs*   rcr) LiltPoolercs*tt|j|j|_t|_dSr)rrrrar"rTanh activationr2r5r7r8rs zLiltPooler.__init__rrcCs(|dddf}||}||}|SNr)rr)r3rfirst_token_tensor pooled_outputr7r7r8rGs  zLiltPooler.forwardrr7r7r5r8rs rcs2eZdZUeed<dZdZgZfddZZ S)LiltPreTrainedModelr4liltTcs@t|t|trt|jt|jj d ddSdS)Nrr) r _init_weightsrrinitcopy_rr/r0shaper1)r3moduler5r7r8rs  &z!LiltPreTrainedModel._init_weights) rRrSrTr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modulesrrUr7r7r5r8rs rcseZdZdfdd ZddZddZe         dd ejdBd ejdBd ejdBd ejdBd ejdBdejdBde dBde dBde dBde eje BfddZ Z S) LiltModelTcsNt|||_t||_t||_t||_|rt |nd|_ | dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) rrr4rrFrVlayout_embeddingsrencoderrpooler post_init)r3r4add_pooling_layerr5r7r8rs     zLiltModel.__init__cCs|jjSrrFr$)r3r7r7r8get_input_embeddings szLiltModel.get_input_embeddingscCs ||j_dSrr)r3rwr7r7r8set_input_embeddingss zLiltModel.set_input_embeddingsNrBrfrrCrrDrrrrc Ks|dur|n|jj}|dur|n|jj}| dur| n|jj} |dur*|dur*td|dur9||||} n|durF|dd} ntd| \} } |durU|jn|j}|durgtj | dtj |d}|durttj | | f|d}|durt |j dr|j jddd| f}|| | }|}n tj | tj |d}||| }|j ||||d \}}|j||d }|j|||||| d }|d }|jdur||nd}| s||f|d dSt|||j|jdS)a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. Examples: ```python >>> from transformers import AutoTokenizer, AutoModel >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModel.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embeds)r9)r;rC)rBrrCrD)rfr)rrrrrr)r pooler_outputrr)r4rruse_return_dictrr%warn_if_padding_and_no_attention_maskr?r;r/r@rAonesrqrFrCr1get_extended_attention_maskrrrr rr)r3rBrfrrCrrDrrrkwargsrE batch_size seq_lengthr;buffered_token_type_ids buffered_token_type_ids_expandedextended_attention_maskembedding_outputlayout_embedding_outputencoder_outputssequence_outputrr7r7r8rGsd(      zLiltModel.forward)T) NNNNNNNNN)rRrSrTrrrrr/rrrr rGrUr7r7r5r8rsF   rz LiLT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ) custom_introcseZdZfddZe          ddejdBdejdBdejdBdejdBdejdBd ejdBd ejdBd e dBd e dBd e dBde eje BfddZ Z S)LiltForSequenceClassificationcs>t||j|_||_t|dd|_t||_|dSNF)r) rr num_labelsr4rrLiltClassificationHead classifierrr2r5r7r8rs   z&LiltForSequenceClassification.__init__NrBrfrrCrrDlabelsrrrrc  Kst| dur| n|jj} |j|||||||| | d } | d} || }d}|dur||j}|jjdurW|jdkr=d|j_n|jdkrS|jt j ksN|jt j krSd|j_nd|j_|jjdkrut }|jdkro|| | }n+|||}n%|jjdkrt}||d|j|d}n|jjdkrt}|||}| s|f| d d}|dur|f|S|St||| j| jd S) a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForSequenceClassification >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForSequenceClassification.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> predicted_class_idx = outputs.logits.argmax(-1).item() >>> predicted_class = model.config.id2label[predicted_class_idx] ```NrfrrCrrDrrrrr regressionsingle_label_classificationmulti_label_classificationrrclosslogitsrr)r4rrrr=r; problem_typerr:r/rArKrsqueezerr}rr rrr3rBrfrrCrrDrrrrrrrrrloss_fctrr7r7r8rGsV.     "       z%LiltForSequenceClassification.forward NNNNNNNNNN)rRrSrTrrr/ LongTensorrrrrr rGrUr7r7r5r8rsH      rcseZdZfddZe          ddejdBdejdBdejdBdejdBdejdBd ejdBd ejdBd edBd edBd edBde ej e BfddZ Z S)LiltForTokenClassificationcsbt||j|_t|dd|_|jdur|jn|j}t||_ t |j |j|_ | dSr)rrrrrclassifier_dropoutr,rr+r-rar"rrr3r4r&r5r7r8rs   z#LiltForTokenClassification.__init__NrBrfrrCrrDrrrrrc  Ks| dur| n|jj} |j|||||||| | d } | d} || } || }d}|durB||j}t}||d|j |d}| sX|f| dd}|durV|f|S|St ||| j | j dS)a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForTokenClassification >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForTokenClassification.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> predicted_class_indices = outputs.logits.argmax(-1) ```Nrrrrcr) r4rrr-rr=r;rr}rrrrr!r7r7r8rGs:+   z"LiltForTokenClassification.forwardr#)rRrSrTrrr/r$rrrrrrGrUr7r7r5r8r%sH     r%cs(eZdZdZfddZddZZS)rz-Head for sentence-level classification tasks.csTtt|j|j|_|jdur|jn|j}t||_ t|j|j |_ dSr) rrrrar"rr&r,r+r-rout_projr'r5r7r8r[s  zLiltClassificationHead.__init__cKsL|dddddf}||}||}t|}||}||}|Sr)r-rr/tanhr()r3featuresrrr7r7r8rGds     zLiltClassificationHead.forward)rRrSrT__doc__rrGrUr7r7r5r8rXs  rcseZdZfddZe           ddejdBdejdBdejdBdejdBdejdBd ejdBd ejdBd ejdBd edBd edBdedBde ej e BfddZ Z S)LiltForQuestionAnsweringcs@t||j|_t|dd|_t|j|j|_| dSr) rrrrrrrar" qa_outputsrr2r5r7r8rqs  z!LiltForQuestionAnswering.__init__NrBrfrrCrrDstart_positions end_positionsrrrrc  KsH| dur| n|jj} |j||||||| | | d } | d}||}|jddd\}}|d}|d}d}|dur|durt|dkrO|d}t|dkr\|d}|d}| d|}| d|}t |d}|||}|||}||d}| s||f| dd}|dur|f|S|St |||| j | j d S) a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForQuestionAnswering >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForQuestionAnswering.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = encoding.input_ids[0, answer_start_index : answer_end_index + 1] >>> predicted_answer = tokenizer.decode(predict_answer_tokens) ```NrrrrrH) ignore_indexrc)r start_logits end_logitsrr)r4rrr-splitr rlenr?clamprr rr)r3rBrfrrCrrDr.r/rrrrrrrr1r2 total_loss ignored_indexr" start_lossend_lossrr7r7r8rG{sP/          z LiltForQuestionAnswering.forward) NNNNNNNNNNN)rRrSrTrrr/r$rrrrr rGrUr7r7r5r8r,nsN       r,)r,rr%rr)3r+rr/rtorch.nnrrrrr activationsrmodeling_layersr modeling_outputsr r r r rmodeling_utilsr pytorch_utilsrutilsrrconfiguration_liltr get_loggerrRloggerModulerrVrmrrrrrrrrrrr%rr,__all__r7r7r7r8sN         V8j:8 pb n