o ik@sdZddlZddlmZmZddlZddlmZddlmZm Z m Z ddl m Z ddl mZdd lmZmZmZmZmZdd lmZdd lmZmZmZdd lmZmZd dlmZe e!Z"Gdddej#Z$Gdddej#Z%Gdddej#Z&Gdddej#Z'Gdddej#Z(Gdddej#Z)Gdddej#Z*GdddeZ+Gdd d ej#Z,Gd!d"d"ej#Z-eGd#d$d$eZ.eGd%d&d&e.Z/ed'd(Gd)d*d*e.Z0eGd+d,d,e.Z1Gd-d.d.ej#Z2eGd/d0d0e.Z3gd1Z4dS)2zPyTorch LiLT model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging) LiltConfigcs>eZdZfddZ    d ddZddZdd ZZS) LiltTextEmbeddingscsttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_|jdt|jdddt|dd|_|j|_tj|j|j|jd|_ dS) N padding_idxeps position_ids)rF) persistentposition_embedding_typeabsolute)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandgetattrr rselfconfig __class__c/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/transformers/models/lilt/modeling_lilt.pyr#+s   zLiltTextEmbeddings.__init__Nc Cs|dur|dur|||j|j}n||}|dur"|}n|dd}|dur9tj|tj|j jd}|durB| |}| |}||}|j dkrY| |}||7}||}||}||fS)Nrdtypedevicer!)"create_position_ids_from_input_idsrtor@&create_position_ids_from_inputs_embedssizer3zeroslongrr(r,r r*r-r1) r8 input_idstoken_type_idsr inputs_embeds input_shaper, embeddingsr*r<r<r=forwardBs*        zLiltTextEmbeddings.forwardcCs2||}tj|dd||}||S)a Args: Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. x: torch.Tensor x: Returns: torch.Tensor rdim)neintr3cumsumtype_asrF)r8rGrmaskincremental_indicesr<r<r=rAfs  z5LiltTextEmbeddings.create_position_ids_from_input_idscCsN|dd}|d}tj|jd||jdtj|jd}|d|S)z Args: We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids.: inputs_embeds: torch.Tensor Returns: torch.Tensor Nrrr>r)rDr3r4rrFr@ unsqueezer5)r8rIrJsequence_lengthrr<r<r=rCss z9LiltTextEmbeddings.create_position_ids_from_inputs_embeds)NNNN)__name__ __module__ __qualname__r#rLrArC __classcell__r<r<r:r=r*s  $ rcs&eZdZfddZdddZZS)LiltLayoutEmbeddingscstt|j|jd|_t|j|jd|_t|j|jd|_t|j|jd|_ |j |_ tj|j |j|j |j d|_tj|j|j|j d|_tj|j|j |jd|_t|j|_dS)Nr) in_features out_featuresr)r"r#rr$max_2d_position_embeddingsr&x_position_embeddingsy_position_embeddingsh_position_embeddingsw_position_embeddingsr'rr)channel_shrink_ratiobox_position_embeddingsLinearbox_linear_embeddingsr-r.r/r0r1r7r:r<r=r#s  zLiltLayoutEmbeddings.__init__Nc CsJz:||dddddf}||dddddf}||dddddf}||dddddf}WntyK}ztd|d}~ww||dddddf|dddddf}||dddddf|dddddf} tj|||||| gdd} || } ||} | | } | | } | | } | S)Nrrrz;The `bbox` coordinate values should be within 0-1000 range.rrM) r`ra IndexErrorrbrcr3catrgrer-r1) r8bboxrleft_position_embeddingsupper_position_embeddingsright_position_embeddingslower_position_embeddingserbrcspatial_position_embeddingsrer<r<r=rLs6  22   zLiltLayoutEmbeddings.forwardNN)rWrXrYr#rLrZr<r<r:r=r[s r[cs8eZdZd fdd Zd ddZ   d dd ZZS) LiltSelfAttentionNcs^t|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|j|j|j|_t |j|j|j|j|_t |j|j|j|j|_t |j|_|pt|dd|_|jdks|jd kr|j|_t d |jd |j|_|j|_||_dS) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r r! relative_keyrelative_key_queryrhr)r"r#r&num_attention_headshasattr ValueErrorrPattention_head_size all_head_sizerrfquerykeyvaluerd layout_query layout_key layout_valuer/attention_probs_dropout_probr1r6r r)r$distance_embedding layer_idx)r8r9r rr:r<r=r#s>   zLiltSelfAttention.__init__rcCs:|dd|j|j|f}|j|}|ddddS)Nrrrhrr)rDrxr{viewpermute)r8xr new_x_shaper<r<r=transpose_for_scoress  z&LiltSelfAttention.transpose_for_scoresFcCs|j|||jd}|j|||jd}|j|||jd}||} |||} |||} || } t | | dd} t || dd}|j dksY|j dkr| d}tj |tj|jddd}tj |tj|jddd}||}|||jd}|j| jd}|j dkrtd | |}| |} n|j dkrtd | |}td | |}| ||} | t|j}|t|j|j}||} ||}|dur||}tjdd |}||}|dur||}t ||}|d d dd}| dd|j|jf}|j|}|dur| |} tjdd | }||}|dur5||}t || }|d d dd}| dd|jf}|j|}|ra||f|f}|S||ff}|S)N)rrrvrwrr>)r?zbhld,lrd->bhlrzbhrd,lrd->bhlrrMrrhr)rrrdrrr}r~rr3matmul transposer rDr4rFr@rrr)rBr?einsummathsqrtr{rSoftmaxr1r contiguousr|)r8 hidden_states layout_inputsattention_mask head_maskoutput_attentionslayout_value_layerlayout_key_layerlayout_query_layermixed_query_layer key_layer value_layer query_layerattention_scoreslayout_attention_scores seq_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keytmp_attention_scorestmp_layout_attention_scoreslayout_attention_probslayout_context_layernew_context_layer_shapeattention_probs context_layeroutputsr<r<r=rLsp                zLiltSelfAttention.forwardrr)rNNF)rWrXrYr#rrLrZr<r<r:r=rss % rsc8eZdZfddZdejdejdejfddZZS)LiltSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNr) r"r#rrfr&denser-r.r/r0r1r7r:r<r=r#G zLiltSelfOutput.__init__r input_tensorreturncC&||}||}|||}|SNrr1r-r8rrr<r<r=rLM  zLiltSelfOutput.forwardrWrXrYr#r3TensorrLrZr<r<r:r=rF $rcsjeZdZdfdd ZddZ   ddejdejd eejd eejd ee d e ejf d dZ Z S) LiltAttentionNcsTtt|||d|_t||_t|_|j}|j|j |_t||_ ||_dS)N)r r) r"r#rsr8routputset pruned_headsr&rd layout_output)r8r9r rori_hidden_sizer:r<r=r#Us    zLiltAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)NrrrM)lenrr8rxr{rrr}r~rrrr|union)r8headsindexr<r<r= prune_headsas zLiltAttention.prune_headsFrrrrrrc CsT||||||}||dd|}||dd|}||ff|dd} | S)Nrr)r8rr) r8rrrrr self_outputsattention_outputlayout_attention_outputrr<r<r=rLsszLiltAttention.forwardrrr) rWrXrYr#rr3rr FloatTensorbooltuplerLrZr<r<r:r=rTs& rc2eZdZfddZdejdejfddZZS)LiltIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSr) r"r#rrfr&intermediate_sizer isinstance hidden_actstrr intermediate_act_fnr7r:r<r=r#s   zLiltIntermediate.__init__rrcCs||}||}|Sr)rr)r8rr<r<r=rLs  zLiltIntermediate.forwardrr<r<r:r=rs rcr) LiltOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r"r#rrfrr&rr-r.r/r0r1r7r:r<r=r#rzLiltOutput.__init__rrrcCrrrrr<r<r=rLrzLiltOutput.forwardrr<r<r:r=rrrcsreZdZdfdd Z   ddejdejdeejdeejd eed e ejf d d Z d dZ ddZ Z S) LiltLayerNcst|j|_d|_t||d|_t||_t||_ |j }|j }|j |j |_ |j |j |_ t||_ t||_||_ ||_ dS)Nrr)r"r#chunk_size_feed_forward seq_len_dimr attentionr intermediaterrr&rrdlayout_intermediater)r8r9rrori_intermediate_sizer:r<r=r#s      zLiltLayer.__init__Frrrrrrc Csr|j|||||d}|dd}|dd}|dd} t|j|j|j|} t|j|j|j|} | | ff| } | S)N)rrr)rrfeed_forward_chunkrrlayout_feed_forward_chunk) r8rrrrrself_attention_outputsrrr layer_outputlayout_layer_outputr<r<r=rLs$   zLiltLayer.forwardcC||}|||}|Sr)rrr8rintermediate_outputrr<r<r=r  zLiltLayer.feed_forward_chunkcCrr)rrrr<r<r=rrz#LiltLayer.layout_feed_forward_chunkrr)rWrXrYr#r3rrrrrrLrrrZr<r<r:r=rs( rcs~eZdZdfdd Z     ddejdejdeejd eejd eed eed eed e e eje ffddZ Z S) LiltEncoderNcs:t|_tfddtjD|_d|_dS)Ncsg|]}t|dqS)r)r).0ir9r<r= sz(LiltEncoder.__init__..F) r"r#r9r ModuleListrangenum_hidden_layerslayergradient_checkpointing)r8r9rr:rr=r#s   zLiltEncoder.__init__FTrrrrroutput_hidden_states return_dictrcCs|rdnd}|r dnd} t|jD]2\} } |r||f}|dur$|| nd} | |||| |} | dd}| dd}|rC| | df} q|rK||f}|sYtdd||| fDSt||| dS)Nr<rrcss|] }|dur|VqdSrr<)rvr<r<r= sz&LiltEncoder.forward..)last_hidden_stater attentions) enumeraterrr )r8rrrrrrrall_hidden_statesall_self_attentionsr layer_modulelayer_head_mask layer_outputsr<r<r=rLs@      zLiltEncoder.forwardr)NNFFT)rWrXrYr#r3rrrrrrr rLrZr<r<r:r=rs0  rcr) LiltPoolercs*tt|j|j|_t|_dSr)r"r#rrfr&rTanh activationr7r:r<r=r#"s zLiltPooler.__init__rrcCs(|dddf}||}||}|SNr)rr)r8rfirst_token_tensor pooled_outputr<r<r=rL's  zLiltPooler.forwardrr<r<r:r=r!s rc@s*eZdZUeed<dZdZgZddZdS)LiltPreTrainedModelr9liltTcCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|tj rX|jj |jj ddSdS)zInitialize the weightsg)meanstdNg?)rrrfweightdatanormal_r9initializer_rangebiaszero_r$rr-fill_)r8moduler<r<r= _init_weights7s      z!LiltPreTrainedModel._init_weightsN) rWrXrYr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modulesrr<r<r<r=r0s  rcseZdZdfdd ZddZddZdd Ze dd ee j d ee j d ee j dee j dee j dee j dee j dee dee dee de e e j effddZZS) LiltModelTcsNt|||_t||_t||_t||_|rt |nd|_ | dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r"r#r9rrKr[layout_embeddingsrencoderrpooler post_init)r8r9add_pooling_layerr:r<r=r#Js     zLiltModel.__init__cCs|jjSrrKr()r8r<r<r=get_input_embeddings[szLiltModel.get_input_embeddingscCs ||j_dSrr)r8rr<r<r=set_input_embeddings^s zLiltModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r8heads_to_prunerrr<r<r= _prune_headsaszLiltModel._prune_headsNrGrkrrHrrrIrrrrc  Cs|dur|n|jj}| dur| n|jj} | dur| n|jj} |dur*|dur*td|dur9||||} n|durF|dd} ntd| \} } |durU|jn|j}|durgtj | dtj |d}|durttj | | f|d}|durt |j dr|j jddd| f}|| | }|}n tj | tj |d}||| }|||jj}|j ||||d \}}|j||d }|j|||||| | d }|d }|jdur||nd}| s||f|d dSt|||j|jdS)a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. Examples: ```python >>> from transformers import AutoTokenizer, AutoModel >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModel.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embeds)r>)r@rH)rGrrHrI)rkr)rrrrrrr)r pooler_outputrr)r9rruse_return_dictrz%warn_if_padding_and_no_attention_maskrDr@r3rErFonesryrKrHr5get_extended_attention_mask get_head_maskrrrrr rr)r8rGrkrrHrrrIrrrrJ batch_sizerr@buffered_token_type_ids buffered_token_type_ids_expandedextended_attention_maskembedding_outputlayout_embedding_outputencoder_outputssequence_outputrr<r<r=rLish(       zLiltModel.forward)T) NNNNNNNNNN)rWrXrYr#r r!r$rrr3rrrrr rLrZr<r<r:r=rHsN    rz LiLT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ) custom_introcseZdZfddZe           ddeejdeejdeej deejdeejd eej d eej d eejd ee d ee dee de e eje ffddZZS)LiltForSequenceClassificationcs>t||j|_||_t|dd|_t||_|dSNF)r) r"r# num_labelsr9rr LiltClassificationHead classifierrr7r:r<r=r#s   z&LiltForSequenceClassification.__init__NrGrkrrHrrrIlabelsrrrrc  Csv| dur| n|jj} |j|||||||| | | d } | d} || }d}|dur||j}|jjdurX|jdkr>d|j_n|jdkrT|jt j ksO|jt j krTd|j_nd|j_|jjdkrvt }|jdkrp|| | }n+|||}n%|jjdkrt}||d|j|d}n|jjdkrt}|||}| s|f| d d}|dur|f|S|St||| j| jd S) a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForSequenceClassification >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForSequenceClassification.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> predicted_class_idx = outputs.logits.argmax(-1).item() >>> predicted_class = model.config.id2label[predicted_class_idx] ```N rkrrHrrrIrrrrr regressionsingle_label_classificationmulti_label_classificationrrhlosslogitsrr)r9r'r r9rBr@ problem_typer7r?r3rFrPrsqueezerrrrrrr8rGrkrrHrrrIr:rrrrr3rAr@loss_fctrr<r<r=rLsX.     "       z%LiltForSequenceClassification.forward NNNNNNNNNNN)rWrXrYr#rrr3 LongTensorrrrrrrrLrZr<r<r:r=r5sN       r5cseZdZfddZe           ddeejdeejdeejdeejdeejd eejd eejd eejd ee d ee dee de e ej e ffddZZS)LiltForTokenClassificationcsbt||j|_t|dd|_|jdur|jn|j}t||_ t |j |j|_ | dSr6)r"r#r7rr classifier_dropoutr0rr/r1rfr&r9rr8r9rIr:r<r=r#Ws   z#LiltForTokenClassification.__init__NrGrkrrHrrrIr:rrrrc  Cs| dur| n|jj} |j|||||||| | | d } | d} || } || }d}|durC||j}t}||d|j |d}| sY|f| dd}|durW|f|S|St ||| j | j dS)a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForTokenClassification >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForTokenClassification.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> predicted_class_indices = outputs.logits.argmax(-1) ```Nr;rrrhr?) r9r'r r1r9rBr@rrr7rrrrDr<r<r=rLes<+   z"LiltForTokenClassification.forwardrF)rWrXrYr#rrr3rGrrrrrrrLrZr<r<r:r=rHTsN      rHcs(eZdZdZfddZddZZS)r8z-Head for sentence-level classification tasks.csTtt|j|j|_|jdur|jn|j}t||_ t|j|j |_ dSr) r"r#rrfr&rrIr0r/r1r7out_projrJr:r<r=r#s  zLiltClassificationHead.__init__cKsL|dddddf}||}||}t|}||}||}|Sr)r1rr3tanhrK)r8featureskwargsrr<r<r=rLs     zLiltClassificationHead.forward)rWrXrY__doc__r#rLrZr<r<r:r=r8s  r8cseZdZfddZe            ddeejdeejdeejdeejdeejd eejd eejd eejd eejd ee dee dee de e ej e ffddZZS)LiltForQuestionAnsweringcs@t||j|_t|dd|_t|j|j|_| dSr6) r"r#r7rr rrfr& qa_outputsrr7r:r<r=r#s  z!LiltForQuestionAnswering.__init__NrGrkrrHrrrIstart_positions end_positionsrrrrc  CsJ| dur| n|jj} |j|||||||| | | d } | d}||}|jddd\}}|d}|d}d}|dur| durt|dkrP|d}t| dkr]| d} |d}| d|}| d|} t |d}|||}||| }||d}| s||f| dd}|dur|f|S|St |||| j | j d S) a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForQuestionAnswering >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForQuestionAnswering.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = encoding.input_ids[0, answer_start_index : answer_end_index + 1] >>> predicted_answer = tokenizer.decode(predict_answer_tokens) ```Nr;rrrrM) ignore_indexrh)r@ start_logits end_logitsrr)r9r'r rQsplitrCrrrDclamprr rr)r8rGrkrrHrrrIrRrSrrrrr3rArUrV total_loss ignored_indexrE start_lossend_lossrr<r<r=rLsR/          z LiltForQuestionAnswering.forward) NNNNNNNNNNNN)rWrXrYr#rrr3rGrrrrrr rLrZr<r<r:r=rPsT       rP)rPr5rHrr)5rOrtypingrrr3rtorch.nnrrr activationsr modeling_layersr modeling_outputsr r r rrmodeling_utilsr pytorch_utilsrrrutilsrrconfiguration_liltr get_loggerrWloggerModulerr[rsrrrrrrrrrr5rHr8rP__all__r<r<r<r=sP      Y8 5<>qc o