o ei@s8dZddlZddlmZddlZddlmZddlmZddlm Z ddl m Z dd l mZdd lmZmZmZdd lmZdd lmZdd lmZmZmZmZddlmZeeZ eeddGdddeZ!Gdddej"Z#Gdddej"Z$Gdddej"Z%Gdddej"Z&Gdddej"Z'Gdddej"Z(Gd d!d!ej"Z)Gd"d#d#ej"Z*Gd$d%d%ej"Z+Gd&d'd'eZ,Gd(d)d)ej"Z-Gd*d+d+ej"Z.Gd,d-d-ej"Z/eGd.d/d/eZ0eGd0d1d1e0Z1eGd2d3d3e0Z2ed4dGd5d6d6e0Z3ed7dGd8d9d9e0Z4gd:Z5dS);zPyTorch Bros model.N) dataclass)nn)CrossEntropyLoss)initialization)ACT2FN)GradientCheckpointingLayer)"BaseModelOutputWithCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentionsTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward) ModelOutputauto_docstringcan_return_tuplelogging) BrosConfigz@ Base class for outputs of token classification models. ) custom_introc@steZdZUdZdZejdBed<dZejdBed<dZ ejdBed<dZ e ejdBed<dZ e ejdBed<dS)BrosSpadeOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification loss. initial_token_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_labels)`): Classification scores for entity initial tokens (before SoftMax). subsequent_token_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, sequence_length+1)`): Classification scores for entity sequence tokens (before SoftMax). Nlossinitial_token_logitssubsequent_token_logits hidden_states attentions) __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rrrtuplerr#r#d/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/bros/modeling_bros.pyr(s  rc2eZdZfddZdejdejfddZZS)BrosPositionalEmbedding1Dcs@t|j|_ddtd|jd|j}|d|dS)Nr'@inv_freq)super__init__dim_bbox_sinusoid_emb_1drarangeregister_buffer)selfconfigr* __class__r#r$r,Bs z"BrosPositionalEmbedding1D.__init__pos_seqreturncCsX|}|\}}}||||d|jddd|jd}tj||gdd}|S)Nrdim)sizeviewr*r-rcatsincos)r0r4seq_sizeb1b2b3 sinusoid_inppos_embr#r#r$forwardLs  (z!BrosPositionalEmbedding1D.forwardrrrr,rTensorrE __classcell__r#r#r2r$r&?s  r&cr%)BrosPositionalEmbedding2Dcs*t|j|_t||_t||_dSN)r+r,dim_bboxr& x_pos_emb y_pos_embr0r1r2r#r$r,Us  z"BrosPositionalEmbedding2D.__init__bboxr5cCsdg}t|jD]!}|ddkr|||d|fq|||d|fqtj|dd}|S)Nr6r.r7r8)rangerKappendrLrMrr<)r0rOstacki bbox_pos_embr#r#r$rE\s z!BrosPositionalEmbedding2D.forwardrFr#r#r2r$rITs rIcs,eZdZfddZdejfddZZS)BrosBboxEmbeddingscs.tt||_tj|j|jdd|_dS)NF)bias) r+r,rIbbox_sinusoid_embrLineardim_bbox_sinusoid_emb_2ddim_bbox_projectionbbox_projectionrNr2r#r$r,hs  zBrosBboxEmbeddings.__init__rOcCs\|dd}|dddddddf|dddddddf}||}||}|S)Nrr) transposerWr[)r0rObbox_tbbox_posrTr#r#r$rEms 8  zBrosBboxEmbeddings.forwardrFr#r#r2r$rUgs rUc sbeZdZdZfddZ    d dejdBdejdBdejdBdejdBd ejf d d ZZS) BrosTextEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_|dt|jd|jdtj|jtj|jjddddS) N) padding_idxeps position_idsrr7token_type_idsdtypedeviceF) persistent)r+r,r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutr/rr.expandzerosrcr:longrhrNr2r#r$r,ys  zBrosTextEmbeddings.__init__N input_idsrerc inputs_embedsr5c Cs|dur |}n|dd}|d}|dur$|jddd|f}|durNt|drC|jddd|f}||d|}|}n tj|tj|jjd}|durW| |}| |} || } | |} | | 7} | | } | | } | S)Nr7rrerrf)r:rchasattrrerxrryrzrhrnrrrprsrw) r0r{rercr| input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedrr embeddingsrpr#r#r$rEs*       zBrosTextEmbeddings.forward)NNNN) rrrrr,rrGrErHr#r#r2r$r_vs$ r_csneZdZfddZ    ddejdejdejdBdejdBd ejdBd ejdBd eejfd d ZZS)BrosSelfAttentioncst|j|jdkrt|dstd|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_|j|_dS)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ())r+r,rlnum_attention_headsr} ValueErrorintattention_head_size all_head_sizerrXquerykeyvalueruattention_probs_dropout_probrw is_decoderrNr2r#r$r,s   zBrosSelfAttention.__init__NFrrTattention_maskencoder_hidden_statesencoder_attention_maskoutput_attentionsr5cCs|jdd|j|jf}|||dd}|du} | r8|||dd} |||dd} |}n|||dd} |||dd} t || dd} |j\} }}}|||| |}| gd}t d||f}| |} | t |j} |dur| |} tjdd| }||}t || }| dddd }|dd|jf}|j|}|r||fn|f}|jr|d }|S) Nrr7rr6)r6rrrzbnid,bijd->bnijr8rrJ)shaperrrr;r\rrrmatmulpermuteeinsummathsqrtrSoftmaxrw contiguousr:rr)r0rrTrrrr hidden_shape query_layeris_cross_attention key_layer value_layerattention_scores batch_sizen_headrd_headbbox_pos_scoresattention_probs context_layernew_context_layer_shapeoutputsr#r#r$rEs8    zBrosSelfAttention.forwardNNNF) rrrr,rrGr"rErHr#r#r2r$rs* rc8eZdZfddZdejdejdejfddZZS)BrosSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNra) r+r,rrXrldensersrtrurvrwrNr2r#r$r, zBrosSelfOutput.__init__r input_tensorr5cC&||}||}|||}|SrJrrwrsr0rrr#r#r$rE  zBrosSelfOutput.forwardrFr#r#r2r$r $rcsleZdZfddZ    ddejdejdejdBdejdBd ejdBd edBd eejfd d ZZ S) BrosAttentioncs"tt||_t||_dSrJ)r+r,rr0routputrNr2r#r$r,s  zBrosAttention.__init__NFrrTrrrrr5c Cs<|j||||||d}||d|}|f|dd} | S)NrrTrrrrrr)r0r) r0rrTrrrr self_outputsattention_outputrr#r#r$rEs zBrosAttention.forwardr) rrrr,rrGboolr"rErHr#r#r2r$rs*  rcr%)BrosIntermediatecsDtt|j|j|_t|jt rt |j|_ dS|j|_ dSrJ) r+r,rrXrlintermediate_sizer isinstance hidden_actstrrintermediate_act_fnrNr2r#r$r,3s   zBrosIntermediate.__init__rr5cCs||}||}|SrJ)rr)r0rr#r#r$rE;s  zBrosIntermediate.forwardrFr#r#r2r$r2s rcr) BrosOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r+r,rrXrrlrrsrtrurvrwrNr2r#r$r,BrzBrosOutput.__init__rrr5cCrrJrrr#r#r$rEHrzBrosOutput.forwardrFr#r#r2r$rArrcsteZdZfddZ    ddejdejdejdBdejdBd ejdBd edBd eejfd d Z ddZ Z S) BrosLayercsnt|j|_d|_t||_|j|_|j|_|jr+|js&t|dt||_ t ||_ t ||_ dS)Nrz> should be used as a decoder model if cross attention is added)r+r,chunk_size_feed_forward seq_len_dimr attentionradd_cross_attention Exceptioncrossattentionr intermediaterrrNr2r#r$r,Ps    zBrosLayer.__init__NFrrTrrrrr5c Cs|j||||d}|d}|jr|dd} n|dd} |jrG|durGt|dr1td|d|j|||||d} | d}| | dd} t|j|j|j|} | f| } |jr]| d } | S) N)rTrrrrr7rz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`)rrrrrJ) rrr}rrr feed_forward_chunkrr) r0rrTrrrrself_attention_outputsrrcross_attention_outputs layer_outputr#r#r$rE^sD     zBrosLayer.forwardcCs||}|||}|SrJ)rr)r0rintermediate_outputrr#r#r$rs  zBrosLayer.feed_forward_chunkr) rrrr,rrGr rr"rErrHr#r#r2r$rOs,  5rcseZdZfddZe      ddejdejdejdBd ejdBd ejdBd edBd edBd edBde eje BfddZ Z S) BrosEncodercs4t|_tfddtjD|_dS)Ncsg|]}tqSr#)r).0_r1r#r$ sz(BrosEncoder.__init__..)r+r,r1r ModuleListrPnum_hidden_layerslayerrNr2rr$r,s $zBrosEncoder.__init__NFTrrTrrrroutput_hidden_states return_dictr5c  Cs|rdnd} |r dnd} |r|jjrdnd} t|jD]-\} } |r&| |f} | ||||||d}|d}|rH| |df} |jjrH| |df} q|rP| |f} t|| | | dS)Nr#rrrr6)last_hidden_staterrcross_attentions)r1r enumeraterr )r0rrTrrrrrrall_hidden_statesall_self_attentionsall_cross_attentionsrS layer_module layer_outputsr#r#r$rEs8    zBrosEncoder.forward)NNNFFT) rrrr,rrrGr rr"r rErHr#r#r2r$rs8   rcr%) BrosPoolercs*tt|j|j|_t|_dSrJ)r+r,rrXrlrTanh activationrNr2r#r$r,s zBrosPooler.__init__rr5cCs(|dddf}||}||}|S)Nr)rr)r0rfirst_token_tensor pooled_outputr#r#r$rEs  zBrosPooler.forwardrFr#r#r2r$rs rcr%)BrosRelationExtractorcst|j|_|j|_|j|_|j|_t|j|_ t |j|j|j|_ t |j|j|j|_ t td|j|_dS)Nr)r+r, n_relationsrlbackbone_hidden_sizehead_hidden_sizeclassifier_dropout_probrrudroprXrr Parameterrry dummy_noderNr2r#r$r,s zBrosRelationExtractor.__init__rrc Cs|||}|jdd|dd}tj||gdd}|||}| |d|d|j |j }| |d|d|j |j }t | dddd| dddd}|S)Nrraxisr6r)rrr unsqueezerepeatr:rr<rr;rrrr)r0rr dummy_vecrelation_scorer#r#r$rEs zBrosRelationExtractor.forwardrFr#r#r2r$rs rcs:eZdZUeed<dZedej ffdd Z Z S)BrosPreTrainedModelr1brosmodulecst||jj}t|trtj|j|ddSt|t r7t |j t |j jddt|jdSt|trTddt d|jd|j}t |j|dSdS) zInitialize the weights)stdr7rdrr'r(r)N)r+ _init_weightsr1initializer_rangerrinitnormal_rr_copy_rcrr.rrxzeros_rer&r-r*)r0rrr*r2r#r$rs   " z!BrosPreTrainedModel._init_weights) rrrrr!base_model_prefixrno_gradrModulerrHr#r#r2r$rs rcseZdZdfdd ZddZddZee           dd ej dBd ej dBd ej dBd ej dBd ej dBdej dBdej dBdej dBde dBde dBde dBde ej e BfddZ ZS) BrosModelTcsNt|||_t||_t||_t||_|rt |nd|_ | dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r+r,r1r_rrUbbox_embeddingsrencoderrpooler post_init)r0r1add_pooling_layerr2r#r$r,s     zBrosModel.__init__cCs|jjSrJrrn)r0r#r#r$get_input_embeddings'szBrosModel.get_input_embeddingscCs ||j_dSrJr )r0rr#r#r$set_input_embeddings*s zBrosModel.set_input_embeddingsNr{rOrrercr|rrrrrr5c  Ks| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur*|dur*td|dur3|} n|dur@|dd} ntd|durLtd| \}}|durW|jn|j}|duretj| |d}|durt |j dr|j j ddd|f}| ||}|}n tj | tj|d}||| }|jjr|dur|\}}}||f}|durtj||d}||}nd}|j ||||d }|jdd kr|ddddgd f}||jj}||}|j|||||| | d d }|d}|jdur||nd}t|||j|j|jdS)a bbox ('torch.FloatTensor' of shape '(batch_size, num_boxes, 4)'): Bounding box coordinates for each token in the input sequence. Each bounding box is a list of four values (x1, y1, x2, y2), where (x1, y1) is the top left corner, and (x2, y2) is the bottom right corner of the bounding box. Examples: ```python >>> import torch >>> from transformers import BrosProcessor, BrosModel >>> processor = BrosProcessor.from_pretrained("jinho8345/bros-base-uncased") >>> model = BrosModel.from_pretrained("jinho8345/bros-base-uncased") >>> encoding = processor("Hello, my dog is cute", add_special_tokens=False, return_tensors="pt") >>> bbox = torch.tensor([[[0, 0, 1, 1]]]).repeat(1, encoding["input_ids"].shape[-1], 1) >>> encoding["bbox"] = bbox >>> outputs = model(**encoding) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer7z5You have to specify either input_ids or inputs_embedszYou have to specify bbox)rhrerf)r{rcrer|)rrr6rr6rrrT)rTrrrrrrr)r pooler_outputrrr)r1rruse_return_dictrr:rhronesr}rrerxryrzget_extended_attention_maskrinvert_attention_maskr bbox_scalerrrr rrr)r0r{rOrrercr|rrrrrkwargsr~rrrhrrextended_attention_maskencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeencoder_extended_attention_maskembedding_output scaled_bboxbbox_position_embeddingsencoder_outputssequence_outputrr#r#r$rE-sx(        zBrosModel.forward)T NNNNNNNNNNN)rrrr,r r rrrrGrr"r rErHr#r#r2r$rsT     rceZdZdgZfddZee           ddejdBdejdBdejdBdejdBd ejdBd ejdBd ejdBd ejdBd e dBde dBde dBde eje BfddZ Z S)BrosForTokenClassificationrcs^t||j|_t||_t|dr|jn|j}t ||_ t |j |j|_ |dSNclassifier_dropout)r+r, num_labelsrrr}r"rvrrurwrXrl classifierrr0r1r"r2r#r$r,s    z#BrosForTokenClassification.__init__Nr{rOrbbox_first_token_maskrercr|labelsrrrr5c  Ks| dur| n|jj} |j||||||| | dd } | d}||}||}d}|durWt}|durJ|d}||d|j||d|}n ||d|j|d}t||| j | j dS)a bbox ('torch.FloatTensor' of shape '(batch_size, num_boxes, 4)'): Bounding box coordinates for each token in the input sequence. Each bounding box is a list of four values (x1, y1, x2, y2), where (x1, y1) is the top left corner, and (x2, y2) is the bottom right corner of the bounding box. bbox_first_token_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to indicate the first token of each bounding box. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. Examples: ```python >>> import torch >>> from transformers import BrosProcessor, BrosForTokenClassification >>> processor = BrosProcessor.from_pretrained("jinho8345/bros-base-uncased") >>> model = BrosForTokenClassification.from_pretrained("jinho8345/bros-base-uncased") >>> encoding = processor("Hello, my dog is cute", add_special_tokens=False, return_tensors="pt") >>> bbox = torch.tensor([[[0, 0, 1, 1]]]).repeat(1, encoding["input_ids"].shape[-1], 1) >>> encoding["bbox"] = bbox >>> outputs = model(**encoding) ```NT)rOrrercr|rrrrr7rlogitsrr) r1rrrwr$rr;r#r rr)r0r{rOrr&rercr|r'rrrrrrr)rloss_fctr#r#r$rEs<-   z"BrosForTokenClassification.forwardrrrr"_keys_to_ignore_on_load_unexpectedr,rrrrGrr"r rErHr#r#r2r$r sR       r a Bros Model with a token classification head on top (initial_token_layers and subsequent_token_layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. The initial_token_classifier is used to predict the first token of each entity, and the subsequent_token_classifier is used to predict the subsequent tokens within an entity. Compared to BrosForTokenClassification, this model is more robust to serialization errors since it predicts next token from one token. cseZdZdgZfddZee            ddejdBdejdBdejdBdejdBd ejdBd ejdBd ejdBd ejdBd ejdBde dBde dBde dBde eje BfddZ Z S)!BrosSpadeEEForTokenClassificationrc st|||_|j|_|j|_|j|_t||_t |dr"|j n|j }t t |t |j|jt |t |j|j|_t||_|dSr!)r+r,r1r#rrlrrrr}r"rvr SequentialrurXinitial_token_classifierrsubsequent_token_classifierrr%r2r#r$r,s    z*BrosSpadeEEForTokenClassification.__init__Nr{rOrr&rercr|initial_token_labelssubsequent_token_labelsrrrr5c  Ks| dur| n|jj} |j||||||| | dd }|d}|dd}||dd}|||d}d|}|j\}}|j }t j |t |dg |gdd}||dddddft |jj}t ||dj |t jd}||dddddft |jj}|d}d}|dur| durt}|d}|dur|d}||d|j|||}n ||d|j|}| d} ||d|d|| |}||}t||||j|jd S) a> bbox ('torch.FloatTensor' of shape '(batch_size, num_boxes, 4)'): Bounding box coordinates for each token in the input sequence. Each bounding box is a list of four values (x1, y1, x2, y2), where (x1, y1) is the top left corner, and (x2, y2) is the bottom right corner of the bounding box. bbox_first_token_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to indicate the first token of each bounding box. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. initial_token_labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for the initial token classification. subsequent_token_labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for the subsequent token classification. Examples: ```python >>> import torch >>> from transformers import BrosProcessor, BrosSpadeEEForTokenClassification >>> processor = BrosProcessor.from_pretrained("jinho8345/bros-base-uncased") >>> model = BrosSpadeEEForTokenClassification.from_pretrained("jinho8345/bros-base-uncased") >>> encoding = processor("Hello, my dog is cute", add_special_tokens=False, return_tensors="pt") >>> bbox = torch.tensor([[[0, 0, 1, 1]]]).repeat(1, encoding["input_ids"].shape[-1], 1) >>> encoding["bbox"] = bbox >>> outputs = model(**encoding) ```NT r{rOrrercr|rrrrrrrhrgr7)rrrrr)r1rrr\rr/r0squeezerrhrr<rytor masked_fillfinforgmineyer;rr#rrr)r0r{rOrr&rercr|r1r2rrrrrlast_hidden_statesrrinv_attention_maskrmax_seq_lengthrhinvalid_token_maskself_token_masksubsequent_token_maskrr*initial_token_losssubsequent_token_lossr#r#r$rE0sh2  &     z)BrosSpadeEEForTokenClassification.forward) NNNNNNNNNNNN)rrrr,r,rrrrGrr"rrErHr#r#r2r$r- sX       r-z Bros Model with a token classification head on top (a entity_linker layer on top of the hidden-states output) e.g. for Entity-Linking. The entity_linker is used to predict intra-entity links (one entity to another entity). cr)!BrosSpadeELForTokenClassificationrcsxt|||_|j|_|j|_|j|_t||_t |dr"|j n|j t ||_ |dSt ||_ |dSr!)r+r,r1r#rrlrrrr}r"rvr entity_linkerrrNr2r#r$r,s      z*BrosSpadeELForTokenClassification.__init__Nr{rOrr&rercr|r'rrrr5c  KsP| dur| n|jj} |j||||||| | dd } | d}|dd}|||d}d}|durt}|j\}}|j }t ||dj |t j d}|d}t j|t j|dgt j |dgdd }||dddddft |jj}||dddddft |jj}||d|d||d|}t||| j| jd S) a bbox ('torch.FloatTensor' of shape '(batch_size, num_boxes, 4)'): Bounding box coordinates for each token in the input sequence. Each bounding box is a list of four values (x1, y1, x2, y2), where (x1, y1) is the top left corner, and (x2, y2) is the bottom right corner of the bounding box. bbox_first_token_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to indicate the first token of each bounding box. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. Examples: ```python >>> import torch >>> from transformers import BrosProcessor, BrosSpadeELForTokenClassification >>> processor = BrosProcessor.from_pretrained("jinho8345/bros-base-uncased") >>> model = BrosSpadeELForTokenClassification.from_pretrained("jinho8345/bros-base-uncased") >>> encoding = processor("Hello, my dog is cute", add_special_tokens=False, return_tensors="pt") >>> bbox = torch.tensor([[[0, 0, 1, 1]]]).repeat(1, encoding["input_ids"].shape[-1], 1) >>> encoding["bbox"] = bbox >>> outputs = model(**encoding) ```NTr3rrr4r7rfrr()r1rrr\rrDr5rrrhrr:r6rr;r<ryr7r8rgr9r rr)r0r{rOrr&rercr|r'rrrrrr;r)rr*rr=rhr?maskr#r#r$rEsJ,   (($z)BrosSpadeELForTokenClassification.forwardrr+r#r#r2r$rCsR      rC)rrr r-rC)6rr dataclassesrrrtorch.nnrrr activationsrmodeling_layersrmodeling_outputsr r r modeling_utilsr pytorch_utilsr utilsrrrrconfiguration_brosr get_loggerrloggerrrr&rIrUr_rrrrrrrrrrrr r-rC__all__r#r#r#r$sd           >SJ5"c  n