o ei@stdZddlZddlZddlmZddlmZddlmZddlm Z m Z m Z ddl m Z dd lmZmZdd lmZdd lmZmZdd lmZdd lmZmZddlmZeeZGdddej Z!Gdddej Z"Gdddej#Z$Gdddej#Z%GdddeZ&eGdddeZ'Gddde'Z(eddGd d!d!e'Z)ed"dGd#d$d$e'e Z*d$dgZ+dS)%z/PyTorch TrOCR decoder model (based on RoBERTa).N)nn)CrossEntropyLoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions)PreTrainedModel)auto_docstringlogging) TrOCRConfigcsPeZdZdZdedeffdd Z d dejd ed ejdBffd d ZZ S)TrOCRLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcsd|_t||j|dSN)offsetsuper__init__)selfrr __class__f/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.pyr*sz(TrOCRLearnedPositionalEmbedding.__init__rN input_idspast_key_values_length position_idscs\|dur |jdd\}}tj|||tj|jjd|d}n|d}t ||j S)z3`input_ids' shape is expected to be [bsz x seqlen].Nr)dtypedevicer) shapetorcharangelongweightr%expand unsqueezerforwardr)rr!r"r#bszseq_lenrrr r.0s z'TrOCRLearnedPositionalEmbedding.forward)rN) __name__ __module__ __qualname____doc__intrr(Tensorr. __classcell__rrrr r%src sLeZdZdZd dededededBffdd Zd ejffd d Z Z S)TrOCRScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. ?rr padding_idx embed_scaleNcst|||||_dSN)rrr;)rrrr:r;rrr rFs z!TrOCRScaledWordEmbedding.__init__r!cst||jSr<)rr.r;)rr!rrr r.Jsz TrOCRScaledWordEmbedding.forward)r9) r1r2r3r4r5floatrr(r6r.r7rrrr r8As$r8c seZdZdZddedededBffdd ZeddedededBfd d Ze dd ej d efddZ dd ej ded edBfddZ Z S)"TrOCRSinusoidalPositionalEmbeddingzDThis module produces sinusoidal positional embeddings of any length.N num_positionsrr:cs0td|_||_||_|||||_dSr)rrrrr: get_embeddingweights)rr?rr:rrr rQs z+TrOCRSinusoidalPositionalEmbedding.__init__rcCs|d}td|d}ttj|tjd| }tj|tjdd|d}tjt |t |gdd |d}|ddkrUtj|t |dgdd}|durad||ddf<| tS) z Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the description in Section 3.5 of "Attention Is All You Need". ri'r)r$rdimr&N)mathlogr(expr)int64r=r-catsincosviewzerostoget_default_dtype)rrr:half_dimembrrr r@Xs $& z0TrOCRSinusoidalPositionalEmbedding.get_embeddingrr!r"cCs|\}}|||j||j}|jd|}|jdus&||jdkr0|||j|j|_|jd| d ||d }|S)Nrrr&) size"create_position_ids_from_input_idsr:rMr%rAr@r index_selectrKdetach)rr!r"r/r0r#max_posxrrr r.ks "z*TrOCRSinusoidalPositionalEmbedding.forwardcCs6||}tj|dd|||}||S)z Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. rrB)ner5r(cumsumtype_asr*)rr!r:r"maskincremental_indicesrrr rR}s zETrOCRSinusoidalPositionalEmbedding.create_position_ids_from_input_idsr<)r)r1r2r3r4r5r staticmethodr@r(no_gradr6r.rRr7rrrr r>Ns r>cseZdZdZ       ddedededBd edBd edBd edBd edBd edBdedBffdd Z     ddej dej dBde dBdej dBdedBdej dBde ej ej dBe ej dBffddZ Z S)TrOCRAttentionz>Multi-headed attention from 'Attention Is All You Need' paper.NFT embed_dim num_headskdimvdimdropout is_decoderbiasis_cross_attention layer_idxc st||_|dur|n||_|dur|n||_||_||_|||_|j||jks9td|jd|d|jd|_ ||_ | |_ t j |j||d|_t j |j||d|_t j |||d|_t j |||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩rf)rrr`rbrcrardhead_dim ValueErrorscalingrerhrLineark_projv_projq_projout_proj) rconfigr`rarbrcrdrerfrgrhrrr rs(    zTrOCRAttention.__init__ hidden_stateskey_value_statespast_key_valuesattention_maskoutput_attentionscache_positionreturncCs$|du}|\}} } |||j} d} |dur1t|tr/|j|j} |r+|j} n|j } n|} |r5|n|}|rN|durN| rN| j |jj }| j |jj }nJ| |}||}||d|j|jdd}||d|j|jdd}|dur|s||nd}| |||jd|i\}}|rt|trd|j|j<||jd|jf}| || |j|jdd} | j|} |j|}|j|}|d}t| |dd}|||j| |fkrtd||j| |fd ||dur ||d| |fkr td |d| |fd ||||j| ||}|||j| |}tjj|dd }|r?|||j| |}|||j| |}nd}tjj||j|jd }t||}|||j| |jfkrrtd ||j| |jfd ||||j| |j}|dd}||| | }||}||fS)z#Input shape: Batch x Time x ChannelNFr&rrrxTz$Attention weights should be of size z , but is z!Attention mask should be of size rBptrainingz `attn_output` should be of size )rQrprl isinstancer is_updatedgetrhcross_attention_cacheself_attention_cachelayerskeysvaluesrnrorKrarj transposeupdatereshaper(bmmrkr functionalsoftmaxrdr|rq)rrsrtrurvrwrxrgr/tgt_lenr` query_statesr~curr_past_key_valuescurrent_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputrrr r.s               zTrOCRAttention.forward)NNr_FTFN)NNNFN)r1r2r3r4r5r=boolrr(r6rtupler.r7rrrr r^s`   &r^cseZdZddeffdd Z       ddejdejdBd ejdBd ejdBd edBd edBd edBdejdBfddZ Z S)TrOCRDecoderLayerNrrc st|j|_t||j|j|jd|d|_|j|_t |j |_ |j |_ t |j|_|jrIt||j|j|j|j|jdd|d |_t |j|_t |j|j|_t |j|j|_t |j|_dS)NT)r`rardrerh)r`rarbrcrdrergrh)rr hidden_sizer`r^decoder_attention_headsattention_dropout self_attnrdractivation_function activation_fnactivation_dropoutr LayerNormself_attn_layer_normrecross_attention_hidden_size encoder_attnencoder_attn_layer_normrmdecoder_ffn_dimfc1fc2final_layer_norm)rrrrhrrr rs<   zTrOCRDecoderLayer.__init__FTrsrvencoder_hidden_statesencoder_attention_maskrurw use_cacherxc Cs |} |j|||||d\}} tjj||j|jd}| |}||}d} |durK|} |j||||||d\}} tjj||j|jd}| |}||}|} || |}tjj||j |jd}| |}tjj||j|jd}| |}| |}|f} |r| | | f7} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. past_key_values (`Cache`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rsrurvrwrxrzN)rsrtrvrurwrx) rrrrdr|rrrrrrrr) rrsrvrrrurwrrxresidualself_attn_weightscross_attn_weightsoutputsrrr r.?sH        zTrOCRDecoderLayer.forwardr<)NNNNFTN) r1r2r3rrr(r6rrr.r7rrrr rs4' rc@s$eZdZUeed<dZdZdgZdS)TrOCRPreTrainedModelrrmodelTrN)r1r2r3r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modulesrrrr rs  rcsFeZdZdZdeffdd Z           dddZZS) TrOCRDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`TrOCRDecoderLayer`] Args: config: TrOCRConfig rrcstj|_j|_j|_jrt j nd}t j j |j|d|_ jr5tjj |_ntj|jdj |j|_jrOtj |_nd|_tfddtjD|_d|_|dS)Nr9)r;rcsg|]}t|dqS))rh)r).0irrrr sz)TrOCRDecoder.__init__..F)rrrddecoder_layerdrop layerdrop pad_token_idr:scale_embeddingrDsqrtrr8 vocab_size embed_tokensuse_learned_position_embeddingsrmax_position_embeddingsembed_positionsr>layernorm_embeddingrr ModuleListrangedecoder_layersrgradient_checkpointing post_init)rrrr;rrr rs*   zTrOCRDecoder.__init__Nc  Ks|dur|n|jj}| dur| n|jj} |dur|n|jj}| dur$| n|jj} |dur4|dur4td|durD|} |d| jd}n|durT|dddddf} ntd|jrg|j rg|rgt dd}|r|dur|dusu|jj rt t|jdt|jdnt|jd}|dur|nd}|dur||}|jjr|j| |d }n|j||d }||}|jdur||}tjj||j|j d }| durtj|||jd |jd } t|j||| |d }|dur|durt|j|||d}| rdnd}|rdnd}|r |dur dnd}t|jD]D\}}| r||f7}|j r.tg}||jkr.q||||||||| d}|d}|rT||d f7}|durT||df7}q| r^||f7}| sot dd|||||fDSt!|||||dS)aU Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer&zEYou have to specify either decoder_input_ids or decoder_inputs_embedsz^`use_cache = True` is incompatible with gradient checkpointing. Setting `use_cache = False`...Frr)r"rzr)r%)rr inputs_embedsrvrxru)rrrrvrr)rrurwrrxrcss|] }|dur|VqdSr<r)rvrrr gsz'TrOCRDecoder.forward..)last_hidden_staterurs attentionscross_attentions)"rrrwoutput_hidden_statesruse_return_dictrkrKr'rr|logger warning_onceis_encoder_decoderrrget_seq_lengthrrrrrrrdr(r)r%r r enumeraterrandrrr )rr!rvrrrurrrwr return_dictrxkwargsinputr" embed_posrsall_hidden_statesall_self_attnsall_cross_attentionsidx decoder_layerdropout_probability layer_outputsrrr r.sA                zTrOCRDecoder.forward) NNNNNNNNNNN)r1r2r3r4rrr.r7rrrr rs!ra The TrOCR Model with a language modeling head. Can be used for summarization. This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. ) custom_introcs$eZdZfddZddZZS)TrOCRDecoderWrappercs"t|t||_|dSr<)rrrdecoderrrrrrrr r}s   zTrOCRDecoderWrapper.__init__cOs|j|i|Sr<)r)rargsrrrr r.szTrOCRDecoderWrapper.forward)r1r2r3rr.r7rrrr rus rzy The TrOCR Decoder with a language modeling head. Can be used as the decoder part of [`EncoderDecoderModel`] and cseZdZddiZfddZddZddZd d Zd d Ze dde j d Bde j d Bde j d Bde j d Bded Bde j d Bde j d Bded Bded Bded Bded Bde j d BdeeBfddZZS)TrOCRForCausalLMzoutput_projection.weightz!model.decoder.embed_tokens.weightcsDd|_d|_t|t||_tj|j|j dd|_ | dS)NTFri) rerrrrrrrmrroutput_projectionrrrrr rs    zTrOCRForCausalLM.__init__cCs |jjjSr<rrrrrrr get_input_embeddings z%TrOCRForCausalLM.get_input_embeddingscCs||jj_dSr<r)rvaluerrr set_input_embeddingssz%TrOCRForCausalLM.set_input_embeddingscCs|jSr<rrrrr get_output_embeddingssz&TrOCRForCausalLM.get_output_embeddingscCs ||_dSr<r)rnew_embeddingsrrr set_output_embeddingsrz&TrOCRForCausalLM.set_output_embeddingsNr!rvrrrurlabelsrrwrrrxryc  Ks| dur| n|jj} | dur| n|jj} | dur| n|jj} |jj|||||||| | | | d }||d}d}|durMt}||d|jj |d}| sc|f|dd}|dura|f|S|St |||j |j |j |jdS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import ( ... TrOCRConfig, ... TrOCRProcessor, ... TrOCRForCausalLM, ... ViTConfig, ... ViTModel, ... VisionEncoderDecoderModel, ... ) >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> # TrOCR is a decoder model and should be used within a VisionEncoderDecoderModel >>> # init vision2text model with random weights >>> encoder = ViTModel(ViTConfig()) >>> decoder = TrOCRForCausalLM(TrOCRConfig()) >>> model = VisionEncoderDecoderModel(encoder=encoder, decoder=decoder) >>> # If you want to start from the pretrained model, load the checkpoint with `VisionEncoderDecoderModel` >>> processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-handwritten") >>> model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten") >>> # load image from the IAM dataset >>> url = "https://fki.tic.heia-fr.ch/static/img/a01-122-02.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())).convert("RGB") >>> pixel_values = processor(image, return_tensors="pt").pixel_values >>> text = "industry, ' Mr. Brown commented icily. ' Let us have a" >>> # training >>> model.config.decoder_start_token_id = processor.tokenizer.eos_token_id >>> model.config.pad_token_id = processor.tokenizer.pad_token_id >>> model.config.vocab_size = model.config.decoder.vocab_size >>> labels = processor.tokenizer(text, return_tensors="pt").input_ids >>> outputs = model(pixel_values, labels=labels) >>> loss = outputs.loss >>> round(loss.item(), 2) 5.30 >>> # inference >>> generated_ids = model.generate(pixel_values) >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> generated_text 'industry, " Mr. Brown commented icily. " Let us have a' ```N) r!rvrrrurrrwrrrxrr&r)losslogitsrursrr)rrrwrrrrrrrKrrrursrr)rr!rvrrrurrrrwrrrxrrrrloss_fctoutputrrr r.sBIzTrOCRForCausalLM.forward) NNNNNNNNNNNN)r1r2r3_tied_weights_keysrrrrrrr( LongTensorr6 FloatTensorrrrrr.r7rrrr rs^       r),r4rDr(rtorch.nnr activationsr cache_utilsrrr generationr masking_utilsr r modeling_layersr modeling_outputsr rmodeling_utilsrutilsrrconfiguration_trocrr get_loggerr1r Embeddingrr8Moduler>r^rrrrr__all__rrrr sF          <qc