o eiM%@sHdZddlZddlZddlZddlmZddlmZmZmZddl m Z ddl m Z ddlmZmZmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZmZmZdd l m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*e'+e,Z-Gdddej.Z/Gdddej.Z0Gdddej.Z1Gdddej.Z2Gdddej.Z3Gdddej.Z4Gdddej.Z5Gdd d eZ6Gd!d"d"ej.Z7e%Gd#d$d$e!Z8Gd%d&d&e8Z9e%Gd'd(d(e8Z:e%d)d*Gd+d,d,e8eZ;e%Gd-d.d.e8Ze%Gd4d5d5e8Z?gd6Z@dS)7zPyTorch UMT5 model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_causal_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) DUMMY_INPUTS DUMMY_MASKauto_docstringis_torchdynamo_compilingloggingtorch_compilable_check) UMT5Configcs&eZdZdfdd ZddZZS) UMT5LayerNormư>cs&ttt||_||_dS)ze Construct a layernorm module in the UMT5 style. No bias and no subtraction of mean. N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__d/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/umt5/modeling_umt5.pyr"7s  zUMT5LayerNorm.__init__cCs\|tjdjddd}|t||j}|jjtj tj fvr)||jj}|j|S)NT)keepdim) tor$float32powmeanrsqrtr'r&dtypefloat16bfloat16)r( hidden_statesvariancer-r-r.forward?s  zUMT5LayerNorm.forward)r )__name__ __module__ __qualname__r"r< __classcell__r-r-r+r.r6src*eZdZdeffdd ZddZZS)UMT5DenseActDenseconfigcsTttj|j|jdd|_tj|j|jdd|_t|j |_ t |j |_ dSNFbias)r!r"rLineard_modeld_ffwiwoDropout dropout_ratedropoutr dense_act_fnactr(rCr+r-r.r"Qs zUMT5DenseActDense.__init__cCsl||}||}||}t|jjtjr/|j|jjjkr/|jjjtj kr/| |jjj}||}|SN) rJrPrN isinstancerKr&r$Tensorr7int8r2r(r:r-r-r.r<Xs    zUMT5DenseActDense.forwardr=r>r?rr"r<r@r-r-r+r.rBPsrBcrA)UMT5DenseGatedActDenserCcsjttj|j|jdd|_tj|j|jdd|_tj|j|jdd|_t |j |_ t |j |_dSrD)r!r"rrGrHrIwi_0wi_1rKrLrMrNrrOrPrQr+r-r.r"hs zUMT5DenseGatedActDense.__init__cCsz|||}||}||}||}t|jjtjr6|j |jjj kr6|jjj tj kr6| |jjj }||}|SrR) rPrYrZrNrSrKr&r$rTr7rUr2)r(r: hidden_gelu hidden_linearr-r-r.r<ps   zUMT5DenseGatedActDense.forwardrWr-r-r+r.rXgsrXcrA) UMT5LayerFFrCcsJt|jrt||_nt||_t|j|jd|_ t |j |_ dS)Nr*)r!r" is_gated_actrXDenseReluDenserBrrHlayer_norm_epsilon layer_normrrLrMrNrQr+r-r.r"s   zUMT5LayerFF.__init__cCs&||}||}|||}|SrR)rbr`rN)r(r:forwarded_statesr-r-r.r<s  zUMT5LayerFF.forwardrWr-r-r+r.r]s r]c seZdZdZddedBffdd Zdejdejfd d Zd d Z dd dZ    ddejdejdBde dBdejdBdejdBf ddZ Z S) UMT5Attentionz7 T5's attention using relative_attention_bias. FN layer_idxcst|j|_||_|j|_|j|_|j|_|j|_|j |_ |j |_ |j |j|_ ||_|dur@|jr@td|jjdtj|j|j dd|_tj|j|j dd|_tj|j|j dd|_tj|j |jdd|_|jrzt|j|j |_dSdS)NzInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.FrE)r!r" is_decoderhas_relative_attention_biasrelative_attention_num_bucketsrelative_attention_max_distancerHd_kvkey_value_proj_dim num_headsn_headsrMrN inner_dimrelogger warning_oncer,r=rrGqkvo Embeddingrelative_attention_bias)r(rCrgrer+r-r.r"s, zUMT5Attention.__init__ projectionreturncCs6|dd|j|jf}||dddd}|S)Nr0rr/rr)sizermrkviewpermute)r(rwnew_projection_shapenew_projectionr-r-r._shapeszUMT5Attention._shapec Csd}|j}|j}|js!|d}||dktj|7}t|}n t|t| }|d}||k}t | |t ||}|||}||tj}t|t ||d}|t |||7}|S)a Adapted from Mesh Tensorflow: https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transformer/transformer_layers.py#L593 Translate relative position to a bucket number for relative attention. The relative position is defined as memory_position - query_position, i.e. the distance in tokens from the attending position to the attended-to position. If bidirectional=False, then positive relative positions are invalid. We use smaller buckets for small absolute relative_position and larger buckets for larger absolute relative_positions. All relative positions >=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rr/r)rhrirfr2r$longabsmin zeros_likelogfloatmath full_likewhere) r(relative_positionrelative_buckets num_buckets max_distance max_exactis_small log_ratiorelative_position_if_larger-r-r._relative_position_buckets$   z'UMT5Attention._relative_position_bucketc Cs|dur |jjj}|durtj|tj|ddddf}n|dddf}tj|tj|ddddf}||}||}||} | gdd} | S)z%Compute binned relative position biasN)r7device)r/rrr) rvr&rr$arangerrr{ unsqueeze) r( query_length key_lengthrcache_positioncontext_positionmemory_positionrrelative_position_bucketvaluesr-r-r. compute_biass    zUMT5Attention.compute_biasr:encoder_hidden_statespast_key_valuesattention_maskrcCsd|jdd\}}|du}||} | |d|j|jdd} d} |durr?__doc__intr"r$rTr~rrr r<r@r-r-r+r.rds* /rdcs8eZdZddedBffdd Z   dddZZS) UMT5LayerSelfAttentionNrec>tt|d|d|_t|j|jd|_t |j |_ dS)NTrgrer^) r!r"rd SelfAttentionrrHrarbrrLrMrNr(rCrer+r-r.r"P zUMT5LayerSelfAttention.__init__cCsD||}|j||||d}|||d}|f|dd}|S)Nrrrrr)rbrrN)r(r:rrrnormed_hidden_statesattention_outputoutputsr-r-r.r<Vs zUMT5LayerSelfAttention.forwardrR)NNNr=r>r?rr"r<r@r-r-r+r.rOs  rcs:eZdZddedBffdd Z    dddZZS) UMT5LayerCrossAttentionNrecr)NFrr^) r!r"rdEncDecAttentionrrHrarbrrLrMrNrr+r-r.r"jrz UMT5LayerCrossAttention.__init__c CsF||}|j|||||d}|||d}|f|dd} | S)Nrrrrrr)rbrrN) r(r:rrrrrr layer_outputrr-r-r.r<ps zUMT5LayerCrossAttention.forwardrRrrr-r-r+r.ris rcs@eZdZddedBffdd Z       d ddZZS) UMT5BlockNrecs^t|j|_t|_|jt||d|jr%|jt||d|jt |dS)Nre) r!r"rfr ModuleListlayerappendrrr]rr+r-r.r"s  zUMT5Block.__init__Fc CsL|jd||||d\}} |jtjkr1t|jj} tt|| d| } tj || | d}d} |j o9|du} | rn|jd|||||d\}} |jtjkrnt|jj} tt|| d| } tj || | d}|jd|}|jtjkrt|jj} tt|| d| } tj || | d}|f}|r|| | f7}|S)Nrri)rmaxrrr0) rr7r$r8finforrisinfanyclamprf)r(r:rrencoder_attention_maskr use_cacheoutput_attentionsrself_attn_weights max_dtype clamp_valuecross_attn_weightsdo_cross_attentionrr-r-r.r<sB     zUMT5Block.forwardrR)NNNNFFNrr-r-r+r.rs rcs<eZdZdZdeffdd ZdejdejfddZZ S) UMT5ClassificationHeadz-Head for sentence-level classification tasks.rCcsBtt|j|j|_tj|jd|_t|j|j |_ dS)N)r) r!r"rrGrHdenserLclassifier_dropoutrN num_labelsout_projrQr+r-r.r"s zUMT5ClassificationHead.__init__r:rxcCs6||}||}t|}||}||}|SrR)rNrr$tanhrrVr-r-r.r<s     zUMT5ClassificationHead.forward) r=r>r?rrr"r$rTr<r@r-r-r+r.rsrc@sReZdZUeed<dZdZdZdgZdgZ e ddZ e dd Zd d Zd S) UMT5PreTrainedModelrC transformerTrrKcCs$tt}tt}|||d}|S)N)decoder_input_ids input_idsdecoder_attention_mask)r$tensorrr)r(r input_mask dummy_inputsr-r-r.rs  z UMT5PreTrainedModel.dummy_inputscCs |jj}t|trt|j|dd St|ttt t fr^tj |j jd|ddt |dr>|jjs>tj |jjd|ddt |dr\tj |jjd||jjddt|jjd Sd St|trt |dr}tj |jjd|ddt|jjd Sd St|trtj |jjd||jjddt |jdr|jjd urt|jjtj |jjd||jjddt |jdr|jjd urt|jjd Sd Sd St|tr$tj |jjd||jjddt |jdr|jjd urt|jjtj |jjd||jjddt |jdr |jjd ur"t|jjd Sd Sd St|trtj |jjd||jjddt |jdrO|jjd urOt|jjtj |jjd||jjddt |jdrt|jjd urtt|jjtj |jjd||jjddt |jdr|jjd urt|jjd Sd Sd St|t r|jj}|jj!}|jj"}tj |j#jd|||ddtj |j$jd||ddtj |j%jd||ddtj |j&jd|||dd|j'rtj |j(jd||ddd Sd Sd S) zInitialize the weights?g)r5stdlm_head qa_outputs classifierrFN))rCinitializer_factorrSrinit constant_r& UMT5ModelUMT5ForConditionalGenerationUMT5EncoderModelUMT5ForQuestionAnsweringnormal_sharedhasattrtie_word_embeddingsrrrHzeros_rFUMT5ForTokenClassificationrrrrrBrJrKrIrXrYrZrdrjrlrqrrrsrtrgrv)r(modulefactorrHrkrmr-r-r. _init_weightss|                     z!UMT5PreTrainedModel._init_weightscCsx|jj}|jj}|durtd||j}|dddf|dddf<||d<|dur2td||dk||S)Nzself.model.config.decoder_start_token_id has to be defined. In UMT5 it is usually set to the pad_token_id. See UMT5 docs for more information..r0r).rz1self.model.config.pad_token_id has to be defined.)rCdecoder_start_token_id pad_token_id ValueError new_zerosrclone masked_fill_)r(rr r shifted_input_idsr-r-r. _shift_right5s  z UMT5PreTrainedModel._shift_rightN)r=r>r?r__annotations__base_model_prefixsupports_gradient_checkpointing_can_compile_fullgraph_no_split_modules_keep_in_fp32_modulespropertyrr$no_gradr rr-r-r-r.rs    BrcsDeZdZfddZddZ           dddZZS) UMT5Stackcsxttjj|_j|_tfddt j D|_ t jj d|_tj|_d|_|dS)Ncsg|]}t|dqS)r)r).0irCr-r. Psz&UMT5Stack.__init__..r^F)r!r"rru vocab_sizerH embed_tokensrfrrange num_layersblockrrafinal_layer_normrLrMrNgradient_checkpointing post_initrQr+rr.r"Ls   zUMT5Stack.__init__cCs ||_dSrR)r!r(new_embeddingsr-r-r.set_input_embeddingsX zUMT5Stack.set_input_embeddingsNc Ks|dur|n|jj}|dur|n|jj}| dur| n|jj} | dur$| n|jj} |durB|durB|jr5dnd} td| d| d|durS|}|d|d}n|dur`|dd}n|jrednd} td| d| d |j r|j r|rt d d }|dur|j durtd | |}|\}}|d ur|jstd|d|jr|r|dur|jjrtt|jdt|jd}n t|jd}n|jsd}|dur|nd}| durtj||||jd} |durts||}tj|||jd}|jr t|j||| |d}n&|dur-|ddddddf}|j|jd}d|t|jj}nd}|jrV|durV|\}}}||f}|durPtj||jd}||}nd}| r]dnd}|rddnd}|ro|jrodnd}||}t|jD]2\}}| r||f}||||||||| d}|d}|r||df7}|jr||df7}q{| |}||}| r||f}| st!dd|||||fDSt"|||||dS)Ndecoder_zYou cannot specify both zinput_ids and zinputs_embeds at the same timer0zYou have to specify either z input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fzsz$UMT5Stack.forward..)last_hidden_staterr: attentionscross_attentions)#rCrroutput_hidden_statesuse_return_dictrfr ryrzr&rrorpr!is_encoder_decoderr r rr$rrrr%r r2r7rrinvert_attention_maskrN enumerater$r%tupler) r(rrrrr.rrrr3 return_dictrkwargserr_msg_prefix input_shaperrpast_key_values_lengthmask_seq_length causal_maskencoder_batch_sizeencoder_sequence_length_encoder_hidden_shapeencoder_extended_attention_maskall_hidden_statesall_attentionsall_cross_attentionsr:r layer_module layer_outputsr-r-r.r<[s                zUMT5Stack.forward) NNNNNNNNNNN)r=r>r?r"r*r<r@r-r-r+r.rKs  rcseZdZUdZdZeed<dddZfddZdd Z d d Z e dd e j d Bde jd Bde j d Bde jd Bdeee jd Bded Bde jd Bde jd Bded Bded Bded Bded Bde j d Bdee jeBfddZZS)rao Examples: ```python >>> from transformers import UMT5Model, AutoTokenizer >>> model = UMT5Model.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> noisy_text = "UN Offizier sagt, dass weiter werden muss in Syrien." >>> label = " verhandelt" >>> inputs = tokenizer(inputs, return_tensors="pt") >>> labels = tokenizer(label=label, return_tensors="pt") >>> outputs = model(input_ids=inputs["input_ids"], decoder_input_ids=labels["input_ids"]) >>> hidden_states = outputs.last_hidden_state ```umt5rC shared.weightencoder.embed_tokens.weightdecoder.embed_tokens.weightcslt|t|j|j|_t|}d|_ d|_ t ||_ t|}d|_ |j |_t ||_|dSNFT)r!r"rrur rHrcopydeepcopyrfrrencodernum_decoder_layersr#decoderr'r(rCencoder_configdecoder_configr+r-r.r"s      zUMT5Model.__init__cC|jSrRrr(r-r-r.get_input_embeddings(zUMT5Model.get_input_embeddingscC"||_|j||j|dSrRrrRr*rTr(r-r-r.r*, zUMT5Model.set_input_embeddingsNrrrrencoder_outputsrr.decoder_inputs_embedsrrr3r9rrxc Ks| dur| n|jj} | dur| n|jj} |dur$|j|||| | | d}n$| rHt|tsHt|dt|dkr9|dndt|dkrD|dndd}|d}|j||||||| | | | | d }| sb||St|j |j |j |j |j |j |j |j dS) ah input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Example: ```python >>> from transformers import AutoTokenizer, UMT5Model >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5Model.from_pretrained("google/umt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> # preprocess: Prepend decoder_input_ids with start token which is pad token for UMT5Model. >>> # This is not needed for torch's UMT5ForConditionalGeneration as it does this internally using labels arg. >>> decoder_input_ids = model._shift_right(decoder_input_ids) >>> # forward pass >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state ```Nrrr.rr3r9rrr/r0r:r1 rrr.rrrrrr3r9r)r0rdecoder_hidden_statesdecoder_attentionsr2encoder_last_hidden_staterencoder_attentions)rCrr4rRrSrlenrTrr0rr:r1r2)r(rrrrr`rr.rarrr3r9rr:r:decoder_outputsr-r-r.r<1sVCzUMT5Model.forward NNNNNNNNNNNNN)r=r>r?r model_typerr_tied_weights_keysr"r[r*rr$ LongTensor FloatTensor BoolTensorr8r rTboolrr<r@r-r-r+r.rsj        rz< UMT5 Model with a `language modeling` head on top. ) custom_introc!seZdZdZdZddddZfddZddZd d Ze dd e j d Bd e j d Bde j d Bde j d Bdeee jd Bded Bde j d Bde j d Bde j d Bded Bded Bded Bded Bde j d Bdee j eBfddZde jfddZZS) ra Examples: ```python >>> from transformers import UMT5ForConditionalGeneration, AutoTokenizer >>> model = UMT5ForConditionalGeneration.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> summary = "Weiter Verhandlung in Syrien." >>> inputs = tokenizer(article, text_target=summary, return_tensors="pt") >>> outputs = model(**inputs) >>> loss = outputs.loss ```rJrK)rMrNzlm_head.weightcst||j|_t|j|j|_t |}d|_ d|_ t ||_ t |}d|_ |j|_t ||_tj|j|jdd|_|dS)NFTrE)r!r"rH model_dimrrur rrPrQrfrrrRrSr#rTrGrr'rUr+r-r.r"s      z%UMT5ForConditionalGeneration.__init__cCrXrRrYrZr-r-r.r[r\z1UMT5ForConditionalGeneration.get_input_embeddingscCr]rRr^r(r-r-r.r*r_z1UMT5ForConditionalGeneration.set_input_embeddingsNrrrrr`rr.ralabelsrrr3r9rrxc Ks| dur| n|jj} | dur| n|jj} |dur$|j|||| | | d}n$| rHt|tsHt|dt|dkr9|dndt|dkrD|dndd}|d}| dur]|dur]|dur]|| }|j||||||| | | | |d }|d}|jj r|||j d}| |}d}| durt d d }| |j} ||d |d | d }| s|f|dd|}|dur|f|S|St|||j|j|j|j|j|j|jd S) a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoTokenizer, UMT5ForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5ForConditionalGeneration.from_pretrained("google/umt5-small") >>> # training >>> input_ids = tokenizer("The walks in park", return_tensors="pt").input_ids >>> labels = tokenizer(" cute dog the ", return_tensors="pt").input_ids >>> outputs = model(input_ids=input_ids, labels=labels) >>> loss = outputs.loss >>> logits = outputs.logits >>> # inference >>> input_ids = tokenizer("Studies have shown that good for you", return_tensors="pt").input_ids >>> outputs = model.generate(input_ids) >>> tokenizer.decode(outputs[0], skip_special_tokens=True) ```Nrbrrr/rcrdrr  ignore_indexr0 losslogitsrrerfr2rgrrh)rCrr4rRrSrrirrTrrsrrr2rrzryrrr:r1r2r0)r(rrrrr`rr.rartrrr3r9rr:r:rjsequence_output lm_logitsrxloss_fctoutputr-r-r.r<spG    z$UMT5ForConditionalGeneration.forwardcCs ||SrR)r)r(rtr-r-r.%prepare_decoder_input_ids_from_labelsvr+zBUMT5ForConditionalGeneration.prepare_decoder_input_ids_from_labels)NNNNNNNNNNNNNN)r=r>r?rrlrmr"r[r*rr$rnrorpr8rTr rqrr<r~r@r-r-r+r.rst        rcseZdZdZdZddiZfddZddZd d Ze dd e j d Bd e j d Bde j d Bde d Bde d Bde d Bdee j eBfddZZS)ra Examples: ```python >>> from transformers import UMT5EncoderModel, AutoTokenizer >>> model = UMT5EncoderModel.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> input_ids = tokenizer(article, return_tensors="pt").input_ids >>> outputs = model(input_ids) >>> hidden_state = outputs.last_hidden_state ```rJrMrKcsJt|t|j|j|_t|}d|_ d|_ t ||_ | dS)NF)r!r"rrur rHrrPrQrr5rrRr')r(rCrVr+r-r.r"s    zUMT5EncoderModel.__init__cCrXrRrYrZr-r-r.r[r\z%UMT5EncoderModel.get_input_embeddingscCs||_|j|dSrR)rrRr*r(r-r-r.r*sz%UMT5EncoderModel.set_input_embeddingsNrrr.rr3r9rxc Ks.|dur|n|jj}|j||||||d}|S)aQ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). Example: ```python >>> from transformers import AutoTokenizer, UMT5EncoderModel >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5EncoderModel.from_pretrained("google/umt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model(input_ids=input_ids) >>> last_hidden_states = outputs.last_hidden_state ```Nrb)rCr4rR) r(rrr.rr3r9r:r`r-r-r.r<s# zUMT5EncoderModel.forward)NNNNNN)r=r>r?rrlrmr"r[r*rr$rnrorqr8rr<r@r-r-r+r.rzs<   rz UMT5 model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cseZdZdgZdeffdd Ze            ddejdBdej dBdejdBd ejdBd e ej dBd ej dBd ej dBd ejdBde dBde dBde dBde dBde eBfddZZS)UMT5ForSequenceClassificationFdecoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weightrCcs,t|t||_t||_|dSrR)r!r"rrrclassification_headr'rQr+r-r.r"s    z&UMT5ForSequenceClassification.__init__Nrrrrr`r.rartrrr3r9rxc  Ks`| dur| n|jj} |durd} |dur!|dur!td|jj|dur6|dur6|dur1td||}|j|||||||| | | | d }|d}||jj  |j }t t |ddkd|j\}}}||ddf|d |ddd ddf}||}d}|dur| |j }|jjdur|jjdkrd |j_n|jjdkr|jt jks|jt jkrd |j_nd |j_|jjd krt}|jjdkr|||}n-|||}n'|jjd krt}||d |jj|d }n|jjd krt}|||}| s|f|dd}|dur|f|S|St|||j|j|j |j!|j"|j#|j$d S)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for If no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.) rrrr`r.rarrr3r9rrz7All examples must have the same number of tokens.r0 regressionsingle_label_classificationmulti_label_classificationrw)%rCr4NotImplementedErrorr,r=r rreq eos_token_idr2rrr$unique_consecutivesumnumelrrzr problem_typerr7rrrsqueezerrrrrerfr2rgrrh)r(rrrrr`r.rartrrr3r9r:rrzeos_maskrrBr)sentence_representationryrxr|r}r-r-r.r<s0    ,      $      z%UMT5ForSequenceClassification.forward) NNNNNNNNNNNN)r=r>r?"_keys_to_ignore_on_load_unexpectedrr"rr$rnrTlistrorqr8rr<r@r-r-r+r.rsV      rcseZdZdgZdeffdd Ze       ddejdBdejdBdejdBd ejdBd e dBd e dBd e dBd e eje BfddZ Z S)rrrCcsJt||j|_t||_t|j|_t |j |j|_ | dSrR) r!r"rrrrrLrrNrGr)rr'rQr+r-r.r"rs   z#UMT5ForTokenClassification.__init__Nrrr.rtrr3r9rxcKs|dur|n|jj}|j||||||d} | d} || } || } d} |dur9t} | | d|j|d} |sN| | ddf}| durL| f|S|St| | | j | j dS)aB input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. N)rr.rr3r9rr0r/)rxryr:r1) rCr4rrNrrrzrrr:r1)r(rrr.rtrr3r9r:rr:ryrxr|r}r-r-r.r<}s2  z"UMT5ForTokenClassification.forward)NNNNNNN)r=r>r?rrr"rr$rTrqr8rr<r@r-r-r+r.rms8  rcseZdZdddZfddZddZddZe dd ej d Bd ej d Bd ej d Bd ej d Bde e ej d Bdej d Bdej d Bdej d Bdej d Bded Bded Bded Bded Bde ej eBfddZZS)rrKrLcst||j|_t|j|j|_t |}d|_ d|_ t ||_ t |}d|_ |j|_t ||_|j|_t|j|j|_|dSrO)r!r"rHrsrrur rrPrQrfrrrRrSr#rTrrGrr'rUr+r-r.r"s      z!UMT5ForQuestionAnswering.__init__cCrXrRrYrZr-r-r.r[r\z-UMT5ForQuestionAnswering.get_input_embeddingscCr]rRr^r(r-r-r.r*r_z-UMT5ForQuestionAnswering.set_input_embeddingsNrrrrr`start_positions end_positionsr.rarrr3r9rxc Ksd| dur| n|jj} | dur| n|jj} |dur|durd} |dur3| dur3|dur.td||}| dur9| n|jj} | durC| n|jj} |durW|j|||| | | d}n$| r{t|ts{t|dt|dkrl|dndt|dkrw|dndd}|d}|j ||| d||| | | | d }|d}| |}|j dd d \}}| d  }| d  }d}|dur|durt|dkr| d |j}t|dkr| d |j}|d}|d|}|d|}t|d }|||}|||}||d}| s||f|dd|}|dur|f|S|St||||j|j|j|j|j|j|jd S)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. NFrrbrrr/rc) rrr.rrrrrr3r9r0rru) rx start_logits end_logitsrrerfr2rgrrh)rCr4rr rrRrSrrirTrsplitrrryr2rrrrrr:r1r2r0)r(rrrrr`rrr.rarrr3r9r:r:rjrzryrr total_loss ignored_indexr| start_lossend_lossr}r-r-r.r<s.         z UMT5ForQuestionAnswering.forwardrk)r=r>r?rmr"r[r*rr$rnrorpr8rTrqrr<r@r-r-r+r.rsd       r)rrrrrrr)ArrPrr$rtorch.nnrrrr-rr activationsr cache_utilsr r r generationr masking_utilsr modeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrutilsrrrrrrconfiguration_umt5r get_loggerr=roModulerrBrXr]rdrrrrrrrrrrrr__all__r-r-r-r.sh       $   9Fm2,N[J 4