o eiJ@sdZddlZddlZddlZddlmZddlmZmZmZddl m Z ddl m Z ddlmZmZmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZmZm Z dd l!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*e'+e,Z-Gdddej.Z/zddl0m1Z1e1Z/e-2dWne3yYne4ye-5dYnwGdddej.Z6Gdddej.Z7Gdddej.Z8Gdddej.Z9Gdddej.Z:Gd d!d!ej.Z;Gd"d#d#eZGd(d)d)e>Z?e&Gd*d+d+e>Z@e&d,d-Gd.d/d/e>eZAe&Gd0d1d1e>ZBe&d2d-Gd3d4d4e>ZCe&Gd5d6d6e>ZDe&Gd7d8d8e>ZEgd9ZFdS):zPyTorch T5 model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) DUMMY_INPUTS DUMMY_MASKauto_docstringloggingtorch_compilable_check)T5Configcs&eZdZdfdd ZddZZS) T5LayerNormư>cs&ttt||_||_dS)zc Construct a layernorm module in the T5 style. No bias and no subtraction of mean. N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__`/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.pyr"/s  zT5LayerNorm.__init__cCs\|tjdjddd}|t||j}|jjtj tj fvr)||jj}|j|S)NT)keepdim) tor$float32powmeanrsqrtr'r&dtypefloat16bfloat16)r( hidden_statesvariancer-r-r.forward7s  zT5LayerNorm.forward)r )__name__ __module__ __qualname__r"r< __classcell__r-r-r+r.r.sr) FusedRMSNormzODiscovered apex.normalization.FusedRMSNorm - will use it instead of T5LayerNormzBdiscovered apex but it failed to load, falling back to T5LayerNormc*eZdZdeffdd ZddZZS)T5DenseActDenseconfigcsTttj|j|jdd|_tj|j|jdd|_t|j |_ t |j |_ dSNFbias)r!r"rLineard_modeld_ffwiwoDropout dropout_ratedropoutr dense_act_fnactr(rDr+r-r.r"Us zT5DenseActDense.__init__cCsl||}||}||}t|jjtjr/|j|jjjkr/|jjjtj kr/| |jjj}||}|SN) rKrQrO isinstancerLr&r$Tensorr7int8r2r(r:r-r-r.r<\s    zT5DenseActDense.forwardr=r>r?rr"r<r@r-r-r+r.rCTsrCcrB)T5DenseGatedActDenserDcsjttj|j|jdd|_tj|j|jdd|_tj|j|jdd|_t |j |_ t |j |_dSrE)r!r"rrHrIrJwi_0wi_1rLrMrNrOrrPrQrRr+r-r.r"ks zT5DenseGatedActDense.__init__cCsz|||}||}||}||}t|jjtjr6|j |jjj kr6|jjj tj kr6| |jjj }||}|SrS) rQrZr[rOrTrLr&r$rUr7rVr2)r(r: hidden_gelu hidden_linearr-r-r.r<ss   zT5DenseGatedActDense.forwardrXr-r-r+r.rYjsrYcrB) T5LayerFFrDcsJt|jrt||_nt||_t|j|jd|_ t |j |_ dS)Nr*)r!r" is_gated_actrYDenseReluDenserCrrIlayer_norm_epsilon layer_normrrMrNrOrRr+r-r.r"s   zT5LayerFF.__init__cCs&||}||}|||}|SrS)rcrarO)r(r:forwarded_statesr-r-r.r<s  zT5LayerFF.forwardrXr-r-r+r.r^s r^csbeZdZ  ddededBffdd Zedd d Zdd d Z        dddZ Z S) T5AttentionFNrD layer_idxcst|j|_||_|j|_|j|_|j|_|j|_|j |_ |j |_ |j |j|_ ||_|dur@|jr@td|jjdtj|j|j dd|_tj|j|j dd|_tj|j|j dd|_tj|j |jdd|_|jrxt|j|j |_d|_dS)NzInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.FrF)r!r" is_decoderhas_relative_attention_biasrelative_attention_num_bucketsrelative_attention_max_distancerId_kvkey_value_proj_dim num_headsn_headsrNrO inner_dimrflogger warning_oncer,r=rrHqkvo Embeddingrelative_attention_biasgradient_checkpointingr(rDrhrfr+r-r.r"s,  zT5Attention.__init__T cCsd}|r|d}||dktj|7}t|}n t|t| }|d}||k}|t||t||||tj}t|t ||d}|t |||7}|S)a Adapted from Mesh Tensorflow: https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transformer/transformer_layers.py#L593 Translate relative position to a bucket number for relative attention. The relative position is defined as memory_position - query_position, i.e. the distance in tokens from the attending position to the attended-to position. If bidirectional=False, then positive relative positions are invalid. We use smaller buckets for small absolute relative_position and larger buckets for larger absolute relative_positions. All relative positions >=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rr/r) r2r$longabsmin zeros_likelogfloatmath full_likewhere)relative_position bidirectional num_buckets max_distancerelative_buckets max_exactis_smallrelative_position_if_larger-r-r._relative_position_buckets*  z%T5Attention._relative_position_bucketc Cs|dur |jjj}|durtj|tj|ddddf}n |dddf|}tj|tj|ddddf}||}|j||j |j |j d}||} | gd d} | S)z%Compute binned relative position biasN)r7device)rrr)r/rrr) rwr&rr$aranger|r2rrgrirjpermute unsqueeze) r( query_length key_lengthrcache_positioncontext_positionmemory_positionrrelative_position_bucketvaluesr-r-r. compute_biass    zT5Attention.compute_biasc Cs|jdd\} } |du} ||} | | d|j|jdd} d}t|tr8|j |j }| r4|j }n|j }n|}| r>|n|}| rW|durW|rW|j |j j}|j |j j}nJ||}||}|| d|j|jdd}|| d|j|jdd}|dur| s| nd} ||||j d| i\}}| rt|trd|j|j <t| |dd}|dur|jd }|dur|n| dd}|jstjd|j| |f|j|jd }|jr|jrd|_n|j|||j| d }|dddd| dddf}|dur|ddddddd|jd f}||}|}||7}tjj| dd !|}tjj"||j"|jd }t||}|dd#}|| d|j$}|%|}||f}|rY||f}|S)z Self-attention (if key_value_states is None) or attention over source sentence (provided by key_value_states). Nr/r0rFrTr)rr7)rrdim)ptraining)&shaperrviewrnrl transposerTr is_updatedgetrfcross_attention_cacheself_attention_cachelayerskeysrrsrtupdater$matmulrhzerosrr7rxr requires_gradrr functionalsoftmaxrtype_asrO contiguousroru)r(r:maskkey_value_states position_biaspast_key_valuesr use_cacheoutput_attentionsr batch_size seq_lengthis_cross_attention query_statesrcurr_past_key_valuescurrent_states key_states value_statesscoresrreal_seq_length causal_maskposition_bias_masked attn_weights attn_outputoutputsr-r-r.r<sp           " &   zT5Attention.forwardFN)Trzr{)NN)NNNNNFFN) r=r>r?rintr" staticmethodrrr<r@r-r-r+r.res("  /recs>eZdZddedBffdd Z      d ddZZS) T5LayerSelfAttentionFNrfcs>tt|||d|_t|j|jd|_t |j |_ dS)Nrhrfr_) r!r"re SelfAttentionrrIrbrcrrMrNrOryr+r-r.r"fs zT5LayerSelfAttention.__init__c CsJ||}|j|||||||d} ||| d}|f| dd} | S)N)rrrrrrrr)rcrrO) r(r:attention_maskrrrrrnormed_hidden_statesattention_outputrr-r-r.r<ns  zT5LayerSelfAttention.forwardr)NNNFFNr=r>r?rr"r<r@r-r-r+r.res rcs@eZdZddedBffdd Z       d ddZZS) T5LayerCrossAttentionNrfcs>tt|d|d|_t|j|jd|_t |j |_ dS)NFrr_) r!r"reEncDecAttentionrrIrbrcrrMrNrO)r(rDrfr+r-r.r"s zT5LayerCrossAttention.__init__Fc  CsN||} |j| |||||||| d } ||| d} | f| dd} | S)N)rrrrrrrrrr)rcrrO)r(r:rrrrrrrrrr layer_outputrr-r-r.r<s  zT5LayerCrossAttention.forwardrS)NNNFNFNrr-r-r+r.rs rcsFeZdZd dedBffdd Z          d ddZZS) T5BlockFNrfcs`t|j|_t|_|jt|||d|jr&|jt||d|jt |dS)Nr)rf) r!r"rgr ModuleListlayerappendrrr^ryr+r-r.r"s   zT5Block.__init__Tc  Cs|jd|||||| | d} | d}| dd} |jtjkr?tt|t|jjdt|jj}tj || |d}|j oE|du}|r|jd|||||| dd|| d}|d}|jtjkrtt|t|jjdt|jj}tj || |d}| |dd} |jd|}|jtjkrtt|t|jjdt|jj}tj || |d}|f}|| S)Nr)rrrrrrri)r~maxr0)rrrrrrr) rr7r$r8risinfanyfinforclamprg)r(r:rrencoder_hidden_statesencoder_attention_maskencoder_decoder_position_biasrrr return_dictrself_attention_outputsattention_outputs clamp_valuedo_cross_attentioncross_attention_outputsrr-r-r.r<sd           zT5Block.forwardr) NNNNNNFFTNrr-r-r+r.rsrcs<eZdZdZdeffdd ZdejdejfddZZ S) T5ClassificationHeadz-Head for sentence-level classification tasks.rDcsBtt|j|j|_tj|jd|_t|j|j |_ dS)N)r) r!r"rrHrIdenserMclassifier_dropoutrO num_labelsout_projrRr+r-r.r" s zT5ClassificationHead.__init__r:returncCs6||}||}t|}||}||}|SrS)rOrr$tanhrrWr-r-r.r<s     zT5ClassificationHead.forward) r=r>r?__doc__rr"r$rUr<r@r-r-r+r.rsrc@sReZdZUeed<dZdZdZdgZdgZ e ddZ e dd Zd d Zd S) T5PreTrainedModelrD transformerTrrLcCs$tt}tt}|||d}|S)N)decoder_input_ids input_idsdecoder_attention_mask)r$tensorrr)r(r input_mask dummy_inputsr-r-r.r$s  zT5PreTrainedModel.dummy_inputscCs |jj}t|trt|j|dd St|ttt t fr^tj |j jd|ddt |dr>|jjs>tj |jjd|ddt |dr\tj |jjd||jjddt|jjd Sd St|trt |dr}tj |jjd|ddt|jjd Sd St|trtj |jjd||jjddt |jdr|jjd urt|jjtj |jjd||jjddt |jdr|jjd urt|jjd Sd Sd St|tr$tj |jjd||jjddt |jdr|jjd urt|jjtj |jjd||jjddt |jdr |jjd ur"t|jjd Sd Sd St|trtj |jjd||jjddt |jdrO|jjd urOt|jjtj |jjd||jjddt |jdrt|jjd urtt|jjtj |jjd||jjddt |jdr|jjd urt|jjd Sd Sd St|t r|jj}|jj!}|jj"}tj |j#jd|||ddtj |j$jd||ddtj |j%jd||ddtj |j&jd|||dd|j'rtj |j(jd||ddd Sd Sd S) zInitialize the weightsg?g)r5stdlm_head qa_outputs classifierrGN))rDinitializer_factorrTrinit constant_r&T5ModelT5ForConditionalGenerationT5EncoderModelT5ForQuestionAnsweringnormal_sharedhasattrtie_word_embeddingsrrrIzeros_rGT5ForTokenClassificationrrrrrCrKrLrJrYrZr[rerkrmrrrsrtrurhrw)r(modulefactorrIrlrnr-r-r. _init_weights/st                    zT5PreTrainedModel._init_weightscCsx|jj}|jj}|durtd||j}|dddf|dddf<||d<|dur2td||dk||S)Nzself.model.config.decoder_start_token_id has to be defined. In T5 it is usually set to the pad_token_id. See T5 docs for more information..r0r).rz1self.model.config.pad_token_id has to be defined.)rDdecoder_start_token_id pad_token_id ValueError new_zerosrclone masked_fill_)r(rr r shifted_input_idsr-r-r. _shift_rightfs  zT5PreTrainedModel._shift_rightN)r=r>r?r__annotations__base_model_prefixsupports_gradient_checkpointing_can_compile_fullgraph_no_split_modules_keep_in_fp32_modulespropertyrr$no_gradrrr-r-r-r.rs    6rcsDeZdZfddZddZ           dddZZS) T5Stackcsxttjj|_j|_tfddt j D|_ t jj d|_tj|_|d|_dS)Ncs"g|] }tt|dk|dqS)rr)rbool).0irDr-r. s"z$T5Stack.__init__..r_F)r!r"rrv vocab_sizerI embed_tokensrgrrange num_layersblockrrbfinal_layer_normrMrNrO post_initrxrRr+rr.r"}s  zT5Stack.__init__cCs ||_dSrS)r!r(new_embeddingsr-r-r.set_input_embeddings zT5Stack.set_input_embeddingsNc Ksh|dur|n|jj}|dur|n|jj}| dur| n|jj} | dur$| n|jj} |durB|durB|jr5dnd} td| d| d|durS|}|d|d}n|dur`|dd}n|jrednd} td| d| d |j r|j r|rt d d }|dur|j durtd | |}|\}}|d ur|jstd|d|jr|r|dur|jjrtt|jdt|jd}n t|jd}n|jsd}|dur|nd}| durtj||||jd} |jjrt|j||| t|tr|jn|d}nt|j||d}d}|jr|durt|j|||d}| rdnd}|r%dnd}|r0|jr0dnd}d}d}||}|jD]I}| rH||f}||||||||||| | d }|d}|d}|jrq|durq||rndnd}|r||df}|jr||df}q>||}||}| r||f}| stdd|||||fDSt|||||dS)Ndecoder_zYou cannot specify both zinput_ids and zinputs_embeds at the same timer0zYou have to specify either z input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fz#sz"T5Stack.forward..)last_hidden_staterr: attentionscross_attentions)rDrroutput_hidden_statesuse_return_dictrgr sizerrxrrprqr!is_encoder_decoderr r get_seq_lengthr$rrrrTrr rOr$r%tupler)r(rrrrr-rrrr3rrkwargserr_msg_prefix input_shaperrpast_key_values_lengthencoder_extended_attention_maskall_hidden_statesall_attentionsall_cross_attentionsrrr: layer_module layer_outputsr-r-r.r<s              zT5Stack.forward) NNNNNNNNNNN)r=r>r?r"r)r<r@r-r-r+r.r|s rcseZdZdgZdddZdeffdd ZddZd d Ze dd e j d Bd e j d Bde j d Bde j d Bdeee j d Bded Bde jd Bde jd Bded Bded Bded Bded Bde j d Bdee j eBfddZZS)rFdecoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weight shared.weightencoder.embed_tokens.weightdecoder.embed_tokens.weightrDcslt|t|j|j|_t|}d|_ d|_ t ||_ t|}d|_ |j |_t ||_|dSNFT)r!r"rrvr rIrcopydeepcopyrgrrencodernum_decoder_layersr#decoderr&r(rDencoder_configdecoder_configr+r-r.r"As      zT5Model.__init__cC|jSrSrr(r-r-r.get_input_embeddingsRzT5Model.get_input_embeddingscC"||_|j||j|dSrSrrKr)rMr'r-r-r.r)U zT5Model.set_input_embeddingsNrrrrencoder_outputsrr-decoder_inputs_embedsrrr3rrrc Ks| dur| n|jj} | dur| n|jj} |dur$|j|||| | | d}n$| rHt|tsHt|dt|dkr9|dndt|dkrD|dndd}|d}|j||||||| | | | | d }| sb||St|j |j |j |j |j |j |j |j dS) aV input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Example: ```python >>> from transformers import AutoTokenizer, T5Model >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") >>> model = T5Model.from_pretrained("google-t5/t5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> # preprocess: Prepend decoder_input_ids with start token which is pad token for T5Model. >>> # This is not needed for torch's T5ForConditionalGeneration as it does this internally using labels arg. >>> decoder_input_ids = model._shift_right(decoder_input_ids) >>> # forward pass >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state ```Nrrr-rr3rrrr/r0r:r1 rrr-rrrrrr3rr)r0rdecoder_hidden_statesdecoder_attentionsr2encoder_last_hidden_staterencoder_attentions)rDrr4rKrTrlenrMrr0rr:r1r2)r(rrrrrYrr-rZrrr3rrr9r:decoder_outputsr-r-r.r<ZsVCzT5Model.forward NNNNNNNNNNNNN)r=r>r?"_keys_to_ignore_on_load_unexpected_tied_weights_keysrr"rTr)rr$ LongTensor FloatTensor BoolTensorr8r rUrrr<r@r-r-r+r.r7sh      rz: T5 Model with a `language modeling` head on top. ) custom_introc!seZdZdgZddddZdeffdd ZddZd d Ze dd e j d Bd e j d Bde j d Bde j d Bdeee jd Bded Bde j d Bde j d Bde j d Bded Bded Bded Bded Bde j d Bdee j eBfddZde jfddZZS) rrCrD)zlm_head.weightrFrGrDcst||j|_t|j|j|_t |}d|_ d|_ t ||_ t |}d|_ |j|_t ||_tj|j|jdd|_|dS)NFTrF)r!r"rI model_dimrrvr rrIrJrgrrrKrLr#rMrHrr&rNr+r-r.r"s      z#T5ForConditionalGeneration.__init__cCrQrSrRrSr-r-r.rTrUz/T5ForConditionalGeneration.get_input_embeddingscCrVrSrWr'r-r-r.r)rXz/T5ForConditionalGeneration.set_input_embeddingsNrrrrrYrr-rZlabelsrrr3rrrc Ks| dur| n|jj} | dur| n|jj} |dur$|j|||| | | d}n$| rHt|tsHt|dt|dkr9|dndt|dkrD|dndd}|d}| dur]|dur]|dur]|| }|j||||||| | | | |d }|d}|jj r|||j d}| |}d}| durt d d }| |j} ||d |d | d }| s|f|dd|}|dur|f|S|St|||j|j|j|j|j|j|jd S) a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoTokenizer, T5ForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") >>> model = T5ForConditionalGeneration.from_pretrained("google-t5/t5-small") >>> # training >>> input_ids = tokenizer("The walks in park", return_tensors="pt").input_ids >>> labels = tokenizer(" cute dog the ", return_tensors="pt").input_ids >>> outputs = model(input_ids=input_ids, labels=labels) >>> loss = outputs.loss >>> logits = outputs.logits >>> # inference >>> input_ids = tokenizer( ... "summarize: studies have shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model.generate(input_ids) >>> print(tokenizer.decode(outputs[0], skip_special_tokens=True)) >>> # studies have shown that owning a dog is good for you. ```Nr[rrr/r\r]rr  ignore_indexr0 losslogitsrr^r_r2r`rra)rDrr4rKrTrrbrrMscale_decoder_outputsrkrrr2rrr5rrr:r1r2r0)r(rrrrrYrr-rZrlrrr3rrr9r:rcsequence_output lm_logitsrploss_fctoutputr-r-r.r<spJ    z"T5ForConditionalGeneration.forwardcCs ||SrS)r)r(rlr-r-r.%prepare_decoder_input_ids_from_labelsr*z@T5ForConditionalGeneration.prepare_decoder_input_ids_from_labels)NNNNNNNNNNNNNN)r=r>r?rerfrr"rTr)rr$rgrhrir8rUr rrr<rwr@r-r-r+r.rst       rcseZdZddiZdgZdeffdd ZddZd d Ze dd e j d Bd e j d Bde j d Bde d Bde d Bde d Bdee j eBfddZZS)rrFrDrMrDcsDt|t|j|j|_|}d|_d|_t ||_ | dS)NF) r!r"rrvr rIrrr6rrKr&)r(rDrOr+r-r.r"s   zT5EncoderModel.__init__cCrQrSrRrSr-r-r.rTrUz#T5EncoderModel.get_input_embeddingscCs||_|j|dSrS)rrKr)r'r-r-r.r)sz#T5EncoderModel.set_input_embeddingsNrrr-rr3rrc Ks.|dur|n|jj}|j||||||d}|S)aI input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). Example: ```python >>> from transformers import AutoTokenizer, T5EncoderModel >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") >>> model = T5EncoderModel.from_pretrained("google-t5/t5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model(input_ids=input_ids) >>> last_hidden_states = outputs.last_hidden_state ```Nr[)rDr4rK) r(rrr-rr3rr9rYr-r-r.r<s" zT5EncoderModel.forward)NNNNNN)r=r>r?rfrerr"rTr)rr$rgrhrr8rr<r@r-r-r+r.rs8  rz T5 model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cseZdZdgZdeffdd Ze            ddejdBdej dBdejdBd ejdBd e ej dBd ej dBd ej dBd ejdBde dBde dBde dBde dBde eBfddZZS)T5ForSequenceClassificationrCrDcs,t|t||_t||_|dSrS)r!r"rrrclassification_headr&rRr+r-r.r"s    z$T5ForSequenceClassification.__init__NrrrrrYr-rZrlrrr3rrc  Ks`| dur| n|jj} |durd} |dur!|dur!td|jj|dur6|dur6|dur1td||}|j|||||||| | | | d }|d}||jj  |j }t t |ddkd|j\}}}||ddf|d |ddd ddf}||}d}|dur| |j }|jjdur|jjdkrd |j_n|jjdkr|jt jks|jt jkrd |j_nd |j_|jjd krt}|jjdkr|||}n-|||}n'|jjd krt}||d |jj|d }n|jjd krt}|||}| s|f|dd}|dur|f|S|St|||j|j|j |j!|j"|j#|j$d S)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for If no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.) rrrrYr-rZrrr3rrrz7All examples must have the same number of tokens.r0 regressionsingle_label_classificationmulti_label_classificationro)%rDr4NotImplementedErrorr,r=r rreq eos_token_idr2rrr$unique_consecutivesumnumelrrry problem_typerr7r|rrsqueezerrrrr^r_r2r`rra)r(rrrrrYr-rZrlrrr3rr9rrseos_maskr_r)sentence_representationrqrprurvr-r-r.r<s0    ,      $      z#T5ForSequenceClassification.forward) NNNNNNNNNNNN)r=r>r?rerr"rr$rgrUlistrhrr8rr<r@r-r-r+r.rxsV      rxcseZdZdeffdd Ze       ddejdBdejdBdejdBdejdBd edBd edBd edBd e eje Bfd dZ Z S)rrDcsJt||j|_t||_t|j|_t |j |j|_ | dSrS) r!r"rrrrrMrrOrHr)rr&rRr+r-r.r"ss   z!T5ForTokenClassification.__init__Nrrr-rlrr3rrcKs|dur|n|jj}|j||||||d} | d} || } || } d} |dur9t} | | d|j|d} |sN| | ddf}| durL| f|S|St| | | j | j dS)a< input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. N)rr-rr3rrr0r/)rprqr:r1) rDr4rrOrrrrrr:r1)r(rrr-rlrr3rr9rr:rqrprurvr-r-r.r<~s2  z T5ForTokenClassification.forward)NNNNNNN) r=r>r?rr"rr$rUrr8rr<r@r-r-r+r.rqs6  rcseZdZdgZdddZdeffdd ZddZd d Ze dd e j d Bd e j d Bde j d Bde j d Bdeee jd Bde j d Bde j d Bde j d Bde j d Bded Bded Bded Bded Bdee j eBfddZZS)rrCrDrErDcst||j|_t|j|j|_t |}d|_ d|_ t ||_ t |}d|_ |j|_t ||_|j|_t|j|j|_|dSrH)r!r"rIrkrrvr rrIrJrgrrrKrLr#rMrrHr)rr&rNr+r-r.r"s      zT5ForQuestionAnswering.__init__cCrQrSrRrSr-r-r.rTrUz+T5ForQuestionAnswering.get_input_embeddingscCrVrSrWr'r-r-r.r)rXz+T5ForQuestionAnswering.set_input_embeddingsNrrrrrYstart_positions end_positionsr-rZrrr3rrc Ksd| dur| n|jj} | dur| n|jj} |dur|durd} |dur3| dur3|dur.td||}| dur9| n|jj} | durC| n|jj} |durW|j|||| | | d}n$| r{t|ts{t|dt|dkrl|dndt|dkrw|dndd}|d}|j ||| d||| | | | d }|d}| |}|j dd d \}}| d  }| d  }d}|dur|durt|dkr| d |j}t|dkr| d |j}|d}|d|}|d|}t|d }|||}|||}||d}| s||f|dd|}|dur|f|S|St||||j|j|j|j|j|j|jd S)az input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. NFrzr[rrr/r\) rrr-rrrrrr3rr0rrm) rp start_logits end_logitsrr^r_r2r`rra)rDr4rr rrKrTrrbrMrsplitrrr5r2rrrrrr:r1r2r0)r(rrrrrYrrr-rZrrr3rr9r:rcrsrqrr total_loss ignored_indexru start_lossend_lossrvr-r-r.r<s.         zT5ForQuestionAnswering.forwardrd)r=r>r?rerfrr"rTr)rr$rgrhrir8rUrrr<r@r-r-r+r.rsf      r)rrrrrrxr)GrrIrr$rtorch.nnrrrr,rr activationsr cache_utilsr r r generationr masking_utilsr rmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrutilsrrrrrconfiguration_t5r get_loggerr=rpModulerapex.normalizationrAinfo ImportError ExceptionwarningrCrYr^rerrrrrrrrrrxrr__all__r-r-r-r.sz      $      M"$]a<>GF 3