o ߥi T@sddlZddlZddlmZmZmZddlZddlmZddlm Z ddl m Z m Z ddl mZddlmZddlmZmZmZdd lmZdd lmZd d lmZmZd d lmZeZdZej ej!ej"dGdddeZ#dS)N)OptionalTupleUnion)nn)CrossEntropyLoss)assert_device_mapget_device_map)Models)MODELS)AttentionBackboneModelOutputSeq2SeqLMOutputTokenGeneratorOutput)Tasks) get_logger)T5PreTrainedModelT5Stack)T5Configa_ The input argument `head_mask` was split into two arguments `head_mask` and `decoder_head_mask`. Currently, `decoder_head_mask` is set to copy `head_mask`, but this feature is deprecated and will be removed in future versions. If you do not want to use any `decoder_head_mask` now, please set `decoder_head_mask = torch.ones(num_layers, num_heads)`. ) group_key module_namec%seZdZgdZdgZd2deffdd Zd2ddZd d Zd d Z d dZ ddZ ddZ ddZ ddZ                d3deejdeejdeejdeejdeejdeejdeejdeeeejdeeeejd eejd!eejd"eejd#eed$eed%eed&eed'eeejeff"d(d)Z       d4d*d+Zd"ejfd,d-Zfd.d/Zd0d1ZZS)5T5ForConditionalGeneration)zencoder\.embed_tokens\.weightzdecoder\.embed_tokens\.weightzlm_head\.weightzMdecoder\.block\.0\.layer\.1\.EncDecAttention\.relative_attention_bias\.weightNconfigc st||j|_t|j|j|_t |}d|_ d|_ d|_ t ||j|_t |}d|_ d|_ |j|_t ||j|_tj|j|jdd|_|d|_|dkrZ|dSdS)NFT)biasauto)super__init__d_model model_dimr Embedding vocab_sizesharedcopydeepcopy is_decoder use_cacheis_encoder_decoderrencodernum_decoder_layers num_layersdecoderLinearlm_head post_initmodel_parallel parallelize)selfr device_mapkwargsencoder_configdecoder_config __class__a/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/nlp/T5/text2text_generation.pyr<s&    z#T5ForConditionalGeneration.__init__cCsv|durtt|jjttjn||_t |jt|jj|j |j|j |j|j |j j|_ d|_dS)NT)rlenr&blockrangetorchcuda device_countr0rr.r)r+to first_devicer-)r/r0r6r6r7r.Xs z&T5ForConditionalGeneration.parallelizecCsX|j|j|jd|_|jd|_|jd|_d|_d|_tj dS)NcpuF) r& deparallelizer)r>r+r-r0r;r< empty_cacher/r6r6r7rAcs  z(T5ForConditionalGeneration.deparallelizecC|jSN)r rCr6r6r7get_input_embeddingsmz/T5ForConditionalGeneration.get_input_embeddingscCs"||_|j||j|dSrE)r r&set_input_embeddingsr)r/new_embeddingsr6r6r7rHps z/T5ForConditionalGeneration.set_input_embeddingscCs ||_dSrEr+rIr6r6r7set_output_embeddingsu z0T5ForConditionalGeneration.set_output_embeddingscCrDrErKrCr6r6r7get_output_embeddingsxrGz0T5ForConditionalGeneration.get_output_embeddingscCrDrE)r&rCr6r6r7 get_encoder{rGz&T5ForConditionalGeneration.get_encodercCrDrE)r)rCr6r6r7 get_decoder~rGz&T5ForConditionalGeneration.get_decoder input_idsattention_maskdecoder_input_idsdecoder_attention_mask head_maskdecoder_head_maskcross_attn_head_maskencoder_outputspast_key_values inputs_embedsdecoder_inputs_embedslabelsr$output_attentionsoutput_hidden_states return_dictreturncKs|| dur| n|jj} |dur|n|jj}|dur,|dur,|jj|jjkr,ttt|}|dur=|j ||| ||||d}n$|rat |t sat |dt |dkrR|dndt |dkr]|dndd}|d}|j rptj|jj| dur|dur| dur|| }|j rtj|jj||jj}|dur||jj}|dur||jj}|dur||jj}|j||| | ||||| |||d }|d}|j rtj|j j|j|j j|_||jjj}|jjr||jd}||}d}| durtd d }||d |d | d }|s*|f|dd|}|dur(|f|S|St|||j|j|j |j!|j"|j|j d S) aH Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`T5Tokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). attention_mask (`torch.FloatTensor` of shape `(batch_size,sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`T5Tokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the encoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*): Tuple consists of (`last_hidden_state`, `optional`: *hidden_states*, `optional`: *attentions*) `last_hidden_state` of shape `(batch_size, sequence_length, hidden_size)` is a sequence of hidden states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. decoder_inputs_embeds (`torch.FloatTensor` of shape `(batch_size, target_sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `decoder_input_ids` you can choose to directly pass an embedded representation. If `past_key_values` is used, optionally only the last `decoder_inputs_embeds` have to be input (see `past_key_values`). This is useful if you want more control over how to convert `decoder_input_ids` indices into associated vectors than the model's internal embedding lookup matrix. If `decoder_input_ids` and `decoder_inputs_embeds` are both unset, `decoder_inputs_embeds` takes the value of `inputs_embeds`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Returns: Examples: >>> from transformers import T5Tokenizer, T5ForConditionalGeneration >>> tokenizer = T5Tokenizer.from_pretrained("t5-small") >>> model = T5ForConditionalGeneration.from_pretrained("t5-small") >>> # training >>> input_ids = tokenizer("The walks in park", return_tensors="pt").input_ids >>> labels = tokenizer(" cute dog the ", return_tensors="pt").input_ids >>> outputs = model(input_ids=input_ids, labels=labels) >>> loss = outputs.loss >>> logits = outputs.logits >>> # inference >>> input_ids = tokenizer( ... "summarize: studies have shown that owning a dog is good for you", return_tensors="pt" >>> ).input_ids # Batch size 1 >>> outputs = model.generate(input_ids) >>> print(tokenizer.decode(outputs[0], skip_special_tokens=True)) >>> # studies have shown that owning a dog is good for you. N)rQrRrZrUr]r^r_rr)last_hidden_state hidden_states attentions) rQrRrZrYencoder_hidden_statesencoder_attention_maskrUrWr$r]r^r_gi) ignore_index) losslogitsrYdecoder_hidden_statesdecoder_attentionscross_attentionsencoder_last_hidden_statereencoder_attentions)#rr$use_return_dictr(r'warningswarn2_T5ForConditionalGeneration__HEAD_MASK_WARNING_MSG FutureWarningr& isinstancer r8r-r;r< set_devicer)r? _shift_rightr>r+weightdevicetie_word_embeddingsrrviewsizer rYrcrdrmrb)r/rQrRrSrTrUrVrWrXrYrZr[r\r$r]r^r_r1rcdecoder_outputssequence_output lm_logitsriloss_fctoutputr6r6r7forwards!        z"T5ForConditionalGeneration.forwardc Ks2|dur|ddddf}||||||||dS)Nrh)rSrYrXrRrUrVrWr$r6) r/rQpastrRrUrVrWr$rXr1r6r6r7prepare_inputs_for_generations z8T5ForConditionalGeneration.prepare_inputs_for_generationcCs ||SrE)rw)r/r\r6r6r7%prepare_decoder_input_ids_from_labelsrMz@T5ForConditionalGeneration.prepare_decoder_input_ids_from_labelscs4tj|i|}tt|tjr|dS|ddS)Nr) sequences)rgenerater rur;Tensor)r/argsr1rr4r6r7rs z#T5ForConditionalGeneration.generatec Cs|dur td|Sd}|D]1}d}|D]}||d||jf}q|dj|djks1Jt|t|ks;J||f}q|S)NzHYou might want to consider setting `use_cache=True` to speed up decodingr6r)loggerwarning index_selectr>ryshaper8)r/rbeam_idxreordered_decoder_pastlayer_past_statesreordered_layer_past_stateslayer_past_stater6r6r7_reorder_caches0  z)T5ForConditionalGeneration._reorder_cacherE)NNNNNNNNNNNNNNNN)NNNNNNN)__name__ __module__ __qualname___keys_to_ignore_on_load_missing"_keys_to_ignore_on_load_unexpectedrrr.rArFrHrLrNrOrPrr; LongTensor FloatTensor BoolTensorrrboolrr rrrrr __classcell__r6r6r4r7r.s            r)$r!rqtypingrrrr;rtorch.nnr'transformers.utils.model_parallel_utilsrrmodelscope.metainfor modelscope.models.builderr modelscope.outputsr r r modelscope.utils.constantrmodelscope.utils.loggerrbackbonerr configurationrr__HEAD_MASK_WARNING_MSGregister_moduletext2text_generationT5rr6r6r6r7s*