o ei@sdZddlmZddlZddlmZddlmZmZmZddl m Z ddl m Z m Z dd lmZdd lmZdd lmZdd lmZdd lmZmZmZmZddlmZmZddlmZddl m!Z!m"Z"m#Z#m$Z$ddl%m&Z&e$'e(Z)Gdddej*Z+ d2dej,dej-dej-dej-dej-dBde.de.fddZ/Gdd d ej,Z0Gd!d"d"eZ1e"Gd#d$d$eZ2Gd%d&d&e2Z3e"Gd'd(d(e2Z4Gd)d*d*e2eZ5e"d+d,Gd-d.d.e2Z6e"Gd/d0d0e2Z7gd1Z8dS)3zPyTorch OPT model.)CallableN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache DynamicCache)GenerationMixin)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPastQuestionAnsweringModelOutput SequenceClassifierOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging) OPTConfigcsReZdZdZdedeffdd Z  d dejd ed ejdBffd d ZZ S)OPTLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcsd|_t||j|dSN)offsetsuper__init__)selfrr __class__b/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.pyr#0sz&OPTLearnedPositionalEmbedding.__init__rNattention_maskpast_key_values_length position_idscsL|durtj|dd}||d}|dd|df}t||jS)z3`input_ids_shape` is expected to be [bsz x seqlen].Nrdim)torchcumsumlongr"forwardr!)r$r)r*r+r%r'r(r16s z%OPTLearnedPositionalEmbedding.forward)rN) __name__ __module__ __qualname____doc__intr#r. LongTensorr1 __classcell__r'r'r%r(r+s rmodulequerykeyvaluer)scalingdropoutc Ks|t||dd|}|dur||}tjj|dtjd|j}tjj |||j d}t||} | dd } | |fS)N)r-dtypeptrainingrr ) r.matmul transposer functionalsoftmaxfloat32torBr?rE contiguous) r:r;r<r=r)r>r?kwargs attn_weights attn_outputr'r'r(eager_attention_forwardHs  rPcseZdZdZ ddededBffdd Z    ddejd e dBd ejdBd e d ejdBd e ejejdBe dBff ddZ Z S) OPTAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNconfig layer_idxc st||_|j|_|j|_|j|_|j |_ ||_ |dur*t d|j jd|j|j|_d|_|j|j|jkrJtd|jd|jd|jd|_tj|j|j|j d|_tj|j|j|j d|_tj|j|j|j d|_tj|j|j|j d|_dS) NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.Tz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩bias)r"r#rR hidden_size embed_dimnum_attention_heads num_headsattention_dropoutr? enable_biasrSlogger warning_oncer&r2head_dim is_causal ValueErrorr>rLineark_projv_projq_projout_proj)r$rRrSrMr%r'r(r#bs0   zOPTAttention.__init__F hidden_statespast_key_valuesr)output_attentionscache_positionreturncKs|\}}} |||j} | |d|j|jdd} ||} ||} | |d|j|jdd} | |d|j|jdd} |durT| | | |j d|i\} } t |j jt} | || | | |f|jshdn|jdd|\}}|||d}||}|sd}||fS) z#Input shape: Batch x Time x Channelr@rr Nrir9g?)r?r>)sizerdr>viewrYr^rGrbrcupdaterSr get_interfacerR_attn_implementationrPrEr?reshaperLre)r$rfrgr)rhrirMbsztgt_len_ query_states key_states value_statesattention_interfacerOrNr'r'r(r1s>     zOPTAttention.forwardN)NNFN)r2r3r4r5rr6r#r.Tensorr booltupler1r8r'r'r%r(rQ_s2&rQcseZdZddededBffdd Z      ddejdejdBd edBd e dBd e dBd ej dBd ejdBde e de eje ejejfdBffddZZS)OPTDecoderLayerNrRrScst|j|_t||d|_|j|_|j|_t|j |_ t j |j|j d|_t j|j|j|jd|_t j|j|j|jd|_t j |j|j d|_dS)N)rRrSelementwise_affinerT)r"r#rVrWrQ self_attndo_layer_norm_beforer?ractivation_function activation_fnr LayerNormlayer_norm_elementwise_affineself_attn_layer_normraffn_dimr[fc1fc2final_layer_norm)r$rRrSr%r'r(r#s  zOPTDecoderLayer.__init__Frfr)rgrh use_cacher+rirMrjc Ks|} |jr ||}|jd||||||d|\}} tjj||j|jd}| |}|js2||}|j} |d| d}|} |jrH| |}| |}| |}| |}tjj||j|jd}| || }|jsq| |}|f} |r{| | f7} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_values (`Cache`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence.. )rfrgr+r)rhrirCr@Nr')rrrrrHr?rEshaperprkrrrrrl) r$rfr)rgrhrr+rirMresidualself_attn_weightshidden_states_shapeoutputsr'r'r(r1sB          zOPTDecoderLayer.forwardrx)NNFFNN)r2r3r4rr6r#r.ryr rzr7rr r{ FloatTensorr1r8r'r'r%r(r|s6  r|c@s8eZdZUeed<dZdZdgZdZdZ dZ dZ dZ dS)OPTPreTrainedModelrRmodelTr|N) r2r3r4r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraphr'r'r'r(rs rcseZdZdZdeffdd Ze          ddejdBdej dBde dBd ej dBd e dBd e dBd e dBd e dBdejdBdej dBde edeeBfddZZS) OPTDecoderz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`OPTDecoderLayer`] Args: config: OPTConfig rRcstj|_j|_j|_j|_j|_t jj |j|_ t jj|_j jkr?t jjj dd|_nd|_j jkrTt jj jdd|_nd|_jrhjsht jjjd|_nd|_t fddtjD|_d|_|dS)NFrTr}csg|]}t|dqS))rS)r|).0irRr'r( Qsz'OPTDecoder.__init__..)r"r#r? layerdrop pad_token_id padding_idxmax_position_embeddingsmax_target_positions vocab_sizer Embeddingword_embed_proj_dim embed_tokensrrVembed_positionsra project_out project_inr_remove_final_layer_normrrr ModuleListrangenum_hidden_layerslayersgradient_checkpointing post_initr$rRr%rr(r#2s,       zOPTDecoder.__init__N input_idsr)rg inputs_embedsrrhoutput_hidden_states return_dictr+rirMrjc  Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur$|n|jj}|du|duAr4td|jrC|jrC|rCt dd}|durP| d|j d}|durY| |}|re|duret |jd}|durm|nd} | durtj| | |j d|jd } |dur| |j d} tj|j d| |jd }| durtj|dd } | |d} | dd| df} t|j||| |d }|j|| | d }|jdur||}|||j}|rd nd}|rd nd}t|jD]8\}}|r||f7}|jrtg}||jkrq||f|| |||| d| }|d}|r"||df7}q|jdur.||}|jdur9||}|rA||f7}t||||dS)aU Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. for padding use -1. [What are position IDs?](../glossary#position-ids) cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length. Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.Fr@rrrdevicer,)rRrr)rirg)r+r')r)r+rgrhrrilast_hidden_statergrf attentions) rRrhrruse_return_dictr`rrEr\r]rlrrr get_seq_lengthr.arangeronesr/r0r rrrK enumeraterrandrrrr)r$rr)rgrrrhrrr+rirMpast_seen_tokens seq_length causal_mask pos_embedsrfall_hidden_statesall_self_attnsidx decoder_layerdropout_probability layer_outputsr'r'r(r1Ws@                zOPTDecoder.forward NNNNNNNNNN)r2r3r4r5rr#rr.r7ryr rrzrr r{rr1r8r'r'r%r(r*sN%     rcseZdZdeffdd ZddZddZee          dd e j dBd e j dBd e dBd e j dBd edBdedBdedBdedBde j dBde j dBdeedeeBfddZZS)OPTModelrRcs"t|t||_|dSrx)r"r#rdecoderrrr%r'r(r#s   zOPTModel.__init__cCs|jjSrxrrr$r'r'r(get_input_embeddings szOPTModel.get_input_embeddingscCs ||j_dSrxrr$r=r'r'r(set_input_embeddings s zOPTModel.set_input_embeddingsNrr)rgrrrhrrr+rirMrjc Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur$|n|jj}|jd||| |||||d| d | } t| j| j| j | j dS)NT rr)r+rgrrrhrrrirr') rRrhrrrrrrrgrfr) r$rr)rgrrrhrrr+rirMdecoder_outputsr'r'r(r1s2 zOPTModel.forwardr)r2r3r4rr#rrrrr.r7ryr rrzrr r{rr1r8r'r'r%r(rsR     rc seZdZddiZfddZddZddZee dd e j d Bd e j d Bd e d Bde j d Bde j d Bded Bded Bded Bded Bde j d Bde j d Bdee j BdeedeeBfddZZS)OPTForCausalLMzlm_head.weightz!model.decoder.embed_tokens.weightcs8t|t||_tj|j|jdd|_| dSNFrT) r"r#rrrrarrlm_headrrr%r'r(r#@s   zOPTForCausalLM.__init__cC |jjjSrxrrrrr'r'r(rJ z#OPTForCausalLM.get_input_embeddingscC||jj_dSrxrrr'r'r(rMz#OPTForCausalLM.set_input_embeddingsNrrr)rgrlabelsrrhrrr+rilogits_to_keeprMrjc  Ks|dur|n|jj}|dur|n|jj}| dur| n|jj} |jjd||| |||||d| d | }|j}t| tr@t | dn| }| |dd|ddf }d}|durf|j d|||jj d| }t|||j|j|jdS)an labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, OPTForCausalLM >>> model = OPTForCausalLM.from_pretrained("facebook/opt-350m") >>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious. I'm just a little bit of a weirdo." ```NTr)logitsrrlossrrgrfrr')rRrhrrrrr isinstancer6slicerrL loss_functionrrrgrfr)r$rr)rgrrrrhrrr+rirrMrrf slice_indicesrrr'r'r(r1Ps>)  zOPTForCausalLM.forward) NNNNNNNNNNNr)r2r3r4_tied_weights_keysr#rrrrr.r7ryr rrzr6rrr{rr1r8r'r'r%r(r=s`       ra The OPT Model transformer with a sequence classification head on top (linear layer). [`OPTForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ) custom_introcseZdZdeffdd Ze          ddejdBdejdBde dBdejdBd ejdBd e dBd e dBd e dBd e dBdejdBde e BfddZ ddZddZZS)OPTForSequenceClassificationrRcs@t||j|_t||_tj|j|jdd|_| dSr) r"r# num_labelsrrrrarscorerrr%r'r(r#s   z%OPTForSequenceClassification.__init__Nrr)rgrrrrhrrr+rjc  KsF| dur| n|jj} |j|||| ||||| d } | d} || }|dur/|jdd\}}n |jdd\}}|jjdurF|dkrFtd|jjdurOd}n1|durt||jjk|jt j }t j |jd|jt j d}|| d}n d}t |jjd |t j ||jd |f}d}|dur|jjdur|jdkrd |j_n|jdkr|jt jks|jt jkrd |j_nd |j_|jjd krt}|jdkr|||}n+|||}n%|jjd krt}||d|j|d}n|jjd krt}|||}| s|f| dd}|dur|f|S|St||| j| j| jdS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrgr)r+rrrhrrrr rz=Cannot handle batch sizes > 1 if no padding token is defined.r@)rrBz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r regressionsingle_label_classificationmulti_label_classificationr)rRrrrrrr`rKrr.int32rargmaxr\r]r&r2 problem_typerrBr0r6rsqueezerrlrrrgrfr)r$rr)rgrrrrhrrr+rMtransformer_outputsrfr batch_sizesequence_lengthlast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctoutputr'r'r(r1sv      "       z$OPTForSequenceClassification.forwardcCrrxrrr'r'r(rrz1OPTForSequenceClassification.get_input_embeddingscCrrxrrr'r'r(rrz1OPTForSequenceClassification.set_input_embeddingsr)r2r3r4rr#rr.r7rr rzr{rr1rrr8r'r'r%r(rsL     ]rcseZdZdeffdd Ze           ddejdBdejdBde dBdejdBd ejdBd ejdBd e dBd e dBd e dBde dBdejdBde e BfddZ ddZddZZS)OPTForQuestionAnsweringrRcs2t|t||_t|jd|_|dSr) r"r#rrrrar qa_outputsrrr%r'r(r#s   z OPTForQuestionAnswering.__init__Nrr)rgrstart_positions end_positionsrrhrrr+rjc  KsX| dur| n|jj} |j|||| |||| | d } | d}||}|jddd\}}|d}|d}d}|dur|durt|dkrO|d}t|dkr\|d}|d}| d| |j }| d| |j }t |d}|||}|||}||d}| s||f| dd}|dur|f|S|St |||| j| jd S) a Example: ```python >>> from transformers import AutoTokenizer, OPTForQuestionAnswering >>> import torch >>> torch.manual_seed(4) # doctest: +IGNORE_RESULT >>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m") >>> # note: we are loading a OPTForQuestionAnswering from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random >>> model = OPTForQuestionAnswering.from_pretrained("facebook/opt-350m") >>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet" >>> inputs = tokenizer(question, text, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> answer_offset = len(tokenizer(question)[0]) >>> predict_answer_tokens = inputs.input_ids[ ... 0, answer_offset + answer_start_index : answer_offset + answer_end_index + 1 ... ] >>> predicted = tokenizer.decode(predict_answer_tokens) >>> predicted ' a nice puppet' ```Nrrrr@r,) ignore_indexr )r start_logits end_logitsrfr)rRrrrsplitrrLlenrkclamprKrrrrfr)r$rr)rgrrrrrhrrr+rMrrfrrr total_loss ignored_indexr start_lossend_lossrr'r'r(r1&sP0        zOPTForQuestionAnswering.forwardcCrrxrrr'r'r(rrz,OPTForQuestionAnswering.get_input_embeddingscCrrxrrr'r'r(rrz,OPTForQuestionAnswering.set_input_embeddings) NNNNNNNNNNN)r2r3r4rr#rr.r7rr rzr{rr1rrr8r'r'r%r(rsR     `r)rrrrr)r9)9r5collections.abcrr.rtorch.nnrrr activationsr cache_utilsr r generationr masking_utilsr modeling_flash_attention_utilsr modeling_layersrmodeling_outputsrrrrmodeling_utilsrrprocessing_utilsrutilsrrrrconfiguration_optr get_loggerr2r\rrModuleryfloatrPrQr|rrrrrr__all__r'r'r'r(sd          $ ]a X;bo q