o iǼ@s dZddlmZmZmZddlZddlmZddlmZm Z m Z ddl m Z ddl mZmZdd lmZdd lmZdd lmZdd lmZdd lmZmZmZmZddlmZmZddl m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)ddl*m+Z+e&rddl,m-Z-ddl.m/Z/e'0e1Z2Gdddej3Z4 d5dej5dej6dej6dej6deej6de7de7fd d!Z8Gd"d#d#ej5Z9Gd$d%d%eZ:e$Gd&d'd'eZ;Gd(d)d)e;Ze$d.d/Gd0d1d1e;Z?e$Gd2d3d3e;Z@gd4ZAdS)6zPyTorch OPT model.)CallableOptionalUnionN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache DynamicCache)GenerationMixin)AttentionMaskConverter)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPastQuestionAnsweringModelOutput SequenceClassifierOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleis_torch_flex_attn_availablelogging)deprecate_kwarg) OPTConfig) BlockMask)make_flex_block_causal_maskcsReZdZdZdedeffdd Z  d dejd ed eejffd d Z Z S)OPTLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcsd|_t||j|dSN)offsetsuper__init__)selfr#r$ __class__X/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.pyr)8sz&OPTLearnedPositionalEmbedding.__init__rNattention_maskpast_key_values_length position_idscsL|durtj|dd}||d}|dd|df}t||jS)z3`input_ids_shape` is expected to be [bsz x seqlen].Nrdim)torchcumsumlongr(forwardr')r*r/r0r1r+r-r.r7>s z%OPTLearnedPositionalEmbedding.forward)rN) __name__ __module__ __qualname____doc__intr)r4 LongTensorrr7 __classcell__r-r-r+r.r"3s r"modulequerykeyvaluer/scalingdropoutc Ks|t||dd|}|dur||}tjj|dtjd|j}tjj |||j d}t||} | dd } | |fS)N)r3dtypeptrainingrr&) r4matmul transposer functionalsoftmaxfloat32torHrErK contiguous) r@rArBrCr/rDrEkwargs attn_weights attn_outputr-r-r.eager_attention_forwardPs  rVcseZdZdZ ddedeeffdd Zeddd d     dd e j dee d ee j dee j de dee j de e j ee j ee ffddZZS) OPTAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNconfig layer_idxc st||_|j|_|j|_|j|_|j |_ ||_ |dur*t d|j jd|j|j|_d|_|j|j|jkrJtd|jd|jd|jd|_tj|j|j|j d|_tj|j|j|j d|_tj|j|j|j d|_tj|j|j|j d|_dS) NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.Tz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩bias)r(r)rX hidden_size embed_dimnum_attention_heads num_headsattention_dropoutrE enable_biasrYlogger warning_oncer,r8head_dim is_causal ValueErrorrDrLineark_projv_projq_projout_proj)r*rXrYrSr+r-r.r)js0   zOPTAttention.__init__past_key_valuepast_key_values4.58new_nameversionF hidden_statesr/layer_head_maskoutput_attentionscache_positionreturncKs|\}} } |||j} | |d|j|jdd} ||} ||} | |d|j|jdd} | |d|j|jdd} |durT| | | |j d|i\} } t }|j j dkrbt|j j }||| | | |f|jsndn|jdd |\}}||| d}||}|sd}||fS) z#Input shape: Batch x Time x ChannelrFrr&Nrueagerr??)rErD)sizerjrDviewr_rdrMrhriupdaterYrVrX_attn_implementationrrKrEreshaperRrk)r*rrrmr/rsrtrurSbsztgt_len_ query_states key_states value_statesattention_interfacerUrTr-r-r.r7s>       zOPTAttention.forwardN)NNNFN)r8r9r:r;rrr<r)rr4Tensorr booltupler7r>r-r-r+r.rWgs: # rWcseZdZddedeeffdd Zedddd      dd ej d eej d eej dee dee dee deej deej de edeejeeejejfffddZZS)OPTDecoderLayerNrXrYcst|j|_t||d|_|j|_|j|_t|j |_ t j |j|j d|_t j|j|j|jd|_t j|j|j|jd|_t j |j|j d|_dS)N)rXrYelementwise_affinerZ)r(r)r\r]rW self_attndo_layer_norm_beforerEr activation_function activation_fnr LayerNormlayer_norm_elementwise_affineself_attn_layer_normrgffn_dimrafc1fc2final_layer_norm)r*rXrYr+r-r.r)s  zOPTDecoderLayer.__init__rlrmrnroFrrr/rsrt use_cacher1rurSrvc  Ks|} |jr ||}|jd|||||||d| \}} tjj||j|jd}| |}|js3||}|j} |d| d}|} |jrI| |}| |}| |}| |}tjj||j|jd}| || }|jsr| |}|f} |r|| | f7} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`, *optional*): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_values (`Cache`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence.. )rrrmr1r/rsrtrurIrFNr-)rrrrrNrErKshaper}ryrrrrrz)r*rrr/rsrmrtrr1rurSresidualself_attn_weightshidden_states_shapeoutputsr-r-r.r7sD          zOPTDecoderLayer.forwardr)NNNFFNN)r8r9r:rrr<r)rr4rr rr=rrr FloatTensorr7r>r-r-r+r.rs>    rc@s@eZdZUeed<dZdZdgZdZdZ dZ dZ dZ ddZ dS)OPTPreTrainedModelrXmodelTrcCs|jj}t|tjr"|jjjd|d|jdur |jj dSdSt|tj rC|jjjd|d|j durA|jj|j  dSdSt|tj rX|jj d|jj dSdS)Nr?)meanstdrx)rXinit_std isinstancerrgweightdatanormal_r[zero_ Embedding padding_idxrfill_)r*r@rr-r-r. _init_weights:s     z OPTPreTrainedModel._init_weightsN)r8r9r:r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraphrr-r-r-r.r-s  rcs eZdZdZdeffdd Z d deejdfdejd ejd e d e f d d Z e dejde de dejd ejde f ddZe           d!deejdeejdeejd ee deejdee d ee dee dee deejd eejdeedeeeffddZZS)" OPTDecoderz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`OPTDecoderLayer`] Args: config: OPTConfig rXcstj|_j|_j|_j|_j|_t jj |j|_ t jj|_j jkr?t jjj dd|_nd|_j jkrTt jj jdd|_nd|_jrhjsht jjjd|_nd|_t fddtjD|_d|_|dS)NFrZrcsg|]}t|dqS))rY)r).0irXr-r. psz'OPTDecoder.__init__..)r(r)rE layerdrop pad_token_idrmax_position_embeddingsmax_target_positions vocab_sizerrword_embed_proj_dim embed_tokensr"r\embed_positionsrg project_out project_inr_remove_final_layer_normrrr ModuleListrangenum_hidden_layerslayersgradient_checkpointing post_initr*rXr+rr.r)Qs,       zOPTDecoder.__init__Fr/r input_tensorrurmrtc Cs:|jjdkr|dur|dkr|SdS|jjdkr&t|tjr$t|}|S|dur.|nd}|dur7|jnd}|jjdkrO|sO|sOt j ||||j drOdS|j }|j d} |r^|} nt|tjri|j d n|| d} |j|| | |||j dd } |jjdkr|dur|jjd vr|st|j} t | | } | S) Nflash_attention_2r?flex_attentionrFsdpa) inputs_embedsr0 is_trainingrrF)sequence_length target_lengthrHru batch_size)cudaxpunpu)rXr|anyrr4rr!get_seq_lengthis_compileabler_ignore_causal_mask_sdparKrHrget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positiondevicetypefinfomin_unmask_unattended) r*r/rrurmrtpast_seen_tokensusing_compilable_cacherHrr causal_mask min_dtyper-r-r._update_causal_maskwsT           zOPTDecoder._update_causal_maskrrrHrc KsD|dur|dkr|}|St|j}tj||f|||jd}|dkr+tj|dd}|tj||jd|ddk9}|ddddddf |ddd}|dur| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuerHrr)diagonalrrFr)r3r4rrfullrtriuaranger}expandclonerrQ masked_fill) r/rrrHrurrSrr mask_length padding_maskr-r-r.rs,  $ 6  z@OPTDecoder._prepare_4d_causal_attention_mask_with_cache_positionN input_ids head_maskrroutput_hidden_states return_dictr1rSrvc  Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}| dur$| n|jj} |du|duAr4td|jrC|jrC|rCt dd}|durP| d|j d}|durY| |}|re|duret |jd}|durm|nd} | durtj| | |j d|jd } |dur| |j d}tj|j d||jd }|||| ||}| durtj|dd } | |d} | dd| df} |j|| | d }|jdur||}|||j}|rd nd}|rd nd}t|gd gD]*\}}|dur|dt|jkrtd|dt|jd|ddqt|jD]E\}}|r'||f7}|jr8tg}||jkr8q||f|| |durF||nd|||| d| }|d}|r_||df7}q|j durl| |}|j!durw|!|}|r||f7}t"||||dS)a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(num_hidden_layers, num_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. for padding use -1. [What are position IDs?](../glossary#position-ids) cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length. Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.FrFrrrrr2)r1r-rzThe `z` should be specified for z layers, but it is for .)r/r1rsrmrtrrulast_hidden_statermrr attentions)#rXrtrruse_return_dictrfrrKrbrcrzrrr rr4rronesrr5r6rrrQziprylenr enumeraterandrrrr)r*rr/rrmrrrtrrr1rurSr seq_lengthr pos_embedsrrall_hidden_statesall_self_attns attn_mask mask_nameidx decoder_layerdropout_probability layer_outputsr-r-r.r7sG                    zOPTDecoder.forward)F NNNNNNNNNNN)r8r9r:r;rr)rr4rr rr staticmethodr<rHrrrr=rrrrrr7r>r-r-r+r.rIs,  D 6      rcseZdZdeffdd ZddZddZee           dd e e j d e e j d e e j d e e d e e jde ede ede ede ede e j de e j deedeeeffddZZS)OPTModelrXcs"t|t||_|dSr)r(r)rdecoderrrr+r-r.r)s   zOPTModel.__init__cC|jjSrrrr*r-r-r.get_input_embeddingszOPTModel.get_input_embeddingscC ||j_dSrrr*rCr-r-r.set_input_embeddings zOPTModel.set_input_embeddingsNrr/rrmrrrtrrr1rurSrvc Ks|dur|n|jj}|dur|n|jj}|dur|n|jj}| dur$| n|jj} |jd||| ||||||d| d | } t| j| j| j | j dS)NT rr/r1rrmrrrtrrrurr-) rXrtrrrrrrrmrrr)r*rr/rrmrrrtrrr1rurSdecoder_outputsr-r-r.r7s4 zOPTModel.forwardr )r8r9r:rr)rrrrrr4r=rr rrrrrrrr7r>r-r-r+r.rsX      rc!seZdZdgZfddZddZddZdd Zd d Ze e dd e e j de e jde e jde ede e jde e j de ede ede ede ede e j de e jdeedeeeffddZZS)OPTForCausalLMzlm_head.weightcs8t|t||_tj|j|jdd|_| dSNFrZ) r(r)rrrrgrrlm_headrrr+r-r.r)s   zOPTForCausalLM.__init__cC |jjjSrrrrrr-r-r.r z#OPTForCausalLM.get_input_embeddingscC||jj_dSrrrr-r-r.rz#OPTForCausalLM.set_input_embeddingscCrrrr)r*rr-r-r. set_decoderrzOPTForCausalLM.set_decodercCrrr#rr-r-r. get_decoderrzOPTForCausalLM.get_decoderNrr/rrmrlabelsrrtrrr1rurSrvc Ks|dur|n|jj}| dur| n|jj} | dur| n|jj} |jjd||| |||||| d| d | }||d}d}|durV||j }|j ||fd|jj i| }t |||j |j|jdS)an labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, OPTForCausalLM >>> model = OPTForCausalLM.from_pretrained("facebook/opt-350m") >>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious. I'm just a little bit of a weirdo." ```NTrrrlosslogitsrmrrrr-)rXrtrrrrrrRrQr loss_functionrrrmrrr)r*rr/rrmrr&rrtrrr1rurSrr)r(r-r-r.r7sL)  zOPTForCausalLM.forward NNNNNNNNNNNN)r8r9r:_tied_weights_keysr)rrr$r%rrrr4r=rr rrrrrrrr7r>r-r-r+r.rsd        ra The OPT Model transformer with a sequence classification head on top (linear layer). [`OPTForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ) custom_introcseZdZdeffdd Ze           ddeejdeej deej dee d eej d eejd ee d ee d ee dee deejde e effddZddZddZZS)OPTForSequenceClassificationrXcs@t||j|_t||_tj|j|jdd|_| dSr) r(r) num_labelsrrrrgrscorerrr+r-r.r)fs   z%OPTForSequenceClassification.__init__Nrr/rrmrr&rrtrrr1rvc  CsL| dur| n|jj} |j|||| ||||| | d } | d} || }|dur0|jdd\}}n |jdd\}}|jjdurG|dkrGtd|jjdurPd}n1|duru||jjk|jt j }t j |jd|jt j d}|| d}n d}t |jjd |t j ||jd |f}d}|dur|jjdur|jdkrd |j_n|jdkr|jt jks|jt jkrd |j_nd |j_|jjd krt}|jdkr|||}n,|||}n&|jjd krt}||d|j|d}n|jjd krt}|||}| s|f| dd}|dur|f|S|St||| j| j| jdS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N rmr/r1rrrrtrrrr&rz=Cannot handle batch sizes > 1 if no padding token is defined.rF)rrHz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r regressionsingle_label_classificationmulti_label_classificationr')rXrrr0rrrfrQrr4int32rargmaxrbrcr,r8 problem_typer/rHr6r<rsqueezerrzrrrmrrr)r*rr/rrmrr&rrtrrr1transformer_outputsrrr)rrlast_non_pad_token non_pad_mask token_indices pooled_logitsr(loss_fctoutputr-r-r.r7osx       "      z$OPTForSequenceClassification.forwardcCrrrrr-r-r.rr z1OPTForSequenceClassification.get_input_embeddingscCr!rrrr-r-r.rr"z1OPTForSequenceClassification.set_input_embeddingsr )r8r9r:rr)rrr4r=rr rrrrr7rrr>r-r-r+r.r.WsR      ^r.cseZdZdeffdd Ze            ddeejdeej deej dee d eej d eejd eejd ee d ee dee dee deejde e effddZddZddZZS)OPTForQuestionAnsweringrXcs2t|t||_t|jd|_|dSr%) r(r)rrrrgr qa_outputsrrr+r-r.r)s   z OPTForQuestionAnswering.__init__Nrr/rrmrstart_positions end_positionsrrtrrr1rvc  CsZ| dur| n|jj} |j|||| |||| | | d } | d}||}|jddd\}}|d}|d}d}|dur|durt|dkrP|d}t|dkr]|d}|d}| d| |j }| d| |j }t |d}|||}|||}||d}| s||f| dd}|dur|f|S|St |||| j| jd S) a Example: ```python >>> from transformers import AutoTokenizer, OPTForQuestionAnswering >>> import torch >>> torch.manual_seed(4) # doctest: +IGNORE_RESULT >>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m") >>> # note: we are loading a OPTForQuestionAnswering from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random >>> model = OPTForQuestionAnswering.from_pretrained("facebook/opt-350m") >>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet" >>> inputs = tokenizer(question, text, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> answer_offset = len(tokenizer(question)[0]) >>> predict_answer_tokens = inputs.input_ids[ ... 0, answer_offset + answer_start_index : answer_offset + answer_end_index + 1 ... ] >>> predicted = tokenizer.decode(predict_answer_tokens) >>> predicted ' a nice puppet' ```Nr1rrrFr2) ignore_indexr&)r( start_logits end_logitsrrr)rXrrrAsplitr8rRrryclamprQrrrrrr)r*rr/rrmrrBrCrrtrrr1r9rrr)rErF total_loss ignored_indexr> start_lossend_lossr?r-r-r.r7sR0        zOPTForQuestionAnswering.forwardcCrrrrr-r-r.rAr z,OPTForQuestionAnswering.get_input_embeddingscCr!rrrr-r-r.rDr"z,OPTForQuestionAnswering.set_input_embeddingsr+)r8r9r:rr)rrr4r=rr rrrrr7rrr>r-r-r+r.r@sX       ar@)rrrr.r@)r?)Br;typingrrrr4rtorch.nnrrr activationsr cache_utilsr r generationr modeling_attn_mask_utilsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrmodeling_utilsrrprocessing_utilsrutilsrrrrrutils.deprecationrconfiguration_optr!torch.nn.attention.flex_attentionr integrations.flex_attentionr! get_loggerr8rbrr"ModulerfloatrVrWrrrrrr.r@__all__r-r-r-r.sn            $ `fd=np r