o ۷ik@sdZddlZddlmZmZmZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZmZmZmZmZdd lmZmZdd lmZdd lmZmZdd lmZmZmZm Z dd l!m"Z"m#Z#ddl$m%Z%e &e'Z(ddZ)ddZ*ddZ+ddZ,ddZ-Gdddejj.Z/Gdddej.Z0Gdddej.Z1 dOd!ej.d"ej2d#ej2d$ej2d%eej2d&e3d'e3d(eej2d)eefd*d+Z4Gd,d-d-ej.Z5Gd.d/d/ej.Z6Gd0d1d1ej.Z7Gd2d3d3ej.Z8Gd4d5d5ej.Z9Gd6d7d7e Z:Gd8d9d9ej.Z;Gd:d;d;ej.Zd?d?e=Z>eGd@dAdAe=Z?GdBdCdCej.Z@edDdEGdFdGdGe=ZAeGdHdIdIe=ZBGdJdKdKej.ZCdLdMZDgdNZEdS)PzPyTorch ESM model.N)CallableOptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)GradientCheckpointingLayer)"BaseModelOutputWithCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentionsMaskedLMOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) find_pruneable_heads_and_indicesprune_linear_layer)TransformersKwargsauto_docstringcan_return_tuplelogging)OutputRecordercheck_model_inputs) EsmConfigcCs&|jddd\}}tj| |fddS)Ndim)chunktorchcat)xx1x2r'Z/home/ubuntu/vllm_env/lib/python3.10/site-packages/transformers/models/esm/modeling_esm.py rotate_half,sr)cCs`|ddddd|jdddf}|ddddd|jdddf}||t||S)N)shaper))r$cossinr'r'r(apply_rotary_pos_emb1s&&r.cCs |ddt|tdS)zo This is the gelu implementation from the original ESM repo. Using F.gelu yields subtly wrong results. g??g@)r"erfmathsqrtr$r'r'r(gelu8s r4cCs||ddS)zJMake layer symmetric in final two dimensions, used for contact prediction.rr*) transposer3r'r'r( symmetrize?sr6cCsH|jddd}|jddd}|jddd}||}||||}|S)z=Perform average product correct, used for contact prediction.rT)keepdimsr*)rr*)sumdiv_)r$a1a2a12avg normalizedr'r'r(average_product_correctDs r?csbeZdZUdZejed<deffdd ZdddZ d ejd ejd e ejejffd d Z Z S)RotaryEmbeddingz Rotary position embeddings based on those in [RoFormer](https://huggingface.co/docs/transformers/model_doc/roformer). Query and keys are transformed by rotation matrices which depend on their relative positions. inv_freqr csPtddtjd|dtjd|}|d|d|_d|_d|_ dS)Nr/i'rrdtyperA) super__init__r"arangeint64floatregister_buffer_seq_len_cached _cos_cached _sin_cached)selfr rA __class__r'r(rEYs $  zRotaryEmbedding.__init__rcCs|j|}||jks|jj|jkrU||_tj|j||jd|j}t||j}tj ||fdd |j}| ddddddf|_| ddddddf|_ |j|j fS)Ndevicerr)r+rJrKrQr"rFtype_asrAouterr#tor,r-rL)rMr$ seq_dimensionseq_lentfreqsembr'r'r(_update_cos_sin_tablescs  z&RotaryEmbedding._update_cos_sin_tablesqkreturncCsJ|j|dd\|_|_t||j|jj|jdt||j|jj|jdfS)Nr*)rUrB)rZrKrLr.rTrC)rMr[r\r'r'r(forwardsszRotaryEmbedding.forward)r) __name__ __module__ __qualname____doc__r"Tensor__annotations__intrErZtupler^ __classcell__r'r'rNr(r@Ps   .r@cs8eZdZdZ  d dedeffdd Zdd ZZS) EsmContactPredictionHeadzWPerforms symmetrization, apc, and computes a logistic regression on the output featuresTr in_featureseos_idxcs4t||_||_t|d||_t|_dS)Nr) rDrErirjrLinear regressionSigmoid activation)rMribiasrjrNr'r(rEs z!EsmContactPredictionHead.__init__c Cs||j|}|d|d}||ddddddddf}|dddddf}|dddddf}|\}}}}}||||||}||jjj}t t |}| dddd}| || dS)Nrr.rrr )nerjrT unsqueezesizeviewrlweightrQr?r6permuternsqueeze) rMtokens attentionseos_mask batch_sizelayersheadsseqlen_r'r'r(r^s" z EsmContactPredictionHead.forward)Tr)r_r`rarbrerEr^rgr'r'rNr(rh|s rhcs:eZdZdZfddZ    d ddZddZZS) EsmEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. csttj|j|j|jd|_|jrtj |j|j d|_ nd|_ t |j |_t|dd|_|jdt|jddd|j|_|jdkrTtj|j|j|jd|_|j|_|j|_dS) N) padding_idxepsposition_embedding_typeabsolute position_ids)rrF) persistent)rDrEr Embedding vocab_size hidden_size pad_token_idword_embeddingsemb_layer_norm_before LayerNormlayer_norm_eps layer_normDropouthidden_dropout_probdropoutgetattrrrIr"rFmax_position_embeddingsexpandrposition_embeddings token_dropout mask_token_idrMconfigrNr'r(rEs"    zEsmEmbeddings.__init__Nc Cs|dur|durt||j}n||}|dur||}|}|jrc|durc|||jkdd}d}|dur=|dn|j d}||jkd |}|d|d|ddddf |j }|j dkrq||} || }|jdur{||}|dur||d |j }|S)NrgQ?rr)"create_position_ids_from_input_idsr&create_position_ids_from_inputs_embedsrr masked_fillrrqr8r+rHrTrCrrr) rM input_idsattention_maskr inputs_embeds embeddingsmask_ratio_train src_lengthsmask_ratio_observedrr'r'r(r^s.   "    zEsmEmbeddings.forwardcCsN|dd}|d}tj|jd||jdtj|jd}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor NrrrCrQr)rrr"rFrlongrQrqr)rMr input_shapesequence_lengthrr'r'r(rs  z4EsmEmbeddings.create_position_ids_from_inputs_embedsNNNN)r_r`rarbrEr^rrgr'r'rNr(rs  1rrmodulequerykeyvaluerscalingr head_maskkwargscKst||dd|} t|drt|jdvrt|jd} tj| tj| jd dd} tj| tj| jd dd} | | } | | |j d}|j |j d}|jd krYtd ||}n|jd krptd ||}td ||}||}| |} |dur|ddddddd|jd f}| |} tjj| dtjd |j } tjj| ||jd} |dur| |} t| |}|dd}|| fS)Nrr r relative_keyrelative_key_queryrrrrBrzbhld,lrd->bhlrrzbhrd,lrd->bhlrr*)r rC)ptraining)r"matmulr5hasattrrr+rFrrQrsdistance_embeddingrrTrCeinsumr functionalsoftmaxfloat32rr contiguous)rrrrrrrrr attn_weights seq_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_key causal_mask attn_outputr'r'r(eager_attention_forwards2    & rcsreZdZdfdd Z    ddejdeejdeejdeejd eejd ee d e ejfd d Z Z S)EsmSelfAttentionNFcs8t||_|j|jdkr"t|ds"td|jd|jd|j|_t|j|j|_|j|j|_ t |j|j |_ t |j|j |_ t |j|j |_|j|_|p\t|dd|_d|_|jdksk|jd kr}|j|_t d |jd |j|_n |jd krt|jd |_d|_|j|_||_|jo| |_dS)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()rrrrrrrotaryrr/)rDrErrnum_attention_headsr ValueErrorreattention_head_size all_head_sizerrkrrrattention_probs_dropout_probrrrrotary_embeddingsrrrr@r is_decoder layer_idx is_causal)rMrrris_cross_attentionrNr'r(rE1s8   zEsmSelfAttention.__init__ hidden_statesrrencoder_hidden_statesencoder_attention_maskrr]cKs>|jdd\}}||d|jf} ||| dd} |du} | r$|n|} | r*|n|}|| | dd} || | dd}| |jd} |jdkrX|| | \} } t }|j j dkry|jdvrst d|j j d |jd t |j j }||| | ||f|jsd n|j|j|d |\}}|||d}||fS) NrrrgreagerrzESM z attention does not support z^ embeddings. Set attention explicitly to 'eager' with `model.set_attn_implementation('eager')`r)rrr)r+rrrsr5rrrrrr_attn_implementationrrrrrreshaper)rMrrrrrrrzr hidden_shape query_layerrcurrent_states key_layer value_layerattention_interfacerrr'r'r(r^SsB         zEsmSelfAttention.forward)NNFr) r_r`rarEr"rcr FloatTensorrrrfr^rgr'r'rNr(r0s*%rc$eZdZfddZddZZS) EsmSelfOutputcs.tt|j|j|_t|j|_dSN) rDrErrkrdenserrrrrNr'r(rE zEsmSelfOutput.__init__cC ||}||}||}|SrrrrMr input_tensorr'r'r(r^  zEsmSelfOutput.forwardr_r`rarEr^rgr'r'rNr(r rcsBeZdZd fdd ZddZ    d deefdd ZZS) EsmAttentionNFcsDtt|||d|_t||_t|_tj |j |j d|_ dS)N)rrr) rDrErrMroutputset pruned_headsrrrr)rMrrrrNr'r(rEs  zEsmAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr)lenrrMrrrrrrrrrrunion)rMr|indexr'r'r( prune_headss zEsmAttention.prune_headsrc Ks:||}|j|f||||d|\}} |||}|S)Nrrrr)rrMr) rMrrrrrrhidden_states_lnrr~r'r'r(r^s  zEsmAttention.forward)NFr) r_r`rarErrrr^rgr'r'rNr(rsrc2eZdZfddZdejdejfddZZS)EsmIntermediatecs tt|j|j|_dSr)rDrErrkrintermediate_sizerrrNr'r(rEs zEsmIntermediate.__init__rr]cCs||}t|}|Sr)rr4)rMrr'r'r(r^s zEsmIntermediate.forwardr_r`rarEr"rcr^rgr'r'rNr(rs rcr) EsmOutputcs.tt|j|j|_t|j|_ dSr) rDrErrkrrrrrrrrNr'r(rErzEsmOutput.__init__cCrrrrr'r'r(r^rzEsmOutput.forwardrr'r'rNr(rrrcs@eZdZfddZ    d deefddZddZZS) EsmLayercst|j|_d|_t||_|j|_|j|_|jr-|js&t|dt|dd|_ t ||_ t ||_ tj|j|jd|_dS)Nrz> should be used as a decoder model if cross attention is addedT)rr)rDrEchunk_size_feed_forward seq_len_dimr attentionradd_cross_attention RuntimeErrorcrossattentionr intermediaterrrrrrrrNr'r(rEs    zEsmLayer.__init__Nrc Ksj|j|f||d|}|jr.|dur.t|ds td|d|j|f||||d|}||}|S)N)rrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r)rrrAttributeErrorrfeed_forward_chunk) rMrrrrrrattention_output layer_outputr'r'r(r^s2    zEsmLayer.forwardcCs$||}||}|||}|Sr)rrr)rMr attention_output_lnintermediate_outputr r'r'r(r s   zEsmLayer.feed_forward_chunkr) r_r`rarErrr^r rgr'r'rNr(rs  #rcs<eZdZfddZe    ddeefddZZS) EsmEncodercsNt|_tfddtjD|_tjj j d|_ d|_ dS)Ncsg|]}tqSr')r).0r~rr'r( sz'EsmEncoder.__init__..rF) rDrErr ModuleListrangenum_hidden_layerslayerrrremb_layer_norm_aftergradient_checkpointingrrNrr(rEs   zEsmEncoder.__init__Nrc Ks\t|jD]\}}|dur||nd} ||f|| ||d|}q|jr)||}t|dS)Nr)last_hidden_state) enumeraterrr ) rMrrrrrri layer_modulelayer_head_maskr'r'r(r^ s   zEsmEncoder.forwardr) r_r`rarErrrr^rgr'r'rNr(rs rcr) EsmPoolercs*tt|j|j|_t|_dSr)rDrErrkrrTanhrnrrNr'r(rE=s zEsmPooler.__init__rr]cCs(|dddf}||}||}|SNr)rrn)rMrfirst_token_tensor pooled_outputr'r'r(r^Bs  zEsmPooler.forwardrr'r'rNr(r<s rc@steZdZUeed<dZdZdZgdZdgZ dZ dZ dZ dZ eeeddd geedd d gd Zd d ZddZdS)EsmPreTrainedModelresmTF)r#EsmFoldTriangularSelfAttentionBlockrzposition_embeddings.weightrr)r layer_namer)rrxcross_attentionscCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|tj rX|jj |jj ddSt|tre|jj dSdS)zInitialize the weightsr)meanstdNr/) isinstancerrkrtdatanormal_rinitializer_rangerozero_rrrfill_ EsmLMHead)rMrr'r'r( _init_weightsas       z EsmPreTrainedModel._init_weightscCsdSrr'rMr'r'r(get_output_embeddingsssz(EsmPreTrainedModel.get_output_embeddingsN)r_r`rarrdbase_model_prefixsupports_gradient_checkpointingaccepts_loss_kwargs_no_split_modules"_keys_to_ignore_on_load_unexpected_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrrr_can_record_outputsr1r3r'r'r'r(r#Ks$   r#cseZdZdZdfdd ZddZddZd d Zee dd e e j d e e j de e j de e j de e j de e j de e j de edeee j effddZddZZS)EsmModela  The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. TcsZt|||_t||_t||_|rt|nd|_t |j |j dd|_ | dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer NT)riro)rDrErrrrencoderrpoolerrhrr contact_head post_init)rMradd_pooling_layerrNr'r(rEs     zEsmModel.__init__cC|jjSrrrr2r'r'r(get_input_embeddingszEsmModel.get_input_embeddingscC ||j_dSrrE)rMrr'r'r(set_input_embeddings zEsmModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr?rrr)rMheads_to_prunerr|r'r'r( _prune_headsszEsmModel._prune_headsNrrrrrrrrr]cKs |du|duAr td|dur|j||d}|jjdkr=|jdd\} } |dur4tj| | f|jd}|j|| | fd}|jj rb|durb| \} } } | | f}|dur\tj||jd}| |}nd}| ||jj }|j|f||||d|}|d }|jdur||nd}t||d S) aV input_ids (`torch.LongTensor` of shape `((batch_size, sequence_length))`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) position_ids (`torch.LongTensor` of shape `((batch_size, sequence_length))`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `((batch_size, sequence_length), hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nz:You must specify exactly one of input_ids or inputs_embeds)rrflash_attention_2rrP)rrr)r pooler_output)rrrrr+r"onesrQget_extended_attention_maskrrrinvert_attention_mask get_head_maskrr?r@r )rMrrrrrrrrrzrencoder_batch_sizeencoder_sequence_lengthr~encoder_hidden_shapeencoder_extended_attention_maskencoder_outputssequence_outputr"r'r'r(r^sL  zEsmModel.forwardcCs`|||dddj}tj|dd}||ddd9}||ddd9}|||S)NT)r return_dictoutput_attentionsrrrr )rxr"stackrqrA)rMrwrattnsr'r'r(predict_contactss  zEsmModel.predict_contacts)T)NNNNNNN)r_r`rarbrErFrIrMrrrr"rcrrrrfr r^r_rgr'r'rNr(r>ysF   Qr>cseZdZdgZfddZddZddZee        dd e e j d e e j d e e j d e e j d e e j de e j de e j de e j deedeeeffddZddZZS)EsmForMaskedLMzlm_head.decoder.weightcsHt||jrtdt|dd|_t||_| | dS)NzjIf you want to use `EsmForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.FrC) rDrErloggerwarningr>r$r0lm_head init_weightsrBrrNr'r(rE s   zEsmForMaskedLM.__init__cCrDrrddecoderr2r'r'r(r3rGz$EsmForMaskedLM.get_output_embeddingscCrHrrf)rMnew_embeddingsr'r'r(set_output_embeddingsrJz$EsmForMaskedLM.set_output_embeddingsNrrrrrrrlabelsrr]c  Ks|j|f||||||d| } | d} || } d} |dur6t}|| j}|| d|jj|d} t| | | j | j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` )rrrrrrrNrlosslogitsrrx) r$rdrrTrQrsrrr rrx)rMrrrrrrrrjroutputsrYprediction_scoresmasked_lm_lossloss_fctr'r'r(r^"s2  zEsmForMaskedLM.forwardcCs|jj||dS)N)r)r$r_)rMrwrr'r'r(r_RszEsmForMaskedLM.predict_contacts)NNNNNNNN)r_r`ra_tied_weights_keysrEr3rirrrr" LongTensorrcrrrrrfr r^r_rgr'r'rNr(r`sJ    .r`c(eZdZdZfddZddZZS)r0z&ESM Head for masked language modeling.cs^tt|j|j|_tj|j|jd|_tj|j|j dd|_ t t |j |_dS)NrF)ro)rDrErrkrrrrrrrg Parameterr"zerosrorrNr'r(rEYs zEsmLMHead.__init__cKs0||}t|}||}|||j}|Sr)rr4rrgrorMfeaturesrr$r'r'r(r^as  zEsmLMHead.forwardr_r`rarbrEr^rgr'r'rNr(r0Vs r0z ESM Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ) custom_introceZdZfddZee      ddeejdeej deejdeej deej d eejd e e d e eeffd d ZZS)EsmForSequenceClassificationcsFt||j|_||_t|dd|_t||_|| dSNFra) rDrE num_labelsrr>r$EsmClassificationHead classifierrerBrrNr'r(rErs   z%EsmForSequenceClassification.__init__Nrrrrrrjrr]c Ks4|j|f||||d|}|d} || } d} |dur|| j}|jjdurM|jdkr3d|j_n|jdkrI|jtj ksD|jtj krId|j_nd|j_|jjdkrkt } |jdkre| | | } n+| | |} n%|jjdkrt } | | d|j|d} n|jjdkrt} | | |} t| | |j|jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rrrrrNrrlsingle_label_classificationmulti_label_classificationrrk)r$rrTrQr problem_typer~rCr"rrerrvrrsrrrrx rMrrrrrrjrrnrYrmrlrqr'r'r(r^~sL     "       z$EsmForSequenceClassification.forwardNNNNNN)r_r`rarErrrr"rsrcrrrrrfrr^rgr'r'rNr(r|ks6   r|cr{)EsmForTokenClassificationcsVt||j|_t|dd|_t|j|_t |j |j|_ | | dSr})rDrEr~r>r$rrrrrkrrrerBrrNr'r(rEs  z"EsmForTokenClassification.__init__Nrrrrrrjrr]c Ks|j|f||||d|}|d} || } || } d} |dur8t} || j}| | d|j|d} t| | |j |j dS)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. rrNrrk) r$rrrrTrQrsr~rrrxrr'r'r(r^s0   z!EsmForTokenClassification.forwardr)r_r`rarErrrr"rsrcrrrrrfrr^rgr'r'rNr(rs6   rcrt)rz-Head for sentence-level classification tasks.cs@tt|j|j|_t|j|_t|j|j |_ dSr) rDrErrkrrrrrr~out_projrrNr'r(rEs zEsmClassificationHead.__init__cKsL|dddddf}||}||}t|}||}||}|Sr )rrr"tanhrrwr'r'r(r^s     zEsmClassificationHead.forwardryr'r'rNr(rs rcCs2||}tj|dd||}||S)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rr)rprer"cumsumrRr)rrmaskincremental_indicesr'r'r(r s  r)r`r|rr>r#)rN)Frbr1typingrrrr"rtorch.nnrrrmodeling_layersr modeling_outputsr r r rrmodeling_utilsrrprocessing_utilsr pytorch_utilsrrutilsrrrr utils.genericrrconfiguration_esmr get_loggerr_rbr)r.r4r6r?Moduler@rhrrcrHrrrrrrrrrr#r>r`r0r|rrr__all__r'r'r'r(s      ,#f 2Y 0  :$-ML;