o i,@sbdZddlZddlmZmZddlmZddlmZddl m Z gd Z Gd d d eZ d gZ dS) z Processor class for EVOLLA. N)OptionalUnion) BatchFeature)ProcessorMixin) AutoTokenizer)aa_seqfoldseekmsac seZdZdZddgZdgZdZdZdZd!fd d Z d"d d Z d#d e fddZ    d$de eeeefde eeeeeefde e d e e fddZddZddZddZddZfddZefdd ZZS)%EvollaProcessoran Constructs a EVOLLA processor which wraps a LLama tokenizer and SaProt tokenizer (EsmTokenizer) into a single processor. [`EvollaProcessor`] offers all the functionalities of [`EsmTokenizer`] and [`LlamaTokenizerFast`]. See the docstring of [`~EvollaProcessor.__call__`] and [`~EvollaProcessor.decode`] for more information. Args: protein_tokenizer (`EsmTokenizer`): An instance of [`EsmTokenizer`]. The protein tokenizer is a required input. tokenizer (`LlamaTokenizerFast`, *optional*): An instance of [`LlamaTokenizerFast`]. The tokenizer is a required input. protein_max_length (`int`, *optional*, defaults to 1024): The maximum length of the sequence to be generated. text_max_length (`int`, *optional*, defaults to 512): The maximum length of the text to be generated. protein_tokenizer tokenizersequence_max_lengthrNc sF|durtd|durtdt||d|j_||_||_dS)Nz+You need to specify an `protein_tokenizer`.z"You need to specify a `tokenizer`.z<|reserved_special_token_0|>) ValueErrorsuper__init__r pad_tokenprotein_max_lengthtext_max_length)selfr rrrkwargs __class__`/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/evolla/processing_evolla.pyr;s zEvollaProcessor.__init__c Cs`g}|D]}|d}|d}dddt||D}||q|jj|dd|dd}|S) Nr r cSs g|] \}}||qSr)upperlower).0sfrrr Ls z4EvollaProcessor.process_proteins..ptT)return_tensors truncation max_lengthpadding)getjoinzipappendr batch_encode_plus) rproteinsr sa_sequencesproteinr r sa_sequence sa_tokensrrrprocess_proteinsGs    z EvollaProcessor.process_proteinsrcCsDg}|D]}|jj|ddd}||q|j|dddd|d}|S)NFT)tokenizeadd_generation_promptr%longest)add_special_tokensr&r)r'r()rapply_chat_templater-)rtextsrpromptsmessagesprompt prompt_inputsrrr process_textTs" zEvollaProcessor.process_textr/ messages_listrc Ks|dus|dur td|dur|n|j}|dur|n|j}t|tr&|g}t|ttfr9t|dttfs9|g}t|ttfrMtdd|DsMtdt|ttfrjtdd|Dsjtdd t d |t|ttfr|D]9}t|ttfstd t |d td d|Dstdt dd|Dst dd|Drtd|qsn tdt |d | ||}| ||}t|d|d|d|dddS)avThis method takes batched or non-batched proteins and messages_list and converts them into format that can be used by the model. Args: proteins (`Union[List[dict], dict]`): A list of dictionaries or a single dictionary containing the following keys: - `"aa_seq"` (`str`) -- The amino acid sequence of the protein. - `"foldseek"` (`str`) -- The foldseek string of the protein. messages_list (`Union[List[List[dict]], List[dict]]`): A list of lists of dictionaries or a list of dictionaries containing the following keys: - `"role"` (`str`) -- The role of the message. - `"content"` (`str`) -- The content of the message. protein_max_length (`int`, *optional*, defaults to 1024): The maximum length of the sequence to be generated. text_max_length (`int`, *optional*, defaults to 512): The maximum length of the text. Return: a dict with following keys: - `protein_input_ids` (`torch.Tensor` of shape `(batch_size, sequence_length)`) -- The input IDs for the protein sequence. - `protein_attention_mask` (`torch.Tensor` of shape `(batch_size, sequence_length)`) -- The attention mask for the protein sequence. - `text_input_ids` (`torch.Tensor` of shape `(batch_size, sequence_length)`) -- The input IDs for the text sequence. - `text_attention_mask` (`torch.Tensor` of shape `(batch_size, sequence_length)`) -- The attention mask for the text sequence. Nz3You need to specify `messages_list` and `proteins`.rcs|]}t|tVqdSN isinstancedictr!prrr z+EvollaProcessor.__call__..zUThe proteins should be a list of dictionaries, but not all elements are dictionaries.css&|]}tdd|DVqdS)css|]}|tvVqdSrB)PROTEIN_VALID_KEYS)r!krrrrHsz5EvollaProcessor.__call__...N)allkeysrFrrrrHs z2There should be a list of dictionaries with keys: z, z for each protein.But got: z;Each messages in messages_list should be a list instead of .csrArBrCr!mrrrrHrIzfEach message in messages_list should be a list of dictionaries, but not all elements are dictionaries.css |] }t|dkVqdS)rN)lenrMrOrrrrHscss$|] }t|ddhkVqdS)rolecontentN)setrMrOrrrrHs zlEach message in messages_list should be a list of dictionaries with two keys: 'role' and 'content'.But got: zFThe messages_list should be a list of lists of dictionaries, but it's input_idsattention_mask)protein_input_idsprotein_attention_maskrUrV)data)rrrrDrElisttuplerLr+rJtypeanyr4r?r) rr/r@rrrr<r3 text_tokensrrr__call__lsd!     zEvollaProcessor.__call__cO|jj|i|SrB)r batch_decoderargsrrrrrazEvollaProcessor.batch_decodecOr`rB)rdecoderbrrrrerdzEvollaProcessor.decodecOr`rB)r rarbrrrprotein_batch_decoderdz$EvollaProcessor.protein_batch_decodecOr`rB)r rerbrrrprotein_decoderdzEvollaProcessor.protein_decodec s|jtj||jd|jv}|r|jdnd}|r'|dur'|jdt j|fi|}|r>|dur>|j |d|S)Nr ) r save_pretrainedospathr+protein_tokenizer_dir_name attributesindexremoverinsert)rsave_directoryrprotein_tokenizer_presentprotein_tokenizer_indexoutputsrrrrhs    zEvollaProcessor.save_pretrainedc s@tj|fi|}t|tr|d}tj||jd}||_|S)Nr) subfolder)rfrom_pretrainedrDr[rrkr )clspretrained_model_name_or_pathr processorr rrrrus zEvollaProcessor.from_pretrained)Nrr)r)r)NNNN)__name__ __module__ __qualname____doc__rl valid_kwargsprotein_tokenizer_classtokenizer_classrkrr4intr?rrrZrEr_rarerfrgrh classmethodru __classcell__rrrrr sB   Y r )r|ritypingrrfeature_extraction_utilsrprocessing_utilsrautorrJr __all__rrrrs    X