o ei' @sNdZddlmZddlmZddlmZgdZeGdddeZdgZ dS) z Processor class for EVOLLA. ) BatchFeature)ProcessorMixin)auto_docstring)aa_seqfoldseekmsac seZdZdfdd ZdddZ ddefd d Ze    dd ee e BdBd eee ee BdBd edBdedBfddZ ddZ ddZ ddZ ddZZS)EvollaProcessorNc sF|durtd|durtdt||d|j_||_||_dS)a protein_tokenizer (`EsmTokenizer`): An instance of [`EsmTokenizer`]. The protein tokenizer is a required input. protein_max_length (`int`, *optional*, defaults to 1024): The maximum length of the sequence to be generated. text_max_length (`int`, *optional*, defaults to 512): The maximum length of the text to be generated. Nz+You need to specify an `protein_tokenizer`.z"You need to specify a `tokenizer`.z<|reserved_special_token_0|>) ValueErrorsuper__init__ tokenizer pad_tokenprotein_max_lengthtext_max_length)selfprotein_tokenizerrrrkwargs __class__j/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/transformers/models/evolla/processing_evolla.pyr s  zEvollaProcessor.__init__c Cs^g}|D]}|d}|d}dddt||D}||q|j|dd|dd}|S) NrrcSs g|] \}}||qSr)upperlower).0sfrrr 7s z4EvollaProcessor.process_proteins..ptT)return_tensors truncation max_lengthpadding)getjoinzipappendr) rproteinsr sa_sequencesproteinrr sa_sequence sa_tokensrrrprocess_proteins2s    z EvollaProcessor.process_proteinsrcCsDg}|D]}|jj|ddd}||q|j|dddd|d}|S)NFT)tokenizeadd_generation_promptr longest)add_special_tokensr!r$r"r#)rapply_chat_templater()rtextsrpromptsmessagesprompt prompt_inputsrrr process_text?s" zEvollaProcessor.process_textr) messages_listrc Ks|dus|dur td|dur|n|j}|dur|n|j}t|tr&|g}t|ttfr9t|dttfs9|g}t|ttfrMtdd|DsMtdt|ttfrjtdd|Dsjtdd t d |t|ttfr|D]9}t|ttfst d t |d td d|Dstdt dd|Dst dd|Drtd|qsn tdt |d | ||}|||}t|d|d|d|dddS)a proteins (`Union[List[dict], dict]`): A list of dictionaries or a single dictionary containing the following keys: - `"aa_seq"` (`str`) -- The amino acid sequence of the protein. - `"foldseek"` (`str`) -- The foldseek string of the protein. messages_list (`Union[List[List[dict]], List[dict]]`): A list of lists of dictionaries or a list of dictionaries containing the following keys: - `"role"` (`str`) -- The role of the message. - `"content"` (`str`) -- The content of the message. protein_max_length (`int`, *optional*, defaults to 1024): The maximum length of the sequence to be generated. text_max_length (`int`, *optional*, defaults to 512): The maximum length of the text. Return: a dict with following keys: - `protein_input_ids` (`torch.Tensor` of shape `(batch_size, sequence_length)`) -- The input IDs for the protein sequence. - `protein_attention_mask` (`torch.Tensor` of shape `(batch_size, sequence_length)`) -- The attention mask for the protein sequence. - `text_input_ids` (`torch.Tensor` of shape `(batch_size, sequence_length)`) -- The input IDs for the text sequence. - `text_attention_mask` (`torch.Tensor` of shape `(batch_size, sequence_length)`) -- The attention mask for the text sequence. Nz3You need to specify `messages_list` and `proteins`.cs|]}t|tVqdSN isinstancedictrprrr z+EvollaProcessor.__call__..zUThe proteins should be a list of dictionaries, but not all elements are dictionaries.css&|]}tdd|DVqdS)css|]}|tvVqdSr=)PROTEIN_VALID_KEYS)rkrrrrCsz5EvollaProcessor.__call__...N)allkeysrArrrrCs z2There should be a list of dictionaries with keys: z, z for each protein.But got: z;Each messages in messages_list should be a list instead of .csr<r=r>rmrrrrCrDzfEach message in messages_list should be a list of dictionaries, but not all elements are dictionaries.css |] }t|dkVqdS)N)lenrHrJrrrrCscss$|] }t|ddhkVqdS)rolecontentN)setrHrJrrrrCs zlEach message in messages_list should be a list of dictionaries with two keys: 'role' and 'content'.But got: zFThe messages_list should be a list of lists of dictionaries, but it's input_idsattention_mask)protein_input_idsprotein_attention_maskrQrR)data)r rrr?r@listtuplerGr&rE TypeErrortypeanyr.r9r) rr)r:rrrr6r- text_tokensrrr__call__Wsd     zEvollaProcessor.__call__cO|jj|i|Sr=)r batch_decoderargsrrrrr^zEvollaProcessor.batch_decodecOr]r=)rdecoder_rrrrbrazEvollaProcessor.decodecOr]r=)rr^r_rrrprotein_batch_decoderaz$EvollaProcessor.protein_batch_decodecOr]r=)rrbr_rrrprotein_decoderazEvollaProcessor.protein_decode)Nr r )r )r )NNNN)__name__ __module__ __qualname__r r.intr9rrVr@r\r^rbrcrd __classcell__rrrrrs2   VrN) __doc__feature_extraction_utilsrprocessing_utilsrutilsrrEr__all__rrrrs