o ॵi[ @sVdZddlZddlZddlZddlZddlZddlmZmZm Z m Z m Z m Z m Z e e eZejddGdddZejddGdd d Zd ed e e ee effd d Zded efddZded efddZde eded eefddZ  dEdede eded efddZdede ed efdd Zd!eded efd"d#Zd!ed efd$d%Zd!ed efd&d'Zd(eded e e efd)d*Zd+ed,ed-e efd.d/Zd0e ed efd1d2Z d3ed e efd4d5Z!d6ed eee"ffd7d8Z#d+ed9ed e efd:d;Z$ejddGded e%fd?d@Z& dFdAededBed e efdCdDZ'dS)Gz+Functions for parsing various file formats.N)DictIterableListOptionalSequenceSetTupleT)frozenc@sPeZdZUdZeeed<eed<eeed<ddZddZ d e fd d Z d S) Msaz%Class representing a parsed MSA file. sequencesdeletion_matrix descriptionscCsZt|jt|jkrt|jks+ntdt|jdt|jdt|jddS)Nz5All fields for an MSA must have the same length. Got z sequences, z! rows in the deletion matrix and z descriptions.)lenr r r ValueErrorselfra/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/models/science/unifold/msa/parsers.py __post_init__!s zMsa.__post_init__cCs t|jSN)rr rrrr__len__*s z Msa.__len__max_seqscCs,t|jd||jd||jd|dS)Nr r r )r r r r )rrrrrtruncate-s    z Msa.truncateN) __name__ __module__ __qualname____doc__rstr__annotations__DeletionMatrixrrintrrrrrr s    r c@s^eZdZUdZeed<eed<eed<eeed<eed<eed<e eed<e eed <d S) TemplateHitz"Class representing a template hit.indexname aligned_cols sum_probsquery hit_sequence indices_query indices_hitN) rrrrr!rrrfloatrrrrrr"5s   r" fasta_stringreturncCspg}g}d}|D])}|}|dr(|d7}||dd|dq |s+q |||7<q ||fS)a]Parses FASTA string and returns list of strings with amino-acid sequences. Arguments: fasta_string: The string contents of a FASTA file. Returns: A tuple of two lists: * A list of sequences. * A list of sequence descriptions taken from the comment lines. In the same order as the sequences. >N) splitlinesstrip startswithappend)r,r r r#linerrr parse_fastaCs    r7stockholm_stringcs,t}|D]$}|}|r|drq|\}||vr$d||<||7<qg}g}d}g}t|D]O\}|dkrN}ddt|D}dfdd|D} | | g} d} t |D]\} } | dkst| dkr| dkr}| d7} qh| | d} qh| | q;t ||t | d S) aParses sequences and deletion matrix from stockholm format alignment. Args: stockholm_string: The string contents of a stockholm file. The first sequence in the file should be the query sequence. Returns: A tuple of: * A list of sequences that have been aligned to the query. These might contain duplicates. * The deletion matrix for the alignment as a list of lists. The element at `deletion_matrix[i][j]` is the number of residues deleted from the aligned sequence i at residue position j. * The names of the targets matched, including the jackhmmer subsequence suffix. #//r1rcSsg|] \}}|dkr|qS-r).0iresrrr z#parse_stockholm..csg|]}|qSrrr>csequencerrrAr=r0r) collections OrderedDictr2r3r4split enumeratevaluesjoinr5zipr listkeys)r8name_to_sequencer6r$msar r' keep_columns seq_indexaligned_sequence deletion_vecdeletion_countseq_res query_resrrErparse_stockholm`sD       rZ a3m_stringc st|\}}g}|D] }g}d}|D]}|r|d7}q||d}q||q tddtjfdd|D}t|||dS)aParses sequences and deletion matrix from a3m format alignment. Args: a3m_string: The string contents of a a3m file. The first sequence in the file should be the query sequence. Returns: A tuple of: * A list of sequences that have been aligned to the query. These might contain duplicates. * The deletion matrix for the alignment as a list of lists. The element at `deletion_matrix[i][j]` is the number of residues deleted from the aligned sequence i at residue position j. * A list of descriptions, one per sequence, from the a3m file. rr0r1csg|]}|qSr) translate)r>sdeletion_tablerrrAszparse_a3m..r)r7islowerr5r maketransstringascii_lowercaser ) r[r r r msa_sequencerVrWjaligned_sequencesrr^r parse_a3ms$    rgquery_non_gapssto_seqccs8t||D]\}}|r|Vq|dkr|VqdS)Nr=)rNlower)rhriis_query_res_non_gap sequence_resrrr_convert_sto_seq_to_a3ms rmstockholm_format max_sequencesremove_first_row_gapscszii}d}|D].}|ot||k}|r8|ds8|jdd\}}||vr0|r,q d||<|||7<q |D]>}|dddkr{|jd d}|dd \}} t|dkr_|d nd} | d krfq=|rm||vrmq=| |<tt|kr{nq=i|rtt|} d d | D} |D]\}} | d d}|rd t | |}||<qfddD}d |dS)z3Converts MSA in Stockholm format to the A3M format.Fr9r0maxsplitr1N#=GSDEcSg|]}|dkqSr<r)r>r@rrrrArGz,convert_stockholm_to_a3m...c3s2|]}d|d|dd|VqdS)r/ r1 N)get)r>k a3m_sequencesr rr s*z+convert_stockholm_to_a3m..rz) r2rr3r4rJnextiterrLitemsreplacerMrm)rnrorpr reached_max_sequencesr6seqname aligned_seqcolumnsfeaturevaluequery_sequencerh sto_sequence out_sequence fasta_chunksrr}rconvert_stockholm_to_a3msZ       rr6seqnamescCs|sdS|dkrdS|drdS|drdS|dddkr1|jdd \}}}||vS|d r8d S|d d }||vS)z'Function to decide which lines to keep.Tr;z # STOCKHOLM#=GC RFNrsrtrqr:Fryr)r3r4rJ partition)r6r_rrrr _keep_lines    r stockholm_msacCst}g}|D]}|r(|ds(|dd}||t||kr(nq |D] }t||r9||q-d |dS)zr?processed_linesrrrYsz:remove_empty_columns_from_stockholm_msa..) rKr2r4 rpartitionrangerrr5anyrM itertoolscompressr3)runprocessed_linesr?r6reference_annotation_ireference_annotation_linerfirst_alignmentmaskreunprocessed_lineprefix alignment line_indexmasked_alignmentrrr'remove_empty_columns_from_stockholm_msa/sR          rc Cstt}|D]}|r&|ds&|}|\}}|||7<q t}t}tt | }dd|D}| D]\}}d t ||} | |vrRq@|| ||q@g} |D] }t||ro| |qcd | dS)z;Remove duplicate sequences (ignoring insertions wrt query).r9cSrwr<rrCrrrrAmrGz-deduplicate_stockholm_msa..r1rz)rH defaultdictrr2r3r4rJrrrrLrrMrrrrr5) r sequence_dictr6rrseen_sequencesr query_alignrrrrrrdeduplicate_stockholm_msa\s.        r regex_patterncCs*t||}|durtd||S)NzCould not parse query line )rematch RuntimeErrorgroups)rr6rrrr_get_hhr_line_regex_groupss rrF start_index indices_listcCs8|}|D]}|dkr|dq|||d7}qdS)zUComputes the relative indices for each residue with respect to the original sequence.r=r.r0N)r5)rFrrcountersymbolrrr _update_hhr_residue_indices_lists   rdetailed_linesc Cst|dd}|ddd}d}t||d}|dur*td||dfdd |D\}}}}}}}}d }d } g} g} d} |d dD]} | d r| d s| ds| dsd}t|| dd}t|dd}|d}t|d}tdd |D}|||} | t|ksJ||7}t ||| qK| dr| ds| ds| dsd}t|| dd}t|dd}|d}| t|ksJ| |7} t ||| qKt ||t|||| | | dS)aParses the detailed HMM HMM comparison section for a single Hit. This works on .hhr files generated from both HHBlits and HHSearch. Args: detailed_lines: A list of lines from a single comparison section between 2 sequences (which each have their own HMM's) Returns: A dictionary with the information from that detailed comparison section Raises: RuntimeError: If a certain line cannot be processed rr.r0NzProbab=(.*)[ ]*E-value=(.*)[ ]*Score=(.*)[ ]*Aligned_cols=(.*)[ ]*Identities=(.*)%[ ]*Similarity=(.*)[ ]*Sum_probs=(.*)[ ]*Template_Neff=(.*)rzCCould not parse section: %s. Expected this: %s to contain summary.cSsg|]}t|qSr)r+r>xrrrrArGz"_parse_hhr_hit..r1ruzQ z Q ss_dsspz Q ss_predz Q Consensusz1[\t ]*([0-9]*) ([A-Z-]*)[\t ]*([0-9]*) \([0-9]*\)cSsg|]}|dkr|qSr<rrrrrrAszT z T ss_dsspz T ss_predz T Consensusz/[\t ]*([0-9]*) ([A-Z-]*)[\t ]*[0-9]* \([0-9]*\)r#r$r%r&r'r(r)r*) r!rJrrrrr4rrrr")r number_of_hitname_hitpatternrrr%r&r'r(r)r* length_blockr6pattrstart delta_queryendnum_insertionsdelta_hit_sequencerrr_parse_hhr_hitsz     r hhr_stringc Csl|}ddt|D}g}|r4|t|tt|dD]}|t|||||dq |S)z)Parses the content of an entire HHR file.cSsg|] \}}|dr|qS)zNo )r4)r>r?r6rrrrAs zparse_hhr..r0)r2rKr5rrr)rlines block_startshitsr?rrr parse_hhrsrtbloutcCsLddi}dd|D}|D]}|}|d}|d}t|||<q|S)zDParse target to e-value mapping parsed from Jackhmmer tblout string.r'rcSsg|] }|ddkr|qS)rr:r)r>r6rrrrA rBz.parse_e_values_from_tblout..rs)r2rJr+)re_valuesrr6fieldse_value target_namerrrparse_e_values_from_tblout srrcCsNg}|}|D]}|dkr|dq|r|d7}q|||d7}q|S)zHReturns indices for non-gap/insert residues starting at the given index.r=r.r0)r5r`)rFrindicesrrrrr _get_indicess    rc@s>eZdZUeed<eed<eed<eed<eed<eed<dS) HitMetadatapdb_idchainrrlengthtextN)rrrrrr!rrrrr*s  r descriptioncCs\td|}|std|dt|d|dt|dt|dt|d|d d S) z3Parses the hmmsearch A3M sequence description line.zF^>?([a-z0-9]+)_(\w+)/([0-9]+)-([0-9]+).*protein length:([0-9]+) *(.*)$zCould not parse description: "z".r0rrurs)rrrrrr)rrr3rrr!)rrrrr_parse_hmmsearch_description4s   rr skip_firstc Csttt|}|r|dd}t|dd}g}t|ddD];\}\}}d|vr)qt|} tdd|D} t|| jdd} t|| j d| j | d|| || d } | | q|S) a&Parses an a3m string produced by hmmsearch. Args: query_sequence: The query sequence. a3m_string: The a3m string produced by hmmsearch. skip_first: Whether to skip the first sequence in the a3m string. Returns: A sequence of `TemplateHit` results. r0Nr)rz mol:proteincSsg|] }|o |dkqSr<)isupper)r>rrrrrAdrBz'parse_hmmsearch_a3m..rr) rOrNr7rrKrsumrr"rrupperr5) rr[r parsed_a3mr)rr?r(hit_descriptionmetadatar%r*hitrrrparse_hmmsearch_a3mJs.   r)NT)T)(rrH dataclassesrrrbtypingrrrrrrrr!r dataclassr r"rr7rZrgboolrmrrrrrrrrrr+rrrrrrrrrst $   " > '   9-#    a