o }oi"@sddlZddlmZedZedZGdddZeddZdd d Zd d Z ddZ ddZ ddZ ddZ ddZddZdS)N)cachez^\{_[A-Za-z][A-Za-z0-9_]*_\}$z ({_[^}]+_})c@s eZdZddZeddZdS)ChatTemplateMixincCs|jdusJt|||jSN) chat_templatetokenize_with_chat_template)selfmessagesr j/home/ubuntu/.local/lib/python3.10/site-packages/nemo/collections/common/tokenizers/chat_template_mixin.pyapply_chat_templatesz%ChatTemplateMixin.apply_chat_templatecCs |jduSr)r)rr r r has_chat_templates z#ChatTemplateMixin.has_chat_templateN)__name__ __module__ __qualname__r propertyr r r r r rsrcCs tt|Sr)rematchTEMPLATE_VAR_VALIDATION_PATsr r r is_template_var s rTccs,tt|D] }|r|dkrq|VqdS)N)rsplitTEMPLATE_VAR_SEARCH_PAT)template skip_emptypartr r r extract_template_parts&s  rcCst|s|S|ddS)N)rrr r r strip_template_wrap.s r ccsxg}tt|D](\}}t|r,t|}|dkr!||dq d||fVg}q ||q d|dfVdS)avRenders a chat turn based on template Args: message (Dict) e.g. {'role': ['user'], 'content': ['What is your favourite fruit?']}, template (Str): "[INST] {_content_} [/INST]", Returns: (str, token_id/None): the template formatted message e.g. "[INST] What is your favourite fruit? [/INST]", None contentrN) enumeraterrr appendjoin)messageransi template_partr r r render_chat_turn5s r)cCsg}t|tr|dkr|||7}nt|tr(t|dkr(||d|7}|durAt||s9Jd|d|t||g7}|S)a Tokenizes a string or a list of string into their corresponding token_ids and appends (at the end) a special_token if present. Args: tokenizer: (SPM) inputs: (Str, List[Str]) e.g. "Alex" or ["Alex", "nvidia"] special_token: (Str): e.g. "eos" Returns: (list[int]): list of token_ids e.g. input="Alex", special_token="eos" Alex->[3413] eos->[2] Will return the following: [3413, 2] rrNzSpecial_token z is not part of tokenizer) isinstancestr text_to_idslistlenr$hasattrgetattr) tokenizerinputs special_tokenr&r r r encode_string_with_special_tokenSsr4c st|sJdt|dksJdd|vsJdg}fdd}d|vr9td |dD] \}}||||7}q-g}|D]8}|d |dvsQJ|d |df|d|d } t|| D]\} }|| g7}|durt||||7}g}q^q=|||d7}t|dksJd |S) Nz"Expected input to be chat-templaterzExpected non-empty messagesrolesz&Expected template to have key `roles`.cs t||Sr)r4)xyr1r r zs z-tokenize_with_chat_template..prefixrrolezExpected non-empty output) is_chat_inputr.r)) r1rrr&encoderr3bufferr% msg_templatetemplated_messagesr r8r rus, $ rcs.g}|D]}|fdd|Dq|S)a' a collated messages can have multiple chat messages in each dict, this extracts (vertically) one of them, for example: messages = [ {'role': ['user', 'user'], 'content': ['What is your favourite condiment?', 'What is your favourite fruit?']}, {'role': ['assistant', 'assistant'], 'content': ["Well, I'm quite partial to a ", "good squeeze of fresh lemon"]}, {'role': ['user', 'user'], 'content': ['Do you have mayonnaise recipes?', 'Do you have tomato salad recipes?']} ] ans = extract_turns(messages, axis=1) ans = [ {'role': ['user'], 'content': ['What is your favourite fruit?']}, {'role': ['assistant'], 'content': ["good squeeze of fresh lemon"]}, {'role': ['user'], 'content': ['Do you have tomato salad recipes?']} ] csi|] \}}||qSr r ).0kvaxisr r sz!extract_turns..)r#items)rrEr&turnr rDr extract_turnssrIc#st|ts Jdt|dksJdttdd|s!Jdttdd|s.Jdt|dd ttfd d|sEJd tD] }t||d VqId S)a+ Example input [ {'role': ['user', 'user'], 'content': ['What is your favourite condiment?', 'What is your favourite fruit?']}, {'role': ['assistant', 'assistant'], 'content': ["Well, I'm quite partial to a ", "good squeeze of fresh lemon"]}, {'role': ['user', 'user'], 'content': ['Do you have mayonnaise recipes?', 'Do you have tomato salad recipes?']} ] Notice the 2D axis system of the messages variable, one for the list and one for each item in the list (i.e. the 'content' contains multiple messages). zExpected messages to be a listrzExpected non empty messagescSs t|tSr)r*dictr6r r r r9s z-explode_chat_template_input..z"Expected messages to contain dictscSsd|vod|vS)Nr;r!r rKr r r r9zBExpected messages each dict to contain 'role' and 'content' fieldsr;cst|dkS)Nr;)r.rKnr r r9rLzIExpected all batch messages to contain equal number of roles in all turnsrDN)r*r-r.allmaprangerI)rr'r rMr explode_chat_template_inputs$   rRcCs$t|tot|dkot|dtS)Nr)r*r-r.rJ)rr r r r<s$r<)T)r functoolsrcompilerrrrrr r)r4rrIrRr<r r r r s     "