o Ti @s<ddlZddlmZmZmZddlZddlmZmZddlm Z ddl m Z e ej e dkrUddlmZeeddd eddd d eddd eddd d d Zneddd eddd d geddd eddd d d Z dd e deddeefddZd e fddZ dded e deedeefddZdS)N)CallableLiteralOptional)DatasetValue)version) AutoTokenizerz4.0.0)Liststring)dtypeid)contentrole) completionprompt)chatml instruction tokenizermessages_field)messages conversationstoolscsfdd}|S)z return a callable function that takes in a "messages" dataset and returns a formatted dataset, based on the tokenizer apply chat template to the dataset along with the schema of the list of functions in the tools list. csbt|dtr'g}tt|D]}|j||ddq|Sj|ddS)NrF)tokenizer isinstancelistrangelenappendapply_chat_template)examples output_textsirrrQ/home/ubuntu/.local/lib/python3.10/site-packages/trl/extras/dataset_formatting.pyformat_dataset.sz9conversations_formatting_function..format_datasetr$)rrrr&r$r#r%!conversations_formatting_function&s r'csfdd}|S)z return a callable function that takes in an "instructions" dataset and returns a formatted dataset, based on the tokenizer apply chat template to the dataset cst|dtr2g}tt|dD]}d|d|dd|d|dg}|j|ddq|Sd|ddd|ddg}j|ddS)Nruser)rr assistantrF)rr)r r!r"converted_samplerr$r%r&Bs  z8instructions_formatting_function..format_datasetr$)rr&r$r+r% instructions_formatting_function<s r,datasetreturncCst|trId|jvr|jdtdkrtdt|d|Sd|jvr9|jdtdkr7tdt|d|SdS|jtdkrItdt|SdS)a Finds the correct formatting function based on the dataset structure. Currently supported datasets are: - `ChatML` with [{"role": str, "content": str}] - `instruction` with [{"prompt": str, "completion": str}] Args: dataset (Dataset): User dataset tokenizer (AutoTokenizer): Tokenizer used for formatting Returns: Callable: Formatting function if the dataset format is supported else None rrz%Formatting dataset with chatml formatrrz*Formatting dataset with instruction formatN)rrfeaturesFORMAT_MAPPINGlogginginfor'r,)r-rrr$r$r% get_formatting_func_from_datasetVs        r3)N)r1typingrrrdatasetsrr packagingr transformersrparse __version__r r0rr'r,r3r$r$r$r%sB