o 7tis@sdZddlZddlZddlmZddlmZmZm Z ddl Z ddl m Z ddl mZdZdd Zd d ZGd d d eZdZdZddZGdddeZdS)al In the dynamic landscape of generative NLP, traditional text processing pipelines limit research flexibility and reproducibility, as they are tailored to specific dataset, task, and model combinations. The escalating complexity, involving system prompts, model-specific formats, instructions, and more, calls for a shift to a structured, modular, and customizable solution. Addressing this need, we present Unitxt, an innovative library for customizable textual data preparation and evaluation tailored to generative language models. Unitxt natively integrates with common libraries like HuggingFace and LM-eval-harness and deconstructs processing flows into modular components, enabling easy customization and sharing between practitioners. These components encompass model-specific formats, task prompts, and many other comprehensive dataset processing definitions. The Unitxt-Catalog centralizes these components, fostering collaboration and exploration in modern textual data workflows. Beyond being a tool, Unitxt is a community-driven platform, empowering users to build, share, and advance their pipelines collaboratively. N)partial)AnyDictOptional)Instance)ConfigurableTaska @misc{bandel2024unitxt, title={Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative AI}, author={Elron Bandel and Yotam Perlitz and Elad Venezian and Roni Friedman-Melamed and Ofir Arviv and Matan Orbach and Shachar Don-Yehyia and Dafna Sheinwald and Ariel Gera and Leshem Choshen and Michal Shmueli-Scheuer and Yoav Katz}, year={2024}, eprint={2401.14019}, archivePrefix={arXiv}, primaryClass={cs.CL} } cCsLtjddur tdddlm}ttt| d}|dkr$tddS)Nunitxtz`Please install unitxt via 'pip install unitxt'. For more information see: https://www.unitxt.ai/r) __version__.)zyPlease install a more recent version of unitxt via 'pip install --upgrade unitxt' to avoid errors due to breaking changes) importlibutil find_spec Exceptionrr tuplemapintsplit)unitxt_versionrM/home/ubuntu/.local/lib/python3.10/site-packages/lm_eval/tasks/unitxt/task.pyassert_unitxt_installeds rcCsPt|\}}tddlm}|D]}|g|d<q|||}|ddddS)Nr)evaluatemetricsscoreglobal)ziprrr)itemsmetric predictions referencesr referenceresultsrrrr.s    rcseZdZdZ d(deeddffdd Zd(deeee fddfdd Z d d Z d d Z ddZ ddZddZddZddZddZddZddZdeffdd Zd d!Zd"d#Zd$d%Zd&d'ZZS))UnitxtrNconfigreturncs\|duri}d|vsJdtjd|ji|dddt|_|jddd|_dS) Nrecipez(Unitxt task must have a 'recipe' string.version)metadata dataset_name)r&testrr)super__init__VERSIONdatasetsImage image_decoderdatasetr)selfr& __class__rrr.<s zUnitxt.__init__dataset_kwargscCs&tddlm}||jdd|_dS)Nr) load_datasetT) use_cache)rrr8 DATASET_NAMEr3)r4r7r8rrrdownloadLs zUnitxt.downloadcC d|jvSNtrainr3r4rrrhas_training_docsR zUnitxt.has_training_docscCr<N validationr?r@rrrhas_validation_docsUrBzUnitxt.has_validation_docscCr<Nr,r?r@rrr has_test_docsXrBzUnitxt.has_test_docscC |jdSr=r?r@rrr training_docs[rBzUnitxt.training_docscCrHrCr?r@rrrvalidation_docs^rBzUnitxt.validation_docscCrHrFr?r@rrr test_docsarBzUnitxt.test_docscC|dS)Nsourcerr4docrrr doc_to_textdzUnitxt.doc_to_textcCsdS)NFrr@rrrshould_decontaminategszUnitxt.should_decontaminatecCrL)NtargetrrNrrr doc_to_targetjrQzUnitxt.doc_to_targetcCs|ddgifS)Nuntil rr4rOctxrrr get_argumentsmszUnitxt.get_argumentsc sTt||tr|dr|d}|||}|Stdtjdd|i|S)Napply_chat_template chat_templateztGot chat template format from Unitxt, but apply_chat_template is false. Add '--apply_chat_template' to command line.rOr) isinstancerPlistgetrr-fewshot_context)r4rOkwargsr[formated_sourcer5rrr_ps  zUnitxt.fewshot_contextcKs:|dd|ddtdd||||dd|gS) aUses RequestFactory to construct Requests and returns an iterable of Requests which will be sent to the LM. :param doc: The document as returned from training_docs, validation_docs, or test_docs. :param ctx: str The context string, generated by fewshot_context. This includes the natural language description, as well as the few shot examples, and the question part of the document for `doc`. rZFr[generate_untilr) request_typerO argumentsidxNr)poprrY)r4rOrXr`rrrconstruct_requests}s  zUnitxt.construct_requestscs&|d}||fdd|jDS)aTake a single document and the LM results and evaluates, returning a dict where keys are the names of submetrics and values are the values of the metric for that one document :param doc: The document as returned from training_docs, validation_docs, or test_docs. :param results: The results of the requests created in construct_requests. rcsi|] }|ddfqS)metrics.replace.0r r!r"rr sz*Unitxt.process_results..r)r4rOr$ continuationrrnrprocess_resultss   zUnitxt.process_resultscCdd|jDS)z :returns: {str: [float] -> float} A dictionary where keys are the names of submetrics and values are functions that aggregate a list of metrics cSs"i|] }|ddtt|dqS)rhri)r )rkrrrlrrrrosz&Unitxt.aggregation..rpr@rrr aggregationszUnitxt.aggregationcCrs)z :returns: {str: bool} A dictionary where keys are the names of submetrics and values are whether a higher value of the submetric is better cSsi|] }|dddqS)rhriTrjrlrrrrosz+Unitxt.higher_is_better..rpr@rrrhigher_is_betterszUnitxt.higher_is_better)N)__name__ __module__ __qualname__r/rdictr.rstrrr;rArErGrIrJrKrPrRrTrYr_rgrrrtru __classcell__rrr5rr%9s0   r%z zrM)r}sub images_regexrNrrrrPszUnitxtMultiModal.doc_to_textcs t|d|}fdd|DS)NrMcsg|]}j|qSr)r2decode_example)rmimager@rr sz1UnitxtMultiModal.doc_to_image..)r)r4rOrrr@r doc_to_imageszUnitxtMultiModal.doc_to_imagecCs|ddgid||ifS)NrUrVvisual)rrWrrrrYszUnitxtMultiModal.get_argumentsN)rvrwrx MULTIMODALrPrrYrrrrrs  r)__doc__importlib.utilrr} functoolsrtypingrrrr0lm_eval.api.instancerlm_eval.api.taskr _CITATIONrrr%rrrrrrrrs"