o Ti@sddlZddlmZmZddlmZddlmZddlm Z ddl m Z ddl m Z eGdd d Zd d Ze d Zed krte eZedZe dddZejegdejdZejdddZejrveejejejdddSdSdS)N) dataclassfield)chain)Optional) load_dataset) ModelCard)HfArgumentParserc@s^eZdZUdZedddidZeed<edddidZe ed <ed dd idZ e e ed <d S) ScriptArgumentsa Arguments for the script. Args: push_to_hub (`bool`, *optional*, defaults to `False`): Whether to push the dataset to the Hugging Face Hub. repo_id (`str`, *optional*, defaults to `"trl-lib/math_shepherd"`): Hugging Face repository ID to push the dataset to. dataset_num_proc (`int` or `None`, *optional*, defaults to `None`): Number of workers to use for dataset processing. Fhelpz4Whether to push the dataset to the Hugging Face Hub.)defaultmetadata push_to_hubztrl-lib/math_shepherdz2Hugging Face repository ID to push the dataset to.repo_idNz0Number of workers to use for dataset processing.dataset_num_proc) __name__ __module__ __qualname____doc__rr bool__annotations__rstrrrintrrS/home/ubuntu/.local/lib/python3.10/site-packages/examples/datasets/math_shepherd.pyr s  r csdddddtdD}tfdd|Ds Jfdd|D}fd dttd g|t|dgD}|dd }|d }|d d}td d|DsXJdd|D}|drk|d d}nLd|vr|d\}}d|}|g|}n5d|vr|d\}}d|}|g|}nd|vr|d\}}|d}|g|}nt d||}t |t |ksJ|||dS)NinputuкиⶻcSsg|]}|qSr)start).0mrrr :sz#process_example..c3s |] }d|dvVqdS)label)+-Nrridxexamplerr =sz"process_example..csg|] }d|dkqS)r r!rr#r%rrr@csg|] \}}||qSrr)rij)inputsrrrCr(rcss|]}|dVqdS)rN) startswithr completionrrrr'MscSsg|] }|ddqS)r-N)stripr/rrrrNr() z Mr. RockyParkerzWhat is the smallest positivez The MythzLet $\mathbf{a}$zFind the arithmeticzDetermine an ordered pairzDetermine the ordered pairz"At the Quill and Scroll stationeryzRound to the nearestzCalculate $\sqrt{10p}zSimplify $\sqrt{28x}zStep 1:zstep 1:?zPrompt can't be processed: )prompt completionslabels) replacerefinditerallziprr.splitr1 ValueErrorlen)r&indexesr6stepsr4r5 first_stepr)r&r+rprocess_example5s<(   rBa --- tags: [trl] --- # Math-Shepherd Dataset ## Summary The Math-Shepherd dataset is a processed version of [Math-Shepherd dataset](peiyi9979/Math-Shepherd), designed to train models using the [TRL library](https://github.com/huggingface/trl) for stepwise supervision tasks. It provides step-by-step solutions to mathematical problems, enabling models to learn and verify each step of a solution, thereby enhancing their reasoning capabilities. ## Data Structure - **Format**: [Standard](https://huggingface.co/docs/trl/main/dataset_formats#standard) - **Type**: [Stepwise supervision](https://huggingface.co/docs/trl/main/dataset_formats#stepwise-supervision) Columns: - `"prompt"`: The problem statement. - `"completions"`: A list of reasoning steps generated to solve the problem. - `"labels"`: A list of booleans or floats indicating the correctness of each corresponding reasoning step. This structure allows models to learn the correctness of each step in a solution, facilitating improved reasoning and problem-solving abilities. ## Generation script The script used to generate this dataset can be found [here](https://github.com/huggingface/trl/blob/main/examples/datasets/math_shepherd.py). __main__zpeiyi9979/Math-Shepherdtrain)r<)rr task)remove_columnsnum_procg?*) test_sizeseeddataset) repo_type)r8 dataclassesrr itertoolsrtypingrdatasetsrhuggingface_hubr transformersrr rB model_cardrparserparse_args_into_dataclasses script_argsrKmaprtrain_test_splitr rrrrrs4     J