o Ti@sddlmZmZddlmZddlmZddlmZddl m Z eGdddZ dd Z ed Z ed krie e ZedZd d ddZededZeje ejgddZejrkeeje jejdddSdSdS)) dataclassfield)Optional) load_dataset) ModelCard)HfArgumentParserc@s^eZdZUdZedddidZeed<edddidZe ed <ed dd idZ e e ed <d S) ScriptArgumentsa Arguments for the script. Args: push_to_hub (`bool`, *optional*, defaults to `False`): Whether to push the dataset to the Hugging Face Hub. repo_id (`str`, *optional*, defaults to `"trl-lib/tldr"`): Hugging Face repository ID to push the dataset to. dataset_num_proc (`int` or `None`, *optional*, defaults to `None`): Number of workers to use for dataset processing. Fhelpz4Whether to push the dataset to the Hugging Face Hub.)defaultmetadata push_to_hubz trl-lib/tldrz2Hugging Face repository ID to push the dataset to.repo_idNz0Number of workers to use for dataset processing.dataset_num_proc) __name__ __module__ __qualname____doc__rr bool__annotations__r strrrintrrJ/home/ubuntu/.local/lib/python3.10/site-packages/examples/datasets/tldr.pyrs  rcCs6d}|j|d|d|dd}d|d}||dS) Nz>SUBREDDIT: r/{subreddit} TITLE: {title} POST: {post} TL;DR: subreddittitlepost)rrr summary)prompt completion)format)exampletldr_format_strrrrrrto_prompt_completion3s  r#a3 --- tags: [trl] --- # TL;DR Dataset ## Summary The TL;DR dataset is a processed version of Reddit posts, specifically curated to train models using the [TRL library](https://github.com/huggingface/trl) for summarization tasks. It leverages the common practice on Reddit where users append "TL;DR" (Too Long; Didn't Read) summaries to lengthy posts, providing a rich source of paired text data for training summarization models. ## Data Structure - **Format**: [Standard](https://huggingface.co/docs/trl/main/dataset_formats#standard) - **Type**: [Prompt-completion](https://huggingface.co/docs/trl/main/dataset_formats#prompt-completion) Columns: - `"prompt"`: The unabridged Reddit post. - `"completion"`: The concise "TL;DR" summary appended by the author. This structure enables models to learn the relationship between detailed content and its abbreviated form, enhancing their summarization capabilities. ## Generation script The script used to generate this dataset can be found [here](https://github.com/huggingface/trl/blob/main/examples/datasets/tldr.py). __main__zghttps://openaipublic.blob.core.windows.net/summarize-from-feedback/datasets/tldr_3_filtered/train.jsonlzghttps://openaipublic.blob.core.windows.net/summarize-from-feedback/datasets/tldr_3_filtered/valid.jsonlzfhttps://openaipublic.blob.core.windows.net/summarize-from-feedback/datasets/tldr_3_filtered/test.jsonl)train validationtestjson) data_files)idrrrr)num_procremove_columnsdataset) repo_typeN) dataclassesrrtypingrdatasetsrhuggingface_hubr transformersrrr# model_cardrparserparse_args_into_dataclasses script_argsr)r-maprr r rrrrs6