o Nim @sdZddlmZddlmZddlmZddlZddlZddlZddlZ ddl Z ddl m m ZddlmZdZdZdZd Zd Zed d d gddddedddZGdddejjZGdddejjZddZdS)?The General Language Understanding Evaluation (GLUE) benchmark.)absolute_import)division)print_functionNa@inproceedings{wang2019glue, title={{GLUE}: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding}, author={Wang, Alex and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel R.}, note={In the Proceedings of ICLR.}, year={2019} } Note that each GLUE dataset has its own citation. Please see the source to see the correct citation for each contained dataset.zGLUE, the General Language Understanding Evaluation benchmark (https://gluebenchmark.com/) is a collection of resources for training, evaluating, and analyzing natural language understanding systems. zhttps://firebasestorage.googleapis.com/v0/b/mtl-sentence-representations.appspot.com/o/data%2Fmrpc_dev_ids.tsv?alt=media&token=ec5c0836-31d5-48f4-b431-7480817f1adczNhttps://dl.fbaipublicfiles.com/senteval/senteval_data/msr_paraphrase_train.txtzMhttps://dl.fbaipublicfiles.com/senteval/senteval_data/msr_paraphrase_test.txt sentence1 sentence2premise hypothesis entailmentneutral contradiction gold_labelzhttps://firebasestorage.googleapis.com/v0/b/mtl-sentence-representations.appspot.com/o/data%2FMNLI.zip?alt=media&token=50329ea1-e339-40e2-809c-10c40afff3ceMNLIa @InProceedings{N18-1101, author = "Williams, Adina and Nangia, Nikita and Bowman, Samuel", title = "A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference", booktitle = "Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)", year = "2018", publisher = "Association for Computational Linguistics", pages = "1112--1122", location = "New Orleans, Louisiana", url = "http://aclweb.org/anthology/N18-1101" } @article{bowman2015large, title={A large annotated corpus for learning natural language inference}, author={Bowman, Samuel R and Angeli, Gabor and Potts, Christopher and Manning, Christopher D}, journal={arXiv preprint arXiv:1508.05326}, year={2015} }z,http://www.nyu.edu/projects/bowman/multinli/) text_features label_classes label_columndata_urldata_dircitationurlcs2eZdZdZejjdddffdd ZZS) GlueConfigzBuilderConfig for GLUE.NcCs|SN)xrrQ/home/ubuntu/.local/lib/python3.10/site-packages/tensorflow_datasets/text/glue.pygszGlueConfig.c  sXtt|jddtjddi| ||_||_||_||_ ||_ ||_ ||_ ||_ dS)aBuilderConfig for GLUE. Args: text_features: `dict[string, string]`, map from the name of the feature dict for each text field to the name of the column in the tsv file label_column: `string`, name of the column in the tsv file corresponding to the label data_url: `string`, url to download the zip file from data_dir: `string`, the path to the folder containing the tsv files in the downloaded zip citation: `string`, citation for the data set url: `string`, url for information about the data set label_classes: `list[string]`, the list of classes if the label is categorical. If not provided, then the label will be of type `tf.float32`. process_label: `Function[string, any]`, function taking in the raw value of the label and processing it to the form required by the label feature **kwargs: keyword arguments forwarded to super. versionz1.0.0z6New split API (https://tensorflow.org/datasets/splits)Nr)superr__init__tfdscoreVersionrrrrrrr process_label) selfrrrrrrrr$kwargs __class__rrr ^s  zGlueConfig.__init__) __name__ __module__ __qualname____doc__r!r"disallow_positional_argsr __classcell__rrr'rr[s rc@seZdZdZededddiddgddd ed d d ed edddiddgdddeddd ededdddddgdddeddd ed ed!d"d#d$d%d&gd'd(d)ed*d+d ed,ed-d.d/dd0d1d2ed3d4ejd5 edbd6ed7d8e edbd9ed:d8e edbd;ed<d8e ed=ed>d?dd@dAdBgddCdDedEdFd edGedHd.d/ddAdBgddIdJedKdLd edMedNd.d/ddBdAgddOdPedQdRd edSedTd.d/dUgdVddWdddXd g Z dYdZZ d[d\Z dcd^d_Z d`daZd]S)dGluercolaa The Corpus of Linguistic Acceptability consists of English acceptability judgments drawn from books and journal articles on linguistic theory. Each example is a sequence of words annotated with whether it is a grammatical English sentence.sentence unacceptable acceptable is_acceptablezhttps://firebasestorage.googleapis.com/v0/b/mtl-sentence-representations.appspot.com/o/data%2FCoLA.zip?alt=media&token=46d5e637-3411-4188-bc44-5809b5bfb5f4CoLAa @article{warstadt2018neural, title={Neural Network Acceptability Judgments}, author={Warstadt, Alex and Singh, Amanpreet and Bowman, Samuel R}, journal={arXiv preprint arXiv:1805.12471}, year={2018} }zhttps://nyu-mll.github.io/CoLA/) name descriptionrrrrrrrsst2a4 The Stanford Sentiment Treebank consists of sentences from movie reviews and human annotations of their sentiment. The task is to predict the sentiment of a given sentence. We use the two-way (positive/negative) class split, and use only sentence-level labels.negativepositivelabelzhttps://firebasestorage.googleapis.com/v0/b/mtl-sentence-representations.appspot.com/o/data%2FSST-2.zip?alt=media&token=aabc5f6b-e466-44a2-b9b4-cf6337f84ac8zSST-2a @inproceedings{socher2013recursive, title={Recursive deep models for semantic compositionality over a sentiment treebank}, author={Socher, Richard and Perelygin, Alex and Wu, Jean and Chuang, Jason and Manning, Christopher D and Ng, Andrew and Potts, Christopher}, booktitle={Proceedings of the 2013 conference on empirical methods in natural language processing}, pages={1631--1642}, year={2013} }z-https://nlp.stanford.edu/sentiment/index.htmlmrpca The Microsoft Research Paraphrase Corpus (Dolan & Brockett, 2005) is a corpus of sentence pairs automatically extracted from online news sources, with human annotations for whether the sentences in the pair are semantically equivalent.)rrnot_equivalent equivalentQualityMRPCaQ @inproceedings{dolan2005automatically, title={Automatically constructing a corpus of sentential paraphrases}, author={Dolan, William B and Brockett, Chris}, booktitle={Proceedings of the Third International Workshop on Paraphrasing (IWP2005)}, year={2005} }z>https://www.microsoft.com/en-us/download/details.aspx?id=52398qqpz The Quora Question Pairs2 dataset is a collection of question pairs from the community question-answering website Quora. The task is to determine whether a pair of questions are semantically equivalent. question1 question2)rCrD not_duplicate duplicate is_duplicatezhttps://firebasestorage.googleapis.com/v0/b/mtl-sentence-representations.appspot.com/o/data%2FQQP.zip?alt=media&token=700c6acf-160d-4d89-81d1-de4191d02cb5QQPaI @online{WinNT, author = {Iyer, Shankar and Dandekar, Nikhil and Csernai, Kornel}, title = {First Quora Dataset Release: Question Pairs}, year = 2017, url = {https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs}, urldate = {2019-04-03} }zAhttps://data.quora.com/First-Quora-Dataset-Release-Question-Pairsstsba* The Semantic Textual Similarity Benchmark (Cer et al., 2017) is a collection of sentence pairs drawn from news headlines, video and image captions, and natural language inference data. Each pair is human-annotated with a similarity score from 1 to 5.rrscorezhttps://firebasestorage.googleapis.com/v0/b/mtl-sentence-representations.appspot.com/o/data%2FSTS-B.zip?alt=media&token=bddb94a7-8706-4e0d-a694-1109e12273b5zSTS-Ban @article{cer2017semeval, title={Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation}, author={Cer, Daniel and Diab, Mona and Agirre, Eneko and Lopez-Gazpio, Inigo and Specia, Lucia}, journal={arXiv preprint arXiv:1708.00055}, year={2017} }z4http://ixa2.si.ehu.es/stswiki/index.php/STSbenchmark) r6r7rrrrrrr$mnlia) The Multi-Genre Natural Language Inference Corpus is a crowdsourced collection of sentence pairs with textual entailment annotations. Given a premise sentence and a hypothesis sentence, the task is to predict whether the premise entails the hypothesis (entailment), contradicts the hypothesis (contradiction), or neither (neutral). The premise sentences are gathered from ten different sources, including transcribed speech, fiction, and government reports. We use the standard test set, for which we obtained private labels from the authors, and evaluate on both the matched (in-domain) and mismatched (cross-domain) section. We also use and recommend the SNLI corpus as 550k examples of auxiliary training data.)r6r7mnli_mismatchedz The mismatched validation and test splits from MNLI. See the "mnli" BuilderConfig for additional information. mnli_matchedz~ The matched validation and test splits from MNLI. See the "mnli" BuilderConfig for additional information.qnlia The Stanford Question Answering Dataset is a question-answering dataset consisting of question-paragraph pairs, where one of the sentences in the paragraph (drawn from Wikipedia) contains the answer to the corresponding question (written by an annotator). We convert the task into sentence pair classification by forming a pair between each question and each sentence in the corresponding context, and filtering out pairs with low lexical overlap between the question and the context sentence. The task is to determine whether the context sentence contains the answer to the question. This modified version of the original task removes the requirement that the model select the exact answer, but also removes the simplifying assumptions that the answer is always present in the input and that lexical overlap is a reliable cue.question)rOr1r not_entailmentzhttps://firebasestorage.googleapis.com/v0/b/mtl-sentence-representations.appspot.com/o/data%2FQNLIv2.zip?alt=media&token=6fdcf570-0fc5-4631-8456-9505272d1601QNLIa? @article{rajpurkar2016squad, title={Squad: 100,000+ questions for machine comprehension of text}, author={Rajpurkar, Pranav and Zhang, Jian and Lopyrev, Konstantin and Liang, Percy}, journal={arXiv preprint arXiv:1606.05250}, year={2016} }z+https://rajpurkar.github.io/SQuAD-explorer/rtea The Recognizing Textual Entailment (RTE) datasets come from a series of annual textual entailment challenges. We combine the data from RTE1 (Dagan et al., 2006), RTE2 (Bar Haim et al., 2006), RTE3 (Giampiccolo et al., 2007), and RTE5 (Bentivogli et al., 2009).4 Examples are constructed based on news and Wikipedia text. We convert all datasets to a two-class split, where for three-class datasets we collapse neutral and contradiction into not entailment, for consistency.zhttps://firebasestorage.googleapis.com/v0/b/mtl-sentence-representations.appspot.com/o/data%2FRTE.zip?alt=media&token=5efa7e85-a0bb-4f19-8ea2-9e1840f077fbRTEa @inproceedings{dagan2005pascal, title={The PASCAL recognising textual entailment challenge}, author={Dagan, Ido and Glickman, Oren and Magnini, Bernardo}, booktitle={Machine Learning Challenges Workshop}, pages={177--190}, year={2005}, organization={Springer} } @inproceedings{bar2006second, title={The second pascal recognising textual entailment challenge}, author={Bar-Haim, Roy and Dagan, Ido and Dolan, Bill and Ferro, Lisa and Giampiccolo, Danilo and Magnini, Bernardo and Szpektor, Idan}, booktitle={Proceedings of the second PASCAL challenges workshop on recognising textual entailment}, volume={6}, number={1}, pages={6--4}, year={2006}, organization={Venice} } @inproceedings{giampiccolo2007third, title={The third pascal recognizing textual entailment challenge}, author={Giampiccolo, Danilo and Magnini, Bernardo and Dagan, Ido and Dolan, Bill}, booktitle={Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing}, pages={1--9}, year={2007}, organization={Association for Computational Linguistics} } @inproceedings{bentivogli2009fifth, title={The Fifth PASCAL Recognizing Textual Entailment Challenge.}, author={Bentivogli, Luisa and Clark, Peter and Dagan, Ido and Giampiccolo, Danilo}, booktitle={TAC}, year={2009} }z9https://aclweb.org/aclwiki/Recognizing_Textual_Entailmentwnlia, The Winograd Schema Challenge (Levesque et al., 2011) is a reading comprehension task in which a system must read a sentence with a pronoun and select the referent of that pronoun from a list of choices. The examples are manually constructed to foil simple statistical methods: Each one is contingent on contextual information provided by a single word or phrase in the sentence. To convert the problem into sentence pair classification, we construct sentence pairs by replacing the ambiguous pronoun with each possible referent. The task is to predict if the sentence with the pronoun substituted is entailed by the original sentence. We use a small evaluation set consisting of new examples derived from fiction books that was shared privately by the authors of the original corpus. While the included training set is balanced between two classes, the test set is imbalanced between them (65% not entailment). Also, due to a data quirk, the development set is adversarial: hypotheses are sometimes shared between training and development examples, so if a model memorizes the training examples, they will predict the wrong label on corresponding development set example. As with QNLI, each example is evaluated separately, so there is not a systematic correspondence between a model's score on this task and its score on the unconverted original task. We call converted dataset WNLI (Winograd NLI).zhttps://firebasestorage.googleapis.com/v0/b/mtl-sentence-representations.appspot.com/o/data%2FWNLI.zip?alt=media&token=068ad0a0-ded7-4bd7-99a5-5e00222e0fafWNLIaZ @inproceedings{levesque2012winograd, title={The winograd schema challenge}, author={Levesque, Hector and Davis, Ernest and Morgenstern, Leora}, booktitle={Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning}, year={2012} }z@https://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WS.htmlaxae A manually-curated evaluation dataset for fine-grained analysis of system performance on a broad range of linguistic phenomena. This dataset evaluates sentence understanding through Natural Language Inference (NLI) problems. Use a model trained on MulitNLI to produce predictions for this dataset.rr zhttps://bit.ly/2BOtOJ7z%https://gluebenchmark.com/diagnosticscCszddt|jjD}|jjrtjj|jjd|d<ntj |d<tj |d<tj j |t tj||jj|jjdtdS)NcSsi|]}|tjqSr)r!featuresText).0 text_featurerrr s zGlue._info..)namesr;idx )builderr7rWhomepager)sixiterkeysbuilder_configrrr!rW ClassLabeltffloat32int32r" DatasetInfo_GLUE_DESCRIPTION FeaturesDictrr_GLUE_CITATION)r%rWrrr_infos     z Glue._infoc Cs|jjdkr||jj}tjjtjj|dddgS|jjdkr-d}|t t t d}n| |jj}t j||jj}d}tjjtjjt j|pKddd |d d}|jjd kry|td |d ddtd|d ddtd|dddtd|dddgS|jjdkrtd|d ddtd|dddgS|jjdkrtd|d ddtd|dddgS|tjjtjjt j|pddd |d dtjjtjjt j|pddd|d dgS)NrVtest) data_filesplitr6 gen_kwargsr<)dev_idstrainrmr=z train.tsvrsrnro mrpc_filesrKvalidation_matcheddevT)matchedvalidation_mismatchedF test_matchedtest_mismatchedrM validationrLzdev.tsvztest.tsv)rcr6downloadrr!r"SplitGeneratorSplitTEST _MRPC_DEV_IDS _MRPC_TRAIN _MRPC_TESTdownload_and_extractospathjoinrTRAIN_mnli_split_generator VALIDATION)r% dl_managerrnrrudl_dir train_splitrrr_split_generatorss~      zGlue._split_generatorsNc#s||jjdkr|j||d}|D] }|d|fVqdS|jj}|jj}|jjdko-|dk}tjj|} t j | dt j d} |rJt j | dt j d} t | D]]\} |r]dd d fd d t|jjD}| |d<|jjvr|jj} |r| |vr| rt| nd} || |d <n|d|d <t|D]} | durnq|d|fVqNWddS1swYdS)Nr<)ruror]r0rm  delimiterquoting)r1r4csi|] \}}||qSrr)rYfeatcolrowrrr[sz+Glue._generate_examples..r;)rcr6_generate_example_mrpc_filesr$rreiogfileGFilecsv DictReader QUOTE_NONEreader enumeratera iteritemsrrint itervalues)r%rnroruexamplesexampler$ris_cola_non_testfrnr;valuerrr_generate_examplessL       "zGlue._generate_examplesc csv|dkr?tjj|d'}tj|dtjd}t|D]\}}|d|dd|dVqWddS1s8wYdStjj|d}tj|dtjd}d d |D}Wdn1scwYtjj|d @}| d tj|dtjd}t|D]$\}}|d |dg|v}||dkkr|d|dt |d|dVqWddS1swYdS)Nrmrrz #1 Stringz #2 Stringr)rrr;r]rrcSsg|] }|d|dgqS)rrr)rYrrrr 6sz5Glue._generate_example_mrpc_files..rsrz#1 IDz#2 IDrwr@) rerrrrrrrrseekr) r%rurorrrrrr is_row_in_devrrrr(s< "    "z!Glue._generate_example_mrpc_filesrr)r)r*r+r,rtextwrapdedentnprf_MNLI_BASE_KWARGSBUILDER_CONFIGSrlrrrrrrrr/s   !3# F 2r/cCs2tjj|tj|d||rdndf|dddS)Nz %s_%s.tsvrx mismatchedrtrp)r!r"r~rrr)r6rrorxrrrrGsr) r, __future__rrrrrrnumpyrratensorflow.compat.v2compatv2retensorflow_datasets.public_api public_apir!rkrirrrdictrrr" BuilderConfigrGeneratorBasedBuilderr/rrrrrsB     $0 ?