o
Ni @ s d Z ddlmZ ddlmZ ddlmZ ddlZddlZddlm m Z
ddl m Z
dZdZdZd Zd
d d ZG d
d de
jjZG dd de
jjZdS )zScientific Papers Dataset. )absolute_import)division)print_functionNap
@article{Cohan_2018,
title={A Discourse-Aware Attention Model for Abstractive Summarization of
Long Documents},
url={http://dx.doi.org/10.18653/v1/n18-2097},
DOI={10.18653/v1/n18-2097},
journal={Proceedings of the 2018 Conference of the North American Chapter of
the Association for Computational Linguistics: Human Language
Technologies, Volume 2 (Short Papers)},
publisher={Association for Computational Linguistics},
author={Cohan, Arman and Dernoncourt, Franck and Kim, Doo Soon and Bui, Trung and Kim, Seokhwan and Chang, Walter and Goharian, Nazli},
year={2018}
}
a
Scientific papers datasets contains two sets of long and structured documents.
The datasets are obtained from ArXiv and PubMed OpenAccess repositories.
Both "arxiv" and "pubmed" have two features:
- article: the body of the document, pagragraphs seperated by "/n".
- abstract: the abstract of the document, pagragraphs seperated by "/n".
- section_names: titles of sections, seperated by "/n".
articleabstractzPhttps://drive.google.com/uc?id=1b3rmCSIoh6VhD4HKWjI4HOW-cSwcwbeC&export=downloadzPhttps://drive.google.com/uc?id=1lvsqvsFi3W-pE1SqNZI0s8NR9rC1tsja&export=download)arxivpubmedc s* e Zd ZdZejjd fdd Z ZS )ScientificPapersConfigz$BuilderConfig for Scientific Papers.Nc s8 t t| jdtjdtjdgd| || _dS )zBuilderConfig for Wikihow.
Args:
filename: filename of different configs for the dataset.
**kwargs: keyword arguments forwarded to super.
z1.1.1z1.1.0)versionsupported_versionsN )superr __init__tfdscoreVersionfilename)selfr kwargs __class__r g/home/ubuntu/.local/lib/python3.10/site-packages/tensorflow_datasets/summarization/scientific_papers.pyr F s
z ScientificPapersConfig.__init__N) __name__
__module__ __qualname____doc__r r disallow_positional_argsr
__classcell__r r r r r C s r c @ sB e Zd ZdZedddedddgZdd Zd d
Zdd d
Zd S )ScientificPaperszScientific Papers.r z Documents from ArXiv repository.)name descriptionr z!Documents from PubMed repository.c C sB t jj| tt jtt j tt j dt j ittfdt dS )N
section_namesz0https://github.com/armancohan/long-summarization)builderr! featuressupervised_keyshomepagecitation)
r r DatasetInfo _DESCRIPTIONr$ FeaturesDict _DOCUMENTText_SUMMARY _CITATION)r r r r _info` s
zScientificPapers._infoc C s | t}tj|| jj | jjd }tjj tj
j dtj|didtjj tj
j dtj|didtjj tj
j
dtj|didgS )zReturns SplitGenerators.z-datasetpathz train.txt)r
gen_kwargszval.txtztest.txt)download_and_extract_URLSosr0 joinbuilder_configr r r SplitGeneratorSplitTRAIN
VALIDATIONTEST)r
dl_managerdl_pathsr0 r r r _split_generatorsn s"
z"ScientificPapers._split_generatorsNc c s t jj|;}|D ]/}t|}d|d }|dddd}|d td|d t |dd|d ifV q W d d S 1 sFw Y d S )
zYields examples.
abstract_textz z
article_id article_textr" N)
tfiogfileGFilejsonloadsr5 replacer+ r- )r r0 flinedsummaryr r r _generate_examples s
"z#ScientificPapers._generate_examplesr ) r r r r r BUILDER_CONFIGSr/ r> rO r r r r r V s r )r
__future__r r r rH r4 tensorflow.compat.v2compatv2rD tensorflow_datasets.public_api
public_apir r. r) r+ r- r3 r
BuilderConfigr GeneratorBasedBuilderr r r r r s"