o NiO@sdZddlmZddlmZddlmZddlZddlZddlmm Z ddl m Z dZdZdZd ZGd d d e jjZd Zd ZGddde jjZdS)z)SCAN tasks with various different splits.)absolute_import)division)print_functionNai @inproceedings{Lake2018GeneralizationWS, title={Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks}, author={Brenden M. Lake and Marco Baroni}, booktitle={ICML}, year={2018}, url={https://arxiv.org/pdf/1711.00350.pdf}, } @inproceedings{Keysers2020, title={Measuring Compositional Generalization: A Comprehensive Method on Realistic Data}, author={Daniel Keysers and Nathanael Sch"{a}rli and Nathan Scales and Hylke Buisman and Daniel Furrer and Sergii Kashubin and Nikola Momchev and Danila Sinopalnikov and Lukasz Stafiniak and Tibor Tihon and Dmitry Tsarkov and Xiao Wang and Marc van Zee and Olivier Bousquet}, note={Additional citation for MCD splits}, booktitle={ICLR}, year={2020}, url={https://arxiv.org/abs/1912.09713.pdf}, } a!SCAN tasks with various splits. SCAN is a set of simple language-driven navigation tasks for studying compositional learning and zero-shot generalization. Most splits are described at https://github.com/brendenlake/SCAN. For the MCD splits please see https://arxiv.org/abs/1912.09713.pdf. Basic usage: ``` data = tfds.load('scan/length') ``` More advanced example: ``` data = tfds.load( 'scan', builder_kwargs=dict( config=tfds.text.ScanConfig( name='simple_p8', directory='simple_split/size_variations'))) ``` z6https://github.com/brendenlake/SCAN/archive/master.zipz=https://storage.googleapis.com/cfq_dataset/scan-splits.tar.gzcs*eZdZdZejjdfdd ZZS) ScanConfigaBuilderConfig for SCAN. Splits can be read in two formats: 1) As a pair of train and test files where each file contains one example input and output per line. 2) With a 'splitfile' which contains for each split the indices into the full (unsplit) dataset. Nc svtt|jd|tjdtd|||_d|vr|d|_|jr+|dur+d|_dS|dur6|d|_dS||_dS) aQBuilderConfig for SCAN. Args: name: Unique name of the split. directory: Which subdirectory to read the data files from. splitfile: If set the samples are read from the original archive (tasks.txt) but the splits are created using this index file. **kwargs: keyword arguments forwarded to super. z1.1.1)nameversion descriptionmcdz.jsonN_split) superr__init__tfdscoreVersion _DESCRIPTION splitfile directory)selfrrrkwargs __class__r Q/home/ubuntu/.local/lib/python3.10/site-packages/tensorflow_datasets/text/scan.pyr]s     zScanConfig.__init__NN) __name__ __module__ __qualname____doc__rrdisallow_positional_argsr __classcell__r r rrrRs rcommandsactionsc@seZdZdZeddedddedddedd ded d ded d ded d ded dedddedddedddedddeddeddeddgZddZddZddZdddZ dS) ScanzCSCAN task / splits as proposed by Brenden M. Lake and Marco Baroni.simple)r addprim_jumpadd_prim_split)rraddprim_turn_left filler_num0 filler_split filler_num1 filler_num2 filler_num3lengthtemplate_around_righttemplate_splittemplate_jump_around_righttemplate_opposite_righttemplate_rightmcd1mcd2mcd3c Cs8tjj|ttjttjttjittfdt dS)Nz#https://github.com/brendenlake/SCAN)builderrfeaturessupervised_keyshomepagecitation) rr DatasetInforr7 FeaturesDict _COMMANDSText_ACTIONS _CITATION)rr r r_infos  z Scan._infoc Cs|tjjttjjjd}tj |d|j j }|j j }|j j }d|vr7|t}tj |d}tj ||}|rTtj |d|d}|}d|d<|}d |d<nd tj |d |d i}d tj |d |d i}tjjtjj|dtjjtjj|dgS)zReturns SplitGenerators.)urlextract_methodz SCAN-masterr z scan-splitsz tasks.txt)datapath splitpathtrain splitnametestrD tasks_train_z.txt tasks_test_)r gen_kwargs)download_and_extractrdownloadResource _DATA_URL ExtractMethodZIPospathjoinbuilder_configrrr_MCD_SPLITS_URLcopyrSplitGeneratorSplitTRAINTEST) r dl_managerdata_dirsplitr split_dirr train_kwargs test_kwargsr r r_split_generatorss@    zScan._split_generatorsccstjj|1}t|D]#\}}|dsq |tdddd\}}|t |t |ifVq WddS1ss"    '