o .iB@sdZddlZddlmZmZddlmZmZddlm Z ddl m Z ddl m Z dd lmZdd lmZmZmZdd lmZdd lmZeeZGd ddeZ     d"dedeeeeefdee deee efdeeeefdeeeeff ddZ     d"dedeeeefdee deee efdeedeeeeeff ddZ!     d"dedeeeefdee deee efdeedeeeeefdeefddZ"      d#dedeedeeeeeeeeeeefffdee deee efdeeeefdeeeefdefddZ#      d#dedeedeeeeeeeeeeefffdee deee efdeeeefdeeeeffd d!Z$dS)$zList and inspect datasets.N)MappingSequence)OptionalUnion)DownloadConfig) DownloadMode)StreamingDownloadManager) DatasetInfo)dataset_module_factoryget_dataset_builder_classload_dataset_builder) get_logger)Versionc@s eZdZdS)SplitsNotFoundErrorN)__name__ __module__ __qualname__rrD/home/ubuntu/.local/lib/python3.10/site-packages/datasets/inspect.pyr&srpath data_filesdownload_config download_moderevisiontokenc s2td}fdd|DS)aGet the meta information about a dataset, returned as a dict mapping config name to DatasetInfoDict. Args: path (`str`): path to the dataset processing script with the dataset builder. Can be either: - a local path to processing script or the directory containing the script (if the script has the same name as the directory), e.g. `'./dataset/squad'` or `'./dataset/squad/squad.py'` - a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`huggingface_hub.list_datasets`]), e.g. `'rajpurkar/squad'`, `'nyu-mll/glue'` or``'openai/webtext'` revision (`Union[str, datasets.Version]`, *optional*): If specified, the dataset module will be loaded from the datasets repository at this version. By default: - it is set to the local version of the lib. - it will also try to load it from the main branch if it's not available at the local version of the lib. Specifying a version that is different from your local version of the lib might cause compatibility issues. download_config ([`DownloadConfig`], *optional*): Specific download configuration parameters. download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`): Download/generate mode. data_files (`Union[Dict, List, str]`, *optional*): Defining the data_files of the dataset configuration. token (`str` or `bool`, *optional*): Optional string or boolean to use as Bearer token for remote files on the Datasets Hub. If `True`, or not specified, will get token from `"~/.huggingface"`. **config_kwargs (additional keyword arguments): Optional attributes for builder class which will override the attributes if supplied. Example: ```py >>> from datasets import get_dataset_infos >>> get_dataset_infos('cornell-movie-review-data/rotten_tomatoes') {'default': DatasetInfo(description="Movie Review Dataset. This is a dataset of containing 5,331 positive and 5,331 negative processed sentences from Rotten Tomatoes movie reviews...), ...} ``` )rrrrrrc s,i|]}|td|dqS))r config_namerrrrrr)get_dataset_config_info).0r config_kwargsrrrrrrrr ^s z%get_dataset_infos..)get_dataset_config_names)rrrrrrr config_namesrrrget_dataset_infos*s, r$dynamic_modules_pathc KsTt|f|||||d|}t|tj|d}t|jp)|j d|j p'dgS)a Get the list of available config names for a particular dataset. Args: path (`str`): path to the dataset processing script with the dataset builder. Can be either: - a local path to processing script or the directory containing the script (if the script has the same name as the directory), e.g. `'./dataset/squad'` or `'./dataset/squad/squad.py'` - a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`huggingface_hub.list_datasets`]), e.g. `'rajpurkar/squad'`, `'nyu-mll/glue'` or `'openai/webtext'` revision (`Union[str, datasets.Version]`, *optional*): If specified, the dataset module will be loaded from the datasets repository at this version. By default: - it is set to the local version of the lib. - it will also try to load it from the main branch if it's not available at the local version of the lib. Specifying a version that is different from your local version of the lib might cause compatibility issues. download_config ([`DownloadConfig`], *optional*): Specific download configuration parameters. download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`): Download/generate mode. dynamic_modules_path (`str`, defaults to `~/.cache/huggingface/modules/datasets_modules`): Optional path to the directory in which the dynamic modules are saved. It must have been initialized with `init_dynamic_modules`. By default the datasets are stored inside the `datasets_modules` module. data_files (`Union[Dict, List, str]`, *optional*): Defining the data_files of the dataset configuration. **download_kwargs (additional keyword arguments): Optional attributes for [`DownloadConfig`] which will override the attributes in `download_config` if supplied, for example `token`. Example: ```py >>> from datasets import get_dataset_config_names >>> get_dataset_config_names("nyu-mll/glue") ['cola', 'sst2', 'mrpc', 'qqp', 'stsb', 'mnli', 'mnli_mismatched', 'mnli_matched', 'qnli', 'rte', 'wnli', 'ax'] ``` rrrr%r dataset_namerdefault) r r osrbasenamelistbuilder_configskeysbuilder_kwargsgetDEFAULT_CONFIG_NAME) rrrrr%rdownload_kwargsdataset_module builder_clsrrrr"ms8 r"returnc Ksjt|f|||||d|}t|tj|d}t|j} | r.t| dkr+| dnd} nd} |j p4| S)a Get the default config name for a particular dataset. Can return None only if the dataset has multiple configurations and no default configuration. Args: path (`str`): path to the dataset processing script with the dataset builder. Can be either: - a local path to processing script or the directory containing the script (if the script has the same name as the directory), e.g. `'./dataset/squad'` or `'./dataset/squad/squad.py'` - a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`huggingface_hub.list_datasets`]), e.g. `'rajpurkar/squad'`, `'nyu-mll/glue'` or `'openai/webtext'` revision (`Union[str, datasets.Version]`, *optional*): If specified, the dataset module will be loaded from the datasets repository at this version. By default: - it is set to the local version of the lib. - it will also try to load it from the main branch if it's not available at the local version of the lib. Specifying a version that is different from your local version of the lib might cause compatibility issues. download_config ([`DownloadConfig`], *optional*): Specific download configuration parameters. download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`): Download/generate mode. dynamic_modules_path (`str`, defaults to `~/.cache/huggingface/modules/datasets_modules`): Optional path to the directory in which the dynamic modules are saved. It must have been initialized with `init_dynamic_modules`. By default the datasets are stored inside the `datasets_modules` module. data_files (`Union[Dict, List, str]`, *optional*): Defining the data_files of the dataset configuration. **download_kwargs (additional keyword arguments): Optional attributes for [`DownloadConfig`] which will override the attributes in `download_config` if supplied, for example `token`. Returns: Optional[str]: the default config name if there is one Example: ```py >>> from datasets import get_dataset_default_config_name >>> get_dataset_default_config_name("openbookqa") 'main' ``` r&r'rrNr)) r r r*rr+r,r-r.lenr1) rrrrr%rr2r3r4r-default_config_namerrrget_dataset_default_config_names"1  r8rc stf||||||d|}|j} | jdurX|r|nt}|dur'||_|t|j|dzfdd| t|j|dD| _W| St yW} zt d| d} ~ ww| S)aGet the meta information (DatasetInfo) about a dataset for a particular config Args: path (``str``): path to the dataset processing script with the dataset builder. Can be either: - a local path to processing script or the directory containing the script (if the script has the same name as the directory), e.g. ``'./dataset/squad'`` or ``'./dataset/squad/squad.py'`` - a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`huggingface_hub.list_datasets`]), e.g. ``'rajpurkar/squad'``, ``'nyu-mll/glue'`` or ``'openai/webtext'`` config_name (:obj:`str`, optional): Defining the name of the dataset configuration. data_files (:obj:`str` or :obj:`Sequence` or :obj:`Mapping`, optional): Path(s) to source data file(s). download_config (:class:`~download.DownloadConfig`, optional): Specific download configuration parameters. download_mode (:class:`DownloadMode` or :obj:`str`, default ``REUSE_DATASET_IF_EXISTS``): Download/generate mode. revision (:class:`~utils.Version` or :obj:`str`, optional): Version of the dataset script to load. As datasets have their own git repository on the Datasets Hub, the default version "main" corresponds to their "main" branch. You can specify a different version than the default "main" by using a commit SHA or a git tag of the dataset repository. token (``str`` or :obj:`bool`, optional): Optional string or boolean to use as Bearer token for remote files on the Datasets Hub. If True, or not specified, will get token from `"~/.huggingface"`. **config_kwargs (additional keyword arguments): optional attributes for builder class which will override the attributes if supplied. )namerrrrrN) base_pathrcsi|] }|j|jdqS))r9r()r9)rsplit_generatorrrrr!)sz+get_dataset_config_info..z>> from datasets import get_dataset_split_names >>> get_dataset_split_names('cornell-movie-review-data/rotten_tomatoes') ['train', 'validation', 'test'] ``` )rrrrrr)rr,r>r.) rrrrrrrr r=rrrget_dataset_split_names4s- rE)NNNNN)NNNNNN)%__doc__r*collections.abcrrtypingrrdownload.download_configrdownload.download_managerr#download.streaming_download_managerr r=r loadr r r utils.loggingr utils.versionrrlogger ValueErrorrstrdictr,boolr$r"r8rrErrrrs       E I E( ?(