o ॵi}@sddlZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZddlmZddlZddlmZmZmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.ddl/m0Z0m1Z1m2Z2dd l3m4Z4m5Z5dd l6m7Z7dd l8m9Z9m:Z:m;Z;mZ>m?Z?m@Z@mAZAmBZBmCZCmDZDmEZEmFZFmGZGmHZHmIZImJZJmKZKmLZLdd lMmNZNdd lOmPZPmQZQmRZRmSZSddlTmUZUddlVmWZWmXZXmYZYmZZZm[Z[m\Z\ddl]m^Z^ddl_m`Z`ddlambZbddlcmdZdddlemfZfddlgmhZhddlimjZjddlkmlZlmmZmddlnmoZpddlnmqZqmrZrmsZsddltmuZuddlvmwZwddlxmyZyddlzm{Z{ddl|m}Z}ddl~mZdd lmZeZed!Zd"ed#ed$efd%d&Zddd'ddd(d)ed*e ed+e ed,ed-e eeefd.e e ed$epfd/d0Z dSd1d'dddd2d)ed3e ed4ed.ed*e ed5e ed-e eeefd$e eeresffd6d7Zd'dddd8d)ed9ee eefd.ed*e ed5e ed-e eeefd$e eeresffd:d;Z  dTded?ed@e e ed#e ed$e ef dAdBZ dSd?ed#e ed$e ee effdCdDZd$e Raises the following errors: - [`~utils.RepositoryNotFoundError`] If the repository to download from cannot be found. This may be because it doesn't exist, or because it is set to `private` and you do not have access. - [`~utils.RevisionNotFoundError`] If the revision to download from cannot be found. rv dataset_name namespacerw)dataset_hub_idrrrDataFilesidFrlrNr`rnrirhrdrfrjrorarcrmrIdSizeTypetree directoryfileSha256)sizesha256 pointerSize) rfilenameblobIdrtypelfsr)rZrget_dataset_id_and_typeget_dataset_infosappend HfDatasetInfo)rrrrrrr_api _namespace _dataset_namer dataset_typedatadata_ddata_file_list file_info_d file_inforrr _dataset_infopsL0  rT) recursiverr repo_typer path_in_reporrc cszt}t|r|ddkr|d\} } n$t|r3|ddkr3td|dtt|} } ntd|dd} d} g} |j| | |pId |pLdd | | d }d |vr\|d d kskt d|d|ddS|dd}| ||D]7}i}|ddkrdnd|d<|d|d<|d|d<|d|d<|ddkrt di|nt di|Vqxt || krdS| d7} qB)NrvrtrzGot a relative path: z+ without namespace, Use default namespace: zInvalid repo_id: z !dTrw)rrr root_pathr page_number page_sizeCodez Get dataset: z file list failed, message: Messagerrrrrrrrpathrrroidr)rZrJcountrloggerwarningr^ ValueErrorlist_repo_treeerrorextendrWrXlen)rrrrrrrrrrrrrtotal_data_listrrr path_inforrr_list_repo_treesL        , r)rrrrpathscCs^t}|d\}} |j| |d\} } |pd}|j| |ddd} | d} | d}d d |DS) NrvrrwFFalse)rrrrrrc Ss8g|]}|ddkrt|d|d|dddddqS)Name README.mdrrzN)rrrr last_commitsecurity)rW).0item_drrr "sz#_get_paths_info..)rZrrr)rrrrrrrrrrrrrrrrrr_get_paths_info s   rcCst|tttfr|stdtt|d}nt|}|pi}|r%||d<t||p*i}i}tt|D].\}}|\}} } |t |dkrOt di| |}q4t di| ||d<| |d<||d<q4|d\} }} t |fi|} | S) Nzempty urlpath sequencerprotocolrttarget_optionstarget_protocolfor) isinstancelisttuplesetrrRrQ enumeratereversedrdictrP)urlpathstorage_optionsrurlpath0chaininkwargsichurlsnested_protocolkwr_fsrrrget_fs_token_paths-s*  rpattern base_pathallowed_extensionsc st|r t||}nt|rtj|dtj}nd}t||d\}}t||d}| dd ddp8|j | dd ddt t t |ht|jtrW|jn|jd}|dkrd|dndi}|d krxtjtd krxd |d <|j|fd di|}fdd|D} durfdd| D} t| t| krtt | t | } td|d| n| } | sd|d} dur| dt7} t| | S)a Resolve the paths and URLs of the data files from the pattern passed by the user. You can use patterns to resolve multiple local files. Here are a few examples: - *.csv to match all the CSV files at the first level - **.csv to match all the CSV files at any level - data/* to match all the files inside "data" - data/** to match all the files inside "data" and its subdirectories The patterns are resolved using the fsspec glob. glob.glob, Path.glob, Path.match or fnmatch do not support ** with a prefix/suffix other than a forward slash /. For instance, this means **.json is the same as *.json. On the contrary, the fsspec glob has no limits regarding the ** prefix/suffix, # noqa: E501 resulting in **.json being equivalent to **/*.json. More generally: - '*' matches any character except a forward-slash (to match just the file or directory name) - '**' matches any character including a forward-slash / Hidden files and directories (i.e. whose names start with a dot) are ignored, unless they are explicitly requested. The same applies to special directories that start with a double underscore like "__pycache__". You can still include one if the pattern explicilty mentions it: - to include a hidden file: "*/.hidden.txt" or "*/.*" - to include a hidden directory: ".hidden/*" or ".*/*" - to include a special directory: "__special__/*" or "__*/*" Example:: >>> from datasets.data_files import resolve_pattern >>> base_path = "." >>> resolve_pattern("docs/**/*.py", base_path) [/Users/mariosasko/Desktop/projects/datasets/docs/source/_config.py'] Args: pattern (str): Unix pattern or paths or URLs of the data files to resolve. The paths can be absolute or relative to base_path. Remote filesystems using fsspec are supported, e.g. with the hf:// protocol. base_path (str): Base path to use when resolving relative paths. allowed_extensions (Optional[list], optional): White-list of file extensions to use. Defaults to None (all extensions). For example: allowed_extensions=[".csv", ".json", ".txt", ".parquet"] Returns: List[str]: List of paths or URLs to the local or remote files that match the patterns. rr|)rz::z://rurhfz0.20.0F expand_infodetailTcszg|]9\}}|ddkr;t|vrttj|tjsttj|tjs|r7|n|qS)rr)r'r"osrrelpathr#r~)rfilepathinfo)files_to_ignore fs_base_path fs_patternprotocol_prefixrrrs$      z$_resolve_pattern..Ncs8g|]}tfddt|dddDr|qS)c3s|] }d|vVqdS).Nr)rsuffixrrr sz._resolve_pattern...rrtN)anyr'r)rrrrrrs z Some files matched the pattern 'z-' but don't have valid data file extensions: zUnable to find ''z with any supported extension )rJr(rIrr splitdrivesepr&rr root_markerrrr'rrr}rHF_HUB_VERSIONrYparseglobitemsrrrrFileNotFoundError) rrrrqrrr glob_kwargstmp_file_paths matched_pathsrinvalid_matched_files error_msgr)rrrrrr_resolve_patternLsP1    r cCs:tt||d}zt|WStytd|ddw)u Get the default pattern from a directory testing all the supported patterns. The first patterns to return a non-empty list of data files is returned. Some examples of supported patterns: Input: my_dataset_repository/ ├── README.md └── dataset.csv Output: {"train": ["**"]} Input: my_dataset_repository/ ├── README.md ├── train.csv └── test.csv my_dataset_repository/ ├── README.md └── data/ ├── train.csv └── test.csv my_dataset_repository/ ├── README.md ├── train_0.csv ├── train_1.csv ├── train_2.csv ├── train_3.csv ├── test_0.csv └── test_1.csv Output: {'train': ['train[-._ 0-9/]**', '**/*[-._ 0-9/]train[-._ 0-9/]**', 'training[-._ 0-9/]**', '**/*[-._ 0-9/]training[-._ 0-9/]**'], 'test': ['test[-._ 0-9/]**', '**/*[-._ 0-9/]test[-._ 0-9/]**', 'testing[-._ 0-9/]**', '**/*[-._ 0-9/]testing[-._ 0-9/]**', ...]} Input: my_dataset_repository/ ├── README.md └── data/ ├── train/ │ ├── shard_0.csv │ ├── shard_1.csv │ ├── shard_2.csv │ └── shard_3.csv └── test/ ├── shard_0.csv └── shard_1.csv Output: {'train': ['train[-._ 0-9/]**', '**/*[-._ 0-9/]train[-._ 0-9/]**', 'training[-._ 0-9/]**', '**/*[-._ 0-9/]training[-._ 0-9/]**'], 'test': ['test[-._ 0-9/]**', '**/*[-._ 0-9/]test[-._ 0-9/]**', 'testing[-._ 0-9/]**', '**/*[-._ 0-9/]testing[-._ 0-9/]**', ...]} Input: my_dataset_repository/ ├── README.md └── data/ ├── train-00000-of-00003.csv ├── train-00001-of-00003.csv ├── train-00002-of-00003.csv ├── test-00000-of-00001.csv ├── random-00000-of-00003.csv ├── random-00001-of-00003.csv └── random-00002-of-00003.csv Output: {'train': ['data/train-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*'], 'test': ['data/test-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*'], 'random': ['data/random-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*']} In order, it first tests if SPLIT_PATTERN_SHARDED works, otherwise it tests the patterns in ALL_DEFAULT_PATTERNS. )rrqzThe directory at z doesn't contain any data filesN)rr r!rr )rrqresolverrrr_get_data_patternss[   rc st}|j}|d\}}|jpd}d|jd|d|jpdd}|j}|jdur1d|_z|j d|||dd }t ||d } t t | j} Wn tyWt} Ynw|jd d} t| } t| } |jdurtt|j}n-| rd tt| vr| dur| | d }n tt| d }t|}nt||jd }tj||t|jd}t||j|jd\}}| t!|}|t"v}|jdur|rz t#||jd }Wn tyd}Ynw|durt$j||j|drtfdd|%D}t&|\}}| rt'|| ||||jd\}}nt(|j)dd |i|g}d}|j*||d|jt+t |jj|d}|j}|jdur@d|_|durRt,| dkrRtt| }|}t-|||| t.| ||ddS)Nrvrwzhf://datasets/rsrzDownloading readmerF file_namerrrextension_filterrprqnamerr|)rrrq)rrrq)rqrcsi|] \}}||qSrr)rrdata_files_listmetadata_data_files_listrr msz-get_module_without_script..)rsupports_metadatadefault_builder_kwargsrqrr)rrrrzDownloading metadatart)metadata_configsbuilder_configsdefault_config_name) dataset_infosbuilder_configs_parametersr)/rZrrrdata_dirrstriprqcopy download_descget_dataset_file_urlrHrSloadrrrrTrgetrMfrom_dataset_card_datar+rr%nextitervaluesrr from_patternsr,r9filter_extensionsrCrBr$rrrDr6r8BUILDER_CONFIG_CLASSget_file_base_pathr@rr/r-)r_ms_api_repo_idrrrrrqrpdataset_readme_pathdataset_card_data subset_namerrpatternssubset_data_filesr module_namerrmetadata_patterns module_pathrrrbuilder_kwargshashrrrget_module_without_scripts                 r;rrrimportscCsg}g}|}|jdurd|_|D]_\}} } } |dkr$|| | fq| |kr9td|d| d| d| d |d krNt} | d } | j| |||d }n |d krU| }ntd t||d}| durjtj || }|| |fqi}|D] \}}zt |}Wqvt y||vs||kr|||<Yqvw|rt |dkrdnd}t |dkrdnd}d|vrd|d<d|vrd|d<t d|d|dd |d|dd |d |S) a Download additional module for a module .py at URL (or local path) /.py The imports must have been parsed first using ``get_imports``. If some modules need to be installed with pip, an error is raised showing how to install them. This function return the list of downloaded modules as tuples (import_name, module_file_path). The downloaded modules can then be moved into an importable directory with ``_copy_script_and_other_resources_in_importable_dir``. NzDownloading extra moduleslibraryz Error in the z script, importing relative z module but z: is the name of the script. Please change relative import zl to another name and add a '# From: URL_OR_PATH' comment pointing to the original relative import file path.internal.py)rrrrexternalzWrong import_typer|rt dependencies dependencythemitsklearnz scikit-learnBio biopythonzTo be able to use z$, you need to install the following : z, z. Please install z using 'pip install  z' for instance.)r"r#rrrZr$rHrrjoin importlib import_module ImportErrorrkeysr*)rrrrr<rq local_importslibrary_imports import_type import_name import_path sub_directoryrrrplocal_import_pathneeds_to_be_installedlibrary_import_namelibrary_import_pathlib_dependencies_str _them_strrrr_download_additional_modulessz      r\c Cst}|jdd}|jdd}|d}|j||||jdd}t||jd}d}|jd|||jdd}t||jd} t|} t|j|||j| |jd } g} |r[| t j |f| re| t j | f|j rk|j nt} t|gd d | D}t| d ||jd }tj|st|j|j}|rt|| | | d ||j|jdn td|jdt| d ||jd \}}t|j||d|jd}t|||S)Nrvrurr?Frrr)rrrrr<rqcSsg|]}|dqS)rtr)rlocrrrr2z*get_module_with_script..datasets)dynamic_modules_pathmodule_namespacesubdirectory_namer) local_pathrOadditional_filesr`rarbr download_modezLoading z requires you to execute the dataset script in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option `trust_remote_code=True` to remove this error.r)rr)rZrrr$rrHrqrNr\rrDATASETDICT_INFOS_FILENAMEREPOCARD_FILENAMEr`r?r:r;rrexistsr<trust_remote_coder=rerr>rKinvalidate_cachesr.r/)rrrrscript_file_name script_urllocal_script_pathdataset_infos_pathdataset_readme_urlr1r<rOrdr`r:importable_file_pathrir8r9rrrget_module_with_scripts      rqc/@sveZdZe                     d dedeedeedeeeeeeeeeeefffdeeee fd eed ee d ee d eee efd eee efdeededeeeefdeeeefdedeedeededeedeeeeeeff(ddZe             d!dedeedeedeeeeeeeeeeefffd eed ee d ee d eee efdeeeefdeeeefdeedeedefddZe          d"dedeeeefd ee d eee efdeedeedeeeeeefd eedeedefddZdS)#DatasetsWrapperHFN deprecatedFrrr rr cache_dirfeaturesrqreverification_modekeep_in_memory save_infosrr streamingnum_procrridataset_info_onlyrrc#Ks|dkr tdt|}| dkr#| rtjntj} td| jdt|dkr.tdtnd}|dur>|s>td|dt|t j  rJtd|rT|durTt d t |pYt j}t| sc| pbtjntj} tjd||||||||| ||||dud |}|ri}t|tr|d rtj |rd d lm}||}dd|D}|S|dust|dstd|d|S|j}|D]%\}}t|}t|dr|jdurddt|j D||<qg||<q|S|r|j!|dS|j"||| d||d| dur| nt#|j$j%} |j&|| | d}|dur6t't(dt|)|}Wdn 1s1wY| r=|*z't+}t,|r_|-ddkrb|.d\} }!|j/|!| dW|SW|SW|St0y}"zt1d|"WYd}"~"|Sd}"~"ww) Nrs'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0. You can remove this warning by passing 'token=' instead.z'ignore_verifications' was deprecated in favor of 'verification_mode' in version 2.9.1 and will be removed in 3.0.0. You can remove this warning by passing 'verification_mode=z ' instead.zF'task' was deprecated in version 2.13.0 and will be removed in 3.0.0. zEmpty 'data_files': 'z3'. It should be either non-empty or None (default).zjYou are trying to load a dataset that was saved using `save_to_disk`. Please use `load_from_disk` instead.zLoading a streaming dataset in parallel with `num_proc` is not implemented. To parallelize streaming, you can wrap the dataset with a PyTorch DataLoader using `num_workers` > 1 instead.) rrr rrtrurqrerrrri_require_default_config_namer?r)get_dataset_config_namescSsi|]}|gqSrr)r_subsetrrrrsz2DatasetsWrapperHF.load_dataset..rzNo builder_configs found for z dataset.rcSsg|]}t|qSr)r})ritemrrrrr^z2DatasetsWrapperHF.load_dataset..)rF)rqrervtry_from_hf_gcsrzr)rrv in_memoryignorervrtrz&Could not record download statistics: r)2warningswarn FutureWarningr NO_CHECKS ALL_CHECKSvaluerrrDATASET_STATE_JSON_FILENAMErhNotImplementedErrorrREUSE_DATASET_IF_EXISTS BASIC_CHECKSrrload_dataset_builderrr}endswithrrr_r~hasattrrrrrrrrNas_streaming_datasetdownload_and_preparerLr dataset_size as_datasetcatch_warnings simplefilterprepare_for_task _save_infosrZrJrrdataset_download_statistics Exceptionr)#rrr rrrtrurqrervignore_verificationsrwrxrruse_auth_tokentaskryrzrrir{ config_kwargsbuilder_instanceret_dictr~ subset_list_tmp_builder_configstmp_config_nametmp_builder_configdsrrrerrr load_dataset^s             zDatasetsWrapperHF.load_datasetTc s| dkr tdt| } t|ptj}| dur#|r|nt}| |_| dur6|r-|nt}|j | t j ||||||| | t ||d }|j }|d|}|d|}|d|p_|jj}|dd}|jrp|j|nd}tvr|dur|jjdjdurd d }fd d tD}|r|d |dd7}t|t||d}|d||||||j||| | d ||}|||S)Nrsr|) rrqrer rrtrir}_require_custom_configsrr r config_namerrz@Please specify the data files or data directory to load for the z dataset builder.csg|] }t|kr|qSr)rA)r extensionrrrrDs  z:DatasetsWrapperHF.load_dataset_builder..z9 For example `data_files={"train": "path/to/data/train/*.z"}`)r) rtrrr rr:rrurrr)rrrrrr"rrrupdaterrdataset_module_factoryboolr9poprrrr&rDrrrArr7r:!_use_legacy_cache_dir_if_possible)rrr rrtrurqrerrrrrir}rdataset_moduler9rrrr example_extensions builder_clsrrrrrs           z&DatasetsWrapperHF.load_dataset_builderr`c Ks| dd} |durtd&i| }|jd| i|r#|jdur#t|_t|p(tj}d|_d|_ |tj k|_ t t dd|tjddd} | dsR| d} tj|| }|tvrht|||||dS|| rtj|r}t||||d Std t|tj|rt||||d Stj|rt||||d St|r|dd krzt z t!j"|||j#d d}Wnft$y*}zYt%|t&t'j(j)t'j(j*frt*d|dt+|j,ddt-|vrd|d}t.|r|d|d|dt-|vr$d|d}|r|d|dn|}t.|d|d}~ww| dd|j/Dvrqd}t0j1rd|rdz t2|||j3dWWSt$yc}z t45|WYd}~nd}~wwt6||||||dWSt7||||||dWSt$y}zUzt8|||dWWYd}~St$yt%|t&rt*d |d!|dt%|t9t.t:fr|dt%|trtd t|d"|d#t+|j,d$|d|dwd}~wwtd t|d%)'NrTcSs|SNr)xrrrsz:DatasetsWrapperHF.dataset_module_factory..rvrur?)r rrqre)rer`riz"Couldn't find a dataset script at )r rrertgY@)rrrrzCouldn't reach 'z' on the Hub ()404z Dataset 'z' doesn't exist on the Hubz at revision 'r401zT. If the repo is private or gated, make sure to log in with `huggingface-cli login`.cSsg|]}|jqSr)r)rsiblingrrrrsz.F)rqr)rrqrer`ri)rr rrqre)r`rtz1Couldn't reach the Hugging Face Hub for dataset 'z': z8 or any data file in the same directory. Couldn't find 'z"' on the Hugging Face Hub either: rHz( or any data file in the same directory.r);rrrrrtr]rrextract_compressed_file force_extractFORCE_REDOWNLOADforce_downloadrfilterreplacerrrrrrJrDr5 get_moduleisfiler4rrKisdirr3rJrrGrV dataset_inforrrrFrequests exceptionsConnectTimeoutConnectionErrorr__name__r}r*rmrUSE_PARQUET_EXPORTr1rnrrr2r0r.r)r )rrrqrer`r rrtrir}rdownload_kwargsr3filename combined_pathrrmsg#can_load_config_from_parquet_exporte1rrrrasl                    z(DatasetsWrapperHF.dataset_module_factory)NNNNNNNNNrsNFNNrsrsFNNNF) NNNNNNNNNrsNNT) NNNNNNNNTF)r __module__ __qualname__ staticmethodr}rr r rrrrrrrrintrrrrrrrrrrrr/rrrrrrr\s(       #      _  rrc osBtj}tj}ttdrtjntj}tj }tj }tj }t j }tj} tj} tt_tt_ttdr4tt_ntt_tt_ tt_ tt_ tt _ tt_tt_z2tj|i|} | VW|t_|t_ttdre|t_n|t_|t_ |t_ |t_ |t _ | t_| t_dS|t_|t_ttdr|t_n|t_|t_ |t_ |t_ |t _ | t_| t_w)N _download)r HF_ENDPOINTrEget_from_cacherrr_download_singlerVrrget_paths_inforresolve_patternr0rr2r[r\rrrrr r;rqrrr) argskwargshf_endpoint_originget_from_cache_origin_download_origindataset_info_originlist_repo_tree_originget_paths_info_originresolve_pattern_origin get_module_without_script_originget_module_with_script_origin dataset_resrrrload_dataset_with_ctx3s`    rr)NN)rK contextlibrr functoolsrpathlibrtypingrrrrrr r r r urllib.parser rr_rrrrrrrrrrrrrrrdatasets.data_filesrrrr r!r"r#r$r%,datasets.download.streaming_download_managerr&r'r(datasets.exceptionsr)r* datasets.infor+ datasets.loadr,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?datasets.namingr@datasets.packaged_modulesrArBrCrDdatasets.utilsrEdatasets.utils.file_utilsrFrGrHrIrJrKdatasets.utils.info_utilsrLdatasets.utils.metadatarMdatasets.utils.py_utilsrNdatasets.utils.trackrOfsspecrP fsspec.corerQ fsspec.utilsrRhuggingface_hubrSrThuggingface_hub.hf_apirUrrVrWrX packagingrY modelscoperZmodelscope.hub.utils.utilsr[)modelscope.msdatasets.utils.hf_file_utilsr\modelscope.utils.config_dsr]modelscope.utils.constantr^modelscope.utils.loggerr_rExpandDatasetProperty_Tr}rfloatrrrrrr rr;r\rqrrcontextmanagerrrrrrs4  , D, X                    d  > % "  j e OdZ