o .i$ @sddlZddlmZddlmZmZddlmZmZm Z m Z m Z m Z m Z ddlmZddlZddlmZddlmZmZddlmZmZdd lmZ   dd ed eed eeeefd eedef ddZ  dd eded eed eeeefdef ddZdddZ dS)N)chain)OptionalUnion) CommitInfoCommitOperationAddCommitOperationDelete DatasetCardDatasetCardDataHfApi HfFileSystem)HfHubHTTPError)DatasetInfosDict)get_dataset_config_namesget_dataset_default_config_name) load_datasetload_dataset_builder)MetadataConfigsrepo_idrevisiontokentrust_remote_codereturnc Csdt|t||||d}td|t||||d}td||r-|}||n|d}td|t||||d}|j||ddd ||d ud }td |j |j } } |D]$}td|t||||d}|j||d |d| |dtd q]t || |d|st t jj|d} z | j|dd|d dWn tyYnwtd| |S)aConvert Hub [script-based dataset](dataset_script) to Parquet [data-only dataset](repository_structure), so that the dataset viewer will be supported. This function: - makes a copy of the script on the "main" branch into a dedicated branch called "script" (if it does not already exist) - creates a pull request to the Hub dataset to convert it to Parquet files (and deletes the script from the main branch) If in the future you need to recreate the Parquet files from the "script" branch, pass the `revision="script"` argument. Note that you should pass the `trust_remote_code=True` argument only if you trust the remote code to be executed locally on your machine. Args: repo_id (`str`): ID of the source Hub dataset repository, in the following format: `/` or `/`. revision (`str`, *optional*): Branch of the source Hub dataset repository. Defaults to the `"main"` branch. token (`bool` or `str`, *optional*): Authentication token for the Hugging Face Hub. trust_remote_code (`bool`, defaults to `False`): Whether you trust the remote code of the Hub script-based dataset to be executed locally on your machine. This option should only be set to `True` for repositories where you have read the code and which you trust. `trust_remote_code` defaults to `False` if not specified. Returns: `huggingface_hub.CommitInfo` )rrrz configs = zdefault_config = rz config = )rrzConvert dataset to ParquetzConvert dataset to Parquet.TN) config_namecommit_messagecommit_description create_prr set_defaultzAdd 'z' config data files)rrrr)rrendpointrscriptdataset)branch repo_typerexist_okz;You can find your PR to convert the dataset to Parquet at: )printrrremovepopr push_to_hubtimesleep pr_revisionpr_url _delete_filesr datasetsconfig HF_ENDPOINT create_branchr ) rrrrconfigsdefault_configr/r! commit_infor+r,apir6@/home/ubuntu/.local/lib/python3.10/site-packages/datasets/hub.pyconvert_to_parquetsZ #      r8rc Csg}ttjj|d}t||||dd}t|jjD]}||}|j |kr0| t |j dqt |} | jddrL|| jdvrL| jd|t| j} | r}| |d} t} | | tjj| vrt| tjj| jtjj<n | jtjjd} t| j} | r| |d} t} | | d| vr| d| jd<n| jdd} | ttjjt| dttjj|d}|j||d |d d |d |d |d d}td|j |S)a=Delete a dataset configuration from a [data-only dataset](repository_structure) on the Hub. Args: repo_id (`str`): ID of the Hub dataset repository, in the following format: `/` or `/`. config_name (`str`): Name of the dataset configuration. revision (`str`, *optional*): Branch to delete the configuration from. Defaults to the `"main"` branch. token (`bool` or `str`, *optional*): Authentication token for the Hugging Face Hub. Returns: `huggingface_hub.CommitInfo` rF)rrr) path_in_repo config_namesN dataset_info)r9path_or_fileobjzDelete 'z' configz ' config.r!T) operationsrrrr#rrz6You can find your PR to delete the dataset config at: )!r r.r/r0rr data_filesvalues resolve_pathrappendrr9rloaddatagetr&rfrom_dataset_card_datar'r to_dataset_card_dataMETADATA_CONFIGS_FIELDr rREPOCARD_FILENAMEstrencoder create_commitr%r,)rrrrr=fsbuilder data_filedata_file_resolved_path dataset_cardmetadata_configs_dataset_card_data dataset_infosr5r4r6r6r7delete_from_hubisZ             rUc Cs|dd}ttjj|d}|j|dd}|rg}g}g}|D]3} | dvr'q | |dkr9|j| |d|dd q | d krC|| q | drN|| q || q |r`|jd |d|d d |rq|D] } |j| |d|d d qd|r|D]} |j| |d|d d qudSdSdS)N/rr!)r#> README.md.gitattributesz.pyzDelete loading script)r#rrzdataset_infos.jsonz Delete legacy dataset_infos.jsonz$Delete loading script auxiliary filezDelete data file) splitr r.r/r0list_repo_files delete_filerAendswith) dataset_idrr dataset_namehf_api repo_fileslegacy_json_file python_filesr>filenamer6r6r7r-sn    &r-)NNN)NN)!r) itertoolsrtypingrrhuggingface_hubrrrrr r r huggingface_hub.utilsr datasets.configr. datasets.infor datasets.inspectrr datasets.loadrrdatasets.utils.metadatarrIboolr8rUr-r6r6r6r7sL $    U H