o ॵi @s0ddlZddlmZddlmZmZddlZddlm Z ddl m Z ddl m Z mZddlmZeZdd Z  d%d ed eed eefd dZde dedededededefddZdefddZd eeefdedefddZd eeefdefddZddZe fd ededed!e d"eef d#d$ZdS)&N) defaultdict)OptionalUnion)HubApi)DatasetContextConfig)DEFAULT_DATASET_REVISIONMetaDataFields) get_loggercCsdd|DS)NcSs*i|]\}}|ds|dr||qS)metafile)get).0kvr]/home/ubuntu/.local/lib/python3.10/site-packages/modelscope/msdatasets/utils/dataset_utils.py sz,format_dataset_structure..)items)dataset_structurerrrformat_dataset_structuresrr subset_namesplitcCs|r||vs|st|dkrtd|d||}|s0tt|}td|t||}|rH||vrHtd|d||rP|||i}||fS)a| Args: dataset_structure (dict): Dataset Structure, like { "default":{ "train":{ "meta":"my_train.csv", "file":"pictures.zip" } }, "subsetA":{ "test":{ "meta":"mytest.csv", "file":"pictures.zip" } } } subset_name (str, optional): Defining the subset_name of the dataset. split (str, optional): Which split of the data to load. Returns: target_subset_name (str): Name of the chosen subset. target_dataset_structure (dict): Structure of the chosen split(s), like { "test":{ "meta":"mytest.csv", "file":"pictures.zip" } } z subset_name z not found. Available: z,No subset_name specified, defaulting to the zsplit )lenkeys ValueErrornextiterloggerinfor)rrrtarget_subset_nametarget_dataset_structurerrrget_target_dataset_structures. !  r"hub_api max_limit is_recursive dataset_name namespaceversionreturnc CsBg}|j||||d|d}|D]}|d} | sq|| q|S)a List all objects for specific dataset. Args: hub_api (class HubApi): HubApi instance. max_limit (int): Max number of objects. is_recursive (bool): Whether to list objects recursively. dataset_name (str): Dataset name. namespace (str): Namespace. version (str): Dataset version. Returns: res (list): List of objects, i.e., ['train/images/001.png', 'train/images/002.png', 'val/images/001.png', ...] T)r&r'r$r% is_filter_dirrevisionKey)list_oss_dataset_objectsr append) r#r$r%r&r'r(resobjectsitem object_keyrrrlist_dataset_objectsRs  r3cCs8d}|D]\}}t|tr|dsd}|Sq|S)a  To check whether input contains at least one directory. Args: file_map (dict): Structure of data files. e.g., {'train': 'train.zip', 'validation': 'val.zip'} Returns: True if input contains at least one directory, False otherwise. Fz.zipT)r isinstancestrendswith)file_mapr/rrrrr contains_dirts r8cCs$t|tr|g}tj|d|S)N_)r4r5ospathjoin)rr(rrrget_subdir_hash_from_splits r=cCs.t|tr|gSt|tr|Sdt|d)z! Unify the split to list-format. z/Expected format of split: str or list, but got .)r4r5listtype)rrrrget_split_lists  rAcCs`i}|D]\}}g||<q|D]}|D]\}}||ddr,|||qq|S)a Get the map between dataset split and oss objects. Args: file_map (dict): Structure of data files. e.g., {'train': 'train', 'validation': 'val'}, both of train and val are dirs. objects (list): List of oss objects. e.g., ['train/001/1_123.png', 'train/001/1_124.png', 'val/003/3_38.png'] Returns: A map of split-objects. e.g., {'train': ['train/001/1_123.png', 'train/001/1_124.png'], 'validation':['val/003/3_38.png']} /)r startswithrstripr.)r7r0r/rrobj_keyrrrget_split_objects_maps  rFsubset_split_intocontext_configr+cCs~tt}tt}tt}tt}t} |jj} |D],\} } | dd|| <| | dd||||| <| dr@| d|| <| d|| <qg} |D]U\} }|r|tj r|| }t || }|j dd}t j |d|d d }|j|jjd }t|d krtd |d|jd }n|d }||} | || <qN| st| dd|||d} t|rt|| }||||fS)a" Return: meta_map: Structure of meta files (.csv), the meta file name will be replaced by url, like { "test": "https://xxx/mytest.csv" } file_map: Structure of data files (.zip), like { "test": "pictures.zip" } customr r args delimiter,F\)iteratorrL escapecharz:FILErzNo column contains ":FILE" in r>T)r#r$r%r&r'r()rdictrdata_meta_configmeta_cache_dirrr get_dataset_file_url_originr ARGS_BIG_DATAfetch_meta_files_from_url config_kwargspdread_csvcolumnsr5containsto_listrrerrorr3r8rF)rGr&r'rHr+meta_mapr7args_mapcustom_type_mapmodelscope_apirTrrr0 args_dictmeta_csv_file_urlmeta_csv_file_path csv_delimitercsv_df target_colrrrget_dataset_filessl         ri)NN)r: collectionsrtypingrrpandasrYmodelscope.hub.apir4modelscope.msdatasets.context.dataset_context_configrmodelscope.utils.constantrrmodelscope.utils.loggerr rrrRr5r"intboolr?r3r8r=rArFrirrrrsP      8 "