o ̳i, @sddlZddlZddlZddlZddlZddlZddlmZmZddl m Z ddl m Z ddl Z ddl mZmZddlmZmZddlmZddlmZdd lmZdd lmZdd lmZmZeeZd Z d Z!dZ"dZ#GdddZ$de%de e%BfddZ&Gddde'Z(de%dee%dee)e%e*e%e*e%ffddZ+d2de%de,de,de,fd d!Z-d"e%d#ede%fd$d%Z.d&d'd(e%dee%d#ed)e/de$f d*d+Z0d"e%d)e/d#ede%dBfd,d-Z1d.e*e%d/e*e%e%BdBde*e%fd0d1Z2dS)3N)IterableSequence)datetime)TemporaryDirectory) ApiBlobTypeApiStartBlobUploadRequest)ApiDatasetNewFileApiUploadDirectoryInfo)Timeout)tqdm)CallbackIOWrapper)build_kaggle_client) BackendError handle_call2z archive.zipiXc@sFeZdZ  d dedeedBdeddBfddZdefdd ZdS) UploadDirectoryInfoNnamefiles directoriescCs4||_|dur |ng|_|dur||_dSg|_dSN)rrr)selfrrrrH/home/ubuntu/.local/lib/python3.10/site-packages/kagglehub/gcs_upload.py__init__szUploadDirectoryInfo.__init__returncCsNt}|j|_g|_|jD]}t}||_|j|q dd|jD|_|S)NcSsg|]}|qSr)to_proto).0drrr 0z0UploadDirectoryInfo.to_proto..)r rrrtokenappendr)rrfilefrrrr(s zUploadDirectoryInfo.to_proto)NN)__name__ __module__ __qualname__strlistrr rrrrrrs    rstringrcCsBgd}|D]}zt|dd|jddWSYq|S)N)z%Y-%m-%dT%H:%M:%Sz%Y-%m-%dT%H:%M:%SZz%Y-%m-%dT%H:%M:%S.%fz%Y-%m-%dT%H:%M:%S.%fZr) microsecond)rstrptimereplace)r* time_formatstrrrparse_datetime_string5s"r1c@s:eZdZdeddfddZed dededefd d Z dS) File init_dictrNcCs"dd|D}|j|dS)NcSsi|] \}}|t|qSr)r1)rkvrrr Asz!File.__init__..)items__dict__update)rr3 parsed_dictrrrr@sz File.__init__rsize precisioncCsVgd}d}|dkr|dkr|d7}|d}|dkr|dks|d|d||S) N)BKBMBGBTBrig@.r$r)r;r<suffixes suffix_indexrrrget_sizeDsz File.get_size)r) r%r&r'dictr staticmethodfloatintr(rGrrrrr2?sr2base_dirignore_patternsc#srt|D]0\}}}t|}g}|D]}|||tfdd|Ds-||q|r6|||fVqdS)aAn `os.walk` like directory tree generator with filtering. This method filters out files matching any ignore pattern. Args: base_dir (str): The base dir to walk in. ignore_patterns (Sequence[str]): The patterns for ignored files. These are standard wildcards relative to base_dir. Yields: Iterable[tuple[str, list[str], list[str]]]: (base_dir_path, list[dir_names], list[filtered_file_names]) c3s"|] }tjt|dVqdS))rpatN)fnmatchr()rrN rel_file_prr `s z filtered_walk..N)oswalkpathlibPath relative_toanyr")rLrMdir_path dir_names file_namesdir_pfiltered_files file_namerrPr filtered_walkNs   r_rC session_uri file_sizebackoff_factorc Csdd|d}d}|tkraz)tj||td}|jdkr5|jd}|r2t|dd d }|WSWdS|WStj t fy\t d |d t |t|d d }|d 7}Ynw|tksdS)z)Check the status of the resumable upload.0zbytes */)Content-Length Content-Ranger)headerstimeout4Range-rCz8Network issue while checking uploaded size, retrying in seconds...<) MAX_RETRIESrequestsputREQUEST_TIMEOUT status_coderfgetrKsplitConnectionErrorr loggerinfotimesleepmin)r`rarbrf retry_countresponse range_headerbytes_uploadedrrr_check_uploaded_sizefs(    r file_path item_typec sztj|}t*t|_tj|_|_t tj |_ t fdd}Wdn1s6wY|j }ddi}d}d}d}t|d} t|d d d d } |tkr|dkse||krza|dkrqd |d<d} n| |d|d|dd||d<t| j| d} tj||| td} | jdvr|jWWdWdS| jdkrt||}nd| jd| j} t| Wn;tjtjfy}z*td|d|dt !|t"|dd}|d7}t||}|| _#WYd}~nd}~ww|tkr|dkse||kseWdn1swYWd|jSWd|jS1s5wY|jS)aUploads a file to a remote server as a blob and returns an upload token. Args: file_path: The path to the file to be uploaded. item_type : The type of the item associated with the file. Returns: A str token of uploaded blob. csjjSr)blobsblob_api_clientstart_blob_uploadr api_clientrrrsz_upload_blob..Nz Content-Typezapplication/octet-streamrrCrb Uploadingr=T)totaldescunit unit_scalercrdzbytes rj/reread)rfdatarg)rhzUpload failed with status code z: zNetwork issue: z, retrying in rkrlrm)$rSpathgetsizer rtypebasenamercontent_lengthrKgetmtimelast_modified_epoch_secondsr create_urlopenr rnseekr r9rorprqrrr!rtextrrur rvrwrxryrzn)rrrar|r`rfr{uploaded_bytesrbr$pbar upload_dataupload_responseupload_failed_exceptionerrr _upload_blobsd       *   rF)quietfolderrc Csd}t||dD] \}}}|t|7}q|tkr|s#tdtdt[}tj|t }t |d+} t||dD]\} }}|D]} tj| | } | | tj | |qBq;Wdn1sbwYddt|||dfD} td | d WdS1swYtd d }tj|rt|||d}|r|j||St||dD]P\} }}tj | |}|}|d kr|tjD]}|jD] }|j|kr|}nqt|d }|j||}q|D]} ttj| | ||d}|r|j|qq|S)Nr)rLrMz More than z* files detected, creating a zip archive...wcSsg|]}|dur|qSrr)rr!rrrrs z0upload_files_and_directories..)rrrarchive)rrroot)rrD)r_lenMAX_FILES_TO_UPLOADrvrwrrSrjoinTEMP_ARCHIVE_FILEzipfileZipFilewriterelpath _upload_filerisfilerr"rtseprr)rrMrr file_count_rtemp_dirzip_pathzipfrr#rtokens root_dictr!r current_dictpartsubdirnew_dirrrrupload_files_and_directoriess`         rcCsl|s td|tj|std|dStj|}t||}|s4td|dt |d|S)aHelper function to upload a single file. Args: full_path: path to the file to upload quiet: suppress verbose output item_type: Type of the item that is being uploaded. Returns: A str token of uploaded file if successful, otherwise None. zStarting upload for file z+Skip uploading %s because it is not a file.NzUpload successful: z ()) rvrwrSrrwarnrrr2rG)rrrrr!rrrrs      rdefault additionalcsFdtdtfdd|durg}nt|tr|g}fdd||DS)zYMerges additional patterns with the default, and normalize the dir pattern with wildcard.patternrcSs|dr |dS|S)Nr*)endswith)rrrradd_wildcard_to_dirsz/normalize_patterns..add_wildcard_to_dirNcsg|]}|qSrr)rrrrrr#r z&normalize_patterns..)r( isinstance)rrrrrnormalize_patternss  r)rC)3rOloggingrSrUrxrcollections.abcrrrtempfilerro&kagglesdk.blobs.types.blob_api_servicerr,kagglesdk.datasets.types.dataset_api_servicerr requests.exceptionsr r tqdm.utilsr kagglehub.clientsr kagglehub.exceptionsrr getLoggerr%rvrrrnrqrr(r1objectr2tupler)r_rKrrboolrrrrrrrsT        0B B.