o eiP'@s dZddlZddlZddlZddlZddlZddlZddlmZm Z ddl m Z m Z ddl m Z ddlmZmZmZmZmZmZddlmZmZdd lmZdd lmZmZmZmZdd lm Z erhdd l!m"Z"e#e$Z%d Z&dZ'de(de)e(e(ffddZ*eGdddZ+eGdddZ,eGdddZ-eddGdddZ.eGdddZ/eGdd d Z0eGd!d"d"Z1de(de)e(e(ffd#d$Z2de(de3fd%d&Z4eGd'd(d(Z5eGd)d*d*Z6Gd+d,d,Z7d-e(de8e)e(e(ffd.d/Z9d0e(dee)e(e:e;ffd1d2Zdd<de(d=ed>d?e:d@e;dAe:dBe;dCe(dDe(dEe3dFe3dGe3dHeede5fdIdJZ? K K K K K  dkdLe(dMe(d3d4dNe3dFe3dEe3dOe3dGe3dPee7dQeede6fdRdSZ@dTe6ddfdUdVZAdTe6dWe(ddfdXdYZBdWe(de6fdZd[ZCdldTe6d3d4d\e3dQeeddf d]d^ZDdTe6ddfd_d`ZE  dmdKdKdKdKdKddddddKdKdKddadLee(dMee(d3d4dNe3dFe3dEe3dOe3dGe3dbee8e(dcee8e(ddee(dTee(deee(dfe3d\e3dge3dhee3e(dfde6f$didjZFdS)nzzShared logic for bucket operations. This module contains the core buckets logic used by both the CLI and the Python API. N) dataclassfield)datetimetimezone)Path) TYPE_CHECKINGAnyIteratorLiteralOptionalUnion) constantslogging)BucketNotFoundError) XetFileDatadisable_progress_barsenable_progress_barsparse_datetime) StatusLineHfApi hf://buckets/pathreturncCsn|dd}t|dks|dr|dstd|d|dd|d}t|dkr1|dnd}||fS)zSplit 'namespace/name(/optional/prefix)' into ('namespace/name', 'prefix'). Returns (bucket_id, prefix) where prefix may be empty string. Raises ValueError if path doesn't contain at least namespace/name. /rr zInvalid bucket path: 'z)'. Expected format: namespace/bucket_name)splitlen ValueError)rparts bucket_idprefixr%V/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/huggingface_hub/_buckets.py_split_bucket_id_and_prefix5s r'c@sBeZdZUdZeed<eed<eed<eed<eed<ddZ d S) BucketInfoa Contains information about a bucket on the Hub. This object is returned by [`bucket_info`] and [`list_buckets`]. Attributes: id (`str`): ID of the bucket. private (`bool`): Is the bucket private. created_at (`datetime`): Date of creation of the bucket on the Hub. size (`int`): Size of the bucket in bytes. total_files (`int`): Total number of files in the bucket. idprivate created_atsize total_filescKsV|d|_|d|_t|d|_|d|_|d|_|jjdi|dS)Nr)r* createdAtr, totalFilesr%) popr)r*rr+r,r-__dict__updateselfkwargsr%r%r&__init__[s    zBucketInfo.__init__N) __name__ __module__ __qualname____doc__str__annotations__boolrintr6r%r%r%r&r(Cs  r(c@seZdZUeeeefed<eed<eddZ e eed<eddZ e e ed<eddZ e ed <eddZe eed <dd d ZdS)_BucketAddFilesource destinationN)defaultxet_hashr,Finitmtime content_typercCsxd|_t|jttfrt|jd|_|jdur"t|jd|_tt|jt s2t j |jdnt d|_dS)Nrr)rG isinstancer@r;r mimetypes guess_typerAr>bytesosrgetmtimetimerF)r4r%r%r& __post_init__ns ( z_BucketAddFile.__post_init__rN)r7r8r9r r;rrKr<rrCr r,r>rFrGrOr%r%r%r&r?ds r?c@seZdZUeed<dS)_BucketDeleteFilerN)r7r8r9r;r<r%r%r%r&rQzs rQT)frozenc@s"eZdZUdZeed<eed<dS)BucketFileMetadataa%Data structure containing information about a file in a bucket. Returned by [`get_bucket_file_metadata`]. Args: size (`int`): Size of the file in bytes. xet_file_data (`XetFileData`): Xet information for the file (hash and refresh route). r, xet_file_dataN)r7r8r9r:r>r<rr%r%r%r&rSs  rSc@sfeZdZUdZeed<dZeed<eddZeed<eddZ eed<eddZ eed <dd d Z d S) BucketUrlaDescribes a bucket URL on the Hub. `BucketUrl` is returned by [`create_bucket`]. At initialization, the URL is parsed to populate properties: - endpoint (`str`) - namespace (`str`) - bucket_id (`str`) - url (`str`) - handle (`str`) Args: url (`str`): String value of the bucket url. endpoint (`str`, *optional*): Endpoint of the Hub. Defaults to . urlrendpointFrD namespacer#handlerNcCs|jptj|_|j|jdd}|dr|tdd}t|\}}|r/t d|j| dd|_ ||_ d|j |_ dS)Nrrzbuckets/zUnable to parse bucket URL: rr)rWrENDPOINTrVreplacestrip startswithr r'r!rrXr#rY)r4url_pathr#r$r%r%r&rOs  zBucketUrl.__post_init__rP) r7r8r9r:r;r<rWrrXr#rYrOr%r%r%r&rUs  rUc@sJeZdZUdZeded<eed<eed<eed<ee ed<dd Z d S) BucketFilez Contains information about a file in a bucket on the Hub. This object is returned by [`list_bucket_tree`]. Similar to [`RepoFile`] but for files in buckets. filetyperr,rCrFcKsX|d|_|d|_|d|_|d|_|dd}|r't||_dSd|_dS)Nrarr,xetHashrF)r0rarr,rCrrF)r4r5rFr%r%r&r6s     zBucketFile.__init__N) r7r8r9r:r r<r;r>r rr6r%r%r%r&r_s    r_c@s6eZdZUdZeded<eed<eed<ddZdS) BucketFolderz Contains information about a directory in a bucket on the Hub. This object is returned by [`list_bucket_tree`]. Similar to [`RepoFolder`] but for directories in buckets. directoryrar uploaded_atcKs,|d|_|d|_t|d|_dS)Nrar uploadedAt)r0rarrrer3r%r%r&r6s  zBucketFolder.__init__N) r7r8r9r:r r<r;rr6r%r%r%r&rcs   rccCs,|tstd|dtt|tS)zParse a bucket path like hf://buckets/namespace/bucket_name/prefix into (bucket_id, prefix). Returns: tuple: (bucket_id, prefix) where bucket_id is "namespace/bucket_name" and prefix may be empty string. zInvalid bucket path: z. Must start with )r] BUCKET_PREFIXr!r' removeprefixrr%r%r&_parse_bucket_paths rjcCs |tS)z!Check if a path is a bucket path.)r]rgrir%r%r&_is_bucket_paths rkc@sreZdZUdZeded<eed<dZee ed<dZ eed<dZ eeed <dZ eeed <dZ eeed <dS) SyncOperationz,Represents a sync operation to be performed.)uploaddownloaddeleteskipactionrNr,rreason local_mtime remote_mtime bucket_file)r7r8r9r:r r<r;r,r r>rrrsrtrur_r%r%r%r&rls   rlc@s^eZdZUdZeed<eed<eed<eedZee ed<de ee e efffdd Z d S) SyncPlanz Represents a complete sync plan.r@dest timestamp)default_factory operationsrcCsttdd|jD}tdd|jD}tdd|jD}tdd|jD}tdd|jD}|||||dS)Ncs|] }|jdkrdVqdS)rmr Nrq.0opr%r%r& z#SyncPlan.summary..csr{)rnr Nr|r}r%r%r&rrcsr{)ror Nr|r}r%r%r&rrcsr{)rpr Nr|r}r%r%r&rrcss$|] }|jdvr|jp dVqdS)rmrnrN)rqr,r}r%r%r&rs")uploads downloadsdeletesskips total_size)sumrz)r4rrrrrr%r%r&summaryszSyncPlan.summaryN)r7r8r9r:r;r<rlistrzrldictr r>rr%r%r%r&rv s "rvc @s`eZdZdZ   d deeedeeedeeeeeffddZded e fd d Z dS) FilterMatcherz4Matches file paths against include/exclude patterns.Ninclude_patternsexclude_patterns filter_rulescCs"|pg|_|pg|_|p g|_dS)aInitialize the filter matcher. Args: include_patterns: Patterns to include (from --include) exclude_patterns: Patterns to exclude (from --exclude) filter_rules: Rules from filter file as list of ("+"/"-", pattern) tuples Nrrr)r4rrrr%r%r&r6)s zFilterMatcher.__init__rrcCsr|jD]\}}t||r|dkSq|jD] }t||r"dSq|jD] }t||r1dSq&|jr7dSdS)zCheck if a path should be included based on the filter rules. Filtering rules: - Filters are evaluated in order, first matching rule decides - If no rules match, include by default (unless include patterns are specified) +FT)rfnmatchrr)r4rsignpatternr%r%r&matches:s      zFilterMatcher.matches)NNN) r7r8r9r:r rr;tupler6r=rr%r%r%r&r&s   r filter_filecCsg}t|G}|D];}|}|r|drq |dr*|d|ddfq |dr=|d|ddfq |d|fq Wd|S1sPwY|S)zParse a filter file and return a list of (sign, pattern) tuples. Filter file format: - Lines starting with "+" are include patterns - Lines starting with "-" are exclude patterns - Empty lines and lines starting with "#" are ignored #rr N-)openr\r]append)rrulesfliner%r%r&_parse_filter_fileWs       r local_pathc cstj|}tj|std|t|D]3\}}}|D]+}tj||}tj||}|tj d}tj |}tj |d}|||fVq qdS)zsList all files in a local directory. Yields: tuple: (relative_path, size, mtime_ms) for each file z Local path must be a directory: rrN) rLrabspathisdirr!walkjoinrelpathr[sepgetsizerM) rroot_filesfilename full_pathrel_pathr,mtime_msr%r%r&_list_local_filests   rapirr#r$ccs|j||pdddD]I}t|trq |j}|r>||dr*|t|dd}n||kr=d|vr:|dddn|}nq |}|jrJ|jdnd}||j ||fVq dS) zList all files in a bucket with a given prefix. Yields: tuple: (relative_path, size, mtime_ms, bucket_file) for each file. bucket_file is the BucketFile object from list_bucket_tree. NT)r$ recursiverr rr) list_bucket_treerHrcrr]r rsplitrFrxr,)rr#r$itemrrrr%r%r&_list_remote_filess rrcCstj|dtjdS)z3Convert mtime in milliseconds to ISO format string.r)tz)r fromtimestamprutc isoformat)rr%r%r& _mtime_to_isosr)rurqr source_size source_mtime dest_size dest_mtimesource_newer_labeldest_newer_label ignore_sizes ignore_timesignore_existingruc Cst|dkr|n|} t|dkr|n|} ||| | d}| r'td ddd|S||k}||tk}|rV|r@td ||| d|S||tk}|rJ|nd}td d|d|S| ro|retd |d| d|Std dd d|S|ss|r|rwdn|}td ||| d|Std dd d|S) aCompare source and dest files and return the appropriate sync operation. This is a unified helper for both upload and download directions. Args: path: Relative file path action: "upload" or "download" source_size: Size of the source file (bytes) source_mtime: Mtime of the source file (milliseconds) dest_size: Size of the destination file (bytes) dest_mtime: Mtime of the destination file (milliseconds) source_newer_label: Label when source is newer (e.g., "local newer" or "remote newer") dest_newer_label: Label when dest is newer (e.g., "remote newer" or "local newer") ignore_sizes: Only compare mtime ignore_times: Only compare size ignore_existing: Skip files that exist on receiver bucket_file: BucketFile object (for downloads only) Returns: SyncOperation describing the action to take rm)rr,rsrtrpz&exists on receiver (--ignore-existing))rqrr)rqrrruz same mtimez size differsz same size identicalNr%)rrl_SYNC_TIME_WINDOW_MS)rrqrrrrrrrrrrulocal_mtime_isoremote_mtime_iso base_kwargs size_differs source_newer dest_newer skip_reasonrrr%r%r&_compare_files_for_syncs2$    rFr@rwroexistingfilter_matcherstatusc "Cs|pt}t| o t|} t|ot| } | s| stdt||ttjd} d} | rt j |}t |\}}t j |sJtd|i}t|D]\}}}||r`||f||<| rm| dt|dqP| r{| dt|di}| rz||j} Wn tyYnwz4t|||D]+\}}}}||r||f||<| r| durd| nd}| d t||dqWntytd |d Ynw| r| d t|dt|t|B}| r| d t|d t|D]}||}||}|rA|sA|r,| jt d||ddt!|ddq| jt d||ddt!|ddq|rd|rd|\}}|\}}| jt"|d||||dd|||d q|s|r|r| jt d||ddt!|ddq| St |\}}t j |}i}i} | rz||j} Wn tyYnwt|||D]3\}}}}!||r||f||<|!| |<| r| durd| nd}| d t||dq| r| d t|di}t j |r t|D] \}}}||r||f||<| r| dt|dq| r.| dt|dt|t|B}| rH| d t|d t|D]}||}||}|r|s|rv| jt d||ddt!|ddqL| jt d||ddt!|d| |dqL|r|r|\}}|\}}| jt"|d||||dd|||| |d qL|s|r|r| jt d||ddt!|ddqL| S)zCompute the sync plan by comparing source and destination. Returns: SyncPlan with all operations to be performed z[One of source or dest must be a bucket path (hf://buckets/...) and the other must be local.r@rwrxNzSource must be a directory: zScanning local directory (z files)rrzScanning remote bucket (zBucket 'z' not found, treating as empty.zComparing files (z paths)rprznew file (--existing)r )rqrr,rrrsrmznew filez local newerz remote newer) rrqrrrrrrrrrroznot in source (--delete))rqrr,rrrtrn)rqrr,rrrtru) rrqrrrrrrrrrru)#rrkr!rvrnowrrrrLrrrjrrrr2r done bucket_infor- Exceptionrrloggerdebugsetkeyssortedgetrzrrlrr)"r@rwrrorrrrrr is_upload is_downloadplan remote_totalrr#r$ local_filesrr,r remote_filesr total_str all_pathsr local_info remote_info local_sizers remote_sizertbucket_file_maprur%r%r&_compute_sync_plans|                   k               rrcCsd|j|j|j|d}|t|d|jD]4}d|j|j |j d}|j dur0|j |d<|j dur:|j |d<|j durD|j |d <|t|dqdS) z1Write a sync plan as JSONL to a file-like object.header)rar@rwrxr operation)rarqrrrNr,rsrt)r@rwrxrwritejsondumpsrzrqrrrr,rsrt)rrrrop_dictr%r%r& _write_plans*       r plan_filecCs:t|d}t||WddS1swYdS)z!Save a sync plan to a JSONL file.wN)rr)rrrr%r%r& _save_plans  "rc Cst| }|}Wdn1swY|s!td|t|d}|ddkr3tdt|d|d|d d }|d dD].}t|}|dd krUqF|jt |d |d|d|dd|d|ddqF|S)z#Load a sync plan from a JSONL file.NzEmpty plan file: rrarz0Invalid plan file: expected header as first liner@rwrxrr rrqrr,rrrrsrt)rqrr,rrrsrt) r readlinesr!rloadsrrvrzrrl)rrlinesrrrrr%r%r& _load_plans8     rverbosec Cst|j o t|j}t|jot|j }|rtj|j}t|j\}}|d}g} g} |jD]n} | j dkretj || j} |rK|d| jn| j} |r]t d| jd| j d| | | fq2| j dkr|rt|d| jn| j} |rt d| jd| j d| | q2| j dkr|rt d | jd| j dq2| s| r|rg}| r| d t| d | r| d t| d |d ||j|| pd| pdddSdS|rt|j\}}|d}tj|j}tj|ddg}g}|jD]} | j dkrStj || j} tjtj| dd|r,t d| jd| j d| jdur<| | j| fq|rG|d| jn| j} | | | fq| j dkrxtj || j} |rqt d| jd| j d| | q| j dkr|rt d | jd| j dqt|dkr|r|dt|d ||||r|r|dt|d|D]9}tj|rt|tj|}||krz t|tj|}Wn tyYnw||ksאqdSdS)zExecute a sync plan.rrmz Uploading: z ()roz Deleting: rpz Skipping: z uploading z filesz deleting z, N)addroT)exist_okrnz Downloading: rz Downloading z Deleting z local files)rkr@rwrLrrrjrstriprzrqrprintrrrr r capitalizebatch_bucket_filesmakedirsdirnamerudownload_bucket_filesexistsremovermdirOSError)rrrrrrrr#r$ add_files delete_pathsr local_file remote_pathr"download_files delete_files file_pathparentr%r%r& _execute_plan1s                  *rcCsl|}td|jd|jtd|dtd|dtd|dtd |d d S) z!Print a summary of the sync plan.z Sync plan: z -> z Uploads: rz Downloads: rz Deletes: rz Skips: rN)rrr@rw)rrr%r%r&_print_plan_summarys r)rorrrrincludeexclude filter_fromrapplydry_runrquiettokenrrrrrrrc Cs|durddlm}||d}| r|s|rtd| dur!td|r'td|r-td|r3td |r9td | r?td | rEtd |rKtd |rQtd| rWtdt| }t| d}|skt|td|rptzt||||dW|rt n|rt ww|std|S|r|stdt |}t |}|r|rtd|s|std|r|rtd|r|rtd| r| rtd|rt j |rt j |std|n t j |std|d}| rt| }t|| |d}t| o| d}t||||||||||d }| rt|tj|S| r3t|| |s1t|td| |S|s:t||}|dd kr\|d!d kr\|d"d kr\|sZtd#|S|sctd$|ritzt||||dW|ryt n|rt ww|std|S)%aSync files between a local directory and a bucket. This is equivalent to the ``hf buckets sync`` CLI command. One of ``source`` or ``dest`` must be a bucket path (``hf://buckets/...``) and the other must be a local directory path. Args: source (`str`, *optional*): Source path: local directory or ``hf://buckets/namespace/bucket_name(/prefix)``. Required unless using ``apply``. dest (`str`, *optional*): Destination path: local directory or ``hf://buckets/namespace/bucket_name(/prefix)``. Required unless using ``apply``. api ([`HfApi`]): The HfApi instance to use for API calls. delete (`bool`, *optional*, defaults to `False`): Delete destination files not present in source. ignore_times (`bool`, *optional*, defaults to `False`): Skip files only based on size, ignoring modification times. ignore_sizes (`bool`, *optional*, defaults to `False`): Skip files only based on modification times, ignoring sizes. existing (`bool`, *optional*, defaults to `False`): Skip creating new files on receiver (only update existing files). ignore_existing (`bool`, *optional*, defaults to `False`): Skip updating files that exist on receiver (only create new files). include (`list[str]`, *optional*): Include files matching patterns (fnmatch-style). exclude (`list[str]`, *optional*): Exclude files matching patterns (fnmatch-style). filter_from (`str`, *optional*): Path to a filter file with include/exclude rules. plan (`str`, *optional*): Save sync plan to this JSONL file instead of executing. apply (`str`, *optional*): Apply a previously saved plan file. When set, ``source`` and ``dest`` are not needed. dry_run (`bool`, *optional*, defaults to `False`): Print sync plan to stdout as JSONL without executing. verbose (`bool`, *optional*, defaults to `False`): Show detailed per-file operations. quiet (`bool`, *optional*, defaults to `False`): Suppress all output and progress bars. token (Union[bool, str, None], optional): A valid user access token. If not provided, the locally saved token will be used. Returns: [`SyncPlan`]: The computed (or loaded) sync plan. Raises: `ValueError`: If arguments are invalid (e.g., both paths are remote, conflicting options). Example: ```python >>> from huggingface_hub import HfApi >>> api = HfApi() # Upload local directory to bucket >>> api.sync_bucket("./data", "hf://buckets/username/my-bucket") # Download bucket to local directory >>> api.sync_bucket("hf://buckets/username/my-bucket", "./data") # Sync with delete and filtering >>> api.sync_bucket( ... "./data", ... "hf://buckets/username/my-bucket", ... delete=True, ... include=["*.safetensors"], ... ) # Dry run: preview what would be synced >>> plan = api.sync_bucket("./data", "hf://buckets/username/my-bucket", dry_run=True) >>> plan.summary() {'uploads': 3, 'downloads': 0, 'deletes': 0, 'skips': 1, 'total_size': 4096} # Save plan for review, then apply >>> api.sync_bucket("./data", "hf://buckets/username/my-bucket", plan="sync-plan.jsonl") >>> api.sync_bucket(apply="sync-plan.jsonl") ``` Nr r)rz,Cannot specify source/dest when using apply.z#Cannot specify both plan and apply.z'Cannot specify delete when using apply.z-Cannot specify ignore_times when using apply.z-Cannot specify ignore_sizes when using apply.z(Cannot specify include when using apply.z(Cannot specify exclude when using apply.z,Cannot specify filter_from when using apply.z)Cannot specify existing when using apply.z0Cannot specify ignore_existing when using apply.z(Cannot specify dry_run when using apply.)enabledzExecuting plan...)rrzSync completed.z7Both source and dest are required (unless using apply).z?Remote to remote sync is not supported. One path must be local.z?One of source or dest must be a bucket path (hf://buckets/...).z2Cannot specify both ignore_times and ignore_sizes.z1Cannot specify both existing and ignore_existing.z%Cannot specify both dry_run and plan.z!Destination must be a directory: z&Source must be an existing directory: r) r@rwrrorrrrrrzPlan saved to: rrrrzNothing to sync.z Syncing...)hf_apirr!rrrrrrrrkrLrrrrrrrsysstdoutrr)r@rwrrorrrrrrrrrrrrrr sync_planrsource_is_bucketdest_is_bucketrrrr%r%r&sync_bucket_internalsc      *r!)FFFFFNN)FN)NN)Gr:rrrIrLrrN dataclassesrrrrpathlibrtypingrrr r r r rrrerrorsrutilsrrrrutils._terminalrrr get_loggerr7rrgrr;rr'r(r?rQrSrUr_rcrjr=rkrlrvrrrr>floatrrrrrrrrrrr!r%r%r%r&sH       (   1 *"      L   l$+c