o -i|B@sdZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZmZmZmZmZmZmZmZddlmZdd lmZd d lmZmZmZmZmZm Z m!Z!m"Z"d d l#m$Z$d d l%m&Z&m'Z'd dl(m)Z)erzd dl*m+Z+e ,e-Z.e/dZ0dZ1dddZ2eGdddZ3e"    dFdee3dee4de4de4dee4dee4deee4e4fdeee4deee5ee5ee4ffdd Z6Gd!d"d"eZ7Gd#d$d$eZ8   dGd%d&d'edee4deee4e4fdee4ddf d(d)Z9d*e5fd+d,Z:d-e5fd.d/Z;d0e5fd1d2Zd3e4ddf d8d9Z?d6ed:e3d7e>dee4fd;d<Z@d=eed>e4de8fd?d@ZAd%d&dAee4d7e>deefdBdCZBd%d&dAee4d7e>deefdDdEZCdS)Hz.Git LFS related type definitions and utilitiesN) dataclass)ceil)getsize)Path) TYPE_CHECKINGBinaryIODictIterableListOptionalTuple TypedDict)unquote) constants)build_hf_headersfix_hf_endpoint_in_url get_sessionhf_raise_for_status http_backoffloggingtqdmvalidate_hf_hub_args) SliceFileObj)sha256 sha_fileobj)is_tqdm_disabled)CommitOperationAddz^[0-9a-f]{40}$zlfs-multipart-uploadzapplication/vnd.git-lfs+json)Acceptz Content-Typec@s`eZdZUdZeed<eed<eed<edefddZ edefd d Z ed e fd d Z dS) UploadInfoad Dataclass holding required information to determine whether a blob should be uploaded to the hub using the LFS protocol or the regular protocol Args: sha256 (`bytes`): SHA256 hash of the blob size (`int`): Size in bytes of the blob sample (`bytes`): First 512 bytes of the blob rsizesamplepathcCs\t|}t|d}|ddd}t|}Wdn1s"wY||||dS)Nrbr rr!)rioopenpeekr)clsr"r filer!shar,L/home/ubuntu/LTX-2/.venv/lib/python3.10/site-packages/huggingface_hub/lfs.py from_pathOs  zUploadInfo.from_pathdatacCs&t|}|t||dd|dS)Nr$)r r!r)rdigestlen)r)r/r+r,r,r- from_bytesWs zUploadInfo.from_bytesfileobjcCsD|d}|dtjt|}|}|dtj||||dS)Nr$rr%)readseekr&SEEK_SETrtell)r)r3r!r+r r,r,r- from_fileobj\s zUploadInfo.from_fileobjN) __name__ __module__ __qualname____doc__bytes__annotations__int classmethodstrr.r2rr8r,r,r,r-r<s  r upload_infostoken repo_typerepo_idrevisionendpointheaders transfersreturncCs|dur|ntj}d}|tjvrtj|}|d||d} d|dur&|nddgdd |Dd d } |dur?d t|i| d <itt|d|pJi}tj| || d} t| | } | dd} t | t slt d| d}t |trx|nd}dd | Ddd | D|fS)a Requests the LFS batch endpoint to retrieve upload instructions Learn more: https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md Args: upload_infos (`Iterable` of `UploadInfo`): `UploadInfo` for the files that are being uploaded, typically obtained from `CommitOperationAdd.upload_info` repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. revision (`str`, *optional*): The git revision to upload to. headers (`dict`, *optional*): Additional headers to include in the request transfers (`list`, *optional*): List of transfer methods to use. Defaults to ["basic", "multipart"]. Returns: `LfsBatchInfo`: 3-tuple: - First element is the list of upload instructions from the server - Second element is a list of errors, if any - Third element is the chosen transfer adapter if provided by the server (e.g. "basic", "multipart", "xet") Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If an argument is invalid or the server response is malformed. [`HTTPError`](https://requests.readthedocs.io/en/latest/api/#requests.HTTPError) If the server returned an error. N/z.git/info/lfs/objects/batchuploadbasic multipartcSsg|] }|j|jdqS)oidr )rhexr ).0rMr,r,r- s z'post_lfs_batch_info..r) operationrIobjects hash_algonameref)rCrHjsonrVzMalformed response from servertransfercSsg|] }d|vrt|qSerror)_validate_batch_actionsrSobjr,r,r-rTcSsg|] }d|vrt|qSr])_validate_batch_errorr`r,r,r-rTrb)rENDPOINTREPO_TYPES_URL_PREFIXESr LFS_HEADERSrrpostrr[get isinstancelist ValueErrorrA)rBrCrDrErFrGrHrI url_prefix batch_urlpayloadresp batch_inforVchosen_transferr,r,r-post_lfs_batch_infofsB,        rrc@seZdZUeed<eed<dS) PayloadPartT partNumberetagN)r9r:r;r?r>rAr,r,r,r-rss  rsc@s&eZdZUdZeed<eeed<dS)CompletionPayloadTz?Payload that will be sent to the Hub when uploading multi-part.rQpartsN)r9r:r;r<rAr>r rsr,r,r,r-rvs rvrUrlfs_batch_actionc CsHt||d}|durtd|jddS|dd}t||dd}|dur2t||di}|d} t|d |d } | durjzt| } Wntt fy`td | d wt ||| | d nt || d|durt|t|d |} t j | t||d|jj|jjdd} t| t|jddS)aw Handles uploading a given object to the Hub with the LFS protocol. Can be a No-op if the content of the file is already present on the hub large file storage. Args: operation (`CommitOperationAdd`): The add operation triggering this upload. lfs_batch_action (`dict`): Upload instructions from the LFS batch endpoint for this object. See [`~utils.lfs.post_lfs_batch_info`] for more details. headers (`dict`, *optional*): Headers to include in the request, including authentication and user agent headers. Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `lfs_batch_action` is improperly formatted [`HTTPError`](https://requests.readthedocs.io/en/latest/api/#requests.HTTPError) If the upload resulted in an error actionsNzContent of file z. is already present upstream - skipping uploadrMverifyheader chunk_sizehref)rGzTMalformed response from LFS batch endpoint: `chunk_size` should be an integer. Got 'z'.)rUr{r| upload_url)rUr~)rCrHrPrZz: Upload successful)r_rhloggerdebug path_in_repo_validate_lfs_actionrr?rk TypeError_upload_multi_part_upload_single_partrrgr upload_inforrRr r) rUrxrCrHrGry upload_action verify_actionr{r|r~ verify_url verify_respr,r,r- lfs_uploadsB        r lfs_actioncCs:t|dtr|ddust|dtstd|S).validates response from the LFS batch endpointr}r{Nz"lfs_action is improperly formatted)rirhrAdictrk)rr,r,r-rs rlfs_batch_actionscCspt|dtrt|dtstd|did}|did}|dur.t||dur6t||S)rrQr z)lfs_batch_actions is improperly formattedryrMrzN)rirhrAr?rkr)rrrr,r,r-r_s r_lfs_batch_errorcCsht|dtrt|dtstd|d}t|tr.t|dtr.t|dts2td|S)rrQr z'lfs_batch_error is improperly formattedr^messagecode)rirhrAr?rkr)r error_infor,r,r-rc's  rcr~cCsH|jdd}td||d}t|WddS1swYdS)a Uploads `fileobj` as a single PUT HTTP request (basic LFS transfer protocol) Args: upload_url (`str`): The URL to PUT the file to. fileobj: The file-like object holding the data to upload. Returns: `requests.Response` Raises: [`HTTPError`](https://requests.readthedocs.io/en/latest/api/#requests.HTTPError) If the upload resulted in an error. T with_tqdmPUTr/N)as_filerr)rUr~r3responser,r,r-r5s "rr{r|cCst||j|d}tj}tjr!t|jts!t|jts!t dd}|r*t |||dnt |||d}t j |t||jjtd}t|dS)z@ Uploads file using HF multipart LFS transfer protocol. )r{rr|zlhf_transfer is enabled but does not support uploading from bytes or BinaryIO, falling back to regular uploadF)rUsorted_parts_urlsr|)r[rHN)_get_sorted_parts_urlsrrHF_HUB_ENABLE_HF_TRANSFERripath_or_fileobjrArwarningswarn_upload_parts_hf_transfer_upload_parts_iterativelyrrg_get_completion_payloadrrRrfr)rUr{r|r~ruse_hf_transferresponse_headerscompletion_resr,r,r-rKs,    rrcCsNddtdd|DdddD}t|}|t|j|kr%td|S)NcSsg|]\}}|qSr,r,)rS_r~r,r,r-rTosz*_get_sorted_parts_urls..cSs2g|]\}}|rt|dkrt|d|fqS)r )isdigitr1r?)rSpart_numr~r,r,r-rTrs  cSs|dS)Nrr,)tr,r,r-wsz(_get_sorted_parts_urls..)keyz0Invalid server response to upload large LFS file)sorteditemsr1rr rk)r{rr|sorted_part_upload_urls num_partsr,r,r-rns rrrQcCsfg}t|D]'\}}|d}|dus|dkr#td|d|d||d|dq||dS)NrurKzInvalid etag (`z`) returned for part r)rtru)rQrw) enumeraterhrkappend)rrQrw part_numberr{rur,r,r-rs  rrc Csg}|jdd<}t|D].\}}t||||d}td||d}t|||jWdn1s6wYq Wd|S1sGwY|S)NTr) seek_from read_limitrr)rrrrrrrH) rUrr|rHr3part_idxpart_upload_url fileobj_slicepart_upload_resr,r,r-rs&   rc Cszddlm}Wn tytdwdt|jv}|s#td|j j }|j }t |dkr9d|dd}t d d |d|ttd d ;}z|d|j||d ddd|r\d|jini}Wntyr} ztd| d} ~ ww|sz|||WdS1swYdS)Nr)multipart_uploadzFast uploading using 'hf_transfer' is enabled (HF_HUB_ENABLE_HF_TRANSFER=1) but 'hf_transfer' package is not available in your environment. Try `pip install hf_transfer`.callbackzYou are using an outdated version of `hf_transfer`. Consider upgrading to latest version to enable progress bars using `pip install -U hf_transfer`.(u(…)iBTzhuggingface_hub.lfs_upload)unit unit_scaletotalinitialdescdisablerX) file_path parts_urlsr| max_filesparallel_failures max_retriesz~An error occurred while uploading using `hf_transfer`. Consider disabling HF_HUB_ENABLE_HF_TRANSFER for better error handling.r,) hf_transferr ImportErrorrkinspect signature parametersrrrr rr1rrrgetEffectiveLevelrupdate Exception RuntimeError) rUrr|rsupports_callbackrrprogressoutputer,r,r-rsb      $r)NNNN)NNN)Dr<rr&rer dataclassesrmathros.pathrpathlibrtypingrrrr r r r r urllib.parserhuggingface_hubrutilsrrrrrrrr utils._lfsr utils.sharr utils.tqdmr _commit_apir get_loggerr9rcompile OID_REGEXLFS_MULTIPART_UPLOAD_COMMANDrfrrArrrrsrvrrr_rcrr?rrrrrr,r,r,r-s    (  (     )  V  F #