o ei@sUdZddlZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z ddl mZddlmZmZmZmZmZmZddlmZddlZddlmZd d lmZd d lmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%d d lm&Z&d dl'm(Z(d dl)m*Z*e&+e,Z-e ddGdddZ.e/dZ0e/dZ1dee2e2fdee.fddZ3dZ4dZ5dZ6ej/dej7dZ8ej/dej7dZ9d ej:ddfd!d"Z;d ej:ddfd#d$Zdej?fd(d)Z@dejAfd*d+ZBegej?fZCegejAfZDeEZFe@aGeCeHd,<eBaIeDeHd-<daJeej?eHd.<d/eCddfd0d1ZKd2eDddfd3d4ZLdej?fd5d6ZMdejAfd7d8ZNdkd9d:ZOePeOeQed;rNejReOd<ejSejTfZUeVeWeXd=feHd><d?ZYeVeZd=feHd@<dAd dBeUeYdCdDdEe*dFe2dGeZdHe[dIe[dJeeWeXeVeWeXd=ffdKeeZeVeZd=ffdLe\deej=ddffdMdNZ]dAd dBeUeYdOdEe*dFe2dGeZdHe[dIe[dJeeWeXeVeWeXd=ffdKeeZeVeZd=ffdej=fdPdQZ^e dAd dBeUeYdOdEe*dFe2dGeZdHe[dIe[dJeeWeXeVeWeXd=ffdKeeZeVeZd=ffdeej=ddffdRdSZ_dCdTdEe*dFe2dUe\dej=fdVdWZ`dFe2dXee2de2fdYdZZadld%ej=d[ee2ddfd\d]ZbecZdd%ej=ddfd^d_Zed`eWe"dae2d%ej=de"fdbdcZfd ej:de2fdddeZge/dfejhZidgee2dheZdee2fdidjZjdS)mz>Contains utilities to handle HTTP requests in huggingface_hub.N)contextmanager) dataclass)quote)AnyCallable GeneratorMappingOptionalUnion)urlparse)OfflineModeIsEnabled) constants)BadRequestErrorBucketNotFoundErrorDisabledRepoErrorGatedRepoErrorHfHubHTTPErrorRemoteEntryNotFoundErrorRepositoryNotFoundErrorRevisionNotFoundError)logging) SliceFileObj) HTTP_METHOD_TT)frozenc@sJeZdZUdZeed<eed<eed<dZeeed<dZ eeed<dS) RateLimitInfoa Parsed rate limit information from HTTP response headers. Attributes: resource_type (`str`): The type of resource being rate limited. remaining (`int`): The number of requests remaining in the current window. reset_in_seconds (`int`): The number of seconds until the rate limit resets. limit (`int`, *optional*): The maximum number of requests allowed in the current window. window_seconds (`int`, *optional*): The number of seconds in the current window. resource_type remainingreset_in_secondsNlimitwindow_seconds) __name__ __module__ __qualname____doc__str__annotations__intr r r!r)r)Y/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/huggingface_hub/utils/_http.pyr6s  rzL\"(?P\w+)\"\s*;\s*r\s*=\s*(?P\d+)\s*;\s*t\s*=\s*(?P\d+)z'q\s*=\s*(?P\d+).*?w\s*=\s*(?P\d+)headersreturnc Csd}d}|D]}|}|dkr||}q|dkr||}q|s"dSt|}|s+dS|d}t|d}t|d}d} d} |rYt|} | rYt| d} t| d} t|||| | d S) a*Parse rate limit information from HTTP response headers. Follows IETF draft: https://www.ietf.org/archive/id/draft-ietf-httpapi-ratelimit-headers-09.html Only a subset is implemented. Example: ```python >>> from huggingface_hub.utils import parse_ratelimit_headers >>> headers = { ... "ratelimit": '"api";r=0;t=55', ... "ratelimit-policy": '"fixed window";"api";q=500;w=300', ... } >>> info = parse_ratelimit_headers(headers) >>> info.remaining 0 >>> info.reset_in_seconds 55 ``` N ratelimitzratelimit-policyrrtqw)rrrr r!)lower_RATELIMIT_REGEXsearchgroupr(_RATELIMIT_POLICY_REGEXr) r+r-policykey lower_keymatchrrrr r! policy_matchr)r)r*parse_ratelimit_headersRs>    r<z x-request-idzX-Amzn-Trace-Idz x-amz-cf-ida  # staging or production endpoint ^https://[^/]+ ( # on /api/repo_type/repo_id /api/(models|datasets|spaces)/(.+) | # or /repo_id/resolve/revision/... /(.+)/resolve/(.+) ) )flagszz # staging or production endpoint ^https?://[^/]+ # on /api/buckets/... /api/buckets/ requestc Cstr td|jdt|jvr!|jtptt |jt<|jt}t d||j |j|jddutjrCt dt||S)z Event hook that will be used to make HTTP requests to the Hugging Face Hub. What it does: - Block requests if offline mode is enabled - Add a request ID to the request headers - Log the request if debug mode is enabled z Cannot reach za: offline mode is enabled. To disable it, please unset the `HF_HUB_OFFLINE` environment variable.z%Request %s: %s %s (authenticated: %s) authorizationNzSend: %s)ris_offline_moder urlX_AMZN_TRACE_IDr+get X_REQUEST_IDr&uuiduuid4loggerdebugmethodHF_DEBUG_curlify)r> request_idr)r)r*hf_request_event_hooks"    rMcs t|S)z3 Async version of `hf_request_event_hook`. )rM)r>r)r)r*async_hf_request_event_hooksrNresponsecsd|jdkr,d|jvr.z t|jd}Wn tyYdSw|dkr0|IdHdSdSdSdS)NzContent-lengthi@B) status_coder+r( ValueErroraread)rOlengthr)r)r*async_hf_response_event_hooks   rUcCstjdtgidddS)zQ Factory function to create a `httpx.Client` with the default transport. r>TN event_hooksfollow_redirectstimeout)httpxClientrMr)r)r)r*default_client_factorys r\cCstjtgtgddddS)zV Factory function to create a `httpx.AsyncClient` with the default transport. )r>rOTNrV)rZ AsyncClientrNrUr)r)r)r*default_async_client_factorys  r^_GLOBAL_CLIENT_FACTORY_GLOBAL_ASYNC_CLIENT_FACTORY_GLOBAL_CLIENTclient_factorycCs4tt|aWddS1swYdS)a Set the HTTP client factory to be used by `huggingface_hub`. The client factory is a method that returns a `httpx.Client` object. On the first call to [`get_client`] the client factory will be used to create a new `httpx.Client` object that will be shared between all calls made by `huggingface_hub`. This can be useful if you are running your scripts in a specific environment requiring custom configuration (e.g. custom proxy or certifications). Use [`get_client`] to get a correctly configured `httpx.Client`. N) _CLIENT_LOCK close_sessionr_)rbr)r)r*set_client_factorys "reasync_client_factorycCs|adS)a Set the HTTP async client factory to be used by `huggingface_hub`. The async client factory is a method that returns a `httpx.AsyncClient` object. This can be useful if you are running your scripts in a specific environment requiring custom configuration (e.g. custom proxy or certifications). Use [`get_async_client`] to get a correctly configured `httpx.AsyncClient`. Contrary to the `httpx.Client` that is shared between all calls made by `huggingface_hub`, the `httpx.AsyncClient` is not shared. It is recommended to use an async context manager to ensure the client is properly closed when the context is exited. Nr`)rfr)r)r*set_async_client_factorysrhcCs8tdurt taWdtS1swYtS)a Get a `httpx.Client` object, using the transport factory from the user. This client is shared between all calls made by `huggingface_hub`. Therefore you should not close it manually. Use [`set_client_factory`] to customize the `httpx.Client`. N)rarcr_r)r)r)r* get_session&s  ricCstS)a Return a `httpx.AsyncClient` object, using the transport factory from the user. Use [`set_async_client_factory`] to customize the `httpx.AsyncClient`. Contrary to the `httpx.Client` that is shared between all calls made by `huggingface_hub`, the `httpx.AsyncClient` is not shared. It is recommended to use an async context manager to ensure the client is properly closed when the context is exited. rgr)r)r)r*get_async_session5s rjc CsZt}da|dur+z|WdSty*}ztd|WYd}~dSd}~wwdS)z Close the global `httpx.Client` used by `huggingface_hub`. If a Client is closed, it will be recreated on the next call to [`get_session`]. Can be useful if e.g. an SSL certificate has been updated. NzError closing client: )raclose ExceptionrGwarning)clienter)r)r*rdEs rdregister_at_fork)after_in_child._DEFAULT_RETRY_ON_EXCEPTIONS)iiii_DEFAULT_RETRY_ON_STATUS_CODESF) max_retriesbase_wait_time max_wait_timeretry_on_exceptionsretry_on_status_codesstreamrIrArwrxryrzr{r|c +st|tr |f}ttrfd|} dd} d|vr-t|dtjtfr-|d} t} d7dzb| durC|d| dt j dt ffdd } |r| j dd |} | | ss| V WdWdSWdn1s}wYn| j dd |} | | s| VWdSWn/|y}z#td |d d t|t jrtkr|WYd}~nd}~wwdurtd}td|dddn| }td|dddt|t|| d} q1)zfInternal implementation of HTTP backoff logic shared between `http_backoff` and `http_stream_backoff`.rNdataTrrOr,csf|jvrdStd|jddkr t|dS|jdkr1t|j}|dur1|jdS)zNHandle response and return True if should retry, False if should return/yield.Fz HTTP Error z thrown while requesting  rsNT)rQrGrmhf_raise_for_statusr<r+r)rOratelimit_inforwrInb_triesratelimit_resetr{rAr)r* _should_retrys   z)_http_backoff_base.._should_retryrIrA'z' thrown while requesting r~zRate limited. Waiting zs before retry [Retry /z].z Retrying in z s [Retry r r)) isinstancetyper(ioIOBasertellriseekrZResponseboolr|r>rGrm ConnectErrorrdfloattimesleepmin)rIrArwrxryrzr{r|kwargs sleep_timeio_obj_initial_posrnrrOerr actual_sleepr)rr*_http_backoff_basedsd   "     r)rwrxryrzr{c Ks$ttd|||||||dd|S)a0 Wrapper around httpx to retry calls on an endpoint, with exponential backoff. Endpoint call is retried on exceptions (ex: connection timeout, proxy error,...) and/or on specific status codes (ex: service unavailable). If the call failed more than `max_retries`, the exception is thrown or `raise_for_status` is called on the response object. Re-implement mechanisms from the `backoff` library to avoid adding an external dependencies to `hugging_face_hub`. See https://github.com/litl/backoff. Args: method (`Literal["GET", "OPTIONS", "HEAD", "POST", "PUT", "PATCH", "DELETE"]`): HTTP method to perform. url (`str`): The URL of the resource to fetch. max_retries (`int`, *optional*, defaults to `5`): Maximum number of retries, defaults to 5 (no retries). base_wait_time (`float`, *optional*, defaults to `1`): Duration (in seconds) to wait before retrying the first time. Wait time between retries then grows exponentially, capped by `max_wait_time`. max_wait_time (`float`, *optional*, defaults to `8`): Maximum duration (in seconds) to wait before retrying. retry_on_exceptions (`type[Exception]` or `tuple[type[Exception]]`, *optional*): Define which exceptions must be caught to retry the request. Can be a single type or a tuple of types. By default, retry on `httpx.TimeoutException` and `httpx.NetworkError`. retry_on_status_codes (`int` or `tuple[int]`, *optional*, defaults to `(429, 500, 502, 503, 504)`): Define on which status codes the request must be retried. By default, retries on rate limit (429) and server errors (5xx). **kwargs (`dict`, *optional*): kwargs to pass to `httpx.request`. Example: ``` >>> from huggingface_hub.utils import http_backoff # Same usage as "httpx.request". >>> response = http_backoff("GET", "https://www.google.com") >>> response.raise_for_status() # If you expect a Gateway Timeout from time to time >>> http_backoff("PUT", upload_url, data=data, retry_on_status_codes=504) >>> response.raise_for_status() ``` > [!WARNING] > When using `requests` it is possible to stream data by passing an iterator to the > `data` argument. On http backoff this is a problem as the iterator is not reset > after a failed call. This issue is mitigated for file objects or any IO streams > by saving the initial position of the cursor (with `data.tell()`) and resetting the > cursor between each call (with `data.seek()`). For arbitrary iterators, http backoff > will fail. If this is a hard constraint for you, please let us know by opening an > issue on [Github](https://github.com/huggingface/huggingface_hub). FrIrArwrxryrzr{r|Nr))nextrrIrArwrxryrzr{rr)r)r* http_backoffsA rc ks,td|||||||dd|EdHdS)a Wrapper around httpx to retry calls on an endpoint, with exponential backoff. Endpoint call is retried on exceptions (ex: connection timeout, proxy error,...) and/or on specific status codes (ex: service unavailable). If the call failed more than `max_retries`, the exception is thrown or `raise_for_status` is called on the response object. Re-implement mechanisms from the `backoff` library to avoid adding an external dependencies to `hugging_face_hub`. See https://github.com/litl/backoff. Args: method (`Literal["GET", "OPTIONS", "HEAD", "POST", "PUT", "PATCH", "DELETE"]`): HTTP method to perform. url (`str`): The URL of the resource to fetch. max_retries (`int`, *optional*, defaults to `5`): Maximum number of retries, defaults to 5 (no retries). base_wait_time (`float`, *optional*, defaults to `1`): Duration (in seconds) to wait before retrying the first time. Wait time between retries then grows exponentially, capped by `max_wait_time`. max_wait_time (`float`, *optional*, defaults to `8`): Maximum duration (in seconds) to wait before retrying. retry_on_exceptions (`type[Exception]` or `tuple[type[Exception]]`, *optional*): Define which exceptions must be caught to retry the request. Can be a single type or a tuple of types. By default, retry on `httpx.TimeoutException` and `httpx.NetworkError`. retry_on_status_codes (`int` or `tuple[int]`, *optional*, defaults to `(429, 500, 502, 503, 504)`): Define on which status codes the request must be retried. By default, retries on rate limit (429) and server errors (5xx). **kwargs (`dict`, *optional*): kwargs to pass to `httpx.request`. Example: ``` >>> from huggingface_hub.utils import http_stream_backoff # Same usage as "httpx.stream". >>> with http_stream_backoff("GET", "https://www.google.com") as response: ... for chunk in response.iter_bytes(): ... print(chunk) # If you expect a Gateway Timeout from time to time >>> with http_stream_backoff("PUT", upload_url, data=data, retry_on_status_codes=504) as response: ... response.raise_for_status() ``` When using `httpx` it is possible to stream data by passing an iterator to the `data` argument. On http backoff this is a problem as the iterator is not reset after a failed call. This issue is mitigated for file objects or any IO streams by saving the initial position of the cursor (with `data.tell()`) and resetting the cursor between each call (with `data.seek()`). For arbitrary iterators, http backoff will fail. If this is a hard constraint for you, please let us know by opening an issue on [Github](https://github.com/huggingface/huggingface_hub). TrNr))rrr)r)r*http_stream_backoffsF r)retry_on_errorsrcKs|rinddd} td||d|ddi|}t|d|jkr)dkrEn |St|jd }|jd krEt|j|jd }q |S) aFPerform an HTTP request with backoff and follow relative redirects only. Used to fetch HEAD /resolve on repo or bucket files. This is useful to follow a redirection to a renamed repository without following redirection to a CDN. A backoff mechanism retries the HTTP call on errors (429, 5xx, timeout, network errors). Args: method (`str`): HTTP method, such as 'GET' or 'HEAD'. url (`str`): The URL of the resource to fetch. retry_on_errors (`bool`, *optional*, defaults to `False`): Whether to retry on errors. If False, no retry is performed (fast fallback to local cache). If True, uses default retry behavior (429, 5xx, timeout, network errors). **httpx_kwargs (`dict`, *optional*): Params to pass to `httpx.request`. r))rzr{TrrXFi,iLocation)path) rrrQr r+netloc_replacergeturl)rIrAr httpx_kwargsno_retry_kwargsrO parsed_targetr)r)r*-_httpx_follow_relative_redirects_with_backoffhs. rendpointcCsD|r|dntj}|tjtjfvr |tj|}|tj|}|S)zReplace the default endpoint in a URL by a custom one. This is useful when using a proxy and the Hugging Face Hub returns a URL with the default endpoint. r)rstriprENDPOINT_HF_DEFAULT_ENDPOINT_HF_DEFAULT_STAGING_ENDPOINTreplace)rArr)r)r*fix_hf_endpoint_in_urls r endpoint_namec Cszt|WntytjdddYnwz|WdStjy}z|jddkr7WYd}~dS|j d}|j d}|d kr]|jd d d |j d }t t ||||dkrw|jd d d|j d }t t ||||dkr|jd d d|j d }t t||||dkr|jd d d|j d dd}t t||||dkr|jdur|jj durtt|jj dur|jd d d|j d dd}t t||||dks |jdkr$|dkr$|jdur$|jj dur$tt|jj dur$|jd d d|j d d}t t||||jdkr>|dur5d|dnd }t t||||jd!kr^d |jd"|d d#|j d d$}t t||||jd%krt|j}|durd&|jd'}|d(|jd)7}|jdur|jdur|d*|jd+|jd,|jd-7}n|d 7}|d.|j d 7}nd/|j d }t t||||jd0kr|jj d1}|d2|d3|j d4d }t t|||t tt|||d}~ww)5a Internal version of `response.raise_for_status()` that will refine a potential HTTPError. Raised exception will be an instance of [`~errors.HfHubHTTPError`]. This helper is meant to be the unique method to raise_for_status when making a call to the Hugging Face Hub. Args: response (`Response`): Response from the server. endpoint_name (`str`, *optional*): Name of the endpoint that has been called. If provided, the error message will be more complete. > [!WARNING] > Raises when the request has failed: > > - [`~utils.RepositoryNotFoundError`] > If the repository to download from cannot be found. This may be because it > doesn't exist, because `repo_type` is not set correctly, or because the repo > is `private` and you do not have access. > - [`~utils.GatedRepoError`] > If the repository exists but is gated and the user is not on the authorized > list. > - [`~utils.RevisionNotFoundError`] > If the repository exists but the revision couldn't be found. > - [`~utils.EntryNotFoundError`] > If the repository exists but the entry (e.g. the requested file) couldn't be > find. > - [`~utils.BadRequestError`] > If request failed with a HTTP 400 BadRequest error. > - [`~utils.HfHubHTTPError`] > If request failed for a reason not listed above. zFailed to parse warning headersT)exc_infodNz X-Error-CodeX-Error-MessageRevisionNotFoundz Client Error. zRevision Not Found for url: . EntryNotFoundzEntry Not Found for url: GatedRepoz!Cannot access gated repo for url z$Access to this resource is disabled.z!Cannot access repository for url  RepoNotFoundzBucket Not Found for url: zG Please make sure you specified the correct bucket id (namespace/name).z; If the bucket is private, make sure you are authenticated.iz+Invalid credentials in Authorization headerzRepository Not Found for url: z Please make sure you specified the correct `repo_id` and `repo_type`. If you are trying to access a private or gated repo, make sure you are authenticated. For more details, see https://huggingface.co/docs/huggingface_hub/authenticationrPz Bad request for z endpoint:z Bad request:iz Forbidden: z Cannot access content at: z2 Make sure your token has the correct permissions.rsz0 429 Too Many Requests: you have reached your 'z ' rate limit.z Retry after z seconds (rz requests remaining in current z s window).z Url: z! 429 Too Many Requests for url: iRangez. Requested range: z. Content-Range: z Content-Range)_warn_on_warning_headersrlrGrHraise_for_statusrZHTTPStatusErrorrQr+rCrA_formatrrrrr>BUCKET_API_REGEXr4r&rREPO_API_REGEXrrrr<rrr r!r)rOrro error_code error_messagemessager range_headerr)r)r*rs!                        rcCsj|jd}|D]*}d|vr|ddnd|f\}}|}|tvr2|}|r2t|t|qdS)a Emit warnings if warning headers are present in the HTTP response. Expected header format: 'X-HF-Warning: topic; message' Only the first warning for each topic will be shown. Topic is optional and can be empty. Note that several warning headers can be present in a single response. Args: response (`httpx.Response`): The HTTP response to check for warning headers. z X-HF-Warning;rrN)r+get_listsplitstrip_WARNED_TOPICSaddrGrm)rOserver_warningsserver_warningtopicrr)r)r*rDs    r error_typecustom_messagec Cs:g}|jd}|dur||z[z|}Wn tjy8z ||}Wn ty5i}YnwYnw|d}|durRt|t rM| |n|||d}|durk|D] }d|vrj||dq]Wn tj y|jdd}|j rd| vr||j Ynwdd |D}t t|}d |} |} | r| | vrd |vr| d | 7} n| d | 7} d} d} td ftd ftdffD]\} }|j| }|rt|} d|d|d} nq| r| | vrd | vr | d }| d|| | |d} n| | 7} || || pddS)Nrerrorerrorsrz Content-TyperhtmlcSs$g|]}t|rt|qSr))r&r).0liner)r)r* s$z_format..rrz Request IDz Amzn Trace IDz Amz CF IDr: ))rOserver_message)r+rCappendjsonrZResponseNotReadread RuntimeErrorrlistextendJSONDecodeErrortextr2dictfromkeysjoinrDrB X_AMZ_CF_IDr&indexr)rrrO server_errors from_headersr}rr content_typerfinal_error_messagerLrequest_id_messageheaderlabelvalue newline_indexr)r)r*r\sx                rcCsdd|jfg}t|jD]\}}|dkrd}|d|d|fg7}qd}z|jdurF|jjdd d }t|d krF|dd d }Wn tj ySd }Ynw|durc|d| ddfg7}|d|j fg7}g}|D]\}}|r~| t t||r| t t|qod|S)zConvert a `httpx.Request` into a curl command (str). Used for debug purposes only. Implementation vendored from https://github.com/ofw/curlify/blob/master/curlify.py. MIT License Copyright (c) 2016 Egor. )curlNz-Xr?zz-HrNzutf-8ignore)riz ... [truncated]zz-drrr~)rIsortedr+itemsr2contentdecodelenrZRequestNotReadrrArrr&r)r>partskvbody flat_partsr)r)r*rKs8      rKz%^\s*bytes\s*=\s*(\d*)\s*-\s*(\d*)\s*$original_range resume_sizecCs|sd|dSd|vrtd|dt|}|s#td|d|\}}|sN|s5td|dt||}d|}|d krLtd |d|St|}||}|rrt|}d|d|}||krptd |d|Sd|dS) zB Adjust HTTP Range header to account for resume position. zbytes=-,zMultiple ranges detected - z, not supported yet.zInvalid range format - rzbytes=-rzEmpty new range - )rR RANGE_REGEXr:rgroupsr()rrr:startend new_suffix new_range new_startr)r)r*_adjust_range_headers2      r )r,N)N)kr%atexitrrosre threadingrrE contextlibr dataclassesrshlexrtypingrrrrr r urllib.parser rZhuggingface_hub.errorsr rrrrrrrrrrrr_lfsr_typingr get_loggerr"rGrcompiler3r6r&r<rDrBrVERBOSErrRequestrMrNrrUr[r\r]r^CLIENT_FACTORY_TASYNC_CLIENT_FACTORY_TLockrcr_r'r`rarerhrirjrdregisterhasattrrpTimeoutException NetworkErrorrrtuplerrlrtr(rrrrrrrrsetrrrrK IGNORECASErr r)r)r)r*s(       (     =  !           e P S 3  ]*"