o wiX9@sdZddlmZddlZddlZddlmZmZmZddl m Z ddl m Z ddl Z ddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZmZddlmZddlm Z m!Z!m"Z"m#Z#erxddl$m%Z%ddlm&Z&m'Z'dZ(GdddedZ)GdddeZ*dS)aqW&B Public API for File objects. This module provides classes for interacting with files stored in W&B. Example: ```python from wandb.apis.public import Api # Get files from a specific run run = Api().run("entity/project/run_id") files = run.files() # Work with files for file in files: print(f"File: {file.name}") print(f"Size: {file.size} bytes") print(f"Type: {file.mimetype}") # Download file if file.size < 1000000: # Less than 1MB file.download(root="./downloads") # Get S3 URI for large files if file.size >= 1000000: print(f"S3 URI: {file.path_uri}") ``` Note: This module is part of the W&B Public API and provides methods to access, download, and manage files stored in W&B. Files are typically associated with specific runs and can include model weights, datasets, visualizations, and other artifacts. ) annotationsN) TYPE_CHECKINGAnyCallable)gql) RetryError)nameof)Attrs)normalize_exceptions)SizedPaginator)utils)RETRY_TIMEDELTA)Run(_server_provides_internal_id_for_project)retry) POW_2_BYTESdownload_file_from_url no_retry_auth to_human_size)Document)ApiRetryingClientafragment RunFilesFragment on Run { files(names: $fileNames, after: $fileCursor, first: $fileLimit, pattern: $pattern) { edges { node { id name url(upload: $upload) directUrl sizeBytes mimetype updatedAt md5 } cursor } pageInfo { endCursor hasNextPage } } }cseZdZdZd(ddZ    d)d*fdd Zd+ddZed,ddZed-ddZ ed.ddZ d+d d!Z d/d#d$Z d0d&d'Z ZS)1FilesaA lazy iterator over a collection of `File` objects. Access and manage files uploaded to W&B during a run. Handles pagination automatically when iterating through large collections of files. Example: ```python from wandb.apis.public.files import Files from wandb.apis.public.api import Api # Example run object run = Api().run("entity/project/run-id") # Create a Files object to iterate over files in the run files = Files(api.client, run) # Iterate over files for file in files: print(file.name) print(file.url) print(file.size) # Download the file file.download(root="download_directory", replace=True) ``` returnrcCs(t|j}td|r dnddtdS)z8Generate query dynamically based on server capabilities.a( query RunFiles($project: String!, $entity: String!, $name: String!, $fileCursor: String, $fileLimit: Int = 50, $fileNames: [String] = [], $upload: Boolean = false, $pattern: String) { project(name: $project, entityName: $entity) { internalIdz run(name: $name) { fileCount ...RunFilesFragment } } } z )rclientr FILE_FRAGMENT)selfwith_internal_idr K/home/ubuntu/.local/lib/python3.10/site-packages/wandb/apis/public/files.py _get_queryos   zFiles._get_queryN2Frrrunrnameslist[str] | Noneper_pageintuploadboolpattern str | NonecsF|r|rtd||_|j|j|j|pg||d}t|||dS)a(Initialize a lazy iterator over a collection of `File` objects. Files are retrieved in pages from the W&B server as needed. Args: client: The run object that contains the files run: The run object that contains the files names (list, optional): A list of file names to filter the files per_page (int, optional): The number of files to fetch per page upload (bool, optional): If `True`, fetch the upload URL for each file pattern (str, optional): Pattern to match when returning files from W&B This pattern uses mySQL's LIKE syntax, so matching all files that end with .json would be "%.json". If both names and pattern are provided, a ValueError will be raised. z{Querying for files by both names and pattern is not supported. Please provide either a list of names or a pattern to match.)projectentityname fileNamesr)r+N) ValueErrorr$r-r.idsuper__init__)rrr$r%r'r)r+ variables __class__r r!r4szFiles.__init__NonecCs|jj||jd|_dS)zHFetch and store the response data for the next page using dynamic query.variable_valuesN)rexecuter"r5 last_responserr r r!_update_responses  zFiles._update_responsecCs |js||jdddS)z[ Returns total number of files. r-r$ fileCount)r< _load_pager=r r r!_lengthsz Files._lengthcCs$|jr|jdddddSdS)zbReturns whether there are more files to fetch. r-r$filespageInfo hasNextPageTr<r=r r r!mores z Files.morecCs(|jr|jddddddSdS)zoReturns the cursor position for pagination of file results. r-r$rBedgescursorNrEr=r r r!rIsz Files.cursorcCs|j|j|jddS)zgUpdates the GraphQL query variables for pagination. ) fileLimit fileCursorN)r5updater'rIr=r r r!update_variablesszFiles.update_variables list[File]cs$fddjddddDS)z[Converts GraphQL edges to File objects. cs g|] }tj|djqS)node)Filerr$).0rr=r r! sz)Files.convert_objects..r-r$rBrGrEr=r r=r!convert_objectss zFiles.convert_objectsstrcCs.dtt|dd|jjdt|dS)N< / (z)>)rtypejoinr$pathlenr=r r r!__repr__s.zFiles.__repr__)rr)Nr#FN) rrr$rr%r&r'r(r)r*r+r,rr8rr(rr*)rr,)rrNrrU)__name__ __module__ __qualname____doc__r"r4r>propertyrArFrIrMrTr^ __classcell__r r r6r!rSs$  )     rrPcseZdZdZ d'd(fd d Zed)d dZed*ddZd+ddZe    d,d-ddZ e d.d!d"Z d*d#d$Z e d/d%d&Z ZS)0rPaxFile saved to W&B. Represents a single file stored in W&B. Includes access to file metadata. Files are associated with a specific run and can include text files, model weights, datasets, visualizations, and other artifacts. You can download the file, delete the file, and access file properties. Specify one or more attributes in a dictionary to fine a specific file logged to a specific run. You can search using the following keys: - id (str): The ID of the run that contains the file - name (str): Name of the file - url (str): path to file - direct_url (str): path to file in the bucket - sizeBytes (int): size of file in bytes - md5 (str): md5 of file - mimetype (str): mimetype of file - updated_at (str): timestamp of last update - path_uri (str): path to file in the bucket, currently only available for S3 objects and reference files Args: client: The run object that contains the file attrs (dict): A dictionary of attributes that define the file run: The run object that contains the file Nrrattrsdict[str, Any]r$ Run | Nonecs2||_||_||_d|_d|_tt|dSN)r_attrsr$+server_supports_delete_file_with_project_id_download_decoratedr3r4dict)rrrir$r6r r!r4s z File.__init__rr(cCs|jd}|dur t|SdS)z&Returns the size of the file in bytes. sizeBytesNr)rmr()r size_bytesr r r!sizes z File.sizerUcCs^|jd}stddS||jdkr|Szt|WSty.tdYdSw)a Returns the URI path to the file in the storage bucket. Returns: str: The S3 URI (e.g., 's3://bucket/path/to/file') if the file is stored in S3, the direct URL if it's a reference file, or an empty string if unavailable. directUrlz!Unable to find direct_url of filerurlz1path_uri is only available for files stored in S3)rmgetwandbtermwarnr parse_s3_url_to_s3_urir1)r direct_urlr r r!path_uris    z File.path_uriCallable[..., io.TextIOWrapper]cs:ddl}tjttt|jfd    ddfdd }|S)Nr)retry_timedeltacheck_retry_fnretryable_exceptions.FrootrUreplacer*exist_okapi Api | Nonerio.TextIOWrappercs\|durt}tj|j}tj|r"|s"|rt|Stdt |j |j t|S)NzgFile already exists, pass replace=True to overwrite or exist_ok=True to leave it as is and don't error.) rwrosr\r[r/existsopenr1rruapi_key)rrrrr\r=r r!_impl6s z+File._build_download_wrapper.._implrFFN rrUrr*rr*rrrr)requestsr retriabler rrRequestException)rrrr r=r!_build_download_wrapper3szFile._build_download_wrapperrFrrr*rrrrcCs$|jdur ||_|||||S)aDownloads a file previously saved by a run from the wandb server. Args: root: Local directory to save the file. Defaults to the current working directory ("."). replace: If `True`, download will overwrite a local file if it exists. Defaults to `False`. exist_ok: If `True`, will not raise ValueError if file already exists and will not re-download unless replace=True. Defaults to `False`. api: If specified, the `Api` instance used to download the file. Raises: `ValueError` if file already exists, `replace=False` and `exist_ok=False`. N)ror)rrrrrr r r!downloadRs  z File.downloadr8cCsXd}d}d|jgi}|r|jj|d<d}d}d||}t|}|jj||ddS) z$Delete the file from the W&B server.rrB projectIdz, $projectId: IntzprojectId: $projectIdz mutation deleteFiles($files: [ID!]!{}) {{ deleteFiles(input: {{ files: $files {} }}) {{ success }} }} r9N)r2*_server_accepts_project_id_for_delete_filer$_project_internal_idformatrrr;)rproject_id_mutation_fragmentproject_id_variable_fragmentr:mutation_stringmutationr r r!deletens     z File.deletec Cs<tt|}t|jtd}d|d|jd|jd|d S)N)unitsrVrWrYz) >)rrZrrsrr/mimetype)r classnamersr r r!r^s "z File.__repr__cCsNd}|jdur$t|}|j|}ddd|didigDv|_|jS)zReturns True if the server supports deleting files with a projectId. This check is done by utilizing GraphQL introspection in the available fields on the DeleteFiles API. a query ProbeDeleteFilesProjectIdInput { DeleteFilesProjectIdInputType: __type(name:"DeleteFilesInput") { inputFields{ name } } } NrcSsg|]}|dqS)r/r )rQxr r r!rSszCFile._server_accepts_project_id_for_delete_file..DeleteFilesProjectIdInputType inputFields)rnrrr;rv)r query_stringqueryresr r r!rs     z/File._server_accepts_project_id_for_delete_filerl)rrrirjr$rkr`rb)rr|rrr_ra)rcrdrerfr4rgrsr{rr rrr^rrhr r r6r!rPs(!     )+rf __future__riortypingrrr wandb_gqlrwandb_gql.clientrrwwandb._strutilsrwandb.apis.attrsr wandb.apis.normalizer wandb.apis.paginatorr wandb.apis.publicr wandb.apis.public.constr wandb.apis.public.runsrr wandb.sdk.libr wandb.utilrrrrwandb_graphql.language.astrrrrrrPr r r r!s0 "