o i3@sVUdZddlmZddlZddlZddlZddlmZmZddl m Z ddl m Z ddl mZmZddlmZdd lmZmZeeZd Zd Zd Zd ZdZdZdZdZdZ dZ!dZ"dZ#dZ$dZ%dZ&dZ'dZ(dZ)dZ*dZ+dZ,dZ-d Z.d!Z/d"Z0d#Z1d$Z2d%Z3d&Z4d'Z5d(Z6d)Z7d*Z8d+Z9d,Z:d-Z;d.Zd1Z?d2Z@d3ZAd4ZBd5ZCd6ZDd7ZEd8ZFd9ZGd:ZHd;ZIdZLd?ZMd@ZNdAZOdBZPdCZQeGdDdEdEeZReGdFdGdGeZSGdHdIdIeZTGdJdKdKeZUGdLdMdMeZVdNZWdOZXdPZYdQZZeYeZgeYeZgeYeZgeYgeYgeYeZgeYgeYgeZeYgeZeYgeZeYgeZeYgeZgdR Z[dSe\dT<ddd[d\Z]d]Z^ded_d`Z_edfdfdbdcZ`dS)gaBase FileIO classes for implementing reading and writing table files. The FileIO abstraction includes a subset of full filesystem implementations. Specifically, Iceberg needs to read or write a file at a given location (as a seekable stream), as well as check if a file exists. An implementation of the FileIO abstract base class is responsible for returning an InputFile instance, an OutputFile instance, and deleting a file given its location. ) annotationsN)ABCabstractmethod)SEEK_SET) TracebackType)Protocolruntime_checkable)urlparse) EMPTY_DICT Propertieszclient.profile-namez client.regionzclient.access-key-idzclient.secret-access-keyzclient.session-tokenzclient.role-arnzclient.role-session-namezs3.profile-namez s3.anonymousz s3.endpointzs3.access-key-idzs3.secret-access-keyzs3.session-tokenz s3.regionzs3.resolve-regionz s3.proxy-urizs3.connect-timeoutzs3.request-timeoutz s3.signerz s3.signer.urizs3.signer.endpointzv1/aws/s3/signz s3.role-arnzs3.role-session-namezs3.force-virtual-addressingzs3.retry-strategy-implz hdfs.hostz hdfs.portz hdfs.userzhdfs.kerberos_ticketzadls.connection-stringzadls.credentialzadls.account-namezadls.account-keyzadls.sas-tokenzadls.tenant-idzadls.client-idzadls.client-secretzadls.account-hostzadls.blob-storage-authorityzadls.dfs-storage-authorityzadls.blob-storage-schemezadls.dfs-storage-schemez adls.tokenz adls.anonzgcs.oauth2.tokenzgcs.oauth2.token-expires-atzgcs.project-idz gcs.accesszgcs.consistencyzgcs.cache-timeoutzgcs.requester-payszgcs.session-kwargszgcs.service.hostzgcs.default-bucket-locationzgcs.version-awarez hf.endpointzhf.tokenc@sfeZdZdZeddddZeefdd d Zed d dZed!ddZ d"ddZ ed#ddZ dS)$ InputStreama A protocol for the file-like object returned by InputFile.open(...). This outlines the minimally required methods for a seekable input stream returned from an InputFile implementation's `open(...)` method. These methods are a subset of IOBase/RawIOBase. rsizeintreturnbytescCdSN)selfr rrS/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/pyiceberg/io/__init__.pyreadpzInputStream.readoffsetwhencecCrrr)rrrrrrseeksrzInputStream.seekcCrrrrrrrtellvrzInputStream.tellNonecCrrrrrrrcloseyrzInputStream.closecCdS)zCProvide setup when opening an InputStream using a 'with' statement.Nrrrrr __enter__|zInputStream.__enter__exctypetype[BaseException] | NoneexcinstBaseException | NoneexctbTracebackType | NonecCrz=Perform cleanup when exiting the scope of a 'with' statement.Nrrr"r$r&rrr__exit__r!zInputStream.__exit__N)r)r rrr)rrrrrrrrrr)rr r"r#r$r%r&r'rr) __name__ __module__ __qualname____doc__rrrrrrr r*rrrrr hs   r c@sHeZdZdZedddZedd d Zedd d ZedddZdS) OutputStreamaA protocol for the file-like object returned by OutputFile.create(...). This outlines the minimally required methods for a writable output stream returned from an OutputFile implementation's `create(...)` method. These methods are a subset of IOBase/RawIOBase. brrrcCrrr)rr3rrrwriterzOutputStream.writercCrrrrrrrrrzOutputStream.closecCr)zDProvide setup when opening an OutputStream using a 'with' statement.Nrrrrrr r!zOutputStream.__enter__r"r#r$r%r&r'cCrr(rr)rrrr*r!zOutputStream.__exit__N)r3rrrr,)rr2r-) r.r/r0r1rr4rr r*rrrrr2s   r2c@sTeZdZdZdddZeddd Zedd d Zedd dZ eddddZ dS) InputFileaA base class for InputFile implementations. Args: location (str): A URI or a path to a local file. Attributes: location (str): The URI or path to a local file for an InputFile instance. exists (bool): Whether the file exists or not. locationstrcC ||_dSr _locationrr6rrr__init__ zInputFile.__init__rrcCrz.Return the total length of the file, in bytes.Nrrrrr__len__r!zInputFile.__len__cC|jS)z/The fully-qualified location of the input file.r9rrrrr6zInputFile.locationboolcCrzCheck whether the location exists. Raises: PermissionError: If the file at self.location cannot be accessed due to a permission error. Nrrrrrexistsr!zInputFile.existsTseekabler cCr)aReturn an object that matches the InputStream protocol. Args: seekable: If the stream should support seek, or if it is consumed sequential. Returns: InputStream: An object that matches the InputStream protocol. Raises: PermissionError: If the file at self.location cannot be accessed due to a permission error. FileNotFoundError: If the file at self.location does not exist. Nr)rrErrropenr!zInputFile.openNr6r7r+rr7rrB)T)rErBrr ) r.r/r0r1r<rr?propertyr6rDrFrrrrr5s    r5c@sbeZdZdZdddZeddd Zedd d Zedd dZ edddZ eddddZ dS) OutputFileaA base class for OutputFile implementations. Args: location (str): A URI or a path to a local file. Attributes: location (str): The URI or path to a local file for an OutputFile instance. exists (bool): Whether the file exists or not. r6r7cCr8rr9r;rrrr<r=zOutputFile.__init__rrcCrr>rrrrrr?r!zOutputFile.__len__cCr@)z0The fully-qualified location of the output file.r9rrrrr6rAzOutputFile.locationrBcCrrCrrrrrrDr!zOutputFile.existsr5cCr)z9Return an InputFile for the location of this output file.Nrrrrr to_input_filer!zOutputFile.to_input_fileF overwriter2cCr)a Return an object that matches the OutputStream protocol. Args: overwrite (bool): If the file already exists at `self.location` and `overwrite` is False a FileExistsError should be raised. Returns: OutputStream: An object that matches the OutputStream protocol. Raises: PermissionError: If the file at self.location cannot be accessed due to a permission error. FileExistsError: If the file at self.location already exists and `overwrite=False`. Nr)rrMrrrcreater!zOutputFile.createNrGr+rHrI)rr5)F)rMrBrr2) r.r/r0r1r<rr?rJr6rDrLrNrrrrrKs     rKc@sReZdZUdZded<efdddZedd d Zedd dZ edddZ dS)FileIOz(A base class for FileIO implementations.r propertiescCr8r)rP)rrPrrrr<r=zFileIO.__init__r6r7rr5cCr)zGet an InputFile instance to read bytes from the file at the given location. Args: location (str): A URI or a path to a local file. Nrr;rrr new_inputr!zFileIO.new_inputrKcCr)zGet an OutputFile instance to write bytes to the file at the given location. Args: location (str): A URI or a path to a local file. Nrr;rrr new_outputr!zFileIO.new_outputstr | InputFile | OutputFilercCr)aDelete the file at the given path. Args: location (Union[str, InputFile, OutputFile]): A URI or a path to a local file--if an InputFile instance or an OutputFile instance is provided, the location attribute for that instance is used as the URI to delete. Raises: PermissionError: If the file at location cannot be accessed due to a permission error. FileNotFoundError: When the file at the provided location does not exist. Nrr;rrrdeleter!z FileIO.deleteN)rPr )r6r7rr5)r6r7rrK)r6rSrr) r.r/r0r1__annotations__r r<rrQrRrTrrrrrOs   rOr6 warehousez"pyiceberg.io.pyarrow.PyArrowFileIOz pyiceberg.io.fsspec.FsspecFileIO) s3s3as3nossgsfilehdfsviewfsabfsabfsswasbwasbshfzdict[str, list[str]]SCHEMA_TO_FILE_IOio_implr7rPr r FileIO | NonecCsz/|d}t|dkrtd|d|dd|d}}t|}t||}||WStyGtj d|t t j dYdSw)N.z;py-io-impl should be full path (module.CustomFileIO), got: Could not initialize FileIO: )exc_info) splitlen ValueErrorjoin importlib import_modulegetattrModuleNotFoundErrorloggerwarning isEnabledForloggingDEBUG)rerP path_parts module_name class_namemoduleclass_rrr_import_file_io=s      r~z py-io-implpathcCs\t|}|jr,t|j}r!|D] }t||}r|SqdStjd|jdddS)Nz-No preferred file implementation for scheme: rh) stacklevel)r schemerdgetr~warningswarn)rrP parsed_urlfile_ios file_io_pathfile_iorrr_infer_file_io_from_schemeNsr str | Nonec Cs|t}rt||}rtd||Std||r(t||}r(|S|t}r8t||}r8|Sztdddlm }||WSt yY}zt d|d}~ww)NzLoaded FileIO: %srjzDefaulting to PyArrow FileIOr) PyArrowFileIOzCould not load a FileIO, please consider installing one: pip3 install "pyiceberg[pyarrow]", for more options refer to the docs.) r PY_IO_IMPLr~rtinfornr WAREHOUSEpyiceberg.io.pyarrowrrs)rPr6rerwarehouse_locationrerrr load_file_ioZs.    r)rer7rPr rrf)rr7rPr rrf)rPr r6rrrO)ar1 __future__rrprwrabcrriortypesrtypingrr urllib.parser pyiceberg.typedefr r getLoggerr.rtAWS_PROFILE_NAME AWS_REGIONAWS_ACCESS_KEY_IDAWS_SECRET_ACCESS_KEYAWS_SESSION_TOKEN AWS_ROLE_ARNAWS_ROLE_SESSION_NAMES3_PROFILE_NAME S3_ANONYMOUS S3_ENDPOINTS3_ACCESS_KEY_IDS3_SECRET_ACCESS_KEYS3_SESSION_TOKEN S3_REGIONS3_RESOLVE_REGION S3_PROXY_URIS3_CONNECT_TIMEOUTS3_REQUEST_TIMEOUT S3_SIGNER S3_SIGNER_URIS3_SIGNER_ENDPOINTS3_SIGNER_ENDPOINT_DEFAULT S3_ROLE_ARNS3_ROLE_SESSION_NAMES3_FORCE_VIRTUAL_ADDRESSINGS3_RETRY_STRATEGY_IMPL HDFS_HOST HDFS_PORT HDFS_USERHDFS_KERB_TICKETADLS_CONNECTION_STRINGADLS_CREDENTIALADLS_ACCOUNT_NAMEADLS_ACCOUNT_KEYADLS_SAS_TOKENADLS_TENANT_IDADLS_CLIENT_IDADLS_CLIENT_SECRETADLS_ACCOUNT_HOSTADLS_BLOB_STORAGE_AUTHORITYADLS_DFS_STORAGE_AUTHORITYADLS_BLOB_STORAGE_SCHEMEADLS_DFS_STORAGE_SCHEME ADLS_TOKEN ADLS_ANON GCS_TOKENGCS_TOKEN_EXPIRES_AT_MSGCS_PROJECT_ID GCS_ACCESSGCS_CONSISTENCYGCS_CACHE_TIMEOUTGCS_REQUESTER_PAYSGCS_SESSION_KWARGSGCS_SERVICE_HOSTGCS_DEFAULT_LOCATIONGCS_VERSION_AWARE HF_ENDPOINTHF_TOKENr r2r5rKrOLOCATIONr ARROW_FILE_IOFSSPEC_FILE_IOrdrUr~rrrrrrrs     /4&