o ˳ix@snddlZddlZddlZddlZddlmZddlmZmZm Z ddl m Z ddl m Z ddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZddlmZ ddlm!Z"dd l#m$Z$m%Z%m&Z&m'Z'dd l(m)Z)dd l*m+Z+m,Z,dd l-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5dd l6m7Z7m8Z8m9Z9m:Z:m;Z;mZ>m?Z?ddl@mAZAmBZBddlCmDZDmEZEmFZFddlGmHZHddlImJZJmKZKm!Z!mLZLddlMmNZNmOZOddlPmQZQmRZRddlSmTZTmUZUmVZVddlWmXZXmYZYmZZZm[Z[m\Z\m]Z]m^Z^m_Z_m`Z`maZambZbmcZcmdZdmeZemfZfmgZgmhZhmiZimjZjmkZkddllmmZmmnZnerddloZpdZqdZrdZsdZtdZudZvdZwdZxdZydZzd Z{d!Z|d"Z}d#Z~d$Zd%ZeeZGd&d'd'ZdHd(eDd)edBd*ed+efd,d-Zd.Zd/Zd0Zd1ZeLjeLjiZ dId0ed1edBd2eVdBd+eeeffd3d4Zd5e d6eVd+e fd7d8ZeYd9e_d:ead;e^deid?egd@ehdAeed?ekd?eXdBe]dBejdCiZGdDdEdEeEeZGdFdGdGe4ZdS)JN) TracebackType) TYPE_CHECKINGAnyUnion)urlparse)Client)AlreadyExistsExceptionCheckLockRequestEnvironmentContext FieldSchemaInvalidOperationException LockComponent LockLevel LockRequest LockResponse LockStateLockType MetaExceptionNoSuchObjectException SerDeInfoStorageDescriptor UnlockRequest)Database)Table)retryretry_if_exception_typestop_after_attemptwait_exponential)TBinaryProtocol)TSocket TTransport)EXTERNAL_TABLEICEBERGLOCATIONMETADATA_LOCATION TABLE_TYPEURIMetastoreCatalogPropertiesUpdateSummary) CommitFailedExceptionNamespaceAlreadyExistsErrorNamespaceNotEmptyErrorNoSuchIcebergTableErrorNoSuchNamespaceErrorNoSuchPropertyExceptionNoSuchTableErrorTableAlreadyExistsErrorWaitingForLockException)UNPARTITIONED_PARTITION_SPEC PartitionSpec)Schema SchemaVisitorvisit) FromInputFile)CommitTableResponse StagedTablerTableProperties)UNSORTED_SORT_ORDER SortOrder)TableRequirement TableUpdate) EMPTY_DICT Identifier Properties) BinaryType BooleanTypeDateType DecimalType DoubleType FixedType FloatType IntegerTypeListTypeLongTypeMapType NestedField PrimitiveType StringType StructType TimestampTypeTimestamptzTypeTimeType UnknownTypeUUIDType)property_as_boolproperty_as_floatcommentownerzhive.hive2-compatibleFzhive.kerberos-authenticationzhive.kerberos-service-namehivezlock-check-min-wait-timezlock-check-max-wait-timezlock-check-retriesg?<DO_NOT_UPDATE_STATStruec @seZdZUdZeed<eedBed<dee fdededBde dBdedBfd d Z d efd d Z d e fddZd e fddZdeedBdedBdedBd dfddZdS) _HiveClientz4Helper class to nicely open and close the transport. _transportN_ugiuriugi kerberos_authkerberos_service_namecCs4||_||_||_|r|dnd|_||_dSN:)_uri_kerberos_auth_kerberos_service_namesplitra_init_thrift_transportr`)selfrbrcrdrernJ/home/ubuntu/.local/lib/python3.10/site-packages/pyiceberg/catalog/hive.py__init__s z_HiveClient.__init__returncCs>t|j}t|j|j}|jst|Stj||j|j dS)N)hostservice) rrhrhostnameportrir TBufferedTransportTSaslClientTransportrj)rm url_partssocketrnrnrorls  z"_HiveClient._init_thrift_transportcCs*t|j}t|}|jr|j|j|SN)rr`rraset_ugi)rmprotocolclientrnrnro_clients  z_HiveClient._clientc Cs||js:z |jW| Sttjfy9z|jWn ty)Ynw||_|jY| Sw| S)z0Make sure the transport is initialized and open.) r`isOpenopen TypeErrorr TTransportExceptionclose Exceptionrlr~)rmrnrnro __enter__s       z_HiveClient.__enter__exctypeexcinstexctbcCs|jr |jdSdS)z!Close transport if it was opened.N)r`rr)rmrrrrnrnro__exit__s z_HiveClient.__exit__)__name__ __module__ __qualname____doc__r __annotations__liststrHIVE_KERBEROS_AUTH_DEFAULTHIVE_KERBEROS_SERVICE_NAMEboolrprlrr~rtype BaseExceptionrrrnrnrnror_s(   .r_schemalocationhive2_compatiblerqcs,tdd}tfdd|jD|dd|dS)Nz2org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe)serializationLibcs(g|]}t|jt|jt|jqSrn)r namer6 field_typeSchemaToHiveConverterdoc).0fieldrrnro sz6_construct_hive_storage_descriptor..z(org.apache.hadoop.mapred.FileInputFormatz)org.apache.hadoop.mapred.FileOutputFormat) serdeInfo)rrfields)rrr ser_de_infornrro"_construct_hive_storage_descriptors  rEXTERNAL table_typemetadata_locationprevious_metadata_locationmetadata_propertiescCsJtdtdt|i}|r||t<|r#|D]\}}||vr"t|||<q|S)NTRUEr") PROP_EXTERNALPROP_TABLE_TYPEPROP_METADATA_LOCATIONPROP_PREVIOUS_METADATA_LOCATIONitemsr)rrr propertieskeyvaluernrnro_construct_parameterss rdatabasercCsHi}|D]\}}|tkr||_q|tkr||_q|||<q||_|Srz)rCOMMENT descriptionr# locationUri parameters)rrparamsrrrnrnro_annotate_namespaces rbooleanintbigintfloatdoubledatestring timestampztimestamp with local time zonebinaryvoidc@seZdZUeed<defddZdededefddZd e d e edefd d Z d e dedefddZ dededefddZ dedededefddZdedefddZdS)rrcCs ||_dSrzr)rmrrnrnrorp s zSchemaToHiveConverter.__init__r struct_resultrqcCs|Srzrn)rmrrrnrnrorzSchemaToHiveConverter.schemastruct field_resultscCsdd|dS)Nzstruct<,>)join)rmrrrnrnrorszSchemaToHiveConverter.structr field_resultcCs|jd|Srfr)rmrrrnrnrorszSchemaToHiveConverter.field list_typeelement_resultcCs d|dS)NzarrayZ/d"eeBdeefd?d@Z0dMd"eeBdeefdBdCZ1d"eeBdefdDdEZ2defd"eeBdFe3edBd.ede4fdGdHZ5deeBddfdIdJZ6dededefdKdLZ7Z8S)N HiveCatalogr~rrc sNtj|fi||||_t|tt|_t|tt |_ t|t t |_ dSrz)superrp_create_hive_clientr~rWLOCK_CHECK_MIN_WAIT_TIME DEFAULT_LOCK_CHECK_MIN_WAIT_TIME_lock_check_min_wait_timeLOCK_CHECK_MAX_WAIT_TIME DEFAULT_LOCK_CHECK_MAX_WAIT_TIME_lock_check_max_wait_timeLOCK_CHECK_RETRIESDEFAULT_LOCK_CHECK_RETRIES_lock_check_retries)rmrr __class__rnrorp,s  zHiveCatalog.__init__rqc Csd}|tdD]+}zt||dt|tt|ttWSt y4}z|}WYd}~q d}~ww|dur;|t d|t)Nrrcz%Unable to connect to hive using uri: ) r&rkr_getrVHIVE_KERBEROS_AUTHrr"HIVE_KERBEROS_SERVICE_NAME_DEFAULTr ValueError)rlast_exceptionrbernrnror8s    zHiveCatalog._create_hive_clienttablec Cs|j}t|vrtd|jd|j|t}|tkr/td|dtd|jd|j|t }r9|}ntdt d|j |d}| |}t |}t|j|jf||| |j||d S) Nz7Property table_type missing, could not determine type: .zProperty table_type is z , expected z: zTable property z is missingr identifiermetadatariocatalog)rr%r.dbName tableNamelowerr"r,rr$ _load_file_io new_inputr7table_metadatarr) rmrrrprop_metadata_locationrrfilerrnrnro_convert_hive_into_icebergJs.      z&HiveCatalog._convert_hive_into_icebergc Cs|}||t\}}ttd}t|||jr$t|jvr$|jtnt |d|dt | | t |jtttt|j|jddS)Ni)rr)rrrY createTimelastAccessTimesd tableTyper)r identifier_to_database_and_tabler/rtime HiveTablerOWNERgetpassgetuserrrrrVHIVE2_COMPATIBLEHIVE2_COMPATIBLE_DEFAULTr!rr)rmridentifier_tuple database_name table_namecurrent_time_millisrnrnro_convert_iceberg_into_hivegs"  z&HiveCatalog._convert_iceberg_into_hive open_client hive_tableNc CsFz||WdSty"}ztd|jd|jd|d}~ww)NzTable r already exists) create_tablerr0rr)rmrrrrnrnro_create_hive_table{s zHiveCatalog._create_hive_tabler rc Cs:z|j||dWSty}ztd||d}~ww)Ndbnametbl_nameTable does not exists: ) get_tablerr/)rmrr rrrnrnro_get_hive_tables zHiveCatalog._get_hive_tablerrz pa.Schemarpartition_spec sort_orderc Csit|}|j||||||d}||\}} ||j|j|j||} |j} | | | | j || d} Wdn1sBwY| | S)aACreate a table. Args: identifier: Table identifier. schema: Table's schema. location: Location for the table. Optional Argument. partition_spec: PartitionSpec for the table. sort_order: SortOrder for the table. properties: Table properties that can be a string based dictionary. Returns: Table: the created table instance. Raises: AlreadyExistsError: If a table with the name already exists. ValueError: If the identifier is invalid. )rrrrrrrN) DEFAULT_PROPERTIES_create_staged_tabler_write_metadatarrrrr~rrr) rmrrrrrr staged_tabler rtblrrrnrnrors"    zHiveCatalog.create_tablerc Cs||\}}|j|d}||}t||ft||||d}||}|j} || || j ||d} Wdn1s@wY| | S)aiRegister a new table using existing metadata. Args: identifier (Union[str, Identifier]): Table identifier for the table metadata_location (str): The location to the metadata Returns: Table: The newly registered table Raises: TableAlreadyExistsError: If the table already exists rrrN) rrrr9r7rrr~rrr) rmrrr rr metadata_filer!r"rrrnrnroregister_tables      zHiveCatalog.register_table namespacecCtrzNotImplementedError)rmr%rnrnro list_viewsrzHiveCatalog.list_viewscCr&rzr'rmrrnrnro view_existsrzHiveCatalog.view_existscCs2ttjtj||dd}t|gtt d}|S)NT)levelrr tablenameisTransactional) componentuserrt) r rTABLEr EXCLUSIVErrr ry gethostname)rmr rlock_component lock_requestrnrnro_create_lock_requests z HiveCatalog._create_lock_requestlockidcsFttttd|j|jdt|jdddtffdd }|S)N) multiplierminmaxT)rwaitstopreraiserqcsjtd}|jtjkr|S|jtjkr'dd}t|t|t ddd|j)Nr7zWait on lock for rzFailed to check lock for , state: ) check_lockr staterACQUIREDWAITINGloggerwarningr1r))responsemsgr r7rrrnro_do_wait_for_locks   z5HiveCatalog._wait_for_lock.._do_wait_for_lock) rrr1rrrrrr)rmr rr7rrJrnrIro_wait_for_locks zHiveCatalog._wait_for_lock requirements.updatesc Cs~|}||t\}}|j}||||}zz|jtjkr>|jtj kr3| |||j |n t d|d|jz| |||} || } Wn tyYd} d} Ynw|| |||} | r| j| jkrt| j| jdWW|t|j dWdS|j| j| j| jd| r| rt| j| j| jd} | j| j} t| jpi}| D]}||dq|| || _t| | t!|jt"t#| _$|j%||| t&t't(idd n|)t*||f| j| j| j|d } |+|| Wnt,y}z t d|d|j|d}~wwW|t|j dn |t|j dwWdn 1s2wYt| j| jdS) aCommit updates to a table. Args: table (Table): The table to be updated. requirements: (Tuple[TableRequirement, ...]): Table requirements. updates: (Tuple[TableUpdate, ...]): Table updates. Returns: CommitTableResponse: The updated metadata. Raises: NoSuchTableError: If a table with the given identifier does not exist. CommitFailedException: Requirement not met, or a conflict with a concurrent commit. zFailed to acquire lock for r@N)rrr?)rr metadata_path)rrrrrrnew_tblenvironment_contextr)-rrr/r~lockr6rBrrCrDrKr7r)rr_update_and_stage_tablerr8runlockrr rrrkeysdictrpopupdaterrrrVr r r$alter_table_with_environment_contextr r]DO_NOT_UPDATE_STATS_DEFAULTrr9rr1)rmrrLrMtable_identifierr rrrSr current_tableupdated_staged_tablenew_iceberg_propertiesdeleted_iceberg_propertiesexisting_hms_parametersrrrnrnro commit_tables    ?    *WzHiveCatalog.commit_tablecCsN||t\}}|j}||||}Wdn1swY||S)aLoad the table's metadata and return the table instance. You can also use this method to check for table existence using 'try catalog.table() except TableNotFoundError'. Note: This method doesn't scan data stored in the table. Args: identifier: Table identifier. Returns: Table: the table instance with its metadata. Raises: NoSuchTableError: If a table with the name does not exist, or the identifier is invalid. N)rr/r~rr)rmrr rrrrnrnro load_tablefs  zHiveCatalog.load_tablec Csz||t\}}z |j}|j||ddWdWdS1s!wYWdSty<}ztd||d}~ww)zDrop a table. Args: identifier: Table identifier. Raises: NoSuchTableError: If a table with the name does not exist, or the identifier is invalid. F)rr deleteDataNr)rr/r~ drop_tabler)rmrr rrrrnrnrore|s &zHiveCatalog.drop_tablecCstd)NzNot yet implementedr'r*rnrnro purge_tableszHiveCatalog.purge_tablefrom_identifier to_identifierc Cs||t\}}||\}}||rtd|z0|j#}|j||d}||_||_|j|||t t t iddWdn1sEwYWn't y_} ztd|| d} ~ wt yr} ztd|| d} ~ ww||S)a2Rename a fully classified table name. Args: from_identifier: Existing table identifier. to_identifier: New table identifier. Returns: Table: the updated table instance with its metadata. Raises: ValueError: When from table identifier is invalid. NoSuchTableError: When a table with the name does not exist. NoSuchNamespaceError: When the destination namespace doesn't exist. TableAlreadyExistsError: When the destination table already exists. zTable already exists: rrOrPNzTable does not exist: Database does not exists: )rr/ table_existsr0r~rrrrZr r]r[rr r-rc) rmrgrhfrom_database_namefrom_table_nameto_database_name to_table_namerr"rrnrnro rename_tables2    zHiveCatalog.rename_tablec Cs||}t||d}z |j}|t||WdWdS1s$wYWdSty@}z td|d|d}~ww)aTCreate a namespace in the catalog. Args: namespace: Namespace identifier. properties: A string dictionary of properties for the given namespace. Raises: ValueError: If the identifier is invalid. AlreadyExistsError: If a namespace with the given name already exists. )rrN Database r)identifier_to_database HiveDatabaser~create_databaserrr*)rmr%rr  hive_databaserrrnrnrocreate_namespaces &zHiveCatalog.create_namespacec Cs||t}z |j}|j|dddWdWdS1swYWdSty;}z td|d|d}~wtyN}ztd||d}~ww)aDrop a namespace. Args: namespace: Namespace identifier. Raises: NoSuchNamespaceError: If a namespace with the given name does not exist, or the identifier is invalid. NamespaceNotEmptyError: If the namespace is not empty. F)rdcascadeNrpz is not emptyri)rqr-r~ drop_databaser r+r)rmr%r rrrnrnrodrop_namespaces &zHiveCatalog.drop_namespacecsZ||t|j}fdd|j|jddDWdS1s&wYdS)aList Iceberg tables under the given namespace in the catalog. When the database doesn't exist, it will just return an empty list. Args: namespace: Database to list. Returns: List[Identifier]: list of table identifiers. Raises: NoSuchNamespaceError: If a namespace with the given name does not exist, or the identifier is invalid. cs,g|]}|jtdtkr|jfqS))rrr%rr"r)rrr rnrors z+HiveCatalog.list_tables..)db_name)r tbl_namesN)rqr-r~get_table_objects_by_nameget_all_tablesrmr%rrnrzro list_tabless   $zHiveCatalog.list_tablesrncCsF|rgS|j}tt|j|WdS1swYdS)zList namespaces from the given namespace. If not given, list top-level namespaces from the catalog. Returns: List[Identifier]: a List of namespace identifiers. N)r~rridentifier_to_tupleget_all_databasesrrnrnrolist_namespacess $zHiveCatalog.list_namespacesc Cs||t}z0|j"}|j|d}|j}|j|t<|j}r"||t<|WdWS1s/wYWdSt yJ}ztd||d}~ww)a2Get properties for a namespace. Args: namespace: Namespace identifier. Returns: Properties: Properties for the given namespace. Raises: NoSuchNamespaceError: If a namespace with the given name does not exist, or identifier is invalid. rNri) rqr-r~ get_databaserrr#rrr)rmr%r rrrrXrrnrnroload_namespace_propertiess    (z%HiveCatalog.load_namespace_propertiesremovalsc Cs |j||d||t}|j_}z ||}|j}Wnty/}ztd||d}~wwt} t} |rK|D]} | |vrJ|| | | q:|r_| D] \} } | || <| | qQ| |t ||Wdn1srwY|p{t | } tt| pgt| pgt| dS)aRemove provided property keys and update properties for a namespace. Args: namespace: Namespace identifier. removals: Set of property keys that need to be removed. Optional Argument. updates: Properties to be updated for the given namespace. Raises: NoSuchNamespaceError: If a namespace with the given name does not exist ValueError: If removals and updates have overlapping keys. )rMrriN)removedupdatedmissing)_check_for_overlaprqr-r~rrrsetrXaddralter_databaser differencer(r)rmr%rrMr rrrrrrrrexpected_to_changernrnroupdate_namespace_propertiess4      "z'HiveCatalog.update_namespace_propertiescCr&rzr'r*rnrnro drop_viewFrzHiveCatalog.drop_viewcCs |||S)zIOverride the default warehouse location to follow Hive-style conventions.)"_get_hive_style_warehouse_location)rmr rrnrnro_get_default_warehouse_locationIs z+HiveCatalog._get_default_warehouse_location)rn)9rrrr_rrrp staticmethodrWrrrrrrrrr2r;r?r@rr4r3r<rArr$rr)rr+rr6rrrKtupler=r>r8rbrcrerfrorurxrrrrr(rrr __classcell__rnrnrror)s~     .    p'   *r)F)NN)rloggingryrtypesrtypingrrr urllib.parser"hive_metastore.ThriftHiveMetastorerhive_metastore.ttypesrr r r r r rrrrrrrrrrrrrrrtenacityrrrrthrift.protocolrthrift.transportrr pyiceberg.catalogr!r"r#r$r%r&r'r(pyiceberg.exceptionsr)r*r+r,r-r.r/r0r1pyiceberg.partitioningr2r3pyiceberg.schemar4r5r6pyiceberg.serializersr7pyiceberg.tabler8r9r:pyiceberg.table.sortingr;r<pyiceberg.table.updater=r>pyiceberg.typedefr?r@rApyiceberg.typesrBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSrTrUpyiceberg.utils.propertiesrVrWpyarrowparrr r rrrrrrrrrrr]r[ getLoggerrrEr_rrrrrrrPARQUET_COMPRESSIONPARQUET_COMPRESSION_DEFAULTrrWrrrrrrnrnrnros   H   (,   X  8