o ˳i#@sUddlmZddlmZddlmZddlmZmZddl m Z m Z m Z m Z ddlmZddlmZddlmZmZmZdd lmZejejejhZeeed <ejejhZeeed <d ed ededeede de e!e ee"ff ddZ#de dee"dedBde$e"eefdBde dBdede%fddZ&d edededBde$e"eefdBdedBdee f dd Z'd edededBdeddf d!d"Z(d edededBde$e"eefdBdedBdee f d#d$Z)d edededBdedBddf d%d&Z*dS)')Iterator)ValidationException)BooleanExpression)ROWS_CANNOT_MATCH_InclusiveMetricsEvaluator)ManifestContent ManifestEntryManifestEntryStatus ManifestFile)Schema)Table) OperationSnapshotancestors_between)Record$VALIDATE_DATA_FILES_EXIST_OPERATIONS$VALIDATE_ADDED_DATA_FILES_OPERATIONStable from_snapshot to_snapshotmatching_operationsmanifest_content_filterreturnc sg}t}d}t|||jD]0}j}|dur!tdd|j|vr'q|j|fdd |j Dq|durM|j|jkrMtd||fS)a<Return newly added manifests and snapshot IDs between the starting snapshot and parent snapshot. Args: table: Table to get the history from from_snapshot: Parent snapshot to get the history from to_snapshot: Starting snapshot matching_operations: Operations to match on manifest_content_filter: Manifest content type to filter Raises: ValidationException: If no matching snapshot is found or only one snapshot is found Returns: List of manifest files and set of snapshots ID's matching conditions NzNo summary found for snapshot !cs&g|]}|jjkr|jkr|qS)added_snapshot_id snapshot_idcontent).0manifestrsnapshotrS/home/ubuntu/.local/lib/python3.10/site-packages/pyiceberg/table/update/validate.py Es z'_validation_history..zNo matching snapshot found.) setrmetadatasummaryr operationaddrextend manifestsio) rrrrrmanifests_files snapshots last_snapshotr&rr r"_validation_history s&    r/entry snapshot_ids data_filterN partition_set entry_statusschemac Cs~|j|vrdS|dur|j|krdS|dur%t||}||jtur%dS|dur=|jj}|jj}||vs;|||vr=dSdS)aFilter manifest entries based on data filter and partition set. Args: entry: Manifest entry to filter snapshot_ids: set of snapshot ids to match data files data_filter: Optional filter to match data files partition_set: Optional set of partitions to match data files entry_status: Optional status to match data files schema: schema for filtering Returns: True if the entry should be included, False otherwise FNT)rstatusreval data_filer partitionspec_id) r0r1r2r3r4r5 evaluatorr9r:rrr"_filter_manifest_entriesRs  r<starting_snapshotparent_snapshotc csf|durdSt|||ttj\}}|D]}|j|jddD]}t||||tj| r/|VqqdS)aFind deleted data files matching a filter since a starting snapshot. Args: table: Table to validate starting_snapshot: Snapshot current at the start of the operation data_filter: Expression used to find deleted data files partition_set: dict of {spec_id: set[partition]} to filter on parent_snapshot: Ending snapshot on the branch being validated Returns: List of conflicting manifest-entries NF)discard_deleted) r/rrDATAfetch_manifest_entryr+r<r DELETEDr5 rr=r2r3r>r*r1rr0rrr"_deleted_data_files{s&rDcC:t|||d|}t|rdd|D}td|ddS)agValidate that no files matching a filter have been deleted from the table since a starting snapshot. Args: table: Table to validate starting_snapshot: Snapshot current at the start of the operation data_filter: Expression used to find deleted data files parent_snapshot: Ending snapshot on the branch being validated NcSsh|]}|jqSrrrr0rrr" sz/_validate_deleted_data_files..z@Deleted data files were found matching the filter for snapshots r)rDanyrrr=r2r>conflicting_entriesconflicting_snapshotsrrr"_validate_deleted_data_files rMc cs`|durdSt|||ttj\}}|D]}||jD]}t||||d|r,|VqqdS)aReturn manifest entries for data files added between the starting snapshot and parent snapshot. Args: table: Table to get the history from starting_snapshot: Starting snapshot to get the history from data_filter: Optional filter to match data files partition_set: Optional set of partitions to match data files parent_snapshot: Parent snapshot to get the history from Returns: Iterator of manifest entries for added data files matching the conditions N)r/rrr@rAr+r<r5rCrrr"_added_data_filess"rOcCrE)aaValidate that no files matching a filter have been added to the table since a starting snapshot. Args: table: Table to validate starting_snapshot: Snapshot current at the start of the operation data_filter: Expression used to find added data files parent_snapshot: Ending snapshot on the branch being validated NcSsh|] }|jdur|jqS)NrFrGrrr"rHsz-_validate_added_data_files..z>Added data files were found matching the filter for snapshots r)rOrIrrJrrr"_validate_added_data_filesrNrP)+collections.abcrpyiceberg.exceptionsrpyiceberg.expressionsrpyiceberg.expressions.visitorsrrpyiceberg.manifestrrr r pyiceberg.schemar pyiceberg.tabler pyiceberg.table.snapshotsr rrpyiceberg.typedefr OVERWRITEREPLACEDELETErr$__annotations__APPENDrtuplelistintr/dictboolr<rDrMrOrPrrrr"s      2 ) '  $