o ioB@sxUdZddlmZddlmZddlmZmZddlm Z ddlm Z ddl m Z ddlmZdd lmZdd lmZdd lmZdd lmZdd lmZmZddlmZddlmZdMddZGdddeZ Gddde Z!Gddde Z"Gddde Z#Gd d!d!e Z$Gd"d#d#e Z%Gd$d%d%e Z&Gd&d'd'e$Z'Gd(d)d)e$Z(Gd*d+d+e$Z)Gd,d-d-e$Z*Gd.d/d/e$Z+Gd0d1d1e$Z,Gd2d3d3e Z-Gd4d5d5e Z.Gd6d7d7e Z/e d8d9Gd:d;d;e Z0Gdd?Gd@dAdAe Z2e d8d9GdBdCdCe Z3GdDdEdEe Z4e d>d>d?GdFdGdGe Z5iZ6dHe7dI<e d>d>d?GdJdKdKe Z8dLS)Na  Classes for building the Reader tree. Constructing a reader tree from the schema makes it easy to decouple the reader implementation from the schema. The reader tree can be changed in such a way that the read schema is different, while respecting the read schema. ) annotations)abstractmethod)CallableMapping) dataclass)field)Decimal)Any)UUID) BinaryDecoder)StructProtocol) StructType)bytes_to_decimaldecimal_required_bytes)LazyDict) Singletondecoderr skip_entryCallable[[], None]returnNonecCsX|}|dkr*|dkr|}||n t|D]}|q|}|dksdSdS)aSkips over an array or map. Both the array and map are encoded similar, and we can reuse the logic of skipping in an efficient way. From the Avro spec: Maps (and arrays) are encoded as a series of blocks. Each block consists of a long count value, followed by that many key/value pairs in the case of a map, and followed by that many array items in the case of an array. A block with count zero indicates the end of the map. Each item is encoded per the map's value schema. If a block's count is negative, its absolute value is used, and the count is followed immediately by a long block size indicating the number of bytes in the block. This block size permits fast skipping through data, e.g., when projecting a record to a subset of its fields. Args: decoder: The decoder that reads the types from the underlying data. skip_entry: Function to skip over the underlying data, element in case of an array, and the key/value in the case of a map. rN)read_intskiprange)rr block_count block_size_rS/home/ubuntu/transcripts/venv/lib/python3.10/site-packages/pyiceberg/avro/reader.py_skip_map_array/s  rc@s2eZdZedddZeddd Zdd d Zd S)Readerrr rr cCdSNrselfrrrrreadTz Reader.readrcCr!r"rr#rrrrWr&z Reader.skipstrcCs|jjdS)z5Return the string representation of the Reader class.z()) __class____name__r$rrr__repr__ZzReader.__repr__N)rr rr rr rrrr')r) __module__ __qualname__rr%rr+rrrrr Ss   r c@ eZdZd ddZd dd Zd S) NoneReaderrr rrcCr!r"rr$rrrrr%`r&zNoneReader.readrcCr!r"rr#rrrrcr&zNoneReader.skipN)rr rrr-r)r/r0r%rrrrrr2_ r2c@s8eZdZUdZded<dddZdd d Zdd dZdS) DefaultReader default_valuer r8rrcCs ||_dSr"r7)r$r8rrr__init__ks zDefaultReader.__init__rr cC|jSr"r7r3rrrr%nszDefaultReader.readrcCr!r"rr#rrrrqr&zDefaultReader.skipN)r8r rr)rr rr r-)r)r/r0 __slots____annotations__r9r%rrrrrr6gs   r6c@r1) BooleanReaderrr rboolcC|Sr") read_booleanr#rrrr%vzBooleanReader.readrcC |dSr") skip_booleanr#rrrry zBooleanReader.skipN)rr rr>r-r4rrrrr=ur5r=c@$eZdZdZd ddZd d d Zd S) IntegerReaderzHLongs and ints are encoded the same way, and there is no long in Python.rr rintcCr?r")rr#rrrr%rAzIntegerReader.readrcCrBr")skip_intr#rrrrrDzIntegerReader.skipN)rr rrGr-r)r/r0__doc__r%rrrrrrF}s rFc@r1) FloatReaderrr rfloatcCr?r") read_floatr#rrrr%rAzFloatReader.readrcCrBr") skip_floatr#rrrrrDzFloatReader.skipNrr rrLr-r4rrrrrKr5rKc@r1) DoubleReaderrr rrLcCr?r") read_doubler#rrrr%rAzDoubleReader.readrcCrBr") skip_doubler#rrrrrDzDoubleReader.skipNrOr-r4rrrrrPr5rPc@eZdZdZdS) DateReaderz_Reads a day granularity date from the stream. The number of days from 1 January 1970. Nr)r/r0rJrrrrrTrTc@rS) TimeReaderzReads a microsecond granularity timestamp from the stream. Long is decoded as an integer which represents the number of microseconds from the unix epoch, 1 January 1970. NrUrrrrrWrVrWc@rS)TimestampReaderzReads a microsecond granularity timestamp from the stream. Long is decoded as python integer which represents the number of microseconds from the unix epoch, 1 January 1970. NrUrrrrrXrVrXc@rS)TimestampNanoReaderzReads a nanosecond granularity timestamp from the stream. Long is decoded as python integer which represents the number of nanoseconds from the unix epoch, 1 January 1970. NrUrrrrrYrVrYc@rS)TimestamptzReaderzReads a microsecond granularity timestamptz from the stream. Long is decoded as python integer which represents the number of microseconds from the unix epoch, 1 January 1970. Adjusted to UTC. NrUrrrrrZrVrZc@rS)TimestamptzNanoReaderzReads a microsecond granularity timestamptz from the stream. Long is decoded as python integer which represents the number of nanoseconds from the unix epoch, 1 January 1970. Adjusted to UTC. NrUrrrrr[rVr[c@r1) StringReaderrr rr'cCr?r") read_utf8r#rrrr%rAzStringReader.readrcCrBr") skip_utf8r#rrrrrDzStringReader.skipN)rr rr'r-r4rrrrr\r5r\c@r1) UUIDReaderrr rr cCst|ddS)N)bytes)r r%r#rrrr%szUUIDReader.readrcCs|ddS)Nr`)rr#rrrrzUUIDReader.skipN)rr rr r-r4rrrrr_r5r_c@s eZdZd ddZd ddZd S) UnknownReaderrr rrcCr!r"rr#rrrr%r&zUnknownReader.readcCr!r"rr#rrrrr&zUnknownReader.skipNr-r4rrrrrcr5rcT)frozenc@sDeZdZUeZded<dddZdd d Zdd d ZdddZ dS) FixedReaderrG_lenrr rracCs|t|Sr")r%lenr#rrrr%rbzFixedReader.readrcCs|t|dSr")rrgr#rrrrszFixedReader.skipcCr:)z:Return the length of an instance of the FixedReader class.rfr*rrr__len__zFixedReader.__len__r'cCsd|jdS)z:Return the string representation of the FixedReader class.z FixedReader()rhr*rrrr+r,zFixedReader.__repr__Nrr rrar-rrGr.) r)r/r0dataclassfieldrfr<r%rrir+rrrrres    rec@rE) BinaryReaderzRead a binary value. First reads an integer, to get the length of the binary value, then reads the binary field itself. rr rracCr?r") read_bytesr#rrrr%rAzBinaryReader.readrcCrBr" skip_bytesr#rrrrrDzBinaryReader.skipNrlr-rIrrrrros roF)rdinitc@s^eZdZUdZeZded<eZded<ded<dddZdd d Z dddZ dddZ dS) DecimalReaderzReads a value as a decimal. Decimal bytes are decoded as signed short, int or long depending on the size of bytes. rG precisionscale_lengthcCs2t|d|t|d|t|dt|dS)Nrurvrw)object __setattr__r)r$rurvrrrr9szDecimalReader.__init__rr rrcCst||j|jSr")rr%rwrvr#rrrr%szDecimalReader.readrcCrBr"rqr#rrrrrDzDecimalReader.skipr'cCsd|jd|jdS)zcCs&t|tr|j|jko|j|jkSdS)z?Return the equality of two instances of the StructReader class.F)rrrr)r$rrrr__eq__ns zStructReader.__eq__r'cCs*dddd|jDdt|jdS)z;Return the string representation of the StructReader class.zStructReader((,css|]}t|VqdSr")repr).0rrrr xsz(StructReader.__repr__..z), rk)joinrrrr*rrrr+vs*zStructReader.__repr__rGcCr:)z9Return a hashed representation of the StructReader class.rr*rrr__hash__zrjzStructReader.__hash__)rrrrrr rr)rr rr r-)rr rr>r.rm)r)r/r0r;r<rr'rGrr r field_reader_functionsr9r%rrr+rrrrrr6s (    rcsJeZdZUdZded<dfdd Zdd d Zdd dZdddZZ S) ListReader)element _is_int_listrr rrrcs.t||_t|j|_t|jt|_dSr")superr9rrrrrFr)r$rr(rrr9s  zListReader.__init__rr list[Any]cCsvg}|}|dkr9|dkr| }|}|jr!|||nt|D] }||j|q%|}|dks |Sr~)rrextend read_intsrrrr%)r$r read_itemsrrrrrr%s  zListReader.readcstfdddS)Ncs jSr")rrrrr$rrs z!ListReader.skip..rr#rrrrszListReader.skiprGcCr:)z7Return a hashed representation of the ListReader class.rr*rrrrrjzListReader.__hash__)rr rr)rr rrr-rm) r)r/r0r;r<r9r%rr __classcell__rrrrrs   rzdict[Any, Any] EMPTY_DICTcs\eZdZUdZded<ded<dfdd Zdd d ZdddZdddZdddZ Z S) MapReader)keyvalue _is_int_int _is_int_bytes _key_reader _value_readerrr rrrrcsvt||_||_t|jtr t|jt|_t|jt|_nd|_d|_|jj |_ |jj |_ t |j|jf|_ dS)NF)rr9rrrrFrrorr%rrrr)r$rrrrrr9s    zMapReader.__init__rr Mapping[int, int]cCsb|}|dkr tSg}|dkr-|dkr| }||||d|}|dkst|S)aRead a mapping from int to int from the decoder. Read a map of ints to ints from the decoder, since this is such a common data type, it is optimized to be faster than the generic map reader, by using a lazy dict. The time it takes to create the python dictionary is much larger than the time it takes to read the data from the decoder as an array, so the lazy dict defers creating the python dictionary until it is actually accessed. r)rrrHrrr)r$rrcontents_arrayrrr _read_int_ints  zMapReader._read_int_intMapping[Any, Any]cCsi}|js|jr3|jr||S|}|dkr1|dkr#| }|}||||}|dks|S|}|dkra|dkrF| }|}t|D]}||}||||<qJ|}|dks;|Sr~)rrrrread_int_bytes_dictrrr)r$rrrrrrrrr%s0      zMapReader.readcsdfdd }t|dS)NrrcsjjdSr")rrrrrrrrs zMapReader.skip..skip)rrr)r$rrrrrrszMapReader.skiprGcCr:)z6Return a hashed representation of the MapReader class.rr*rrrrrjzMapReader.__hash__)rr rr rr)rr rr)rr rrr-rm) r)r/r0r;r<r9rr%rrrrrrrrs   # rN)rr rrrr)9rJ __future__rabcrcollections.abcrr dataclassesrrrndecimalrtypingr uuidr pyiceberg.avro.decoderr pyiceberg.typedefr pyiceberg.typesr pyiceberg.utils.decimalrrpyiceberg.utils.lazydictrpyiceberg.utils.singletonrrr r2r6r=rFrKrPrTrWrXrYrZr[r\r_rcrerortrzrrrr<rrrrrsZ            $      I "