o wiD@sTdZddlZddlZddlZddlZddlZddlZddlZddlm Z m Z m Z m Z m Z ddlZddlmZd,de defd d Zd e fd d Zd e fddZd ejfddZd e eejffddZddZddZddZddZddZeZd e dedefd d!Z d"edefd#d$Z!d%e e"eee ffd&d'Z#Gd(d)d)Z$Gd*d+d+Z%dS)-zAClasses and functions for writing tar files and WebDataset files.N)AnyCallableDictOptionalUnion)gopenPNGimageformatcCsjddl}ddl}t||jjtjfsJt|t|tjrs|jtdtdfvrXt|dks9t |dkrIt dt|dt |t |d d }t |d d }|j d vs_J|j dkrm|jddvsmJ|j|}|dkr|d}n|dvrd}|dvrtdd}ni}t}|j|fd|i||WdS1swYdS)aCCompress an image using PIL and return it as a string. Can handle float or uint8 images. Args: image: ndarray representing an image format: compression format (PNG, JPEG, PPM) Returns: bytes: Compressed image data Raises: ValueError: If image values are out of range rNfdgMbPgjt?zimage values out of range  gg?go@uint8)rr)rrJPGJPEG>IMGIMAGEPPM>rtiffd)qualityr )PIL PIL.Image isinstanceImagenpndarraytypedtypeaminamax ValueErrorcliparrayndimshape fromarrayupperdictioBytesIOsavegetvalue)r r roptsresultr2N/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/webdataset/writer.py imageencoders0         $r4datacCs0t|tr|St|tr|dSt|dS)zConvert data into a bytestring. Uses str and ASCII encoding for data that isn't already in string format. Args: data: Data to be converted Returns: bytes: Converted bytestring ascii)rbytesstrencoder5r2r2r3bytestrBs   r;cCs,ddl}ddl}|}||||S)zDump data into a bytestring using torch.dumps. This delays importing torch until needed. Args: data: Data to be dumped Returns: bytes: Dumped data as bytestring rN)r,torchr-r.r/)r5r,r<streamr2r2r3 torch_dumpsTs   r>cCs0ddl}ddl}|}|jj|||S)zDump data into a bytestring using numpy npy format. Args: data: Data to be dumped Returns: bytes: Dumped data as bytestring rN)r,numpy.lib.formatr-libr write_arrayr/)r5r,numpyr=r2r2r3 numpy_dumpshs  rCcCslddl}t|ts Jt|D]\}}t|tsJt|tjs$Jq|}tj |fi|| S)aDump data into a bytestring using numpy npz format. Args: data: Dictionary of numpy arrays to be dumped Returns: bytes: Dumped data as bytestring Raises: AssertionError: If input is not a dictionary of numpy arrays rN) r,rr+listitemsr8rrr-savez_compressedr/)r5r,kvr=r2r2r3numpy_npz_dumpszs rIcCs4ddlm}t|trt||St||gS)zDump data into a bytestring using tenbin format. Args: x: Data to be dumped (list or single item) Returns: memoryview: Dumped data as memoryview r)tenbin)rJrrD memoryview encode_buffer)xrJr2r2r3 tenbin_dumpss rOcCddl}||S)zDump data into a bytestring using CBOR format. Args: x: Data to be dumped Returns: bytes: Dumped data as bytestring rN)cbordumps)rNrQr2r2r3 cbor_dumps rScCrP)zDump data into a bytestring using MessagePack format. Args: x: Data to be dumped Returns: bytes: Dumped data as bytestring rN)msgpackpackb)rNrUr2r2r3mp_dumpsrTrWcCs(t|tr |}|D]}|||<q dS)zAdd handlers to a dictionary for given keys. Args: d: Dictionary to add handlers to keys: String of space-separated keys or list of keys value: Handler function to be added N)rr8split)r keysvaluerGr2r2r3 add_handlerss  r[cCsi}t|dddt|dddt|dddt|dtjt|d tt|d tt|d tt|d tt|d ddt|dtt|dtt|dddt|dddt|dddt|dddt|dddt|ddd|S)z}Create a list of handlers for encoding data. Returns: dict: Dictionary of handlers for different data types z!cls cls2 class count index inx idcSst|dS)Nr6)r8r9rNr2r2r3szmake_handlers..ztxt text transcriptcS |dSNutf-8r9r\r2r2r3r] zhtml htmcSr^r_rar\r2r2r3r]rbz pyd picklepthnpynpzz ten tenbin tbzjson jsncSst|dSr_)jsonrRr9r\r2r2r3r]szmp msgpack msgrQzjpg jpeg img imagecS t|dS)Njpgr4r:r2r2r3r]rbpngcSrg)Nrjrir:r2r2r3r]rbpbmcSrg)Nrkrir:r2r2r3r]rbpgmcSrg)Nrlrir:r2r2r3r]rbppmcSrg)Nrmrir:r2r2r3r]rbztiff tifcSrg)Nrrir:r2r2r3r]rb) r[picklerRr>rCrIrOrWrShandlersr2r2r3 make_handlerss&      rqtnamerpcCs|ddkrt|tstd|Sd}|dr d}|dd}td d |}t|tr7|r5t |}|St|trJ| d }|rHt |}|S| |}|durZtd |||}|ret |}|S) aAEncode data based on its extension and a dict of handlers. Args: data: Data to be encoded tname: File extension handlers: Dictionary of handlers for different data types Raises: ValueError: If no handler is found for the given extension or if metadata values are not strings r_z-the values of metadata must be of string typeFz.gzTNz.*\.rKr`zno handler found for ) rr8r$endswithresublowerr7gzipcompressr9get)r5rrrprz extensionhandlerr1r2r2r3encode_based_on_extension1s2          r~samplecsfddt|DS)zEncode an entire sample with a collection of handlers. Args: sample: Data sample (a dict) handlers: Handlers for encoding Returns: dict: Encoded sample csi|] \}}|t||qSr2)r~).0rGrHror2r3 sz-encode_based_on_extension..)rDrE)rrpr2ror3encode_based_on_extensions rspeccsdusdur dd}n+tr}n$ttr"fdd}|}ndur1tfdd }|}ntd t|sCtd |S) aMake an encoder function from a specification. Args: spec: Specification for the encoder Returns: Callable: Encoder function Raises: ValueError: If the specification is invalid or doesn't yield a callable encoder FNcS|S)zDo not encode at all.r2r\r2r2r3encoder,szmake_encoder..encoderc t|SzEncode based on extension.rr)rr2r3r 4 zmake_encoder..fTcrrrrror2r3g=rzmake_encoder..gz: unknown decoder specz! did not yield a callable encoder)callablerr+default_handlersr$)rrr rr2)rprr3 make_encoders    rc@seZdZdZ        ddeded ed eeeefd edee fd ed ee de fddZ ddZ ddZddZddZedd eeeeffddZdS) TarWritera4A class for writing dictionaries to tar files. Args: fileobj: File name for tar file (.tgz/.tar) or open file descriptor. encoder: Sample encoding. Defaults to True. compress: Compression flag. Defaults to None. user: User for tar files. Defaults to "bigdata". group: Group for tar files. Defaults to "bigdata". mode: Mode for tar files. Defaults to 0o0444. keep_meta: Flag to keep metadata (entries starting with "_"). Defaults to False. mtime: Modification time. Defaults to None. format: Tar format. Defaults to None. Returns: TarWriter object. Raises: ValueError: If the encoder doesn't yield bytes for a key. `True` will use an encoder that behaves similar to the automatic decoder for `Dataset`. `False` disables encoding and expects byte strings (except for metadata, which must be strings). The `encoder` argument can also be a `callable`, or a dictionary mapping extensions to encoders. The following code will add two file to the tar archive: `a/b.png` and `a/b.output.png`. tarwriter = TarWriter(stream) image = imread("b.jpg") image2 = imread("b.out.jpg") sample = {"__key__": "a/b", "png": image, "output.png": image2} tarwriter.write(sample) bigdata$NTFusergroupmoderzr keep_metamtimer c Cs| rtt| | ntj} ||_|||} t|tr"t|d}||_nd|_t ||_ ||_ ||_ tj || d|_||_||_||_||_dS)aCreate a tar writer. Args: fileobj: Stream to write data to. user: User for tar files. group: Group for tar files. mode: Mode for tar files. compress: Desired compression. encoder: Encoder function. keep_meta: Keep metadata (entries starting with "_"). mtime: Modification time (set this to some fixed value to get reproducible tar files). format: Tar format. wbN)fileobjr)getattrtarfile USTAR_FORMATrtarmoderr8r own_fileobjrrrr=open tarstreamrrrrz) selfrrrrrzrrrr rr2r2r3__init__os     zTarWriter.__init__cCr)zQEnter context. Returns: self: The TarWriter object. r2rr2r2r3 __enter__zTarWriter.__enter__cC |dSz Exit context.Nclose)rexc_typeexc_valexc_tbr2r2r3__exit__ zTarWriter.__exit__cCs,|j|jdur|jd|_dSdS)zClose the tar file.N)rrrrr2r2r3rs    zTarWriter.closec Cs^d}||}d|vrtdt|D]\}}|ddkr qt|tttfs4t|dt|dq|d}t | D]m}|dkrFq?|j sP|ddkrPq?||}t|t r^| d}t}t|d|}t||_|jd urx|jn||_|j|_|j|_|j|_t|tttfstd |d t|t|}|j||||j7}q?|S) a(Write a dictionary to the tar file. Args: obj: Dictionary of objects to be stored. Returns: int: Size of the entry. Raises: ValueError: If the object doesn't contain a __key__ or if a key doesn't map to bytes after encoding. r__key__zobject must contain a __key__rsz( doesn't map to a bytes after encoding ()r`.Nzconverter didn't yield bytes: z, )rr$rDrErr7 bytearrayrLr sortedrYrr8r9timerTarInfolensizerrrunamergnamer,r-raddfile) robjtotalrGrHkeynowtir=r2r2r3writes@       zTarWriter.writecCsx|durdS|dus|dkst|tr|drdS|dks(t|tr*|dr*dS|dks8t|tr:|dr:d SdS) NFzw|Tgzzw|gzbz2zw|bz2xzzw|xz)rr8ru)rrzr2r2r3rs$zTarWriter.tarmode)rrrNTFNN)N)__name__ __module__ __qualname____doc__r8intrrboolrfloatrrrrrr staticmethodrr2r2r2r3rJsB'   +-"rc@s|eZdZdZ      ddeded ed eed ed ed eefddZ ddZ ddZ ddZ ddZ ddZddZdS) ShardWritera3Like TarWriter but splits into multiple shards. Args: pattern: Output file pattern. maxcount: Maximum number of records per shard. Defaults to 100000. maxsize: Maximum size of each shard. Defaults to 3e9. post: Optional callable to be executed after each shard is written. Defaults to None. start_shard: Starting shard number. Defaults to 0. verbose: Verbosity level. Defaults to 1. opener: Optional callable to open output files. Defaults to None. **kw: Other options passed to TarWriter. 順 ZANrrpatternmaxcountmaxsizepost start_shardverboseopenerc KsZ||_||_||_||_||_d|_||_||_d|_d|_ d|_ d|_ ||_ | dS)aCreate a ShardWriter. Args: pattern: Output file pattern. maxcount: Maximum number of records per shard. maxsize: Maximum size of each shard. post: Optional callable to be executed after each shard is written. start_shard: Starting shard number. verbose: Verbosity level. opener: Optional callable to open output files. **kw: Other options passed to TarWriter. Nr)rkwrrrrshardrrcountrfnamer next_stream) rrrrrrrrrr2r2r3rs zShardWriter.__init__cCs||j|j|_|jrtd|j|jd|jd|j|jd7_|j r7t | |jfi|j |_ n t |jfi|j |_ d|_d|_dS)z.Close the current stream and move to the next.z # writingz%.1f GBgeArrN) finishrrrrprintrrrrrrrrr2r2r3rs   zShardWriter.next_streamcCsd|jdus|j|jks|j|jkr||j|}|jd7_|jd7_|j|7_dS)zNWrite a sample. Args: obj: Sample to be written. Nr)rrrrrrrr)rrrr2r2r3r1s " zShardWriter.writecCsF|jdur!|j|jdusJt|jr||jd|_dSdS)z'Finish all writing (use close instead).N)rrrrrrr2r2r3r>s     zShardWriter.finishcCs||`|`|`|`dS)zClose the stream.N)rrrrrrr2r2r3rGs zShardWriter.closecCr)zSEnter context. Returns: self: The ShardWriter object. r2rr2r2r3rOrzShardWriter.__enter__cOrrr)rargsrr2r2r3rWrzShardWriter.__exit__)rrNrrN)rrrrr8rrrrrrrrrrrr2r2r2r3rs: '   r)r )&rryr,rfrnrvrrtypingrrrrrrBrrKrr8r4r;r>rrCrIrOrSrWr[rqrr+r~rrrrrr2r2r2r3s8 +& ,