o SisB@sNdZddlZddlZddlZddlZddlZddlZddlmZddl Z ddl m Z ej dddkZ gd Zd efd d Zd dZddZddZddZddZddZidddddddddd d!dd"d#dd$d%dd&d'dd(d)dd*d+dd,d-dd.ed/ed0ed1d2dd3ed4ed5eeeeed6Zd7d8Zd9d:Zd;d<Zid=d>d?d@dAdBdCdDdEdFdGdHdIdJdKdLdMdNdOdPdQdRdSdRdTdUdVdWdXdYdZdYd[d\ZGd]d^d^Zefd_d`ZdadbZ dcddZ!GdedfdfZ"dgdhZ#e#gZ$egZ%Gdidjdje&Z'GdkdldlZ(e(e$e%Z)dS)mz(Automatically decode webdataset samples.N)partial)utilsWDS_PYTORCH_WEIGHTS_ONLY01)?blpbmpdibbufrcurpcxdcxddspsepsfitfitsfliflcftcftugbrgifgribh5hdfpngapngjp2j2kjpcjpfjpxj2cicnsicoimiimtiftiffjfifjpejpgjpegmpgmpegmsppcdpxrpbmpgmppmpnmpsdbwrgbrgbasgirastgaicbvdavstwebpwmfemfxbmxpmdatacCs8ddl}ddl}tjrtd||}|j|tddS)atFunction: torch_loads Description: This function loads data using torch.loads. It first imports torch only if necessary. Then it decodes the input data using torch.load. Parameters: - data (bytes): The data to be decoded. Returns: It returns the decoded input data. Example: data = b'...' output = torch_loads(data) rNzMtorch.loads is not allowed for security reasons when enforce_security is set.cpu) weights_only map_location)iotorchrenforce_security ValueErrorBytesIOloadpytorch_weights_only)rGrKrLstreamrSI/home/ubuntu/.local/lib/python3.10/site-packages/webdataset/autodecode.py torch_loadshs  rUcCsddlm}||S)z?Load data from tenbin format. Imports tenbin only if necessary.r)tenbin)rV decode_buffer)rGrVrSrSrT tenbin_loadss  rYcCddl}||S)zALoad data from msgpack format. Imports msgpack only if necessary.rN)msgpackunpackb)rGr[rSrSrT msgpack_loads r]cCs ddl}t|}|jj|S)z;Load data from npy format. Imports numpy only if necessary.rN)numpy.lib.formatrKrOlibformat read_array)rGnumpyrRrSrSrT npy_loadss rdcCst|}tt|S)z;Load data from npz format. Imports numpy only if necessary.)rKrOdictnprP)rGrRrSrSrT npz_loadss rgcCrZ)z;Load data from cbor format. Imports cbor only if necessary.rN)cborloads)rGrhrSrSrT cbor_loadsr^rjcCstjrtdt|S)z?Load data from pickle format. Imports pickle only if necessary.zLUnpickling is not allowed for security reasons when enforce_security is set.)rrMrNpicklerirGrSrSrTunpickle_loadss rmtxtcC |dSNutf-8decoderlrSrSrT rttextcCrorprrrlrSrSrTrtru transcriptcCrorprrrlrSrSrTrtruclscCt|SNintrlrSrSrTrtcls2cCryrzr{rlrSrSrTrtr}indexcCryrzr{rlrSrSrTrtr}inxcCryrzr{rlrSrSrTrtr}idcCryrzr{rlrSrSrTrtr}jsoncC t|SrzrrirlrSrSrTrtrujsncCrrzrrlrSrSrTrtrupklrkpydpthcCryrz)rUrlrSrSrTrtr}tentbmp)msgnpynpzrhcCs&tdd|}|tvrt||SdS)aHandle basic file decoding. This function is usually part of the post= decoders. This handles the following forms of decoding: - txt -> unicode string - cls cls2 class count index inx id -> int - json jsn -> JSON decoding - pyd pickle -> pickle decoding - pth -> torch.loads - ten tenbin -> fast tensor loading - mp messagepack msg -> messagepack decoding - npy -> Python NPY decoding :param key: file name extension :param data: binary data to be decoded .*[.]rWN)resubdecoders)keyrG extensionrSrSrT basichandlerss rcCsZ|d}|D]!}|d}t|t|krq |t| d|kr*||Sq dS)zCall the function f with the given data if the key matches the extensions. :param key: actual key found in the sample :param data: binary data :param f: decoder function :param extensions: list of matching extensions .N)lowersplitlen)rrGf extensionsrtargetrSrSrTcall_extension_handlers  rcCs|}tt||dS)aReturn a decoder function for the list of extensions. Extensions can be a space separated list of extensions. Extensions can contain dots, in which case the corresponding number of extension components must be present in the key given to f. Comparisons are case insensitive. Examples: handle_extension("jpg jpeg", my_decode_jpg) # invoked for any file.jpg handle_extension("seg.jpg", special_case_jpg) # invoked only for file.seg.jpg )rr)rrrr)rrrSrSrThandle_extensions rl8)rcuint8lrgb8)rcrr:rgba8)rcrr;r)rcfloatrr:)rcrr:r;)rcrr;torchl8)rLrr torchrgb8)rLrr: torchrgba8)rLrr;torchl)rLrrtorchrgb)rLrr:rL torchrgba)rLrr;pill)pilNrr)rNr:pilrgbpilrgba)rNr;c@s$eZdZdZefddZddZdS) ImageHandleraDecode image data using the given `imagespec`. The `imagespec` specifies whether the image is decoded to numpy/torch/pi, decoded to uint8/float, and decoded to l/rgb/rgba: - l8: numpy uint8 l - rgb8: numpy uint8 rgb - rgba8: numpy uint8 rgba - l: numpy float l - rgb: numpy float rgb - rgba: numpy float rgba - torchl8: torch uint8 l - torchrgb8: torch uint8 rgb - torchrgba8: torch uint8 rgba - torchl: torch float l - torchrgb: torch float rgb - torch: torch float rgb - torchrgba: torch float rgba - pill: pil None l - pil: pil None rgb - pilrgb: pil None rgb - pilrgba: pil None rgba cCs4|ttvrtd|||_t||_dS)zCreate an image handler. :param imagespec: short string indicating the type of decoding :param extensions: list of extensions the image handler is invoked for zUnknown imagespec: %sN)list imagespecskeysrNr imagespecsetr)selfrrrSrSrT__init__?s  zImageHandler.__init__c Csddl}tdd|}||jvrdS|j}t|\}}}t|} |j | } | | | } Wdn1s?wY|dkro|dkrS| d} | S|dkr^| d } | S|d kri| d } | Std |t| } |d kr| tjd} | jdvsJ| j|dvsJ||dkr| jdkrtj| ddddddfdd} nu|dkr| jdkrtj| ddddtjfddd} nY| jddkr| ddddddf} nD|d kr#| jdkrtj| ddddtjfddd} d| dddddf<n| jddkr#tj| dt| jddgdd} |dvs,J||dkr3| S|dkrUddl} | jdkrN| | dddS| | SdS)zjPerform image decoding. :param key: file name extension :param data: binary data rNrrWrrLr:RGBr;RGBAzUnknown mode: %srgo@))rr:r;rr)axis)rcrLrcrLr) PIL.ImagerrrrrrrKrOImageopenrPconvertupperrNrfasarrayastypefloat32ndimshapemeanrepeatnewaxis concatenateonesrL from_numpy transposecopy) rrrGPILrratypeetypemoderRimgresultrLrSrSrT__call__Jsh        $ &  $&   zImageHandler.__call__N)__name__ __module__ __qualname____doc__IMAGE_EXTENSIONSrrrSrSrSrTr$s  rcCs t||S)zCreate an image handler. This is just a lower case alias for ImageHander. :param imagespec: textual image spec :param extensions: list of extensions the handler should be applied for )r)rrrSrSrT imagehandlers rc Cstdd|}|dvrdSddl}t4}tj|d|}t |d }| |Wdn1s7wY|j j |dd WdS1sNwYdS) zxDecode video using the torchvideo library. :param key: file name extension :param data: data to be decoded rrWz'mp4 ogv mjpeg avi mov h264 mpg webm wmvNrfile.wbsec)pts_unit) rrrtorchvision.iotempfileTemporaryDirectoryospathjoinrwriterK read_video)rrGr torchvisiondirnamefnamerRrSrSrT torch_videos    $rc Cstdd|}|dvr dSddl}t1}tj|d|}t|d }| |Wdn1s5wY| |WdS1sIwYdS)zxDecode audio using the torchaudio library. :param key: file name extension :param data: data to be decoded rrW)flacmp3soxwavm4aoggwmaNrrr) rr torchaudiorrrrrrrrP)rrGrrrrrRrSrSrT torch_audios   $rc@seZdZdZddZdS)ContinuezSpecial class for continuing decoding. This is mostly used for decompression, as in: def decompressor(key, data): if key.endswith(".gz"): return Continue(key[:-3], decompress(data)) return None cCs|||_|_dS)z<__init__. :param key: :param data: N)rrG)rrrGrSrSrTrszContinue.__init__NrrrrrrSrSrSrTrs rcCs<ddl}|ds dS|t|}t|dd|S)zDecode .gz files. This decodes compressed files and the continues decoding. :param key: file name extension :param data: binary data rNz.gz)gzipendswithrrKrOreadr)rrGr decompressedrSrSrTgzfilters  rc@seZdZdZdddZdS) DecodingErrorz$Exception class for decoding errors.NcCs||_||_||_||_dSrz)urlrksample)rrrrrrSrSrTrs zDecodingError.__init__)NNNNrrSrSrSrTrsrc@s2eZdZdZd ddZddZdd Zd d ZdS) DecoderzDecode samples using a list of handlers. For each key/data item, this iterates through the list of handlers until some handler returns something other than None. NFcCst|ts Jd|dt|tr|}|dur|nt||_|dur't}|dur-t}tdd|Ds>Jd|dtdd|DsOJd|dtd d|Ds`Jd|d||||_ ||_ dS) aCreate a Decoder. :param handlers: main list of handlers :param pre: handlers called before the main list (.gz handler by default) :param post: handlers called after the main list (default handlers by default) :param only: a list of extensions; when give, only ignores files with those extensions :param partial: allow partial decoding (i.e., don't decode fields that aren't of type bytes) z handlers = z must be a listNcs|]}t|VqdSrzcallable.0hrSrSrT z#Decoder.__init__..zone of z not callablecsrrzrrrSrSrTr r csrrzrrrSrSrTr  r ) isinstancerstrrronlydefault_pre_handlersdefault_post_handlersallhandlersr)rrprepostrrrSrSrTr s """ zDecoder.__init__cCsLd|}|jD]}|||}t|tr|j|j}}q|dur#|Sq|S)zuDecode a single field of a sample. :param key: file name extension :param data: binary data rN)rr rrrG)rrrGrrrSrSrTdecode1$s   zDecoder.decode1c CsXi}t|ts J|t|D]\}}zs|dddkrDt|tr>z|d}Wnty=td|d|Ynw|||<Wq|jdurT||jvrT|||<Wq|dusZJ|j rpt|trk| ||||<n|||<nt|tsJd|d|| ||||<Wqty}z| d d}| d d}t ||||d |d}~ww|S) zeDecode an entire sample. :param sample: the sample, a dictionary of key value pairs Nr__rqzCan't decode v of k = z as utf-8: v = zk,v = z, __key____url__)rrrr) r reritemsbytesrs Exceptionprintrrrgetr)rrrrvexnrrrSrSrTrs4s>       zDecoder.decodecCs$t|ts Jt||f||S)zDDecode an entire sample. :param sample: the sample )r rerrs)rrrSrSrTrWs zDecoder.__call__)NNNF)rrrrrrrsrrSrSrSrTrs   #r)*rrKrrrkrr functoolsrrcrfrWrenvironrrQrrrUrYr]rdrgrjrmrrrrrrrrrrrrrrrrdefault_decoderrSrSrSrTs   G            k Z