o .i)@sddlZddlZddlmZmZmZddlZddlmZm Z m Z m Z ddl m Z ddlmZddlmZddlmZmZd d lmZGd d d eZGd ddZdS)N)BinaryIOOptionalUnion)DatasetFeatures NamedSplitconfig) query_table)Json)tqdm)NestedDataStructureLikePathLike)AbstractDatasetReadercsjeZdZ       ddeedeedeedede de d eed ee ffd d Z d dZ Z S)JsonDatasetReaderNF path_or_pathssplitfeatures cache_dirkeep_in_memory streamingfieldnum_procc s^tj|f||||||d| ||_t|tr|n|j|i}td||||d| |_dS)N)rrrrrr)r data_filesrr)super__init__r isinstancedictrr builder) selfrrrrrrrrkwargs __class__rD/home/ubuntu/.local/lib/python3.10/site-packages/datasets/io/json.pyrs,   zJsonDatasetReader.__init__cCs\|jr |jj|jd}|Sd}d}d}d}|jj|||||jd|jj|j||jd}|S)N)r)download_config download_modeverification_mode base_pathr)rr( in_memory)rr as_streaming_datasetrdownload_and_preparer as_datasetr)r!datasetr&r'r(r)rrr%read0s$ zJsonDatasetReader.read)NNNFFNN)__name__ __module__ __qualname__r rrrrstrboolintrr/ __classcell__rrr#r%rs4  rc @sneZdZ   ddedeeefdeedeedee f ddZ d efd d Z d d Z ded efddZ dS)JsonDatasetWriterNr. path_or_buf batch_sizerstorage_optionscKs\|dur|dkrtd|d||_||_|r|ntj|_||_d|_|p'i|_||_ dS)Nrz num_proc z must be an integer > 0.zutf-8) ValueErrorr.r8r DEFAULT_MAX_BATCH_SIZEr9rencodingr:to_json_kwargs)r!r.r8r9rr:r>rrr%rIs   zJsonDatasetWriter.__init__returncCsV|jdd}|jdd}|jd|dkrdnd}d|jvr)|dvr)d|jd<t|jtttjfr5d nd}|jd |}|d vrJtd |d |sW|j |j j krWtdt|jtttjfrt j |jdfd |i|jpni}|jd|||d|j}Wd|S1swY|S|rtd|d|jd|j||d|j}|S)Nr8orientrecordslinesTFindex)rtableinfer compression)NrEgzipbz2xzz&`datasets` currently does not support z compressionzOutput JSON will not be formatted correctly when lines = False and batch_size < number of rows in the dataset. Use pandas.DataFrame.to_json() instead.wb)file_objr@rBzUThe compression parameter is not supported when writing to a buffer, but compression=z1 was passed. Please provide a local path instead.r)r>poprr8r3bytesosrNotImplementedErrorr9r.num_rowsfsspecopenr:_write)r!_r@rBdefault_compressionrFbufferwrittenrrr%write]sB     zJsonDatasetWriter.writecCsh|\}}}}t|jjt|||j|jjd}|jdd||d|}|ds.|d7}| |j S)N)rDkeyindices)r8r@rB r) r r.dataslicer9_indices to_pandasto_jsonendswithencoder=)r!argsoffsetr@rBr>batchjson_strrrr% _batch_json~s   zJsonDatasetWriter._batch_jsonrKc sd}|jdus |jdkr0ttdt|j|jdddD]}||f}|||7}q|St|j|j}} t |j9} t| |jfddtd|| D|| r^|| dn|| ddd D] }|||7}qfWd|S1s{wY|S) zWrites the pyarrow table as JSON lines to a binary file handle. Caller is responsible for opening and closing the handle. rNrbazCreating json from Arrow format)unitdesccsg|]}|fqSrr).0rdrBr@r>rr% sz,JsonDatasetWriter._write..)totalrirj) rhf_tqdmrangelenr.r9rgrXmultiprocessingPoolimap) r!rKr@rBr>rWrdrfrPr9poolrrlr%rSs6       zJsonDatasetWriter._write)NNN)r0r1r2rrrrrr5rrrXrgrSrrrr%r7Hs,  ! r7)rrrNtypingrrrrQrrrr formattingr packaged_modules.json.jsonr utilsr ro utils.typingr rabcrrr7rrrr%s    9