o 8wi+b @sdZddlZddlZddlZddlZddlmZddlmZddl m Z m Z m Z ddl ZddlmZddlmZddlmZdd lmZmZdd lmZmZmZmZdd lmZdd lm Z!e rndd l"m#Z#ddl$m%Z%m&Z&e'e(Z)dZ*e+dedddej,Z-e+dZ.Gddde/Z0Gddde/Z1eddGdddZ2  d9de3de4dde e3d fd!e e3d"e e3d#e2f d$d%Z5Gd&d'd'Z6Gd(d)d)e6Z7Gd*d+d+e6Z8eddGd,d-d-Z9eddGd.d/d/Z:d0d1Z;d2d3ZGd8d d Z?dS):zArrow ArrowReader.N) dataclass)partial) TYPE_CHECKINGOptionalUnion) thread_map)DownloadConfig) _split_refilenames_for_dataset_split) InMemoryTableMemoryMappedTableTable concat_tables)logging)tqdm) DatasetInfo)Split SplitInfoz=https://storage.googleapis.com/huggingface-nlp/cache/datasetsz ^ (?Pz) (\[ ((?P-?[\d_]+) (?P%)?)? : ((?P-?[\d_]+) (?P%)?)? \])?(\((?P[^\)]*)\))? $ z\s*\+\s*c@eZdZdZdS)DatasetNotOnHfGcsErrorz?When you can't get the dataset from the Hf google cloud storageN__name__ __module__ __qualname____doc__rrR/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/datasets/arrow_reader.pyrArc@r)MissingFilesOnHfGcsErrorz9When some files are missing on the Hf oogle cloud storageNrrrrrr Grr T)frozenc@s&eZdZUdZeed<eeed<dS)FileInstructionsa}The file instructions associated with a split ReadInstruction. Attributes: num_examples: `int`, The total number of examples file_instructions: List[dict(filename, skip, take)], the files information. The filenames contains the relative path, not absolute. skip/take indicates which example read in the file: `ds.slice(skip, take)` num_examplesfile_instructionsN)rrrrint__annotations__listdictrrrrr"Ms  r"name split_infosr instructionReadInstructionfiletype_suffix prefix_pathreturncsttstdtjstddd|D}dd|Dfdd|D}t|ts9t|}||}g}d} |D]} || j } || j } | j } | j dur\dn| j }| j durf| n| j }| dur| D]}||}|dkrzqo| |7} | |||d qoqDd}d}t | | D]D\}}||7}||kr||kr||kr||nd}||kr|||nd }|dkrq| |||d | |d kr||n|7} ||7}qqDt| |d S) aReturns instructions of the split dict. Args: name (`str`): Name of the dataset. split_infos (`list` of `[SplitInfo]`): Dataset splits information. instruction ([`ReadInstruction`] or `str`): Reading instruction for a dataset. filetype_suffix (`str`, *optional*): Suffix of dataset files, e.g. 'arrow' or 'parquet'. prefix_path (`str`, *optional*): Prefix of dataset files, e.g. directory name. Returns: [`FileInstructions`] zExpected str 'name', but got: zExpected non-empty str 'name'cSi|]}|j|jqSr)r)r#.0inforrr sz*make_file_instructions..cSr0r)r) shard_lengthsr1rrrr4tr5c s*i|]}|jt|j|jdqS))path dataset_namesplitr-r6)r)r r1r-r)name2shard_lengthsr.rrr4usrN)filenameskiptaker)r#r$) isinstancestr TypeErrortyper ValueErrorr, from_spec to_absolute splitnamefrom_toappendzipr")r)r*r+r-r.name2lenname2filenamesabsolute_instructionsr$r# abs_instr split_length filenamesr6rGrHr<r> index_start index_end shard_lengthr=rr:rmake_file_instructions\sZ        rTc@szeZdZdZdededfddZddefd d Zddefd d Z d dZ dddZ  dde e dedfddZdS) BaseReaderz@ Build a Dataset object out of Instruction instance(s). r7r3rcCs||_||_d|_dS)zInitializes ArrowReader. Args: path (str): path where tfrecords are stored. info (DatasetInfo): info about the dataset. N)_path_info_filetype_suffixselfr7r3rrr__init__s zBaseReader.__init__Fr/cCst)=Returns a Dataset instance from given (filename, skip, take).)NotImplementedError)rZfilename_skip_take in_memoryrrr_get_table_from_filenamesz#BaseReader._get_table_from_filenamecCst|dkstdd|Dstdt|}|D]}tj|j|d|d<qt t |j |d|t dt|dkp:d d }d d |D}|sU|j d usQ|j jd urUtd |pdtjgt|j jjdg}t|dkrqt|}|S|d}|S)aReturns Dataset for given file instructions. Args: files: List[dict(filename, skip, take)], the files information. The filenames contain the absolute path, not relative. skip/take indicates which example read in the file: `ds.slice(skip, take)` in_memory (bool, default False): Whether to copy the data in-memory. rcss|]}t|tVqdSN)r?r()r2frrr sz)BaseReader._read_files..z&please provide valid file informationsr<r_zLoading dataset shardsN) tqdm_classdescdisablecSsg|] }t|dkr|qS)r)len)r2trrr sz*BaseReader._read_files..zqTried to read an empty table. Please specify at least info.features to create an empty table with the right type.)schemar)riallrCcopydeepcopyosr7joinrVrrr`hf_tqdmrWfeaturesr from_batchesparlrBr)rZfilesr_rb pa_tablespa_tablerrr _read_filess,    zBaseReader._read_filescCs t||||j|jd}|j}|S)z?Return list of dict {'filename': str, 'skip': int, 'take': int})r-r.)rTrXrVr$)rZr)r+r*r$rvrrrget_file_instructionss z BaseReader.get_file_instructionscCs6||||}|sd|d}t||j|||dS)a Returns Dataset instance(s). Args: name (str): name of the dataset. instructions (ReadInstruction): instructions to read. Instruction can be string and will then be passed to the Instruction constructor as it. split_infos (list of SplitInfo proto): the available splits for dataset. in_memory (bool, default False): Whether to copy the data in-memory. Returns: kwargs to build a single Dataset instance. z Instruction "z" corresponds to no data!)rvoriginal_instructionsr_)rzrC read_files)rZr) instructionsr*r_rvmsgrrrreads  zBaseReader.readNrvr{)Nr,rcCsF|j||d}|durddlm}|t|}nd}||j|d}|S)aJReturns single Dataset instance for the set of file instructions. Args: files: List[dict(filename, skip, take)], the files information. The filenames contains the relative path, not absolute. skip/take indicates which example read in the file: `ds.skip().take()` original_instructions: store the original instructions used to build the dataset split in the dataset. in_memory (bool, default False): Whether to copy the data in-memory. Returns: kwargs to build a Dataset instance. rdNr)r) arrow_tabler3r9)rysplitsrr@rW)rZrvr{r_rxrr9dataset_kwargsrrrr|s zBaseReader.read_filesF)NF)rrrrr@rr[rr`ryrzrr'r(rr|rrrrrUs   rUcsReZdZdZdededffdd Zd defd d Ze d defd d Z Z S) ArrowReaderz Build a Dataset object out of Instruction instance(s). This Reader uses either memory mapping or file descriptors (in-memory) on arrow files. r7r3rct||d|_dS)zInitializes ArrowReader. Args: path (str): path where Arrow files are stored. info (DatasetInfo): info about the dataset. arrowNsuperr[rXrY __class__rrr[# zArrowReader.__init__Fr/cCs|dd|vr |dndd|vr|dnd}}}tj||d}|dkr+t||}|durC|durC|dkr=|t|ksC|||}|S)r\r<r=Nr>rdrr)r read_tablerislice)rZr^r_r<r=r>tablerrrr`-s  $ z$ArrowReader._get_table_from_filenamecCs|rtnt}||S)z Read table from file. Args: filename (str): File name of the table. in_memory (bool, default=False): Whether to copy the data in-memory. Returns: pyarrow.Table )r r from_file)r<r_ table_clsrrrr<s zArrowReader.read_tabler) rrrrr@rr[rr` staticmethodr __classcell__rrrrrs  rcs6eZdZdZdededffdd ZddZZS) ParquetReaderzv Build a Dataset object out of Instruction instance(s). This Reader uses memory mapping on parquet files. r7r3rcr)zInitializes ParquetReader. Args: path (str): path where tfrecords are stored. info (DatasetInfo): info about the dataset. parquetNrrYrrrr[RrzParquetReader.__init__cKsv|dd|vr |dndd|vr|dnd}}}tj|dd}|dur9|dur9|dkr3|t|ks9|||}|S)r\r<r=Nr>T) memory_mapr)pqrrir)rZr^kwargsr<r=r>rxrrrr`\s $ z&ParquetReader._get_table_from_filename) rrrrr@rr[r`rrrrrrLs rc@s*eZdZUdZeed<eed<eed<dS)_AbsoluteInstructionz?A machine friendly slice: defined absolute positive boundaries.rFrGrHN)rrrrr@r&r%rrrrrks  rc@sbeZdZUdZeed<dZeeed<dZ eeed<dZ eeed<dZ eeed<dd Z dS) _RelativeInstructionzHRepresents a single parsed slicing instruction, can use % and negatives.rFNrGrHunitroundingcCs|jdur|jdvrtd|jdur|jdvrtd|jdkr*|jdur*td|jdkr?|jdur?t|jdkr?td|jdkrT|jdurTt|jdkrTtd|jdur`|jdkr`d n|j|jd <dS) N)%abszunit must be either % or abs)closestpct1_dropremainderz5rounding must be either closest or pct1_dropremainderrzAIt is forbidden to specify rounding if not using percent slicing.dz2Percent slice boundaries must be > -100 and < 100.rr)rrCrrGrrH__dict__rZrrr __post_init__~s""(z"_RelativeInstruction.__post_init__) rrrrr@r&rGrr%rHrrrrrrrrts  rcCst|}|std||ds|drdnd}t|d|d|dr1t|dnd |d rAt|d |d Sd |d S) z)Returns ReadInstruction for given string.z!Unrecognized instruction format: from_pctto_pctrrr9rfromNrH) split_namerrGrHr) _SUB_SPEC_REmatchrCgroupr,r%)specresrrrr_str_to_read_instructions rcCs&|dkr d}t||t|dS)NrzUsing "pct1_dropremainder" rounding on a split with less than 100 elements is forbidden: it always results in an empty dataset.Y@)rCmathtrunc)boundaryr#r~rrr_pct_to_abs_pct1s rcCstt||dS)Nr)r%round)rr#rrr_pct_to_abs_closestsrcCs|jdkrtnt}|j}||vrtd|dt|d||}|j}|j}|jdkrC|dur2dn|||}|dur=|n|||}n|durIdn|}|durQ|n|}|dkr^t ||d}|dkrit ||d}t ||}t ||}t |||S)zReturns _AbsoluteInstruction instance for given RelativeInstruction. Args: rel_instr: RelativeInstruction instance. name2len: dict {split_name: num_examples}. rzUnknown split "z". Should be one of .rNr) rrrrFrCr'rGrHrmaxminr) rel_instrrK pct_to_absr9r#rGrHrrr_rel_to_abs_instrs&    rc@sbeZdZdZddZeddZdddZed d Zd d Z d dZ ddZ ddZ ddZ dS)r,aReading instruction for a dataset. Examples:: # The following lines are equivalent: ds = datasets.load_dataset('mnist', split='test[:33%]') ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction.from_spec('test[:33%]')) ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction('test', to=33, unit='%')) ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction( 'test', from_=0, to=33, unit='%')) # The following lines are equivalent: ds = datasets.load_dataset('mnist', split='test[:33%]+train[1:-1]') ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction.from_spec( 'test[:33%]+train[1:-1]')) ds = datasets.load_dataset('mnist', split=( datasets.ReadInstruction('test', to=33, unit='%') + datasets.ReadInstruction('train', from_=1, to=-1, unit='abs'))) # The following lines are equivalent: ds = datasets.load_dataset('mnist', split='test[:33%](pct1_dropremainder)') ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction.from_spec( 'test[:33%](pct1_dropremainder)')) ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction( 'test', from_=0, to=33, unit='%', rounding="pct1_dropremainder")) # 10-fold validation: tests = datasets.load_dataset( 'mnist', [datasets.ReadInstruction('train', from_=k, to=k+10, unit='%') for k in range(0, 100, 10)]) trains = datasets.load_dataset( 'mnist', [datasets.ReadInstruction('train', to=k, unit='%') + datasets.ReadInstruction('train', from_=k+10, unit='%') for k in range(0, 100, 10)]) cCs ||_dSra_relative_instructions)rZrelative_instructionsrrr_inits zReadInstruction._initcCs||}|||S)zCReturns ReadInstruction obj initialized with relative_instructions.)__new__r)clsrresultrrr,_read_instruction_from_relative_instructionss  z.rN)r@_ADDITION_SEP_REr9rCrsum)rrsubsr+rrrrDs   zReadInstruction.from_specc Csg}|jD]c}|j}|jdus|jdurc|j}|j}|j}|j}|dkr&|nd}|dur2t||nd}|dur>t||nd}d|d|d}|dkr[|dur[|dkr[d|dnd} ||| 7}||qd |S) Nr[:]r()+) rrFrGrHrrr@rIrq) rZrel_instr_specsrrel_instr_specrGrHrr slice_str rounding_strrrrto_spec7s" &   zReadInstruction.to_speccCsjt|ts d}t||j}|j}|djdkr.|djdkr.|jdj|djkr.td|||S)zEReturns a new ReadInstruction obj, result of appending other to self.zAReadInstruction can only be added to another ReadInstruction obj.rrzPIt is forbidden to sum ReadInstruction instances with different rounding values.)r?r,rArrrrCr)rZotherr~self_ris other_risrrr__add__Ks zReadInstruction.__add__cCs|Sra)rrrrr__str__ZszReadInstruction.__str__cCsd|jdS)NzReadInstruction(rrrrrr__repr__]szReadInstruction.__repr__csfdd|jDS)aZTranslate instruction into a list of absolute instructions. Those absolute instructions are then to be added together. Args: name2len (`dict`): Associating split names to number of examples. Returns: list of _AbsoluteInstruction instances (corresponds to the + in spec). csg|]}t|qSr)r)r2rrKrrrklr5z/ReadInstruction.to_absolute..r)rZrKrrrrE`s zReadInstruction.to_absolute)NNNN)rrrrr classmethodrr[rDrrrrrErrrrr,s&    )NN)@rrnrrpre dataclassesr functoolsrtypingrrrpyarrowrupyarrow.parquetrrtqdm.contrib.concurrentrdownload.download_configr namingr r rr r rrutilsrrrrr3rrrr get_loggerrloggerHF_GCP_BASE_URLcompileXrrConnectionErrorrr r"r@r'rTrUrrrrrrrrr,rrrrsv              Kv/