o wi$@sdZddlZddlZddlZddlmZmZmZmZm Z m Z m Z m Z ddl Z ddlmZmZddlmZdZdZdZd d Zd eeefd efd dZddddZefdeeeefdeegefdeeeffddZdeddfdejde edeegefde eegefde eegefd e eeeff ddZeddifdeeeefdeegefde eegefde eegefde ed e eeeff dd Zed!defdeeeefd"eege eeffd#ed$e e edeegefd e eeeff d%d&Z eddfd'eeeefdeegefde eegefde eegefd eeeeff d(d)Z!e"e!Z#dS)*z/Low level iteration functions for tar archives.N)AnyCallableDictIterableIteratorOptionalSetTuple)filtersgopen)reraise_exceptionF__cCs(td|}|s dS|d|dfS)zSplit off all file extensions. Args: path: Path with extensions. Returns: Tuple containing the base path and all extensions. z^((?:.*/|)[^.]+)[.]([^/]*)$)NNr )rematchgroup)pathrrT/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/webdataset/tariterators.py base_plus_exts rsamplereturncCs4|duot|tott|dko|dd S)zCheck whether a sample is valid. Args: sample: A dictionary representing a sample. Returns: Boolean indicating whether the sample is valid. Nr__bad__F) isinstancedictlenlistkeysget)rrrr valid_sample(s  r )shuffleccsHt|tr t|}nt|}|rt||D]}t|dVqdS)zGenerate a list of URLs, possibly shuffled. Args: urls: A string or list of URLs. shuffle: Whether to shuffle the URLs. Yields: Dictionary containing the URL. )urlN)rstr braceexpandrrandomr!r)urlsr!r"rrr shardlist:s  r'datahandlerkwc ks|D]N}t|tsJ|d|vsJ|d}ztj|fi|}|j|d|VWqtyQ}z|j|f|_||rEWYd}~qWYd}~dSd}~wwdS)zOpen URLs and yield a stream of url+stream pairs. Args: data: Iterator over dict(url=...). handler: Exception handler. **kw: Keyword arguments for gopen.gopen. Yields: A stream of url+stream pairs. r")streamN)rrr update Exceptionargs)r(r)r*rr"r+exnrrr url_openerNs"    r0z__[^/]*__($|/)fileobj skip_meta select_files rename_filesc csBtj|dd}|D]}|j}zM|sWq |durWq d|vr,|tr,|tr,Wq |dur8t ||r8Wq |r>||}|durH||sHWq | | }t ||d} | Vg|_ Wq ty} z5t| drt| jdkrt| jddt|f| jd d| _|| rWYd} ~ q WYd} ~ ~dSd} ~ ww~dS) axIterate over tar file, yielding filename, content pairs for the given tar stream. Args: fileobj: The tar file stream. skip_meta: Regexp for keys that are skipped entirely. handler: Exception handler. select_files: Predicate for selecting files. rename_files: Function to rename files. Yields: A stream of samples. zr|*)r1modeN/)fnamer(r.rz @ r )tarfileopennameisreg startswith meta_prefixendswith meta_suffixrr extractfilereadrmembersr-hasattrrr.r#) r1r2r)r3r4r+tarinfor7r(resultr/rrrtar_file_iteratorms>  ,  rF eof_valuec cs|D]x}|d}|d}zAt|tsJd|vsJt|d|||dD] }t|tr5d|vr5d|vs7J||d<|durC||d <|Vq&|durN|VWqty{} z | j|d|df| _|| roWYd} ~ qWYd} ~ dSd} ~ wwdS) acExpand tar files. Args: data: Iterator over opened tar file streams. handler: Exception handler. select_files: Select files from tarfiles by name (permits skipping files). rename_files: Function to rename files. eof_value: Value to yield at the end of each shard. Yields: A stream of samples. r" local_pathr+r)r3r4r(r7__url__N__local_path__)rrrrFr-r.) r(r)r3r4rGsourcer"rHrr/rrrtar_file_expanders:    rMTrlcasesuffixesc csd}|D]}zt|tsJ|ikrt|r|Vd}Wq|d|d}}||\} } tr>t| | t|tr;|nd| durDWq|rJ| } |dusT| |dkrct|r[|Vt| |dd}| |vrut|d| d||dus}| |vr||| <|d } | dur| |d <Wqt y} z| j |d |d f| _ || rWYd} ~ qWYd} ~ nd} ~ wwt|r|VdSdS) aGroup tarfile contents by keys and yield samples. Args: data: Iterator over tarfile contents. keys: Function that takes a file name and returns a key and a suffix. lcase: Whether to lowercase the suffix. suffixes: List of suffixes to keep. handler: Exception handler. Raises: ValueError: If there are duplicate file names in the tar file. Yields: Iterator over samples. Nr7r(__key__rJ)rPrJz": duplicate file name in tar file  rKr+r") rrr traceprintrlower ValueErrorrr-r.) r(rrNrOr)current_sample filesampler7valueprefixsuffixrHr/rrr group_by_keyssX    r[srccCs,t||d}t||||d}t||d}|S)aGenerate samples from a stream of tar files. Args: src: Stream of tar files. handler: Exception handler. select_files: Function that selects files to be included. rename_files: Function to rename files. Returns: Stream of samples. )r)rI)r0rMr[)r\r)r3r4streamsfilessamplesrrrtarfile_sampless  r`)$__doc__r%rr8typingrrrrrrrr r$r r handlersr rRr=r?rr#boolr r'r-r0TarFilerFrMr[r`pipelinefiltertarfile_to_samplesrrrrs(    !  4  2   B