o Si$ @sdZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl m Z m Z mZmZmZddlZddlZeeejddZddZdd Zd d Zd d ZGdddZde de fddZd8dedefddZ dede!fddZ"e j#e j#fdededefd d!Z$d"ee%e!ffd#d$Z&ddde&fd%ededed&ed'e d(eff d)d*Z'd9d+d,Z(d9d-d.Z)d9d/d0Z*Gd1d2d2e+Z,d9dd3d4d5Z-d6d7Z.dS):z Miscellaneous utility functions.N)AnyCallableIterableIteratorUnion WDS_SECURE0cCst|}dd|DS)zApply glob to patterns with braces by pre-expanding the braces. Args: pattern (str): The glob pattern with braces. Returns: list: A list of file paths matching the expanded pattern. cSs g|] }t|D]}|q qS)glob).0patfr r D/home/ubuntu/.local/lib/python3.10/site-packages/webdataset/utils.py $s z$glob_with_braces..) braceexpand)patternexpandedr r rglob_with_bracess rcs<t|}|D] }t|rdSqtfdd|DS)a*Apply fnmatch to patterns with braces by pre-expanding the braces. Args: filename (str): The filename to match against. pattern (str): The pattern with braces to match. Returns: bool: True if the filename matches any of the expanded patterns, False otherwise. Tc3s|] }t|VqdSN)fnmatch)r r filenamer r 5sz&fnmatch_with_braces..)rrany)rrrr r rrfnmatch_with_braces's  rcGs&d}|D] }|dt|d@}q|S)zGenerate a seed value from the given arguments. Args: *args: Variable length argument list to generate the seed from. Returns: int: A 31-bit positive integer seed value. ri)hash)argsseedargr r r make_seed8s r cCsJt|trdSt|trdSt|trdSt|trdSt|tr#dSdS)zCheck if an object is iterable (excluding strings and bytes). Args: obj: The object to check for iterability. Returns: bool: True if the object is iterable (excluding strings and bytes), False otherwise. FT) isinstancestrbyteslistrr)objr r r is_iterableGs     r&c@seZdZdZddZdS) PipelineStagezBase class for pipeline stages.cOst)zInvoke the pipeline stage. Args: *args: Variable length argument list. **kw: Arbitrary keyword arguments. Raises: NotImplementedError: This method should be implemented by subclasses. )NotImplementedError)selfrkwr r rinvoke`s zPipelineStage.invokeN)__name__ __module__ __qualname____doc__r+r r r rr']s r'xreturncCs|S)zReturn the argument as is. Args: x (Any): The input value. Returns: Any: The input value unchanged. r )r0r r ridentityms r2{}sexprcCs0tdd||krtd|dt||S)a5Evaluate the given expression more safely. Args: s (str): The string to evaluate. expr (str, optional): The expression format. Defaults to "{}". Returns: Any: The result of the evaluation. Raises: ValueError: If the input string contains illegal characters. z [^A-Za-z0-9_]z#safe_eval: illegal characters in: '')resub ValueErrorevalformat)r4r5r r r safe_evalys r=symmodulescCs8|D]}tj|dd}t||d}|dur|SqdS)zLook up a symbol in a list of modules. Args: sym (str): The symbol to look up. modules (list): A list of module names to search in. Returns: Any: The found symbol, or None if not found. webdataset)packageN) importlib import_modulegetattr)r>r?mnamemoduleresultr r r lookup_syms  rHloadernepochsnbatchesccs&t|D] }t||EdHqdS)aeRepeatedly returns batches from a DataLoader. Args: loader (Iterator): The data loader to yield batches from. nepochs (int, optional): Number of epochs to repeat. Defaults to sys.maxsize. nbatches (int, optional): Number of batches per epoch. Defaults to sys.maxsize. Yields: Any: Batches from the data loader. N)rangeittislice)rIrJrK_r r r repeatedly0s rPbatchcCs t|dS)zGuess the batch size by looking at the length of the first element in a tuple. Args: batch (Union[tuple, list]): The batch to guess the size of. Returns: int: The guessed batch size. r)len)rQr r rguess_batchsizes rSsourcensamples batchsize.c cs~d}d}d} |D]%}|V|d7}|dur||krdS|dur/|t|7}||kr/dSq |d7}|dur>||kr>dSq)aRepeatedly yield samples from an iterator. Args: source (Iterator): The source iterator to yield samples from. nepochs (int, optional): Number of epochs to repeat. Defaults to None. nbatches (int, optional): Number of batches to yield. Defaults to None. nsamples (int, optional): Number of samples to yield. Defaults to None. batchsize (Callable[..., int], optional): Function to guess batch size. Defaults to guess_batchsize. Yields: Any: Samples from the source iterator. rTN)rS) rTrJrKrUrVepochrQtotalsampler r r repeatedlys& r[cCsd}d}d}d}dtjvr!dtjvr!ttjd}ttjd}n/z%ddl}|jrE|jrE|p6|jjj}|jj |d}|jj |d}Wn t yOYnwdtjvridtjvrittjd}ttjd}n zddl }|j j}|dur~|j}|j}Wn t yYnw||||fS) aReturn node and worker info for PyTorch and some distributed environments. Args: group (optional): The process group for distributed environments. Defaults to None. Returns: tuple: A tuple containing (rank, world_size, worker, num_workers). rrWRANK WORLD_SIZENgroupWORKER NUM_WORKERS)osenvironinttorch.distributed distributed is_availableis_initializedr_WORLDget_rankget_world_sizeModuleNotFoundErrortorch.utils.datautilsdataget_worker_infoid num_workers)r_rank world_sizeworkerrrtorch worker_infor r rpytorch_worker_infos>     rxcCst|d\}}}}|d|S)zCompute a distinct, deterministic RNG seed for each worker and node. Args: group (optional): The process group for distributed environments. Defaults to None. Returns: int: A deterministic RNG seed. r^i)rx)r_rsrtrurrr r rpytorch_worker_seeds rycs:t|r |}dnd}|fdd}|dur|S||S)Ncstfdd}|S)Ncs@djd}dur|d7}tj|tdd|i|S)NzCall to deprecated function . Reason: )category stacklevel)r,warningswarnDeprecationWarningrkwargsmsgfuncreasonr rnew_func s z/deprecated..decorator..new_func) functoolswraps)rrr)rr decorators zdeprecated..decorator)callable)rrrr rr deprecateds rc@s eZdZdS)ObsoleteExceptionN)r,r-r.r r r rr6srrcs2dur tjtdStfdd}|S)NrcsJttjddsdjd}dur|d7}t||i|S)NALLOW_OBSOLETErzCall to obsolete function z%. Set env ALLOW_OBSOLETE=1 to permit.r{)rdrbrcgetr,rrrr rr>s  zobsolete..new_func)rpartialobsoleter)rrrr rrr:s rcCsBtdd|D}tdd|D}||}|t|}|S)NcSg|]}|dqS)rr r pr r rrKz*compute_sample_weights..cSr)rWr rr r rrLr)nparrayamax) n_w_pairsnswsweightedpsr r rcompute_sample_weightsJs r)r3r)/r/rrr rB itertoolsrMrbr8sysrtypingrrrrrrnumpyrboolrdrcrenforce_securityrrr r&r'r2r"r=r$rHmaxsizerPtuplerSr[rxryr Exceptionrrrr r r rsZ     % + !