o ۾i @sddlZddlmZddlmZerddlmZzddlZWn ey+edZYnwde e de e de e fd d Z de e de e de e fd d Z  dddde de e dBde e fddZ  dddde de e dBde e dBdee e e e ff ddZdS)N) TYPE_CHECKING)PlaceholderModule) BaseClientboto3pathspatternsreturncfdd|DS)Ncs&g|]tfddDrqS)c3|] }t|VqdSNfnmatch.0patternpathT/home/ubuntu/.local/lib/python3.10/site-packages/vllm/transformers_utils/s3_utils.py z+_filter_allow...anyrrrr  z!_filter_allow..rrrrrr _filter_allow rcr )Ncs&g|]tfddDsqS)c3r r r rrrrrrz,_filter_ignore...rrrrrrrz"_filter_ignore..rrrrr_filter_ignorerr s3zBaseClient | Noner allow_patterncsJ|dur td}|ds|d}t|||d\}}fdd|DS)a) List full file names from S3 path and filter by allow pattern. Args: s3: S3 client to use. path: The S3 path to list from. allow_pattern: A list of patterns of which files to pull. Returns: list[str]: List of full S3 paths allowed by the pattern Nr"/)rr#csg|] }dd|qS)s3://r$r)rr bucket_namerrr7szglob..)rclientendswith list_files)r"rr#_rrr&rglob"s   r,rignore_patternc Cs|dd}d|dd}|d}|j||d}dd|d gD}t|d g}|dur6t||}|dur?t||}|||fS) aZ List files from S3 path and filter by pattern. Args: s3: S3 client to use. path: The S3 path to list from. allow_pattern: A list of patterns of which files to pull. ignore_pattern: A list of patterns of which files not to pull. Returns: tuple[str, str, list[str]]: A tuple where: - The first element is the bucket name - The second element is string represent the bucket and the prefix as a dir like string - The third element is a list of files allowed or disallowed by pattern r%r$Nr)BucketPrefixcSsg|]}|dqS)Keyr)robjrrrrVszlist_files..Contentsz*/) removeprefixsplitjoinlist_objects_v2getr r) r"rr#r-partsprefixr'objectsrrrrr*:s    r*)Nr!N)NN)r typingrvllm.utils.import_utilsrbotocore.clientrr ImportErrorliststrrr r,tupler*rrrrsJ      ""