o -i @sddlZddlmZmZddlmZerddlmZzddlZWn e y-edZYnwde e de e de e fd d Z de e de e de e fd d Z  ddedde de e dBde e fddZ  dddde de e dBde e dBdee e e e ff ddZdS)N) TYPE_CHECKINGOptional)PlaceholderModule) BaseClientboto3pathspatternsreturncfdd|DS)Ncs&g|]tfddDrqS)c3|] }t|VqdSNfnmatch.0patternpath]/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/vllm/transformers_utils/s3_utils.py z+_filter_allow...anyrrrr  z!_filter_allow..rrrrrr _filter_allow rcr )Ncs&g|]tfddDsqS)c3r r r rrrrrrz,_filter_ignore...rrrrrrrz"_filter_ignore..rrrrr_filter_ignorer r!s3rr allow_patterncsJ|dur td}|ds|d}t|||d\}}fdd|DS)a) List full file names from S3 path and filter by allow pattern. Args: s3: S3 client to use. path: The S3 path to list from. allow_pattern: A list of patterns of which files to pull. Returns: list[str]: List of full S3 paths allowed by the pattern Nr#/)rr$csg|] }dd|qS)s3://r%r)rr bucket_namerrr7szglob..)rclientendswith list_files)r#rr$_rrr'rglob"s   r-ignore_patternc Cs|dd}d|dd}|d}|j||d}dd|d gD}t|d g}|dur6t||}|dur?t||}|||fS) aZ List files from S3 path and filter by pattern. Args: s3: S3 client to use. path: The S3 path to list from. allow_pattern: A list of patterns of which files to pull. ignore_pattern: A list of patterns of which files not to pull. Returns: tuple[str, str, list[str]]: A tuple where: - The first element is the bucket name - The second element is string represent the bucket and the prefix as a dir like string - The third element is a list of files allowed or disallowed by pattern r&r%Nr)BucketPrefixcSsg|]}|dqS)Keyr)robjrrrrVszlist_files..Contentsz*/) removeprefixsplitjoinlist_objects_v2getr!r) r#rr$r.partsprefixr(objectsrrrrr+:s    r+)Nr"N)NN)rtypingrrvllm.utils.import_utilsrbotocore.clientrr ImportErrorliststrrr!r-tupler+rrrrsJ     ""