o ;i}@sddlmZddlmZmZddlmZddlmZddl m Z ddl m Z ddl mZeGd d d Zd eeeefd eejfd dZe eZdS)) dataclass)OptionalSequence)urlparse)api_pb2)synchronize_api)logger)_Secretc@s~eZdZUdZeed<dZeeed<dZeeed<dZ ee ed<dZ eeed<dZ e ed <dZe ed <dZe ed <dS) _CloudBucketMounta Mounts a cloud bucket to your container. Currently supports AWS S3 buckets. S3 buckets are mounted using [AWS S3 Mountpoint](https://github.com/awslabs/mountpoint-s3). S3 mounts are optimized for reading large files sequentially. It does not support every file operation; consult [the AWS S3 Mountpoint documentation](https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMANTICS.md) for more information. **AWS S3 Usage** ```python import subprocess app = modal.App() secret = modal.Secret.from_name( "aws-secret", required_keys=["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY"] # Note: providing AWS_REGION can help when automatic detection of the bucket region fails. ) @app.function( volumes={ "/my-mount": modal.CloudBucketMount( bucket_name="s3-bucket-name", secret=secret, read_only=True ) } ) def f(): subprocess.run(["ls", "/my-mount"], check=True) ``` **Cloudflare R2 Usage** Cloudflare R2 is [S3-compatible](https://developers.cloudflare.com/r2/api/s3/api/) so its setup looks very similar to S3. But additionally the `bucket_endpoint_url` argument must be passed. ```python import subprocess app = modal.App() secret = modal.Secret.from_name( "r2-secret", required_keys=["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY"] ) @app.function( volumes={ "/my-mount": modal.CloudBucketMount( bucket_name="my-r2-bucket", bucket_endpoint_url="https://.r2.cloudflarestorage.com", secret=secret, read_only=True ) } ) def f(): subprocess.run(["ls", "/my-mount"], check=True) ``` **Google GCS Usage** Google Cloud Storage (GCS) is [S3-compatible](https://cloud.google.com/storage/docs/interoperability). GCS Buckets also require a secret with Google-specific key names (see below) populated with a [HMAC key](https://cloud.google.com/storage/docs/authentication/managing-hmackeys#create). ```python import subprocess app = modal.App() gcp_hmac_secret = modal.Secret.from_name( "gcp-secret", required_keys=["GOOGLE_ACCESS_KEY_ID", "GOOGLE_ACCESS_KEY_SECRET"] ) @app.function( volumes={ "/my-mount": modal.CloudBucketMount( bucket_name="my-gcs-bucket", bucket_endpoint_url="https://storage.googleapis.com", secret=gcp_hmac_secret, ) } ) def f(): subprocess.run(["ls", "/my-mount"], check=True) ``` bucket_nameNbucket_endpoint_url key_prefixsecretoidc_auth_role_arnF read_onlyrequester_paysforce_path_style)__name__ __module__ __qualname____doc__str__annotations__r rrrr rrboolrrrrL/home/ubuntu/.local/lib/python3.10/site-packages/modal/cloud_bucket_mount.pyr s Y  r mountsreturnc Csg}|D]o\}}|jr3t|j}|jdrtjjj}n|jdr(tjjj}nt dtjjj }ntjjj }|j rB|j sBtd|jrO|jdsOtd|j}tj|j|j||j r`|j jnd|j||j ||j|jd }||q|S) zfHelper function to convert `CloudBucketMount` to a list of protobufs that can be passed to the server.zr2.cloudflarestorage.comzstorage.googleapis.comzfCloudBucketMount received unrecognized bucket endpoint URL. Assuming AWS S3 configuration as fallback.z4Credentials required in order to use Requester Pays./zHkey_prefix will be prefixed to all object paths, so it must end in a '/') r r mount_pathcredentials_secret_idr bucket_typerrrr)r rhostnameendswithrCloudBucketMount BucketTypeR2GCPr infoS3rr ValueErrorrr object_idrrrappend)rcloud_bucket_mountspathmount parse_resultr#rcloud_bucket_mountrrrcloud_bucket_mounts_to_proto{s@          r4N) dataclassesrtypingrr urllib.parser modal_protor_utils.async_utilsrconfigr rr r tuplerlistr&r4rrrrs      $m /