o $i,%@shddlZddlZddlmZGdddZedddeDZedd Z e e_ d d Z e e_ dS) N)urlparsec@sreZdZdZeddZeddZeddZedd Zed d Z ed d Z ede de de fddZ dS)ProtocolsProviderz~Note that these must be preinstalled on all nodes in the Ray cluster; it is not sufficient to install them in the runtime_env.cChdS)N> gss3uvgcspipfileabfssazurecondahttpsclsrr^/home/ubuntu/veenaModal/venv/lib/python3.10/site-packages/ray/_private/runtime_env/protocol.py get_protocols zProtocolsProvider.get_protocolscCr)N>rrr r r rrrrrrget_remote_protocols&rz&ProtocolsProvider.get_remote_protocolscCsz ddl}ddlm}Wntytd|jw|}|r)|d}nddlm }ddl m }|jd||dd }d |i}||fS) zSet up S3 protocol handling. Returns: tuple: (open_file function, transport_params) Raises: ImportError: If required dependencies are not installed. rNopenzBYou must `pip install smart_open[s3]` to fetch URIs in s3 bucket. r)UNSIGNED)Config)signature_version)configclient) boto3 smart_openr ImportError_MISSING_DEPENDENCIES_WARNINGSessionget_credentialsrbotocorerbotocore.configr)rr open_filesession s3_clientrrtransport_paramsrrr_handle_s3_protocol*s$     z%ProtocolsProvider._handle_s3_protocolcCs@zddlm}ddlm}W|dfStytd|jw)zSet up Google Cloud Storage protocol handling. Returns: tuple: (open_file function, transport_params) Raises: ImportError: If required dependencies are not installed. r)storagerzTYou must `pip install smart_open[gcs]` to fetch URIs in Google Cloud Storage bucket.N) google.cloudr*rrrr )rr*r%rrr_handle_gs_protocolMs  z%ProtocolsProvider._handle_gs_protocolcCszddlm}ddlm}ddlm}Wnty"td|jwt d}|s.t dd|d }d |||d i}||fS) aSet up Azure blob storage protocol handling. Returns: tuple: (open_file function, transport_params) Raises: ImportError: If required dependencies are not installed. ValueError: If required environment variables are not set. rDefaultAzureCredential)BlobServiceClientrzpYou must `pip install azure-storage-blob azure-identity smart_open[azure]` to fetch URIs in Azure Blob Storage. AZURE_STORAGE_ACCOUNTz`Azure Blob Storage authentication requires AZURE_STORAGE_ACCOUNT environment variable to be set.zhttps://z.blob.core.windows.net/r) account_url credential) azure.identityr.azure.storage.blobr/rrrr osgetenv ValueError)rr.r/r%azure_storage_account_namer1r(rrr_handle_azure_protocolcs,     z(ProtocolsProvider._handle_azure_protocolcsRz ddlddlmWntytd|jwddfdd }|dfS)a&Set up Azure Blob File System Secure (ABFSS) protocol handling. Returns: tuple: (open_file function, transport_params) Raises: ImportError: If required dependencies are not installed. ValueError: If the ABFSS URI format is invalid. rNr-z\You must `pip install adlfs azure-identity` to fetch URIs in Azure Blob File System Secure. r(cst|}|jr d|jvrtd||jdd\}}|s%td||r,|ds3td||dd}|sCtd |j|d }|||S) N@z6Invalid ABFSS URI format - missing container@account: z1Invalid ABFSS URI format - empty container name: z.dfs.core.windows.netzSInvalid ABFSS URI format - invalid hostname (must end with .dfs.core.windows.net): .rz/Invalid ABFSS URI format - empty account name: ) account_namer2)rnetlocr7splitendswithAzureBlobFileSystemr)urimoder(parsedcontainer_part hostname_partr8 filesystemr.adlfsrrr%s0 z;ProtocolsProvider._handle_abfss_protocol..open_file)rJr3r.rr )rr%rrIr_handle_abfss_protocols  &z(ProtocolsProvider._handle_abfss_protocolprotocol source_uri dest_filec CsN||vsJd}d}|dkr |tdd}dddd}nI|dkr+|\}}n>|dkr6|\}}n3|d krA|\}}n(|d krL|\}}nzd d lm}Wntyhtd | d|j w||d|d-}t|d}| | Wdn1swYWddSWddS1swYdS)axDownload file from remote URI to destination file. Args: protocol: The protocol to use for downloading (e.g., 's3', 'https'). source_uri: The source URI to download from. dest_file: The destination file path to save to. Raises: ImportError: If required dependencies for the protocol are not installed. Nr zfile://r:cSs t||SNr)rCrDr(rrrr%s z8ProtocolsProvider.download_remote_uri..open_filerrr r rrz+You must `pip install smart_open` to fetch z URIs. rbwb) rlenr)r,r9rKrrrupperr writeread)rrLrMrNtpr%finfoutrrrdownload_remote_uris@   "z%ProtocolsProvider.download_remote_uriN) __name__ __module__ __qualname__r classmethodrrr)r,r9rKstrrYrrrrrs"   "  ' <rProtocolcCsi|]}||qSrrS.0rLrrr srccsfddtDS)Ncsg|]}|qSrr`rarrr s z%_remote_protocols..)rrrrrr_remote_protocolss recCst|j||SrO)rrYvalue)selfrMrNrrr_download_remote_urisrh) enumr5 urllib.parserrEnumrr_r]reremote_protocolsrhrYrrrrs q