o `۷i*@stddlZddlZddlmZdZdZdZGdddZedd d e DZ e d d Z e e _ d dZee _dS)N)urlparseRAY_RUNTIME_ENV_HTTP_USER_AGENTRAY_RUNTIME_ENV_BEARER_TOKENzray-runtime-env-curl/1.0c@seZdZdZeddZeddZeddZedd Zed d Z ed d Z ede fddZ eddZ edededefddZdS)ProtocolsProviderz~Note that these must be preinstalled on all nodes in the Ray cluster; it is not sufficient to install them in the runtime_env.cChdS)N> gss3uvgcspipfileabfssazurecondahttpsclsrrW/home/ubuntu/vllm_env/lib/python3.10/site-packages/ray/_private/runtime_env/protocol.py get_protocolszProtocolsProvider.get_protocolscCr)N>rrr r rrrrrrrget_remote_protocols*rz&ProtocolsProvider.get_remote_protocolscCsz ddl}ddlm}Wntytd|jw|}|r)|d}nddlm }ddl m }|jd||dd }d |i}||fS) zSet up S3 protocol handling. Returns: tuple: (open_file function, transport_params) Raises: ImportError: If required dependencies are not installed. rNopenzBYou must `pip install smart_open[s3]` to fetch URIs in s3 bucket. r)UNSIGNED)Config)signature_version)configclient) boto3 smart_openr ImportError_MISSING_DEPENDENCIES_WARNINGSessionget_credentialsrbotocorerbotocore.configr)rr open_filesession s3_clientrrtransport_paramsrrr_handle_s3_protocol.s$     z%ProtocolsProvider._handle_s3_protocolcCs@zddlm}ddlm}W|dfStytd|jw)zSet up Google Cloud Storage protocol handling. Returns: tuple: (open_file function, transport_params) Raises: ImportError: If required dependencies are not installed. r)storagerzTYou must `pip install smart_open[gcs]` to fetch URIs in Google Cloud Storage bucket.N) google.cloudr,r rr!r")rr,r'rrr_handle_gs_protocolQs  z%ProtocolsProvider._handle_gs_protocolcCszddlm}ddlm}ddlm}Wnty"td|jwt d}|s.t dd|d }d |||d i}||fS) aSet up Azure blob storage protocol handling. Returns: tuple: (open_file function, transport_params) Raises: ImportError: If required dependencies are not installed. ValueError: If required environment variables are not set. rDefaultAzureCredential)BlobServiceClientrzpYou must `pip install azure-storage-blob azure-identity smart_open[azure]` to fetch URIs in Azure Blob Storage. AZURE_STORAGE_ACCOUNTz`Azure Blob Storage authentication requires AZURE_STORAGE_ACCOUNT environment variable to be set.zhttps://z.blob.core.windows.net/r) account_url credential) azure.identityr0azure.storage.blobr1r rr!r"osgetenv ValueError)rr0r1r'azure_storage_account_namer3r*rrr_handle_azure_protocolgs,     z(ProtocolsProvider._handle_azure_protocolcsRz ddlddlmWntytd|jwddfdd }|dfS)a&Set up Azure Blob File System Secure (ABFSS) protocol handling. Returns: tuple: (open_file function, transport_params) Raises: ImportError: If required dependencies are not installed. ValueError: If the ABFSS URI format is invalid. rNr/z\You must `pip install adlfs azure-identity` to fetch URIs in Azure Blob File System Secure. r*cst|}|jr d|jvrtd||jdd\}}|s%td||r,|ds3td||dd}|sCtd |j|d }|||S) N@z6Invalid ABFSS URI format - missing container@account: z1Invalid ABFSS URI format - empty container name: z.dfs.core.windows.netzSInvalid ABFSS URI format - invalid hostname (must end with .dfs.core.windows.net): .rz/Invalid ABFSS URI format - empty account name: ) account_namer4)rnetlocr9splitendswithAzureBlobFileSystemr)urimoder*parsedcontainer_part hostname_partr: filesystemr0adlfsrrr's0 z;ProtocolsProvider._handle_abfss_protocol..open_file)rLr5r0r!r"rr'rrKr_handle_abfss_protocols  &z(ProtocolsProvider._handle_abfss_protocolreturncCs6tjttdd}tjt}|rd||d<|S)Nz*/*)z User-AgentAcceptzBearer Authorization)r7environget'RAY_RUNTIME_ENV_HTTP_USER_AGENT_ENV_VAR_DEFAULT_HTTP_USER_AGENT$RAY_RUNTIME_ENV_BEARER_TOKEN_ENV_VAR)rheaders bearer_tokenrrr _http_headerss zProtocolsProvider._http_headerscsJzddlmWntytdjwddfdd }|dfS)z6Set up HTTPS protocol handling with curl-like headers.rrz7You must `pip install smart_open` to fetch HTTPS URIs. Nr<cs*dd}|r|||||dS)N<)rWtimeoutr<)rYupdate)rErFr*paramsrsmart_open_openrrr's  z;ProtocolsProvider._handle_https_protocol..open_file)r rr!r"rMrr^r_handle_https_protocols  z(ProtocolsProvider._handle_https_protocolprotocol source_uri dest_filec Csd||vsJd}d}|dkr |tdd}dddd}nT|dkr+|\}}nI|dkr6|\}}n>|d krA|\}}n3|d krL|\}}n(|d krW|\}}nzd d lm}Wnt yst d| d|j w||d|d-}t|d}| | Wdn1swYWddSWddS1swYdS)axDownload file from remote URI to destination file. Args: protocol: The protocol to use for downloading (e.g., 's3', 'https'). source_uri: The source URI to download from. dest_file: The destination file path to save to. Raises: ImportError: If required dependencies for the protocol are not installed. Nr zfile://r<cSs t||SNr)rErFr*rrrr's z8ProtocolsProvider.download_remote_uri..open_filerrrrr rrz+You must `pip install smart_open` to fetch z URIs. rbwb)rlenr`r+r.r;rNr rr!upperr"writeread)rrarbrctpr'finfoutrrrdownload_remote_urisD   "z%ProtocolsProvider.download_remote_uriN)__name__ __module__ __qualname__r" classmethodrrr+r.r;rNdictrYr`strrnrrrrr s*   "  ' < rProtocolcCsi|]}||qSrrh.0rarrr $srycsfddtDS)Ncsg|]}|qSrrvrwrrr ,s z%_remote_protocols..)rrrrrr_remote_protocols(s r{cCst|j||Srd)rrnvalue)selfrbrcrrr_download_remote_uri4sr~)enumr7 urllib.parserrTrVrUrEnumrrurrr{remote_protocolsr~rnrrrrs$