o xi0@sdZddlmZddlZddlZddlmZddlmZddl m Z m Z m Z dd l mZed d d Zed dd ZeeZGdddeZdS)zImplements the AWS environment.) annotationsN) LaunchError) get_module)ARN_PARTITION_RE S3_URI_REevent_loop_thread_exec)AbstractEnvironmentboto3zcAWS environment requires boto3 to be installed. Please install it with `pip install wandb[launch]`.)requiredbotocorezfAWS environment requires botocore to be installed. Please install it with `pip install wandb[launch]`.cseZdZdZd&fd d Zed'd(d dZed)ddZed*ddZ e j d+ddZ d*ddZ d,ddZ d-ddZ d.dd Zd.d!d"Zd/d$d%ZZS)0AwsEnvironmentzAWS environment.regionstr access_key secret_key session_tokenreturnNonecsDttd|d||_||_||_||_d|_d|_ dS)zInitialize the AWS environment. Arguments: region (str): The AWS region. Raises: LaunchError: If the AWS environment is not configured correctly. z'Initializing AWS environment in region .N) super__init___loggerinfo_region _access_key _secret_key_session_token_account _partition)selfrrrr __class__`/home/ubuntu/.local/lib/python3.10/site-packages/wandb/sdk/launch/environment/aws_environment.pyrs  zAwsEnvironment.__init__N str | Nonec Cstdz+t}t|dr|p|j}|ptjd}| }|s&t d|j }|j }|j }WntjjyE}zt d|d}~ww|sLt d|||||dS) a-Create an AWS environment from the default AWS environment. Arguments: region (str, optional): The AWS region. verify (bool, optional): Whether to verify the AWS environment. Defaults to True. Returns: AwsEnvironment: The AWS environment. z2Creating AWS environment from default credentials.r AWS_REGIONz|Could not create AWS environment from default environment. Please verify that your AWS credentials are configured correctly.z}Could not create AWS environment from default environment. Please verify that your AWS credentials are configured correctly. NzPCould not create AWS environment from default environment. Region not specified.)rrrr)rrr Sessionhasattrrosenvirongetget_credentialsrrrtokenr client ClientError)clsrsession credentialsrrrer$r$r% from_default7s<    zAwsEnvironment.from_defaultconfigdict[str, str]cCs(t|dd}|std|j|dS)a.Create an AWS environment from the default AWS environment. Arguments: config (dict): Configuration dictionary. verify (bool, optional): Whether to verify the AWS environment. Defaults to True. Returns: AwsEnvironment: The AWS environment. rzCCould not create AWS environment from config. Region not specified.)r)rr,rr5)r1r6rr$r$r% from_config_szAwsEnvironment.from_configcCs|jS)zThe AWS region.r)r!r$r$r%rvszAwsEnvironment.regioncCs ||_dSNr:)r!rr$r$r%r{s c sz>|IdH}t|jdIdH}t|j}|IdH}|d}|s)tdt|}|s8td|d|d}|WSt j j yU}ztd||d}~ww) z*Set the partition for the AWS environment.NstsArnz;Could not set partition for AWS environment. ARN not found.z1Could not set partition for AWS environment. ARN z is not valid.r z-Could not set partition for AWS environment. ) get_sessionrr/get_caller_identityr,rrmatchgroupr exceptionsr0) r!r2r/r?identityarnmatched_partition partitionr4r$r$r% get_partitions4      zAwsEnvironment.get_partitionc stdz#|IdH}t|jdIdH}t|j}|IdHd|_WdStj j y?}zt d||d}~ww)zVerify that the AWS environment is configured correctly. Raises: LaunchError: If the AWS environment is not configured correctly. zVerifying AWS environment.Nr<AccountzdCould not verify AWS environment. Please verify that your AWS credentials are configured correctly. ) rdebugr>rr/r?r,rr rBr0r)r!r2r/r?r4r$r$r%verifys  zAwsEnvironment.verify boto3.Sessionc sjtd|jzttj}||j|j|j|jdIdHWSt j j y4}zt d|d}~ww)zGet an AWS session. Returns: boto3.Session: The AWS session. Raises: LaunchError: If the AWS session could not be created. zCreating AWS session in region ) region_nameaws_access_key_idaws_secret_access_keyaws_session_tokenNzCould not create AWS session. ) rrIrrr r(rrrr rBr0r)r!r2r4r$r$r%r>s  zAwsEnvironment.get_sessionsource destinationc std|d|d|d|d}tj|s%t|d|dt|}|s6t|d|d|d }|d  d }|sGd }| Id H}zt |j dId H}| |||Wd Stjjy} zt|d|d|d| d } ~ ww)a Upload a file to s3 from local storage. The destination is a valid s3 URI, e.g. s3://bucket/key and will be used as a prefix for the uploaded file. Only the filename of the source is kept in the upload key. So if the source is "foo/bar" and the destination is "s3://bucket/key", the file "foo/bar" will be uploaded to "s3://bucket/key/bar". Arguments: source (str): The path to the file or directory. destination (str): The uri of the storage destination. This should be a valid s3 URI, e.g. s3://bucket/key. Raises: LaunchError: If the copy fails, the source path does not exist, or the destination is not a valid s3 URI, or the upload fails. Uploading  to Error attempting to copy r : Source  does not exist.: Destination  is not a valid s3 URI.r r/r8Ns3$: botocore error attempting to copy . )rrIr*pathisfilerrr@rAlstripr>rr/ upload_filer rBr0) r!rPrQ _err_prefixr@bucketkeyr2r/r4r$r$r%r`s0   zAwsEnvironment.upload_filec std|d|d|d|d}tj|s%t|d|dt|}|s6t|d|d|d }|d  d }|sGd }| Id H}z8t |j dId H}t |D]%\} } } | D]} tj| | } | |d dd  d }|| ||qeq^Wd Stjjy}zt|d|d|d||d }~wty}zt|d|d|d||d }~ww)aUpload a directory to s3 from local storage. The upload will place the contents of the source directory in the destination with the same directory structure. So if the source is "foo/bar" and the destination is "s3://bucket/key", the contents of "foo/bar" will be uploaded to "s3://bucket/key/bar". Arguments: source (str): The path to the file or directory. destination (str): The URI of the storage. recursive (bool, optional): If True, copy the directory recursively. Defaults to False. Raises: LaunchError: If the copy fails, the source path does not exist, or the destination is not a valid s3 URI. rRrSrTrrUrVrWrXr rrYr8NrZ\r[r\z&: Unexpected error attempting to copy )rrIr*r]isdirrrr@rAr_r>rr/walkjoinreplacer`r rBr0 Exception)r!rPrQrar@rbrcr2r/r]_filesfileabs_pathkey_pathr4r$r$r% upload_dirsX    zAwsEnvironment.upload_diruric std|t|}|std|d|d}z|IdH}t|jdIdH}|j |dWdSt j j yv}z1|j dd d krUtd |d |d |j dd dkrftd |dtd|d|j dd}~ww)aVerify that s3 storage is configured correctly. This will check that the bucket exists and that the credentials are configured correctly. Arguments: uri (str): The URI of the storage. Raises: LaunchError: If the storage is not configured correctly or the URI is not a valid s3 URI. Returns: None zVerifying storage z Failed to validate storage uri: rXr NrZ)BucketErrorCode404z!Could not verify AWS storage uri z . Bucket rV403zy. Bucket {bucket} is not accessible. Please check that this client is authenticated with permission to access the bucket.z!Failed to verify AWS storage uri z . Response: zB Please verify that your AWS credentials are configured correctly.)rrIrr@rrAr>rr/ head_bucketr rBr0response)r!rpr@rbr2r/r4r$r$r%verify_storage_uris4    z!AwsEnvironment.verify_storage_uri) rrrrrrrrrrr;)rr&rr)r6r7rr)rr)rrrr)rr)rrK)rPrrQrrr)rprrr)__name__ __module__ __qualname____doc__r classmethodr5r9propertyrsetterrGrJr>r`rorx __classcell__r$r$r"r%rs"'        (5r)r| __future__rloggingr*wandb.sdk.launch.errorsr wandb.utilrutilsrrrabstractr r r getLoggerryrrr$r$r$r%s$