o -wi0@sdZddlZddlZddlmZmZddlmZddlm Z ddl m Z m Z m Z dd lmZe d d d Ze d dd ZeeZGdddeZdS)zImplements the AWS environment.N)DictOptional) LaunchError) get_module)ARN_PARTITION_RE S3_URI_REevent_loop_thread_exec)AbstractEnvironmentboto3zcAWS environment requires boto3 to be installed. Please install it with `pip install wandb[launch]`.)requiredbotocorezfAWS environment requires botocore to be installed. Please install it with `pip install wandb[launch]`.c seZdZdZdededededdf fdd Zed"deeddfd d Zed e eefddfd dZ e defddZ e j deddfddZ defddZd#ddZd$ddZdededdfddZdededdfddZdeddfd d!ZZS)%AwsEnvironmentzAWS environment.region access_key secret_key session_tokenreturnNcsDttd|d||_||_||_||_d|_d|_ dS)zInitialize the AWS environment. Arguments: region (str): The AWS region. Raises: LaunchError: If the AWS environment is not configured correctly. z'Initializing AWS environment in region .N) super__init___loggerinfo_region _access_key _secret_key_session_token_account _partition)selfrrrr __class__i/home/ubuntu/sommelier/.venv/lib/python3.10/site-packages/wandb/sdk/launch/environment/aws_environment.pyrs  zAwsEnvironment.__init__c Cstdz+t}t|dr|p|j}|ptjd}| }|s&t d|j }|j }|j }WntjjyE}zt d|d}~ww|sLt d|||||dS) a-Create an AWS environment from the default AWS environment. Arguments: region (str, optional): The AWS region. verify (bool, optional): Whether to verify the AWS environment. Defaults to True. Returns: AwsEnvironment: The AWS environment. z2Creating AWS environment from default credentials.r AWS_REGIONz|Could not create AWS environment from default environment. Please verify that your AWS credentials are configured correctly.z}Could not create AWS environment from default environment. Please verify that your AWS credentials are configured correctly. NzPCould not create AWS environment from default environment. Region not specified.)rrrr)rrr Sessionhasattrrosenvirongetget_credentialsrrrtokenrclient ClientError)clsrsession credentialsrrrer#r#r$ from_default6s<    zAwsEnvironment.from_defaultconfigcCs(t|dd}|std|j|dS)a.Create an AWS environment from the default AWS environment. Arguments: config (dict): Configuration dictionary. verify (bool, optional): Whether to verify the AWS environment. Defaults to True. Returns: AwsEnvironment: The AWS environment. rzCCould not create AWS environment from config. Region not specified.)r)strr*rr3)r/r4rr#r#r$ from_config^szAwsEnvironment.from_configcCs|jS)zThe AWS region.r)r r#r#r$ruszAwsEnvironment.regioncCs ||_dSNr8)r rr#r#r$rzs c sz>|IdH}t|jdIdH}t|j}|IdH}|d}|s)tdt|}|s8td|d|d}|WSt j j yU}ztd||d}~ww) z*Set the partition for the AWS environment.NstsArnz;Could not set partition for AWS environment. ARN not found.z1Could not set partition for AWS environment. ARN z is not valid.r z-Could not set partition for AWS environment. ) get_sessionr r-get_caller_identityr*rrmatchgroupr exceptionsr.) r r0r-r=identityarnmatched_partition partitionr2r#r#r$ get_partition~s4      zAwsEnvironment.get_partitionc stdz#|IdH}t|jdIdH}t|j}|IdHd|_WdStj j y?}zt d||d}~ww)zVerify that the AWS environment is configured correctly. Raises: LaunchError: If the AWS environment is not configured correctly. zVerifying AWS environment.Nr:AccountzdCould not verify AWS environment. Please verify that your AWS credentials are configured correctly. ) rdebugr<r r-r=r*rrr@r.r)r r0r-r=r2r#r#r$verifys  zAwsEnvironment.verify boto3.Sessionc sjtd|jzttj}||j|j|j|jdIdHWSt j j y4}zt d|d}~ww)zGet an AWS session. Returns: boto3.Session: The AWS session. Raises: LaunchError: If the AWS session could not be created. zCreating AWS session in region ) region_nameaws_access_key_idaws_secret_access_keyaws_session_tokenNzCould not create AWS session. ) rrGrr r r&rrrrr@r.r)r r0r2r#r#r$r<s  zAwsEnvironment.get_sessionsource destinationc std|d|d|d|d}tj|s%t|d|dt|}|s6t|d|d|d }|d  d }|sGd }| Id H}zt |j dId H}| |||Wd Stjjy} zt|d|d|d| d } ~ ww)a Upload a file to s3 from local storage. The destination is a valid s3 URI, e.g. s3://bucket/key and will be used as a prefix for the uploaded file. Only the filename of the source is kept in the upload key. So if the source is "foo/bar" and the destination is "s3://bucket/key", the file "foo/bar" will be uploaded to "s3://bucket/key/bar". Arguments: source (str): The path to the file or directory. destination (str): The uri of the storage destination. This should be a valid s3 URI, e.g. s3://bucket/key. Raises: LaunchError: If the copy fails, the source path does not exist, or the destination is not a valid s3 URI, or the upload fails. Uploading  to Error attempting to copy r : Source  does not exist.: Destination  is not a valid s3 URI.r r/r5Ns3$: botocore error attempting to copy . )rrGr(pathisfilerrr>r?lstripr<r r- upload_filerr@r.) r rNrO _err_prefixr>bucketkeyr0r-r2r#r#r$r^s0   zAwsEnvironment.upload_filec std|d|d|d|d}tj|s%t|d|dt|}|s6t|d|d|d }|d  d }|sGd }| Id H}z8t |j dId H}t |D]%\} } } | D]} tj| | } | |d dd  d }|| ||qeq^Wd Stjjy}zt|d|d|d||d }~wty}zt|d|d|d||d }~ww)aUpload a directory to s3 from local storage. The upload will place the contents of the source directory in the destination with the same directory structure. So if the source is "foo/bar" and the destination is "s3://bucket/key", the contents of "foo/bar" will be uploaded to "s3://bucket/key/bar". Arguments: source (str): The path to the file or directory. destination (str): The URI of the storage. recursive (bool, optional): If True, copy the directory recursively. Defaults to False. Raises: LaunchError: If the copy fails, the source path does not exist, or the destination is not a valid s3 URI. rPrQrRrrSrTrUrVr rrWr5NrX\rYrZz&: Unexpected error attempting to copy )rrGr(r[isdirrrr>r?r]r<r r-walkjoinreplacer^rr@r. Exception)r rNrOr_r>r`rar0r-r[_filesfileabs_pathkey_pathr2r#r#r$ upload_dirsX    zAwsEnvironment.upload_diruric std|t|}|std|d|d}z|IdH}t|jdIdH}|j |dWdSt j j yv}z1|j dd d krUtd |d |d |j dd dkrftd |dtd|d|j dd}~ww)aVerify that s3 storage is configured correctly. This will check that the bucket exists and that the credentials are configured correctly. Arguments: uri (str): The URI of the storage. Raises: LaunchError: If the storage is not configured correctly or the URI is not a valid s3 URI. Returns: None zVerifying storage z Failed to validate storage uri: rVr NrX)BucketErrorCode404z!Could not verify AWS storage uri z . Bucket rT403zy. Bucket {bucket} is not accessible. Please check that this client is authenticated with permission to access the bucket.z!Failed to verify AWS storage uri z . Response: zB Please verify that your AWS credentials are configured correctly.)rrGrr>rr?r<r r- head_bucketrr@r.response)r rnr>r`r0r-r2r#r#r$verify_storage_uris4    z!AwsEnvironment.verify_storage_urir9)rN)rrI)__name__ __module__ __qualname____doc__r6r classmethodrr3rr7propertyrsetterrErHr<r^rmrv __classcell__r#r#r!r$rs>'    (5r)rzloggingr(typingrrwandb.sdk.launch.errorsr wandb.utilrutilsrrr abstractr r r getLoggerrwrrr#r#r#r$s$