Clients and Errors

Which S3 client (and so which AWS account and region) a call uses, how to use several clients, what errors to expect and how to catch them, and the lower level better_client functions.

Setup

This cell only prepares the notebook: it connects to the S3 bucket that the s3pathlib CI uses and empties the folder of this chapter, so every run starts from the same state. In your own code you only need the two lines marked below, with your own AWS profile and bucket.

import os

import boto3
from s3pathlib.api import S3Path
from s3pathlib.api import s3_context

boto_ses = boto3.Session(profile_name=os.environ["S3PATHLIB_TEST_AWS_PROFILE"])
s3_context.attach_s3_client(boto_ses.client("s3"))  # <-- in your code
aws_account_id = boto_ses.client("sts").get_caller_identity()["Account"]
bucket = f"{aws_account_id}-{boto_ses.region_name}-s3pathlib-test"  # <-- your bucket

# every demo of this chapter writes under this folder; start from empty
s3dir_chapter = S3Path(bucket, "projects/s3pathlib/docs/10-Clients-and-Errors/")
s3dir_chapter.delete()
print(s3dir_chapter.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/10-Clients-and-Errors/

1. The default client

Every method that calls AWS uses s3_context.s3_client, unless you pass s3_client=.... The setup cell attached one with s3_context.attach_s3_client(...):

s3_context.s3_client.meta.region_name

'us-east-1'

If you never attach one, s3pathlib creates boto3.client("s3") the first time it needs a client, from the default credentials and region (environment variables, ~/.aws, an instance role, …). It does that on first use, not at import, so environment variables such as AWS_PROFILE set after import s3pathlib are honored.

2. A client for one call

Every method that calls AWS takes s3_client=... for that call only. Use it to work with two AWS accounts, or two regions, in the same program.

To show which client a call uses, make a client with credentials that do not work (the example key of the AWS documentation) and attach it as the default:

good_client = s3_context.s3_client
bad_client = boto3.client(
    "s3",
    aws_access_key_id="AKIAIOSFODNN7EXAMPLE",
    aws_secret_access_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    region_name="us-east-1",
)
s3_context.attach_s3_client(bad_client)

A call without s3_client now uses the bad client and fails:

import botocore.exceptions

p = s3dir_chapter / "clients/" / "file.txt"
try:
    p.write_text("hello")
except botocore.exceptions.ClientError as e:
    print(e.response["Error"]["Code"])

InvalidAccessKeyId

The same call with s3_client=good_client works:

p.write_text("hello", metadata={"owner": "alice"}, s3_client=good_client)

S3Path('s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/10-Clients-and-Errors/clients/file.txt')

p.read_text(s3_client=good_client)

'hello'

A path returned by a call with an explicit client remembers that client, but only to load its own metadata. Here the list call used the good client, so reading the user metadata of a listed path (one head_object call) works, although the default client is the bad one:

s3dir = s3dir_chapter / "clients/"
listed = s3dir.iter_objects(s3_client=good_client).one()
print(f"{listed.metadata = }")

listed.metadata = {'owner': 'alice'}

Every other method uses its own s3_client argument, or else the default, however the path was made. So read_text without s3_client uses the bad client again:

try:
    listed.read_text()
except botocore.exceptions.ClientError as e:
    print(e.response["Error"]["Code"])

InvalidAccessKeyId

Put the good client back as the default:

s3_context.attach_s3_client(good_client)
p.read_text()

'hello'

s3_client must be a boto3 S3 client. Anything else, such as a boto3 session, raises TypeError before calling AWS:

try:
    p.read_text(s3_client=boto_ses)
except TypeError as e:
    print(f"{type(e).__name__}: {e}")

TypeError: s3_client must be a boto3 S3 client, got Session(region_name='us-east-1'); for a BotoSesManager, pass bsm.s3_client

3. Errors from AWS are raised as is

An AWS call that fails raises boto3’s own error, unchanged: NoSuchKey, NoSuchBucket, AccessDenied, PreconditionFailed, … These are the errors the boto3 documentation lists, so you catch them as you would with boto3. Either with ClientError and the error code:

missing = s3dir_chapter / "missing.txt"
try:
    missing.read_text()
except botocore.exceptions.ClientError as e:
    print(e.response["Error"]["Code"])

NoSuchKey

Or with the exception classes of the client:

s3_client = s3_context.s3_client
try:
    missing.read_text()
except s3_client.exceptions.NoSuchKey:
    print("no such key")

no such key

4. Errors of s3pathlib

s3pathlib.exc has the errors of s3pathlib’s own rules: the wrong kind of path, overwrite=False, an invalid URI, … Each one inherits from S3PathlibError, and from the builtin exception that matches its meaning, so you can catch it without knowing s3pathlib:

from s3pathlib.api import exc

p = s3dir_chapter / "errors/" / "file.txt"
p.write_text("hello")
try:
    p.touch()
except FileExistsError as e:
    print(f"{type(e).__name__}: {e}")

S3FileAlreadyExist: S3 object 's3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/10-Clients-and-Errors/errors/file.txt' already exists!

issubclass(exc.S3FileAlreadyExist, exc.S3PathlibError)

True

The HEAD calls (head_object, and exists on a bucket or a file) are the exception to “raised as is”: S3 answers them with a bare 404 or 403 and no error name, which the client’s exception classes cannot catch. s3pathlib turns them into S3FileNotExist (a FileNotFoundError) and S3PermissionDenied (a PermissionError):

try:
    missing.head_object()
except FileNotFoundError as e:
    print(f"{type(e).__name__}: {e}")

S3FileNotExist: S3 object 's3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/10-Clients-and-Errors/missing.txt' does not exist!

A bucket of another AWS account that you cannot access, such as the bucket named aws:

try:
    S3Path("aws").exists()
except PermissionError as e:
    print(f"{type(e).__name__}: {e}")

S3PermissionDenied: no permission to access bucket 's3://aws/': it exists, but it belongs to another AWS account, or you lack the s3:ListBucket permission

The original boto3 error is always the __cause__:

try:
    missing.head_object()
except exc.S3FileNotExist as e:
    error = e
error.__cause__.response["Error"]["Code"]

'404'

5. Deprecation warnings

APIs that will change in 4.0.0 emit a DeprecationWarning that points at your line, such as joining onto a file (S3Path("my-bucket") / "folder" / "file.txt"). Python hides DeprecationWarning outside __main__ by default; run your tests with python -W error::DeprecationWarning (or pytest -W error::DeprecationWarning) to turn them into errors and find them all:

import warnings

with warnings.catch_warnings():
    warnings.simplefilter("error", DeprecationWarning)
    try:
        S3Path("my-bucket") / "folder" / "file.txt"
    except DeprecationWarning as e:
        print(f"{type(e).__name__}: {e}")

DeprecationWarning: 's3://my-bucket/folder' is a file. Joining onto a file is deprecated since 3.0.0, and will raise S3PathIsNotFolderError in 4.0.0. Join onto a dir instead: end the folder with '/' (e.g. ... / "folder/" / "file.txt"), call .to_dir() first, or join all parts at once (e.g. .joinpath("folder", "file.txt")).

6. The lower level better_client

s3pathlib.api.better_client has the functions that S3Path is built on. They work on a client, a bucket and a key, like boto3, and fix its rough edges: a True / False existence check, pagination with a limit, deleting a whole folder in batches, merging tags, … Use them when a path is not the right tool, e.g. to list keys that S3Path cannot represent (see List and Filter). s3pathlib.api.utils has the string helpers behind the paths, such as split_s3_uri and make_s3_console_url.

from s3pathlib.api import better_client

better_client.is_object_exists(s3_client, bucket, p.key)

True

better_client.is_bucket_exists(s3_client, bucket)

True

Common pitfalls

• The client of a listed path. iter_objects(s3_client=...) paths load their metadata with that client, but read_text(), delete(), … use their own s3_client argument or the default client. Pass s3_client to every call when you work with several accounts.

• Catch boto3 errors as boto3 errors. s3pathlib does not wrap NoSuchKey, NoSuchBucket, AccessDenied, …; only the HEAD calls become S3FileNotExist / S3PermissionDenied.

• Pass a client, not a session. s3_client=boto_ses raises TypeError; pass boto_ses.client("s3").

See also

• Quick Start

• Upgrade from 2.x: bsm= and context are gone.