Buckets

Create and delete a bucket, list the buckets of your account, and turn versioning on and off. The bucket methods work on a bucket path, S3Path("my-bucket").

Setup

This cell only prepares the notebook: it connects to AWS with the profile that the s3pathlib CI uses. The demos create and delete a temporary bucket; if a failed run left it behind, the cell deletes it first, so every run starts from the same state.

[1]:

import os

import boto3
from s3pathlib.api import S3Path
from s3pathlib.api import s3_context

boto_ses = boto3.Session(profile_name=os.environ["S3PATHLIB_TEST_AWS_PROFILE"])
s3_context.attach_s3_client(boto_ses.client("s3"))  # <-- in your code
aws_account_id = boto_ses.client("sts").get_caller_identity()["Account"]

# a bucket name is global: put your account id in it to make it unique
bucket_name = f"s3pathlib-test-tmp-docs-{aws_account_id}"
if S3Path(bucket_name).exists():  # left over from a failed run
    S3Path(bucket_name).delete(is_hard_delete=True, skip_prompt=True)
    S3Path(bucket_name).delete_bucket()
print(bucket_name)

s3pathlib-test-tmp-docs-123456789012

1. Create a bucket

from_bucket builds a bucket path from a name (the same as S3Path(bucket_name), but it only accepts a bucket name):

[2]:

s3bucket = S3Path.from_bucket(bucket_name)
s3bucket

[2]:

S3Path('s3://s3pathlib-test-tmp-docs-123456789012/')

[3]:

s3bucket.exists()

[3]:

False

create_bucket creates it, in the region of the S3 client unless you pass region. It returns the raw boto3 response:

[4]:

response = s3bucket.create_bucket()
s3bucket.exists()

[4]:

True

In us-east-1, creating a bucket you already own succeeds again (a legacy behavior of S3); in the other regions it raises BucketAlreadyOwnedByYou. A name owned by another AWS account raises BucketAlreadyExists.

2. List buckets

list_buckets returns every bucket of the account, as bucket paths:

[5]:

buckets = S3Path.list_buckets()
s3bucket in buckets

[5]:

True

Each bucket path carries what S3 returned: last_modified_at is the creation date of the bucket, and the region is in response["BucketRegion"]:

[6]:

for bucket in buckets:
    if bucket == s3bucket:
        print(bucket.response["BucketRegion"])

us-east-1

3. Versioning

A new bucket has no versioning:

[7]:

s3bucket.is_versioning_enabled()

[7]:

False

put_bucket_versioning(True) enables it:

[8]:

response = s3bucket.put_bucket_versioning(True)
s3bucket.is_versioning_enabled()

[8]:

True

get_bucket_versioning returns the raw response:

[9]:

response = s3bucket.get_bucket_versioning()
response["Status"]

[9]:

'Enabled'

Versioning can never be turned off again, only suspended: new writes no longer add versions, but the existing versions are kept.

[10]:

response = s3bucket.put_bucket_versioning(False)
s3bucket.is_versioning_suspended()

[10]:

True

4. Delete a bucket

Write an object, so the bucket is not empty:

[11]:

(s3bucket / "file.txt").write_text("hello")

[11]:

S3Path('s3://s3pathlib-test-tmp-docs-123456789012/file.txt')

S3 only deletes an empty bucket, with no object, no version and no delete marker. Otherwise delete_bucket raises BucketNotEmpty:

[12]:

import botocore.exceptions

try:
    s3bucket.delete_bucket()
except botocore.exceptions.ClientError as e:
    print(e.response["Error"]["Code"])

BucketNotEmpty

Empty it first. delete() on a bucket path deletes everything in it, so it asks you to type YES; skip_prompt=True skips the question. Because this bucket had versioning, is_hard_delete=True also deletes the versions and delete markers.

[13]:

s3bucket.delete(is_hard_delete=True, skip_prompt=True)
s3bucket.count_objects()

[13]:

0

[14]:

response = s3bucket.delete_bucket()
s3bucket.exists()

[14]:

False

The bucket methods only work on a bucket path:

[15]:

from s3pathlib.api import exc

try:
    S3Path("my-bucket", "folder/").create_bucket()
except exc.S3PathIsNotBucketError as e:
    print(f"{type(e).__name__}: {e}")

S3PathIsNotBucketError: 's3://my-bucket/folder/' is not a bucket!

Common pitfalls

  • Bucket names are global. Another AWS account may own the name you want; add something unique, such as your account id.

  • A bucket must be empty to be deleted, including old versions and delete markers in a bucket that ever had versioning.

  • Deleting a bucket path deletes everything in it. S3Path("my-bucket").delete() empties the bucket (after asking); delete_bucket() removes the bucket itself.

See also