Versioning

In a bucket with versioning, S3 keeps every version of an object: a write adds a version, and a delete adds a delete marker instead of removing anything. This chapter writes versions, lists them, reads an old one, restores a deleted object, and deletes versions permanently.

The demos use a bucket where versioning is enabled. Turning versioning on and off is in Buckets.

Setup

This cell only prepares the notebook: it connects to the S3 bucket that the s3pathlib CI uses and empties the folder of this chapter, so every run starts from the same state. In your own code you only need the two lines marked below, with your own AWS profile and bucket.

import os

import boto3
from s3pathlib.api import S3Path
from s3pathlib.api import s3_context

boto_ses = boto3.Session(profile_name=os.environ["S3PATHLIB_TEST_AWS_PROFILE"])
s3_context.attach_s3_client(boto_ses.client("s3"))  # <-- in your code
aws_account_id = boto_ses.client("sts").get_caller_identity()["Account"]
bucket = f"{aws_account_id}-{boto_ses.region_name}-s3pathlib-test-versioning-on"  # <-- your bucket

# every demo of this chapter writes under this folder; start from empty
s3dir_chapter = S3Path(bucket, "projects/s3pathlib/docs/07-Versioning/")
s3dir_chapter.delete(is_hard_delete=True)  # all versions
print(s3dir_chapter.uri)

s3://123456789012-us-east-1-s3pathlib-test-versioning-on/projects/s3pathlib/docs/07-Versioning/

Version ids are random strings that change on every run, so this chapter compares them instead of printing them.

1. Is versioning enabled?

Ask the bucket:

S3Path(bucket).is_versioning_enabled()

True

get_bucket_versioning returns the raw response. "Status" is "Enabled" or "Suspended", and missing if versioning was never enabled:

response = S3Path(bucket).get_bucket_versioning()
response["Status"]

'Enabled'

2. Write versions

Each write adds a version. The returned path knows the version id of what it wrote.

s3dir = s3dir_chapter / "versions/"
p = s3dir / "config.txt"
v1 = p.write_text("version 1")
v2 = p.write_text("version 2 is longer")
v3 = p.write_text("version 3 is the longest one")

v1.version_id == v2.version_id

False

Reading without version_id reads the latest version:

p.read_text()

'version 3 is the longest one'

Pass version_id to read an older one:

p.read_text(version_id=v1.version_id)

'version 1'

3. List versions

list_object_versions lists the versions of a file, newest first. Each path carries its own version id, size and last modified time:

for version in p.list_object_versions():
    print(version.uri, version.size, version.is_delete_marker())

s3://123456789012-us-east-1-s3pathlib-test-versioning-on/projects/s3pathlib/docs/07-Versioning/versions/config.txt 28 False
s3://123456789012-us-east-1-s3pathlib-test-versioning-on/projects/s3pathlib/docs/07-Versioning/versions/config.txt 19 False
s3://123456789012-us-east-1-s3pathlib-test-versioning-on/projects/s3pathlib/docs/07-Versioning/versions/config.txt 9 False

The newest one is v3:

newest = p.list_object_versions().one()
newest.version_id == v3.version_id

True

A listed version describes its version, but methods still work on the latest version unless you pass version_id. Reading the oldest listed version without version_id gives the latest content:

oldest = p.list_object_versions().all()[-1]
oldest.read_text()

'version 3 is the longest one'

Pass its version id to read it:

oldest.read_text(version_id=oldest.version_id)

'version 1'

On a folder, list_object_versions lists the versions of every object under it.

4. Delete adds a delete marker

delete() on a file adds a delete marker on top of the versions, and returns a path for that marker:

marker = p.delete()
marker.is_delete_marker()

True

The object no longer exists:

p.exists()

False

But every version is still there, under the new delete marker:

for version in p.list_object_versions():
    print(version.uri, version.size, version.is_delete_marker())

s3://123456789012-us-east-1-s3pathlib-test-versioning-on/projects/s3pathlib/docs/07-Versioning/versions/config.txt 0 True
s3://123456789012-us-east-1-s3pathlib-test-versioning-on/projects/s3pathlib/docs/07-Versioning/versions/config.txt 28 False
s3://123456789012-us-east-1-s3pathlib-test-versioning-on/projects/s3pathlib/docs/07-Versioning/versions/config.txt 19 False
s3://123456789012-us-east-1-s3pathlib-test-versioning-on/projects/s3pathlib/docs/07-Versioning/versions/config.txt 9 False

An old version can still be read:

p.read_text(version_id=v2.version_id)

'version 2 is longer'

5. Restore a deleted object

Delete the delete marker itself (by its version id), and the latest version is back:

p.delete(version_id=marker.version_id)
p.exists()

True

p.read_text()

'version 3 is the longest one'

To bring back an older version, copy it onto the object: the copy becomes the new latest version.

p.copy_file(p, version_id=v1.version_id, overwrite=True)
p.read_text()

'version 1'

6. Delete permanently

delete(version_id=...) deletes one version permanently:

p.delete(version_id=v2.version_id)

S3Path('s3://123456789012-us-east-1-s3pathlib-test-versioning-on/projects/s3pathlib/docs/07-Versioning/versions/config.txt')

for version in p.list_object_versions():
    print(version.uri, version.size, version.is_delete_marker())

s3://123456789012-us-east-1-s3pathlib-test-versioning-on/projects/s3pathlib/docs/07-Versioning/versions/config.txt 9 False
s3://123456789012-us-east-1-s3pathlib-test-versioning-on/projects/s3pathlib/docs/07-Versioning/versions/config.txt 28 False
s3://123456789012-us-east-1-s3pathlib-test-versioning-on/projects/s3pathlib/docs/07-Versioning/versions/config.txt 9 False

exists(version_id=...) checks one version:

p.exists(version_id=v2.version_id)

False

delete(is_hard_delete=True) deletes every version and every delete marker of the file. Other keys that start with the same name (such as config.txt.bak) are not touched.

backup = s3dir / "config.txt.bak"
backup.write_text("backup")
p.delete(is_hard_delete=True)

S3Path('s3://123456789012-us-east-1-s3pathlib-test-versioning-on/projects/s3pathlib/docs/07-Versioning/versions/config.txt')

for version in s3dir.list_object_versions():
    print(version.uri, version.size, version.is_delete_marker())

s3://123456789012-us-east-1-s3pathlib-test-versioning-on/projects/s3pathlib/docs/07-Versioning/versions/config.txt.bak 6 False

7. Folders

On a folder, delete() adds a delete marker to every object under it:

s3dir = s3dir_chapter / "folder/"
(s3dir / "a.txt").write_text("a")
(s3dir / "b.txt").write_text("b")
s3dir.delete()

S3Path('s3://123456789012-us-east-1-s3pathlib-test-versioning-on/projects/s3pathlib/docs/07-Versioning/folder/')

for version in s3dir.list_object_versions():
    print(version.uri, version.is_delete_marker())

s3://123456789012-us-east-1-s3pathlib-test-versioning-on/projects/s3pathlib/docs/07-Versioning/folder/a.txt True
s3://123456789012-us-east-1-s3pathlib-test-versioning-on/projects/s3pathlib/docs/07-Versioning/folder/a.txt False
s3://123456789012-us-east-1-s3pathlib-test-versioning-on/projects/s3pathlib/docs/07-Versioning/folder/b.txt True
s3://123456789012-us-east-1-s3pathlib-test-versioning-on/projects/s3pathlib/docs/07-Versioning/folder/b.txt False

delete(is_hard_delete=True) deletes every version and delete marker under the folder, which is how you really empty a folder in a versioned bucket:

s3dir.delete(is_hard_delete=True)
s3dir.list_object_versions().all()

[]

Common pitfalls

• Delete is not final. In a bucket with versioning, delete() only adds a delete marker; the data stays (and costs storage) until you delete the versions or a lifecycle rule expires them.

• Methods use the latest version. read_bytes, get_tags, copy_file, delete, … ignore the version a path describes; pass version_id=p.version_id.

• The version id of an object written before versioning was enabled is "null".

• Version ids are not ordered. Compare last_modified_at, or use the order of list_object_versions (newest first).

See also

• Buckets: enable and suspend versioning.

• Metadata and Tags: a listed version keeps the metadata of that version.