Metadata and Tags

What an S3Path knows about its object (size, ETag, last modified time, version id, user metadata), when it asks S3 for it, and how to read and change the tags of an object.

Setup

This cell only prepares the notebook: it connects to the S3 bucket that the s3pathlib CI uses and empties the folder of this chapter, so every run starts from the same state. In your own code you only need the two lines marked below, with your own AWS profile and bucket.

import os

import boto3
from s3pathlib.api import S3Path
from s3pathlib.api import s3_context

boto_ses = boto3.Session(profile_name=os.environ["S3PATHLIB_TEST_AWS_PROFILE"])
s3_context.attach_s3_client(boto_ses.client("s3"))  # <-- in your code
aws_account_id = boto_ses.client("sts").get_caller_identity()["Account"]
bucket = f"{aws_account_id}-{boto_ses.region_name}-s3pathlib-test"  # <-- your bucket

# every demo of this chapter writes under this folder; start from empty
s3dir_chapter = S3Path(bucket, "projects/s3pathlib/docs/05-Metadata-and-Tags/")
s3dir_chapter.delete()
print(s3dir_chapter.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/05-Metadata-and-Tags/

1. Metadata properties

Write an object with some user metadata and an Expires header:

from datetime import datetime, timezone

s3dir = s3dir_chapter / "properties/"
p = s3dir / "report.json"
p.write_text(
    '{"total": 42}',
    metadata={"author": "alice"},
    expires_datetime=datetime(2030, 1, 1, tzinfo=timezone.utc),
)

S3Path('s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/05-Metadata-and-Tags/properties/report.json')

Take a new path to the same object. It knows nothing about the object yet; the first property you read loads the metadata with one head_object call, and the next reads cost nothing.

p = S3Path(bucket, "projects/s3pathlib/docs/05-Metadata-and-Tags/properties/report.json")
print(f"{p.size = }")

p.size = 13

print(f"{p.size_for_human = }")

p.size_for_human = '13 B'

print(f"{p.etag = }")

p.etag = '0f76e9fd645d9902dd96c79bf4d18748'

print(f"{p.metadata = }")

p.metadata = {'author': 'alice'}

print(f"{p.expire_at = }")

p.expire_at = datetime.datetime(2030, 1, 1, 0, 0, tzinfo=tzutc())

last_modified_at is a timezone aware datetime (its value changes on every run, so it is not printed here):

type(p.last_modified_at)

datetime.datetime

version_id is "null": that is the version id S3 gives an object written without versioning (see Versioning).

print(f"{p.version_id = }")

p.version_id = 'null'

response is the raw head_object response the properties read from (a copy; it never calls AWS):

response = p.response
response["ContentType"]

'binary/octet-stream'

write_text does not set a content type, so S3 uses binary/octet-stream. Pass content_type="text/plain" (or "application/json", …) when it matters, e.g. for browsers and CloudFront.

2. A snapshot of one version at one moment

What a path knows is a snapshot: it does not follow later changes in S3. Someone changes the object:

p.write_text('{"total": 42, "status": "final"}')

S3Path('s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/05-Metadata-and-Tags/properties/report.json')

p still has the old size:

print(f"{p.size = }")

p.size = 13

head_object() loads the latest metadata into the path (and returns the response):

response = p.head_object()
print(f"{p.size = }")

p.size = 32

It changes again:

p.write_text("{}")

S3Path('s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/05-Metadata-and-Tags/properties/report.json')

clear_cache() drops the snapshot, so the next read loads it again:

p.clear_cache()
print(f"{p.size = }")

p.size = 2

3. Where a snapshot comes from

A path gets a snapshot from the call that produced it:

• a full snapshot from head_object / get_object: head_object(), exists(), read_bytes() / read_text(), or the first property read;

• a partial snapshot from a write or list call: write_bytes(), copy_file(), iter_objects(), … It only has the fields that call returns.

A write returns a new path with the snapshot of the put call, so the size and the user metadata you just wrote cost no extra call:

p_new = (s3dir / "new.txt").write_text("hello", metadata={"source": "docs"})
print(f"{p_new.size = }")

p_new.size = 5

print(f"{p_new.metadata = }")

p_new.metadata = {'source': 'docs'}

A listed path has the size, ETag and last modified time of the list call:

for p in s3dir.iter_objects():
    print(p.uri, p.size)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/05-Metadata-and-Tags/properties/new.txt 5
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/05-Metadata-and-Tags/properties/report.json 2

A field the list call does not return, such as metadata, is loaded with head_object of the same object version: s3pathlib sends the listed ETag along (IfMatch), so it never mixes the size of one object with the metadata of another.

listed = s3dir.iter_objects().filter(S3Path.basename == "new.txt").one()
print(f"{listed.metadata = }")

listed.metadata = {'source': 'docs'}

If the object is replaced between the list call and the read, that is no longer possible. List it:

listed = s3dir.iter_objects().filter(S3Path.basename == "new.txt").one()

Someone replaces it:

(s3dir / "new.txt").write_text("replaced", metadata={"source": "someone else"})

S3Path('s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/05-Metadata-and-Tags/properties/new.txt')

Reading the metadata of the listed path now raises S3ObjectChangedError, instead of returning the metadata of the new object. List again to get a path to the new object.

from s3pathlib.api import exc

try:
    listed.metadata
except exc.S3ObjectChangedError as e:
    print(f"{type(e).__name__}: {e}")

S3ObjectChangedError: S3 object 's3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/05-Metadata-and-Tags/properties/new.txt' changed since its metadata was loaded (ETag 5d41402abc4b2a76b9719d911017c592), get a new S3Path to read the new object

A path built from another one (parent, /, change, copy(), …) is a new path and has no snapshot, so its response is None:

p = s3dir / "new.txt"
p.head_object()
p_copy = p.copy()
print(f"{p_copy.response = }")

p_copy.response = None

4. User metadata

User metadata (x-amz-meta-* headers) is set when the object is written: write_bytes, write_text, open, upload_file (extra_args={"Metadata": ...}), or a copy. S3 cannot change the metadata of an existing object; to change it, copy the object onto itself with new metadata (see Copy, Move and Delete).

S3 stores the keys in lower case, so s3pathlib warns about an upper case key. The cell below catches the warning to show it:

import warnings

with warnings.catch_warnings(record=True) as records:
    warnings.simplefilter("always")
    p = s3dir / "case.txt"
    p.write_text("x", metadata={"Author": "alice"})

print(records[0].message)

metadata key 'Author' has uppercase letters; S3 stores user-defined metadata keys in lowercase, so it is read back as 'author'. See https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html#UserMetadata

response = p.head_object()
response["Metadata"]

{'author': 'alice'}

5. Tags

Tags are key-value pairs on an object, used by lifecycle rules, IAM policies and cost reports. Unlike user metadata they can be changed at any time, without writing a new object. Each method returns the version id (None in a bucket that never had versioning) and the tags.

Set tags when writing:

p = s3dir / "tagged.txt"
p.write_text("data", tags={"team": "data"})
version_id, tags = p.get_tags()
tags

{'team': 'data'}

update_tags adds and overwrites keys, like dict.update:

version_id, tags = p.update_tags({"owner": "alice"})
tags

{'team': 'data', 'owner': 'alice'}

put_tags replaces all the tags:

version_id, tags = p.put_tags({"stage": "prod"})
tags

{'stage': 'prod'}

put_tags({}) removes them all:

version_id, tags = p.put_tags({})
tags

{}

update_tags reads the tags, merges and writes them back: two calls, not atomic, so a change made by someone else in between is lost. Tags belong to files; a folder path raises:

try:
    s3dir.get_tags()
except exc.S3PathIsNotFileError as e:
    print(f"{type(e).__name__}: {e}")

S3PathIsNotFileError: 's3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/05-Metadata-and-Tags/properties/' is not a file!

Common pitfalls

• A snapshot does not follow S3. A path that read its size keeps that size after the object changes; call head_object() or clear_cache() when you need the latest data.

• Writes return a new path. The path you wrote to keeps its old snapshot; read the metadata of what you wrote from the returned path.

• The snapshot never picks the version. read_bytes(), get_tags(), delete(), … always work on the latest version unless you pass version_id, even on a path that describes an older version (see Versioning).

• Lower case metadata keys. {"Author": ...} comes back as {"author": ...}.

See also

• List and Filter: filter by the listed size, etag, last_modified_at.

• Clients and Errors: which client loads the metadata of a path listed with s3_client=....