Read and Write

Read and write one S3 object: bytes and text, metadata and tags on write, conditional writes, empty objects and folder markers, file objects with open (JSON, CSV, gzip, streaming), and uploading local files.

Setup

This cell only prepares the notebook: it connects to the S3 bucket that the s3pathlib CI uses and empties the folder of this chapter, so every run starts from the same state. In your own code you only need the two lines marked below, with your own AWS profile and bucket.

import os

import boto3
from s3pathlib.api import S3Path
from s3pathlib.api import s3_context

boto_ses = boto3.Session(profile_name=os.environ["S3PATHLIB_TEST_AWS_PROFILE"])
s3_context.attach_s3_client(boto_ses.client("s3"))  # <-- in your code
aws_account_id = boto_ses.client("sts").get_caller_identity()["Account"]
bucket = f"{aws_account_id}-{boto_ses.region_name}-s3pathlib-test"  # <-- your bucket

# every demo of this chapter writes under this folder; start from empty
s3dir_chapter = S3Path(bucket, "projects/s3pathlib/docs/03-Read-and-Write/")
s3dir_chapter.delete()
print(s3dir_chapter.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/03-Read-and-Write/

1. Bytes and text

write_bytes replaces the object if it exists, and returns a new S3Path with a snapshot of what was written, so reading its size costs no AWS call. The path you called it on does not change; see Metadata and Tags for the snapshot model.

s3dir = s3dir_chapter / "bytes-and-text/"
p = s3dir / "hello.txt"
p_new = p.write_bytes(b"Hello, S3!")
print(f"{p_new.size = }")

p_new.size = 10

p.read_bytes()

b'Hello, S3!'

write_text and read_text do the same with a str:

p.write_text("Hello again!")
p.read_text()

'Hello again!'

encoding works as in str.encode / bytes.decode:

p = s3dir / "greeting.txt"
p.write_text("你好", encoding="utf-16")
p.read_text(encoding="utf-16")

'你好'

range reads part of the object, with the syntax of the HTTP Range header (both ends included):

p = s3dir / "hello.txt"
p.read_bytes(range="bytes=0-4")

b'Hello'

Reading a missing object raises boto3’s own error as is (NoSuchKey). Catch it as botocore.exceptions.ClientError and check the error code:

import botocore.exceptions

try:
    (s3dir / "missing.txt").read_text()
except botocore.exceptions.ClientError as e:
    print(e.response["Error"]["Code"])

NoSuchKey

2. Metadata, tags and headers

The other arguments of write_bytes / write_text are the arguments of boto3’s put_object in snake case. metadata and tags take a plain dict.

ContentTypeEnum has the common content types, as strings; with one of them, the “Open” button of the S3 console shows the object in the browser instead of downloading it.

from s3pathlib.api import ContentTypeEnum

s3dir = s3dir_chapter / "metadata-and-tags/"
p = s3dir / "report.json"
p_new = p.write_text(
    '{"total": 42}',
    content_type=ContentTypeEnum.app_json,
    metadata={"author": "alice"},
    tags={"project": "docs"},
)
p_new.metadata

{'author': 'alice'}

The content type is in the head_object response:

response = p.head_object()
response["ContentType"]

'application/json'

get_tags returns the version id (None without versioning) and the tags:

version_id, tags = p.get_tags()
tags

{'project': 'docs'}

S3 stores the keys of user metadata in lower case, so {"Author": ...} comes back as {"author": ...}; s3pathlib warns when you write an upper case key. More in Metadata and Tags.

3. Conditional writes

if_none_match="*" only writes when the object does not exist, in one atomic call: when two processes write at the same time, only one succeeds.

s3dir = s3dir_chapter / "conditional-writes/"
p = s3dir / "lock.txt"
p.write_text("owner: job-1", if_none_match="*")
p.read_text()

'owner: job-1'

The second write fails with PreconditionFailed, and the object keeps the first content:

try:
    p.write_text("owner: job-2", if_none_match="*")
except botocore.exceptions.ClientError as e:
    print(e.response["Error"]["Code"])

PreconditionFailed

p.read_text()

'owner: job-1'

if_match=<etag> only replaces the object if nobody changed it since you last saw it (optimistic locking). Keep the ETag of what you wrote:

p = s3dir / "counter.txt"
p_new = p.write_text("1")
etag = p_new.etag
etag

'c4ca4238a0b923820dcc509a6f75849b'

Someone else writes in between:

p.write_text("100")

S3Path('s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/03-Read-and-Write/conditional-writes/counter.txt')

Our write with the old ETag fails, and the other write is kept:

try:
    p.write_text("2", if_match=etag)
except botocore.exceptions.ClientError as e:
    print(e.response["Error"]["Code"])

PreconditionFailed

p.read_text()

'100'

4. Empty objects and folder markers

touch creates an empty object:

s3dir = s3dir_chapter / "touch-and-mkdir/"
p = s3dir / "_SUCCESS"
p.touch()
p.read_bytes()

b''

It never changes an existing object; it raises S3FileAlreadyExist, or does nothing with exist_ok=True:

from s3pathlib.api import exc

try:
    p.touch()
except exc.S3FileAlreadyExist as e:
    print(f"{type(e).__name__}: {e}")

S3FileAlreadyExist: S3 object 's3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/03-Read-and-Write/touch-and-mkdir/_SUCCESS' already exists!

p.touch(exist_ok=True)

S3 has no folders. A folder “exists” as soon as one key starts with it. mkdir writes a folder marker, the empty object folder/ that “Create folder” makes in the S3 console, so that an empty folder shows up.

s3dir_new = s3dir / "a/b/c/"
s3dir_new.exists()

False

parents=True also writes the markers of the parent folders:

s3dir_new.mkdir(parents=True)
s3dir_new.exists()

True

iter_objects skips folder markers, so it only lists _SUCCESS:

for p in s3dir.iter_objects():
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/03-Read-and-Write/touch-and-mkdir/_SUCCESS

count_objects counts them with include_folder=True: _SUCCESS and the markers touch-and-mkdir/, a/, a/b/ and a/b/c/.

s3dir.count_objects(include_folder=True)

5

Note that parents=True writes the marker of every parent up to the bucket (here also projects/, projects/s3pathlib/, …), not only of the folders that did not exist: in S3 a folder without marker “exists” too.

mkdir on an existing folder raises S3FolderAlreadyExist, unless exist_ok=True:

try:
    s3dir_new.mkdir()
except exc.S3FolderAlreadyExist as e:
    print(f"{type(e).__name__}: {e}")

S3FolderAlreadyExist: S3 folder 's3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/03-Read-and-Write/touch-and-mkdir/a/b/c/' already exists!

5. Exists

exists works on a file:

s3dir = s3dir_chapter / "exists/"
p = s3dir / "data.csv"
p.write_text("a,b")
p.exists()

True

On a folder (any key under it, a folder marker included):

s3dir.exists()

True

(s3dir / "missing/").exists()

False

And on a bucket:

S3Path(bucket).exists()

True

ensure_not_exists raises if the path exists, handy before an operation that must not replace anything:

try:
    p.ensure_not_exists()
except exc.S3FileAlreadyExist as e:
    print(f"{type(e).__name__}: {e}")

S3FileAlreadyExist: S3 object 's3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/03-Read-and-Write/exists/data.csv' already exists!

6. File objects with open

open returns a file object (through smart_open), so any library that reads or writes files works with S3. The content is streamed: a big object is never loaded in memory at once. Always use it in a with block; a write is only finished when the file is closed.

import json

s3dir = s3dir_chapter / "open/"
p = s3dir / "config.json"
with p.open("w") as f:
    json.dump({"name": "alice", "tags": ["a", "b"]}, f)

with p.open("r") as f:
    config = json.load(f)
config

{'name': 'alice', 'tags': ['a', 'b']}

Write a log file line by line:

p = s3dir / "events.log"
with p.open("w") as f:
    for i in range(1000):
        f.write(f"event {i}\n")

And read it line by line; only a small part of the file is in memory at any time:

n_lines = 0
with p.open("r") as f:
    for line in f:
        n_lines += 1
n_lines

1000

The csv module needs newline="", as with local files:

import csv

p = s3dir / "users.csv"
with p.open("w", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["id", "name"])
    writer.writerow([1, "alice"])

with p.open("r", newline="") as f:
    for row in csv.DictReader(f):
        print(row)

{'id': '1', 'name': 'alice'}

Compression follows the extension. Writing *.gz (or .bz2, .zst, …) writes compressed data, and reading it gives the uncompressed content.

p = s3dir / "data.json.gz"
with p.open("w") as f:
    json.dump({"compressed": True}, f)

with p.open("r") as f:
    data = json.load(f)
data

{'compressed': True}

read_bytes shows what is really stored: gzip data starts with the bytes 1f 8b.

p.read_bytes()[:2].hex()

'1f8b'

compression="disable" reads or writes the bytes as they are:

with p.open("rb", compression="disable") as f:
    first_bytes = f.read(2)
first_bytes.hex()

'1f8b'

If an exception is raised inside the with block of a write, the upload is aborted and nothing is written, so there is never a half written object:

p = s3dir / "half-written.txt"
try:
    with p.open("w") as f:
        f.write("first part")
        raise RuntimeError("the job failed")
except RuntimeError:
    print("the job failed")

the job failed

p.exists()

False

open also takes the write arguments of write_bytes (metadata, tags, content_type, …). By default it uploads in parts while you write (multipart upload); multipart_upload=False keeps the data in memory and sends it with one put_object when the file is closed, which is required for the arguments that only put_object has, such as if_none_match.

7. Upload local files

Create a few local files to upload:

import tempfile
from pathlib import Path

tmp = tempfile.TemporaryDirectory()
dir_local = Path(tmp.name)
(dir_local / "report.txt").write_text("local report")
(dir_local / "logs").mkdir()
(dir_local / "logs" / "1.log").write_text("log 1")
(dir_local / "logs" / "2.log").write_text("log 2")

5

upload_file uploads one local file (with boto3’s managed transfer, in parts for a big file) and returns the new S3Path:

s3dir = s3dir_chapter / "upload/"
p = s3dir / "report.txt"
p_new = p.upload_file(dir_local / "report.txt")
p_new.read_text()

'local report'

Like the copy methods, it does not replace an existing object unless you pass overwrite=True:

try:
    p.upload_file(dir_local / "report.txt")
except exc.S3FileAlreadyExist as e:
    print(f"{type(e).__name__}: {e}")

S3FileAlreadyExist: S3 object 's3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/03-Read-and-Write/upload/report.txt' already exists!

upload_dir uploads every file of a local directory (or the files matching a glob pattern), keeping their relative paths, and returns how many it uploaded. With overwrite=False (the default) it checks every target first and uploads nothing if one exists.

s3dir_backup = s3dir / "backup/"
s3dir_backup.upload_dir(dir_local, pattern="**/*.log")

2

for p in s3dir_backup.iter_objects():
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/03-Read-and-Write/upload/backup/logs/1.log
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/03-Read-and-Write/upload/backup/logs/2.log

tmp.cleanup()

Common pitfalls

• Writes return a new path. p.write_text(...) does not update the snapshot of p; use the returned path to read size / etag of what you wrote.

• Opening without a with block. The object is only written when the file is closed.

• Compression by extension. open on data.gz decompresses; pass compression="disable" to get the raw bytes.

• Conditional writes with open need multipart_upload=False, otherwise ValueError.

See also

• Metadata and Tags: the snapshot model, head_object, tags.

• Copy, Move and Delete.

• Versioning: read and write a specific version (version_id).