Quick Start

A ten minute tour of what you use every day: build a path, connect to S3, write and read an object, list a folder, copy and delete. Each section ends with the chapter that covers it in depth.

Setup

This cell only prepares the notebook: it connects to the S3 bucket that the s3pathlib CI uses and empties the folder of this chapter, so every run starts from the same state. In your own code you only need the two lines marked below, with your own AWS profile and bucket.

[1]:

import os

import boto3
from s3pathlib.api import S3Path
from s3pathlib.api import s3_context

boto_ses = boto3.Session(profile_name=os.environ["S3PATHLIB_TEST_AWS_PROFILE"])
s3_context.attach_s3_client(boto_ses.client("s3"))  # <-- in your code
aws_account_id = boto_ses.client("sts").get_caller_identity()["Account"]
bucket = f"{aws_account_id}-{boto_ses.region_name}-s3pathlib-test"  # <-- your bucket

# every demo of this chapter writes under this folder; start from empty
s3dir_chapter = S3Path(bucket, "projects/s3pathlib/docs/01-Quick-Start/")
s3dir_chapter.delete()
print(s3dir_chapter.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/01-Quick-Start/

1. Build a path

An S3Path is a bucket plus a key, like pathlib.Path is a drive plus a path. Building one never calls AWS. The first argument is the bucket, the others are parts of the key.

[2]:

p = S3Path("my-bucket", "folder", "file.txt")
p

[2]:

S3Path('s3://my-bucket/folder/file.txt')

The same path from an S3 URI:

[3]:

S3Path("s3://my-bucket/folder/file.txt")

[3]:

S3Path('s3://my-bucket/folder/file.txt')

Its parts:

[4]:

print(f"{p.bucket = }")

p.bucket = 'my-bucket'

[5]:

print(f"{p.key = }")

p.key = 'folder/file.txt'

[6]:

print(f"{p.uri = }")

p.uri = 's3://my-bucket/folder/file.txt'

[7]:

print(f"{p.basename = }")

p.basename = 'file.txt'

[8]:

print(f"{p.parent = }")

p.parent = S3Path('s3://my-bucket/folder/')

A trailing / makes a folder. S3 itself has no folders, only keys; s3pathlib treats every key that ends with / as a folder, the same as the AWS console.

[9]:

folder = S3Path("my-bucket", "folder/")
folder.is_dir()

[9]:

True

Join with /. Every part on the left of / must be a folder, so it ends with /:

[10]:

S3Path("my-bucket") / "folder/" / "file.txt"

[10]:

S3Path('s3://my-bucket/folder/file.txt')

More in Path Manipulation.

2. Connect to S3

Methods that call AWS use the S3 client attached to s3_context (the setup cell above did that). Without one, s3pathlib creates boto3.client("s3") with your default credentials the first time it needs one. Every such method also takes s3_client=... for a single call, e.g. to work with two AWS accounts. More in Clients and Errors.

3. Write and read

Each demo below uses its own folder under s3dir_chapter.

[11]:

s3dir = s3dir_chapter / "write-and-read/"
p = s3dir / "hello.txt"
p.exists()

[11]:

False

write_text (and write_bytes) returns a new S3Path that already knows the size and the ETag of what it wrote, so reading them costs no extra call.

[12]:

p_new = p.write_text("Hello, S3!")
print(f"{p_new.size = }")

p_new.size = 10

[13]:

print(f"{p_new.etag = }")

p_new.etag = '9176e88e2973b7e2e260c438f56b1cd0'

[14]:

p.exists()

[14]:

True

[15]:

p.read_text()

[15]:

'Hello, S3!'

More in Read and Write: bytes, conditional writes, file objects with open, uploading local files.

4. List a folder

Write three files to list:

[16]:

s3dir = s3dir_chapter / "list/"
(s3dir / "a.txt").write_text("a")
(s3dir / "b.csv").write_text("b")
(s3dir / "sub/" / "c.csv").write_text("c")

[16]:

S3Path('s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/01-Quick-Start/list/sub/c.csv')

iter_objects lists every object under a folder (recursively by default), in the order S3 returns them, sorted by key.

[17]:

for p in s3dir.iter_objects():
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/01-Quick-Start/list/a.txt
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/01-Quick-Start/list/b.csv
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/01-Quick-Start/list/sub/c.csv

iterdir lists one level, folders included, like pathlib.Path.iterdir:

[18]:

for p in s3dir.iterdir():
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/01-Quick-Start/list/a.txt
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/01-Quick-Start/list/b.csv
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/01-Quick-Start/list/sub/

Keep only some files, here by extension:

[19]:

for p in s3dir.iter_objects().filter_by_ext(".csv"):
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/01-Quick-Start/list/b.csv
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/01-Quick-Start/list/sub/c.csv

Count the objects and their total size:

[20]:

s3dir.count_objects()

[20]:

3

[21]:

count, total_size = s3dir.calculate_total_size()
print(f"{count = }")
print(f"{total_size = }")

count = 3
total_size = 3

More in List and Filter.

5. Copy

copy_file copies one object and returns the new S3Path of the copy.

[22]:

s3dir = s3dir_chapter / "copy/"
src = s3dir / "src.txt"
src.write_text("some data")

[22]:

S3Path('s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/01-Quick-Start/copy/src.txt')

[23]:

dst = s3dir / "dst.txt"
src.copy_file(dst)

[23]:

S3Path('s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/01-Quick-Start/copy/dst.txt')

[24]:

dst.read_text()

[24]:

'some data'

It does not replace an existing object unless you pass overwrite=True:

[25]:

from s3pathlib.api import exc

try:
    src.copy_file(dst)
except exc.S3FileAlreadyExist as e:
    print(f"{type(e).__name__}: {e}")

S3FileAlreadyExist: S3 object 's3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/01-Quick-Start/copy/dst.txt' already exists!

copy_dir copies every object under a folder and returns how many it copied:

[26]:

backup = s3dir / "backup/"
(s3dir_chapter / "list/").copy_dir(backup)

[26]:

3

[27]:

for p in backup.iter_objects():
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/01-Quick-Start/copy/backup/a.txt
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/01-Quick-Start/copy/backup/b.csv
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/01-Quick-Start/copy/backup/sub/c.csv

More in Copy, Move and Delete.

6. Delete

delete deletes an object, or every object under a folder. Deleting what does not exist is not an error.

[28]:

p = s3dir_chapter / "delete/" / "file.txt"
p.write_text("bye")

[28]:

S3Path('s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/01-Quick-Start/delete/file.txt')

[29]:

p.delete()

[29]:

S3Path('s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/01-Quick-Start/delete/file.txt')

[30]:

p.exists()

[30]:

False

Delete a whole folder:

[31]:

s3dir = s3dir_chapter / "copy/"
s3dir.delete()

[31]:

S3Path('s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/01-Quick-Start/copy/')

[32]:

s3dir.exists()

[32]:

False

In a bucket with versioning, delete adds a delete marker and keeps the old versions; see Versioning.

Where to go next