List and Filter

List the objects of a folder, recursively or one level at a time, take only what you need, filter by any attribute, and count. Listing is lazy: S3 is called page by page while you iterate.

Setup

This cell only prepares the notebook: it connects to the S3 bucket that the s3pathlib CI uses and empties the folder of this chapter, so every run starts from the same state. In your own code you only need the two lines marked below, with your own AWS profile and bucket.

import os

import boto3
from s3pathlib.api import S3Path
from s3pathlib.api import s3_context

boto_ses = boto3.Session(profile_name=os.environ["S3PATHLIB_TEST_AWS_PROFILE"])
s3_context.attach_s3_client(boto_ses.client("s3"))  # <-- in your code
aws_account_id = boto_ses.client("sts").get_caller_identity()["Account"]
bucket = f"{aws_account_id}-{boto_ses.region_name}-s3pathlib-test"  # <-- your bucket

# every demo of this chapter writes under this folder; start from empty
s3dir_chapter = S3Path(bucket, "projects/s3pathlib/docs/04-List-and-Filter/")
s3dir_chapter.delete()
print(s3dir_chapter.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/

All demos of this chapter list the same small folder. hard-folder/ has a folder marker (an empty object hard-folder/, what mkdir and the S3 console make); soft-folder/ has none, it only exists because keys start with it.

s3dir = s3dir_chapter / "tree/"
(s3dir / "README.txt").write_bytes(b"x" * 100)
(s3dir / "data-2024.csv").write_bytes(b"x" * 2000)
(s3dir / "hard-folder/" / "1.txt").write_bytes(b"x" * 300)
(s3dir / "soft-folder/" / "2.csv").write_bytes(b"x" * 4000)
(s3dir / "soft-folder/" / "sub/" / "3.csv").write_bytes(b"x" * 500)
(s3dir / "soft-folder/" / "sub/" / "4.json").write_bytes(b"x" * 60)

S3Path('s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/soft-folder/sub/4.json')

hard-folder/ already “exists” because a key starts with it, so mkdir needs exist_ok=True to write its marker:

(s3dir / "hard-folder/").mkdir(exist_ok=True)

1. Every object under a folder

iter_objects is recursive by default. The paths come in the order S3 returns them, sorted by key as a string (so data-2024.csv comes before hard-folder/..., because - is before /). Folder markers are skipped.

for p in s3dir.iter_objects():
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/README.txt
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/data-2024.csv
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/hard-folder/1.txt
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/soft-folder/2.csv
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/soft-folder/sub/3.csv
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/soft-folder/sub/4.json

recursive=False keeps the objects directly in the folder:

for p in s3dir.iter_objects(recursive=False):
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/README.txt
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/data-2024.csv

2. One level, folders included

iterdir works like pathlib.Path.iterdir: the files and the folders directly in this folder, with or without a marker, mixed in key order.

for p in s3dir.iterdir():
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/README.txt
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/data-2024.csv
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/hard-folder/
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/soft-folder/

On a sub folder:

for p in (s3dir / "soft-folder/").iterdir():
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/soft-folder/2.csv
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/soft-folder/sub/

Both need a folder. Without the trailing / the path is a file, and listing it raises before calling AWS. To list keys by a string prefix instead, use better_client.paginate_list_objects_v2 (section 7).

from s3pathlib.api import exc

not_a_folder = S3Path(bucket, "projects/s3pathlib/docs/04-List-and-Filter/tree")
try:
    not_a_folder.iter_objects()
except exc.S3PathIsNotFolderError as e:
    print(f"{type(e).__name__}: {e}")

S3PathIsNotFolderError: 's3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree' is not a dir!

3. Take what you need

iter_objects and iterdir return an S3PathIterProxy, an iterator with a few helpers. The pages are fetched while you iterate, so taking the first object of a folder with millions of objects makes one API call.

one() returns the first path:

first = s3dir.iter_objects().one()
first

S3Path('s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/README.txt')

many(n) returns a list of the first n:

for p in s3dir.iter_objects().many(2):
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/README.txt
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/data-2024.csv

skip(n) skips the first n:

for p in s3dir.iter_objects().skip(4):
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/soft-folder/sub/3.csv
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/soft-folder/sub/4.json

one_or_none() returns None for an empty folder, where one() would raise:

empty = s3dir / "empty/"
print(empty.iter_objects().one_or_none())

None

all() returns a list of everything:

paths = s3dir.iter_objects().all()
len(paths)

6

limit stops after that many paths. (batch_size, 1 to 1000, default 1000, is how many keys one API call asks for.)

for p in s3dir.iter_objects(limit=3):
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/README.txt
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/data-2024.csv
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/hard-folder/1.txt

An S3PathIterProxy can be iterated only once, like any iterator. The second loop below prints nothing; call iter_objects() again, or keep .all() in a list.

proxy = s3dir.iter_objects()
for p in proxy:
    pass
for p in proxy:
    print(p.uri)
print("done")

done

4. Filter

filter_by_ext keeps the files with one of the extensions (case insensitive):

for p in s3dir.iter_objects().filter_by_ext(".csv", ".json"):
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/data-2024.csv
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/soft-folder/2.csv
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/soft-folder/sub/3.csv
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/soft-folder/sub/4.json

filter takes conditions written on the class attributes, like an ORM: S3Path.size >= 1000 is a test that filter runs on each path. The filters run in memory, on the paths S3 returns.

for p in s3dir.iter_objects().filter(S3Path.size >= 1000):
    print(p.uri, p.size)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/data-2024.csv 2000
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/soft-folder/2.csv 4000

Several conditions in one filter must all be true:

for p in s3dir.iter_objects().filter(S3Path.ext == ".csv", S3Path.size < 1000):
    print(p.uri, p.size)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/soft-folder/sub/3.csv 500

or_ keeps the paths that pass any of the conditions:

from s3pathlib.api import or_

for p in s3dir.iter_objects().filter(or_(S3Path.ext == ".txt", S3Path.ext == ".json")):
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/README.txt
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/hard-folder/1.txt
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/soft-folder/sub/4.json

and_ keeps the paths that pass all the conditions; it is useful inside an or_. Here: the big CSV files, or any JSON file:

from s3pathlib.api import and_

for p in s3dir.iter_objects().filter(
    or_(
        and_(S3Path.ext == ".csv", S3Path.size >= 1000),
        S3Path.ext == ".json",
    )
):
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/data-2024.csv
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/soft-folder/2.csv
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/soft-folder/sub/4.json

not_ reverses a condition:

from s3pathlib.api import not_

for p in s3dir.iter_objects().filter(not_(S3Path.dirname == "sub")):
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/README.txt
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/data-2024.csv
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/hard-folder/1.txt
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/soft-folder/2.csv

Besides the comparisons (==, !=, <, <=, >, >=, also written as the methods equal_to, not_equal_to, less, less_equal, greater, greater_equal), there are between, startswith, endswith and contains:

for p in s3dir.iter_objects().filter(S3Path.size.between(100, 500)):
    print(p.uri, p.size)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/README.txt 100
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/hard-folder/1.txt 300
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/soft-folder/sub/3.csv 500

for p in s3dir.iter_objects().filter(S3Path.basename.startswith("data")):
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/data-2024.csv

for p in s3dir.iter_objects().filter(S3Path.parts.contains("sub")):
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/soft-folder/sub/3.csv
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/soft-folder/sub/4.json

The attributes you can filter on are bucket, key, uri, arn, parts, basename, dirname, fname, ext, abspath, dirpath, and the metadata size, etag, last_modified_at, version_id, expire_at. For example, the objects written in the last hour:

from datetime import datetime, timedelta, timezone

one_hour_ago = datetime.now(timezone.utc) - timedelta(hours=1)
recent = s3dir.iter_objects().filter(S3Path.last_modified_at >= one_hour_ago).all()
len(recent)

6

Any function that takes a path and returns a bool works too:

def is_small_json(p):
    return p.ext == ".json" and p.size < 100


for p in s3dir.iter_objects().filter(is_small_json):
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/tree/soft-folder/sub/4.json

5. Listed paths already know their size

Each listed path carries what the list call returned (size, etag, last_modified_at), so reading them, or filtering on them, costs no extra call:

first = s3dir.iter_objects().one()
print(f"{first.size = }")

first.size = 100

print(f"{first.etag = }")

first.etag = 'aed563ecafb4bcc5654c597a421547b2'

Other fields, such as the user metadata, are loaded with a head_object call the first time you read them; see Metadata and Tags.

6. Count and total size

count_objects and calculate_total_size go through every object under the folder (recursively), like “Calculate total size” in the S3 console.

s3dir.count_objects()

6

Folder markers only count with include_folder=True (here the marker of hard-folder/):

s3dir.count_objects(include_folder=True)

7

count, total_size = s3dir.calculate_total_size()
print(f"{count = }")
print(f"{total_size = }")

count = 6
total_size = 6960

count, total_size = s3dir.calculate_total_size(for_human=True)
print(f"{total_size = }")

total_size = '6.80 KB'

7. Keys that S3Path cannot represent

S3 allows keys like a//b.txt (two / in a row) or /a.txt (a leading /), but a path collapses repeated /, so the S3Path of a//b.txt would be a/b.txt: another object. Rather than give you a path to the wrong object, listing such a key raises S3ValidationError.

Write such a key with boto3:

s3dir_odd = s3dir_chapter / "odd-keys/"
s3_client = s3_context.s3_client
response = s3_client.put_object(Bucket=bucket, Key=s3dir_odd.key + "a//b.txt", Body=b"odd")

try:
    s3dir_odd.iter_objects().all()
except exc.S3ValidationError as e:
    print(f"{type(e).__name__}: {e}")

S3ValidationError: the S3 key 'projects/s3pathlib/docs/04-List-and-Filter/odd-keys/a//b.txt' in bucket '123456789012-us-east-1-s3pathlib-test' cannot be represented by S3Path, which joins consecutive '/' and drops a leading '/' (its path would be 's3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/04-List-and-Filter/odd-keys/a/b.txt', another object). List this folder with s3pathlib.api.better_client or boto3 instead

List it with better_client.paginate_list_objects_v2, which returns the raw boto3 responses (and also lists by any string prefix):

from s3pathlib.api import better_client

result = better_client.paginate_list_objects_v2(s3_client, bucket, prefix=s3dir_odd.key)
for content in result.contents():
    print(content["Key"])

projects/s3pathlib/docs/04-List-and-Filter/odd-keys/a//b.txt

Methods that work on a whole folder (delete, copy_dir, move_to) work on the keys S3 returns, not on paths, so they handle such keys:

s3dir_odd.delete()
s3dir_odd.exists()

False

Common pitfalls

• The trailing slash. iter_objects and iterdir need a folder path.

• Iterate once. An S3PathIterProxy is consumed by the first .all() or for loop.

• S3 order is not sorted order. Results come in key string order; sorted(paths) puts each folder before its content (see Path Manipulation).

• Filters run on your side. S3 only filters by prefix; a filter on size still lists every key under the folder. Narrow the folder first.

See also

• Metadata and Tags: what a listed path knows, and what triggers a head_object.

• Versioning: list_object_versions.