list_objects

Improve the list_objects_v2 and ListObjectsV2 API.

class s3pathlib.better_client.list_objects.ObjectTypeDefIterproxy(iterable: Iterable)

An iterproxy that yields the “Contents” part of the ListObjectsV2 response.

Added in version 2.0.1.

class s3pathlib.better_client.list_objects.CommonPrefixTypeDefIterproxy(iterable: Iterable)

An iterproxy that yields the “CommonPrefixes” part of the ListObjectsV2 response.

Added in version 2.0.1.

class s3pathlib.better_client.list_objects.ListObjectsV2OutputTypeDefIterproxy(iterable: Iterable)

An iterproxy that yields the original ListObjectsV2 response. It has two utility methods to get the contents and common prefixes.

Added in version 2.0.1.

contents() → ObjectTypeDefIterproxy

Iterate object contents.

Added in version 2.0.1.

common_prefixs() → CommonPrefixTypeDefIterproxy

Iterate folders.

Added in version 2.0.1.

contents_and_common_prefixs() → Tuple[List[ObjectTypeDef], List[CommonPrefixTypeDef]]

Return the full list of object contents and folders.

Added in version 2.0.1.

s3pathlib.better_client.list_objects.paginate_list_objects_v2(s3_client: S3Client, bucket: str, prefix: str, batch_size: int = 1000, limit: int = OPT, delimiter: str = OPT, encoding_type: str = OPT, fetch_owner: bool = OPT, start_after: str = OPT, request_payer: str = OPT, expected_bucket_owner: str = OPT) → ListObjectsV2OutputTypeDefIterproxy

Wrapper of list_objects_v2 and ListObjectsV2. However, it returns a user-friendly ListObjectsV2OutputTypeDefIterproxy object.

Example:

>>> result = paginate_list_objects_v2(
...     s3_client=s3_client,
...     bucket="my-bucket",
...     prefix="my-folder",
... )
>>> for content in result.contents():
...     print(content)
{"Key": "1.json", "ETag": "...", "Size": 123, "LastModified": datetime(2015, 1, 1), "StorageClass": "...", "Owner", {...}}
{"Key": "2.json", "ETag": "...", "Size": 123, "LastModified": datetime(2015, 1, 1), "StorageClass": "...", "Owner", {...}}
{"Key": "3.json", "ETag": "...", "Size": 123, "LastModified": datetime(2015, 1, 1), "StorageClass": "...", "Owner", {...}}
...

Parameters:

• s3_client – boto3.session.Session().client("s3") object.

• bucket – See ListObjectsV2.

• prefix – See ListObjectsV2.

• batch_size – See ListObjectsV2.

• limit – See ListObjectsV2.

• delimiter – See ListObjectsV2.

• encoding_type – See ListObjectsV2.

• fetch_owner – See ListObjectsV2.

• start_after – See ListObjectsV2.

• request_payer – See ListObjectsV2.

• expected_bucket_owner – See ListObjectsV2.

Returns:

a ListObjectsV2OutputTypeDefIterproxy object.

Added in version 2.0.1.

s3pathlib.better_client.list_objects.is_content_an_object(content: ObjectTypeDef) → bool

Return True if the content is an object (not a folder).

Truth table

• ends with “/”, size is 0: False

• ends with “/”, size > 0: False

• ends without “/”, size is 0:

• ends without “/”, size > 0:

s3pathlib.better_client.list_objects.calculate_total_size(s3_client: S3Client, bucket: str, prefix: str, include_folder: bool = False) → Tuple[int, int]

Perform the “Calculate Total Size” action in AWS S3 console.

Parameters:

• s3_client – boto3.session.Session().client("s3") object

• bucket – S3 bucket name

• prefix – The s3 prefix (logic directory) you want to calculate

• include_folder – Default False, whether counting the hard folder (an empty “/” object).

Returns:

Tuple of (count, total_size). First value is number of objects, Second value is total size in bytes.

Added in version 2.0.1.

s3pathlib.better_client.list_objects.count_objects(s3_client: S3Client, bucket: str, prefix: str, include_folder: bool = False) → int

Count number of objects under prefix.

Parameters:

• s3_client – boto3.session.Session().client("s3") object

• bucket – S3 bucket name

• prefix – The s3 prefix (logic directory) you want to calculate

• include_folder – Default False, whether counting the hard folder (an empty “/” object).

Returns:

Number of objects under prefix.

Added in version 2.0.1.