list_objects#
Improve the list_objects_v2 and ListObjectsV2 API.
- class s3pathlib.better_client.list_objects.ObjectTypeDefIterproxy(iterable: Iterable)[source]#
An iterproxy that yields the “Contents” part of the ListObjectsV2 response.
New in version 2.0.1.
- class s3pathlib.better_client.list_objects.CommonPrefixTypeDefIterproxy(iterable: Iterable)[source]#
An iterproxy that yields the “CommonPrefixes” part of the ListObjectsV2 response.
New in version 2.0.1.
- class s3pathlib.better_client.list_objects.ListObjectsV2OutputTypeDefIterproxy(iterable: Iterable)[source]#
An iterproxy that yields the original ListObjectsV2 response. It has two utility methods to get the contents and common prefixes.
New in version 2.0.1.
- contents() ObjectTypeDefIterproxy [source]#
Iterate object contents.
New in version 2.0.1.
- common_prefixs() CommonPrefixTypeDefIterproxy [source]#
Iterate folders.
New in version 2.0.1.
- s3pathlib.better_client.list_objects.paginate_list_objects_v2(s3_client: S3Client, bucket: str, prefix: str, batch_size: int = 1000, limit: int = Sentinel('NOTHING'), delimiter: str = Sentinel('NOTHING'), encoding_type: str = Sentinel('NOTHING'), fetch_owner: bool = Sentinel('NOTHING'), start_after: str = Sentinel('NOTHING'), request_payer: str = Sentinel('NOTHING'), expected_bucket_owner: str = Sentinel('NOTHING')) ListObjectsV2OutputTypeDefIterproxy [source]#
Wrapper of list_objects_v2 and ListObjectsV2. However, it returns a user-friendly
ListObjectsV2OutputTypeDefIterproxy
object.Example:
>>> result = paginate_list_objects_v2( ... s3_client=s3_client, ... bucket="my-bucket", ... prefix="my-folder", ... ) >>> for content in result.contents(): ... print(content) {"Key": "1.json", "ETag": "...", "Size": 123, "LastModified": datetime(2015, 1, 1), "StorageClass": "...", "Owner", {...}} {"Key": "2.json", "ETag": "...", "Size": 123, "LastModified": datetime(2015, 1, 1), "StorageClass": "...", "Owner", {...}} {"Key": "3.json", "ETag": "...", "Size": 123, "LastModified": datetime(2015, 1, 1), "StorageClass": "...", "Owner", {...}} ...
- Parameters:
s3_client –
boto3.session.Session().client("s3")
object.bucket – See ListObjectsV2.
prefix – See ListObjectsV2.
batch_size – See ListObjectsV2.
limit – See ListObjectsV2.
delimiter – See ListObjectsV2.
encoding_type – See ListObjectsV2.
fetch_owner – See ListObjectsV2.
start_after – See ListObjectsV2.
request_payer – See ListObjectsV2.
expected_bucket_owner – See ListObjectsV2.
- Returns:
a
ListObjectsV2OutputTypeDefIterproxy
object.
New in version 2.0.1.
- s3pathlib.better_client.list_objects.is_content_an_object(content: ObjectTypeDef) bool [source]#
Return True if the content is an object (not a folder).
Truth table
ends with “/”, size is 0: False
ends with “/”, size > 0: False
ends without “/”, size is 0:
ends without “/”, size > 0:
- s3pathlib.better_client.list_objects.calculate_total_size(s3_client: S3Client, bucket: str, prefix: str, include_folder: bool = False) Tuple[int, int] [source]#
Perform the “Calculate Total Size” action in AWS S3 console.
- Parameters:
s3_client –
boto3.session.Session().client("s3")
objectbucket – S3 bucket name
prefix – The s3 prefix (logic directory) you want to calculate
include_folder – Default False, whether counting the hard folder (an empty “/” object).
- Returns:
Tuple of
(count, total_size)
. First value is number of objects, Second value is total size in bytes.
New in version 2.0.1.
- s3pathlib.better_client.list_objects.count_objects(s3_client: S3Client, bucket: str, prefix: str, include_folder: bool = False) int [source]#
Count number of objects under prefix.
- Parameters:
s3_client –
boto3.session.Session().client("s3")
objectbucket – S3 bucket name
prefix – The s3 prefix (logic directory) you want to calculate
include_folder – Default False, whether counting the hard folder (an empty “/” object).
- Returns:
Number of objects under prefix.
New in version 2.0.1.