Pure S3 Path Manipulation

Sanhe

Apr 20, 2023

15 min read

What is Pure S3 Path

A Pure S3 Path is a Python object that represents an AWS S3 bucket, object, or folder. However, it’s important to note that a Pure S3 Path object does not make any calls to the AWS API, nor does it imply the existence of the corresponding S3 object. Rather, it’s a lightweight abstraction that allows you to work with S3 paths in a Pythonic, object-oriented manner without incurring any network overhead.

from s3pathlib import S3Path

s3path = S3Path("s3://bucket/folder/file.txt")
print(s3path)

S3Path('s3://bucket/folder/file.txt')

Construct an S3 Path object in Python

With s3pathlib, there are numerous ways to create an S3Path object.

From bucket, and key parts

In a file system, you typically use a file path like C:\\Users\username\file.txt on Windows or /Users/username/file.txt on a POSIX system. It’s similarly intuitive to construct an S3 Path from a string.

# construct from bucket, key parts
s3path = S3Path("bucket", "folder", "file.txt")
s3path

S3Path('s3://bucket/folder/file.txt')

# construct from full path also works
s3path = S3Path("bucket/folder/file.txt")
s3path

S3Path('s3://bucket/folder/file.txt')

S3 uses / as a delimiter to organize and browse your keys hierarchically. With s3pathlib, the delimiter is handled intelligently.

s3path = S3Path("bucket", "/folder/", "/file.txt")
s3path

S3Path('s3://bucket/folder/file.txt')

From S3 URI

S3 URI is the unique resource identifier within the context of the S3 protocol. They follow this naming convention: s3://bucket-name/key-name. You can create an S3 Path from S3 URI.

s3path = S3Path("s3://bucket/folder/file.txt")
s3path

S3Path('s3://bucket/folder/file.txt')

You can also use the from_s3_uri() factory method to create an S3Path object from an URI.

s3path = S3Path.from_s3_uri("s3://bucket/folder/file.txt")
s3path

S3Path('s3://bucket/folder/file.txt')

From S3 ARN

S3 ARN is the Amazon Resource Name of an S3 resources. They follow this naming convention: arn:aws:s3:::bucket_name/key_name. You can create an S3 Path from S3 ARN.

s3path = S3Path("arn:aws:s3:::bucket/folder/file.txt")
s3path

S3Path('s3://bucket/folder/file.txt')

You can use the from_s3_arn() factory method to create an S3Path object from an ARN.

s3path = S3Path.from_s3_arn("arn:aws:s3:::bucket/folder/file.txt")
s3path

S3Path('s3://bucket/folder/file.txt')

S3 Path Types

S3 Path is a logical concept that can represent different types of AWS S3 concepts. Here is the list of S3 Path types:

1. 📜 Classic S3 object: represents an S3 object, such as s3://bucket/folder/file.txt.

2. 📁 Logical S3 directory: represents an S3 directory, such as s3://bucket/folder/.

3. 🪣 S3 bucket: represents an S3 bucket, such as s3://bucket/

4. Void Path: denotes the absence of any bucket or key, essentially representing a blank slate, no bucket, no key, no nothing.

5. Relative Path: represents a path relative to another S3 Path. For example, the relative path from s3://bucket/folder/file.txt to s3://bucket/ is simply folder/file.txt. A relative path can be joined with another S3 Path to create a new S3 Path. Importantly, any concrete path joined with a void path will result in the original concrete path.

6. Concrete Path: represents an S3 Path that refers to a concrete object in the S3 storage system. This includes classic S3 object paths, logical S3 directory paths, and S3 bucket paths. Any concrete path joined with a relative path will result in another concrete path.

Classic S3 object

Similar to a file on your local laptop, an S3 object stores your data. At any given moment, it could be just a pointer, and the object doesn’t have to exist in S3.

s3path = S3Path("s3://bucket/folder/file.txt")
s3path

S3Path('s3://bucket/folder/file.txt')

There are some “is XYZ test” methods can tell you whether the S3 Path object is a “file”, “directory”, “bucket”, “void path”, “relative path”.

• is_dir():

• is_file():

• is_bucket():

• is_void():

• is_relpath():

s3path.is_file()

True

s3path.is_dir()

False

s3path.is_bucket()

False

s3path.is_void()

False

s3path.is_relpath()

False

Logical S3 Directory

Since AWS S3 is an object storage system, not a file system, directories are only a logical concept in AWS S3. AWS uses / as the path delimiter in S3 keys. There are two types of directories in AWS S3:

• Hard directory: When you create a folder in the S3 console, it creates a special object without any content (an empty string) with the / character at the end of the key. You can see the folder as an object in the list_objects API response.

• Soft directory: This type of directory does not actually exist; it is a virtual concept used to help organize your objects in a folder. For example, if you have an S3 object like s3://bucket/folder/file.txt, then the s3://bucket/folder/ path is a soft folder. Although you can see it in the S3 console, it does not actually exist.

You can create a S3 directory from string, URI, ARN.

s3dir = S3Path("bucket", "folder/")
s3dir

S3Path('s3://bucket/folder/')

s3dir = S3Path("s3://bucket/folder/")
s3dir

S3Path('s3://bucket/folder/')

s3dir = S3Path("arn:aws:s3:::bucket/folder/")
s3dir

S3Path('s3://bucket/folder/')

Sometimes, you may be concerned that you forgot to append a trailing slash / to the end of a path to indicate that it refers to a directory. In this case, you can use the to_dir() method to ensure that the path refers to a directory.

s3dir = S3Path("bucket", "folder").to_dir()
s3dir

S3Path('s3://bucket/folder/')

You can also use “is XYZ test” methods on S3 directory too.

s3dir.is_dir()

True

s3dir.is_file()

False

s3dir.is_bucket()

False

s3dir.is_void()

False

s3dir.is_relpath()

False

S3 Bucket

An S3 bucket is a special type of directory that can be thought of as a “root” directory without a key. In other words, it represents the top-level directory of the bucket, and it is both a bucket and a directory in its own right.

s3bkt = S3Path("bucket")
s3bkt

S3Path('s3://bucket/')

s3bkt.is_bucket()

True

s3bkt.is_dir()

True

s3bkt.is_file()

False

You can use root() method to get the S3 bucket of any S3 object or directory.

s3bkt = S3Path("bucket/folder/file.txt").root
s3bkt

S3Path('s3://bucket/')

Void Path

While Void path should not be used in your application, it can serve as an indicator that something is wrong if you accidentally attempt to use a Void path to perform an S3 API operation.

s3path = S3Path()
s3path

S3VoidPath()

s3path.is_void()

True

s3path.is_file()

False

s3path.is_dir()

False

s3path.is_bucket()

False

s3path.is_relpath()

True

Relative Path

Relative paths are very useful for S3 Path calculations. For example, if you want to move all objects in folder A to another folder B, you can use the relative path from each object C to A to calculate the target location in B. Specifically, the target location for each object can be found by joining the relative path from C to A with the folder path B. In other words, the formula for the target path is: Target = B + (C - A).

Even though you can, but I don’t recommend you to construct a relative path manually. You should use path calculation method relative_to() to create it.

# The correct way
s3relpath = S3Path("s3://bucket/folder/file.txt").relative_to(S3Path("s3://bucket/folder"))
s3relpath

S3RelPath('file.txt')

# The manual way (NOT RECOMMENDED)
s3relpath = S3Path.make_relpath("file.txt")
s3relpath

S3RelPath('file.txt')

s3path = S3Path("s3://another-bucket/another-folder").to_dir().joinpath(s3relpath)
s3path

S3Path('s3://another-bucket/another-folder/file.txt')

S3 Path Variable Naming Convention

I recommend the following variable naming convention for different types of S3 Path. So when you read the code, you can easily tell what to expect.

• s3path_xyz: Classic S3 object

• s3dir_xyz: Logical S3 directory

• s3bkt_xyz: S3 bucket

• s3void_xyz: Void Path

• s3relpath_xyz: Relative Path

S3 Path Attributes

S3 Path object has a lot of useful attributes (even though they are property method).

• bucket: Return the bucket name as a string.

• key: return the S3 key as a string.

• parts: Provides sequence-like access to the components in the filesystem path.

• uri: Return the AWS S3 URI.

• arn: Return an AWS S3 Resource ARN.

• console_url: Return an url that can inspect the object, directory details in AWS Console.

• us_gov_cloud_console_url: Return a Gov Cloud url that can inspect the object, directory details in AWS Console.

# create an instance
s3path = S3Path("bucket", "folder", "file.txt")

s3path.bucket

'bucket'

s3path.key

'folder/file.txt'

s3path.parts

['folder', 'file.txt']

The S3Path class is both immutable and hashable. These attributes don’t require any AWS boto3 API calls and are generally available. Because S3Path objects are immutable, you cannot change the value of these attributes once they have been created.

try:
    s3path.bucket = "new-bucket"
except Exception as e:
    print(e)

can't set attribute S3Path.bucket

s3path.uri

's3://bucket/folder/file.txt'

s3path.arn

'arn:aws:s3:::bucket/folder/file.txt'

s3path.console_url

'https://console.aws.amazon.com/s3/object/bucket?prefix=folder/file.txt'

s3path.us_gov_cloud_console_url

'https://console.amazonaws-us-gov.com/s3/object/bucket?prefix=folder/file.txt'

Logically, a S3Path is also a file system like object. So it should have those file system concepts too:

• basename: the file name with extension.

• fname: file name without file extension.

• ext: file extension, if available

• dirname: the basename of the parent directory

• abspath: the absolute path is the full path from the root drive. You can think of S3 bucket as the root drive.

• parent: the parent directory S3 Path

• dirpath: the absolute path of the parent directory. It is equal to s3path.parent.abspath

s3path.basename

'file.txt'

s3path.fname

'file'

s3path.ext

'.txt'

s3path.dirname

'folder'

s3path.abspath

'/folder/file.txt'

s3path.parent

S3Path('s3://bucket/folder/')

s3path.dirpath

'/folder/'

S3 Path Methods

Comparison

Because every S3Path object corresponds to an S3 URI (except for relative paths), it’s often useful to compare these URIs. Therefore, the comparison operator is implemented for S3Path, allowing you to compare one S3Path to another.

S3Path("bucket/file.txt") == S3Path("bucket/file.txt")

True

S3Path("bucket") == S3Path("bucket")

True

S3Path("bucket1") == S3Path("bucket2")

False

S3Path("bucket1") < S3Path("bucket2")

True

S3Path("bucket1") <= S3Path("bucket2")

True

# right one is a prefix of the left one
S3Path("bucket/a/1.txt") > S3Path("bucket/a/")

True

S3Path("bucket/a/1.txt") < S3Path("bucket/a/2.txt")

True

Hash

S3Path is hashable. You can use set data structure to deduplicate them.

p1 = S3Path("bucket", "1.txt")
p2 = S3Path("bucket", "2.txt")
p3 = S3Path("bucket", "3.txt")
set1 = {p1, p2}
set2 = {p2, p3}

# union
set1.union(set2)

{S3Path('s3://bucket/1.txt'),
 S3Path('s3://bucket/2.txt'),
 S3Path('s3://bucket/3.txt')}

# intersection
set1.intersection(set2)

{S3Path('s3://bucket/2.txt')}

# difference
set1.difference(set2)

{S3Path('s3://bucket/1.txt')}

Mutate the immutable S3Path

It’s common to modify existing S3Path objects. However, since S3Path is immutable by design, it cannot be directly edited. Nonetheless, there are numerous utility methods available that enable you to manipulate S3Path objects in various ways.

• copy(): Create a copy of an S3Path object that logically equals to this one, but is actually different identity in memory. Also, the cache data are cleared.

• change(): Create a new S3Path by replacing part of the attributes.

• joinpath(): join with other path parts or relative paths to form another S3Path.

• parent: travel back to the the parent directory S3Path.

Copy

s3path1 = S3Path("bucket", "folder", "file.txt")
s3path2 = s3path1.copy()
s3path2

S3Path('s3://bucket/folder/file.txt')

s3path1 == s3path2

True

s3path1 is s3path2

False

Change

s3path = S3Path("bkt", "a", "b", "c.jpg")

# only change the bucket
s3path.change(new_bucket="new-bkt")

S3Path('s3://new-bkt/a/b/c.jpg')

# only change the absolute path
s3path.change(new_abspath="x/y/z.png")

S3Path('s3://bkt/x/y/z.png')

# only change the file extention
s3path.change(new_ext=".png")

S3Path('s3://bkt/a/b/c.png')

# only change the file name
s3path.change(new_fname="ddd")

S3Path('s3://bkt/a/b/ddd.jpg')

# only change the base name (file name + file extension)
s3path_new = s3path.change(new_basename="ddd.png")
s3path_new

S3Path('s3://bkt/a/b/ddd.png')

s3path_new.is_file()

True

# only change the base name, but this time it becomes a folder
s3path_new = s3path.change(new_basename="ddd/")
s3path_new

S3Path('s3://bkt/a/b/ddd/')

s3path_new.is_dir()

True

# only change the dir name
s3path.change(new_dirname="ddd/")

S3Path('s3://bkt/a/ddd/c.jpg')

# only change the dir name
s3path.change(new_dirname="ddd")

S3Path('s3://bkt/a/ddd/c.jpg')

s3path.change(new_dirpath="xxx/yyy/")

S3Path('s3://bkt/xxx/yyy/c.jpg')

Join

S3Path.joinpath is a very powerful method.

s3path1 = S3Path("bucket", "folder", "subfolder", "file.txt")
s3path1

S3Path('s3://bucket/folder/subfolder/file.txt')

s3path2 = s3path1.parent
s3path2

S3Path('s3://bucket/folder/subfolder/')

relpath1 = s3path1.relative_to(s3path2)
relpath1

S3RelPath('file.txt')

# join concrete path with a relative path
s3path2.joinpath(relpath1)

S3Path('s3://bucket/folder/subfolder/file.txt')

s3path3 = s3path2.parent
s3path3

S3Path('s3://bucket/folder/')

relpath2 = s3path2.relative_to(s3path3)
relpath2

S3RelPath('subfolder/')

s3path3.joinpath(relpath2, relpath1)

S3Path('s3://bucket/folder/subfolder/file.txt')

s3path3.joinpath("subfolder", "file.txt")

S3Path('s3://bucket/folder/subfolder/file.txt')

# it's OK if you mess up with the "/"
s3path3.joinpath("/subfolder/", "/file.txt")

S3Path('s3://bucket/folder/subfolder/file.txt')

The / operator provide a syntax sugar for joinpath method

s3path = S3Path("bucket")
s3path / "file.txt"

S3Path('s3://bucket/file.txt')

s3path / "folder" / "file.txt"

S3Path('s3://bucket/folder/file.txt')

Calculate Relative Path

The relative_to() method is used to calculate the relative path between two paths. The syntax for this method is s3path_from.relative_to(s3path_to). Note that the s3path_to argument must be a shorter path than the s3path_from argument in order for the method to work correctly.

S3Path("bucket", "a/b/c").relative_to(S3Path("bucket", "a")).parts

['b', 'c']

S3Path("bucket", "a").relative_to(S3Path("bucket", "a")).parts

[]

# this won't work
try:
    S3Path("bucket", "a").relative_to(S3Path("bucket", "a/b/c")).parts
except Exception as e:
    print(e)

s3://bucket/a does not start with s3://bucket/a/b/c

The - operator override provide a syntax sugar for relative_to method.

(S3Path("bucket", "a/b/c") - S3Path("bucket", "a")).parts

['b', 'c']

What’s Next

Now that we have established the basics of working with s3pathlib, let’s explore how to use it to interact with the AWS S3 API.