Example App - Sync Two S3 Folder

In this tutorial, you will learn how to create a simple app that can sync files from one folder to another folder in S3 while preserving the same folder structure in real-time.

To get started, you should first configure an S3 put object event trigger that can monitor any changes made to the source S3 folder. After that, you can proceed to create a Lambda function that can handle the event and sync the file to the target S3 folder.

# this is your lambda function code

from s3pathlib import S3Path

s3dir_source = S3Path("s3pathlib/example-app/sync-two-s3-folders/source/")
s3dir_target = S3Path("s3pathlib/example-app/sync-two-s3-folders/target/")

def lambda_handler(event, context):
    # parse s3 put object event
    s3path_source = S3Path(
        event["Records"][0]["s3"]["bucket"]["name"],
        event["Records"][0]["s3"]["object"]["key"],
    )
    # find out the target s3 location
    s3path_target = s3dir_target.joinpath(
        s3path_source.relative_to(s3dir_source)
    )
    # copy data
    s3path_source.copy_to(s3path_target, overwrite=True)

# firstly, we clean up the source and target location
s3dir_source.delete()
s3dir_target.delete()

S3Path('s3://s3pathlib/example-app/sync-two-s3-folders/target/')

# then we create a file in the source location
s3path_source = s3dir_source.joinpath("folder/file.txt")
s3path_source.write_text("hello")

S3Path('s3://s3pathlib/example-app/sync-two-s3-folders/source/folder/file.txt')

# at begin, the s3 target folder doesn't have any file
s3dir_target.count_objects()

0

# we use this code to simulate the s3 put object event
event = {
    "Records": [
        {
            "s3": {
                "bucket": {"name": s3path_source.bucket},
                "object": {"key": s3path_source.key}
            }
        }
    ]
}
lambda_handler(event, None)

# it should have one file now
s3dir_target.iter_objects().all()

[S3Path('s3://s3pathlib/example-app/sync-two-s3-folders/target/folder/file.txt')]