Sync

Sync a folder between your machine and S3, or between two S3 folders, with the aws s3 sync command: it only copies the files that are new or changed. The AWS CLI must be installed.

The command runs with the identity of your S3 client (the one of s3_context, or s3_client=...), not with the default profile of the CLI, so it uses the same account as the rest of s3pathlib.

Setup

This cell only prepares the notebook: it connects to the S3 bucket that the s3pathlib CI uses and empties the folder of this chapter, so every run starts from the same state. In your own code you only need the two lines marked below, with your own AWS profile and bucket.

[1]:

import os

import boto3
from s3pathlib.api import S3Path
from s3pathlib.api import s3_context

boto_ses = boto3.Session(profile_name=os.environ["S3PATHLIB_TEST_AWS_PROFILE"])
s3_context.attach_s3_client(boto_ses.client("s3"))  # <-- in your code
aws_account_id = boto_ses.client("sts").get_caller_identity()["Account"]
bucket = f"{aws_account_id}-{boto_ses.region_name}-s3pathlib-test"  # <-- your bucket

# every demo of this chapter writes under this folder; start from empty
s3dir_chapter = S3Path(bucket, "projects/s3pathlib/docs/09-Sync/")
s3dir_chapter.delete()
print(s3dir_chapter.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/09-Sync/

1. Local to S3

A local folder with three files:

[2]:

import tempfile
from pathlib import Path

tmp = tempfile.TemporaryDirectory()
dir_local = Path(tmp.name) / "data"
(dir_local / "logs").mkdir(parents=True)
(dir_local / "a.csv").write_text("a")
(dir_local / "b.json").write_text("b")
(dir_local / "logs" / "1.log").write_text("1")

[2]:

1

sync_from syncs a local folder (or an S3 folder) into this S3 folder. By default it prints the command line; the local path here is a random temporary folder, so verbose=False hides it.

[3]:

s3dir = s3dir_chapter / "data/"
s3dir.sync_from(dir_local, verbose=False)

[4]:

for p in s3dir.iter_objects():
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/09-Sync/data/a.csv
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/09-Sync/data/b.json
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/09-Sync/data/logs/1.log

2. S3 to local

sync_to syncs this S3 folder into a local folder (or an S3 folder):

[5]:

dir_download = Path(tmp.name) / "download"
s3dir.sync_to(dir_download, verbose=False)

[6]:

for path in sorted(dir_download.rglob("*")):
    if path.is_file():
        print(path.relative_to(dir_download))

a.csv
b.json
logs/1.log

3. S3 to S3

S3Path.sync(src, dst) takes the source and the target. Here both are in S3, so the printed command line is shown; it never contains the credentials, which are passed as environment variables.

[7]:

s3dir_backup = s3dir_chapter / "backup/"
S3Path.sync(s3dir, s3dir_backup)

aws s3 sync s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/09-Sync/data/ s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/09-Sync/backup/ --only-show-errors

[8]:

for p in s3dir_backup.iter_objects():
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/09-Sync/backup/a.csv
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/09-Sync/backup/b.json
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/09-Sync/backup/logs/1.log

4. Delete what is gone

By default, sync never deletes. Delete a local file:

[9]:

(dir_local / "b.json").unlink()

delete=True also deletes the files of the target that are not in the source:

[10]:

s3dir.sync_from(dir_local, delete=True, verbose=False)

[11]:

for p in s3dir.iter_objects():
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/09-Sync/data/a.csv
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/09-Sync/data/logs/1.log

5. Include and exclude

exclude and include take a pattern or a list of patterns. Every include is applied after every exclude, so “exclude everything, include *.csv” syncs the CSV files only:

[12]:

s3dir_csv = s3dir_chapter / "csv-only/"
s3dir_csv.sync_from(dir_local, exclude="*", include="*.csv", verbose=False)

[13]:

for p in s3dir_csv.iter_objects():
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/09-Sync/csv-only/a.csv

[14]:

tmp.cleanup()

6. Errors

When the command fails, sync raises S3SyncError, with the exit code, what the CLI wrote to stderr, and the command line. For example, a bucket that does not exist:

[15]:

from s3pathlib.api import exc

missing = S3Path("s3pathlib-test-tmp-this-bucket-does-not-exist", "data/")
try:
    S3Path.sync(s3dir, missing, verbose=False)
except exc.S3SyncError as e:
    error = e
print(f"{error.returncode = }")

error.returncode = 1

The reason is in stderr:

[16]:

"NoSuchBucket" in error.stderr

[16]:

True

Sync works on folders only. A file path raises before the command runs:

[17]:

try:
    s3dir.joinpath("a.csv").sync_to(s3dir_backup)
except exc.S3PathIsNotFolderError as e:
    print(f"{type(e).__name__}: {e}")

S3PathIsNotFolderError: 's3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/09-Sync/data/a.csv' is not a dir!

Common pitfalls

  • The AWS CLI must be installed; without the aws command, sync raises FileNotFoundError.

  • Long syncs and temporary credentials. The CLI gets the credentials of the moment; with an assumed role or SSO, a sync that outlives them fails.

  • The order of include and exclude. All excludes come first, then all includes; “include then exclude again” cannot be written.

  • Folders only. src and dst are folders (or buckets), and at least one of them is in S3.

See also