Copy, Move and Delete

Copy and move a file or a whole folder, change the metadata of an object while copying it, and delete files and folders. The copies happen inside S3: no data goes through your machine.

Setup

This cell only prepares the notebook: it connects to the S3 bucket that the s3pathlib CI uses and empties the folder of this chapter, so every run starts from the same state. In your own code you only need the two lines marked below, with your own AWS profile and bucket.

import os

import boto3
from s3pathlib.api import S3Path
from s3pathlib.api import s3_context

boto_ses = boto3.Session(profile_name=os.environ["S3PATHLIB_TEST_AWS_PROFILE"])
s3_context.attach_s3_client(boto_ses.client("s3"))  # <-- in your code
aws_account_id = boto_ses.client("sts").get_caller_identity()["Account"]
bucket = f"{aws_account_id}-{boto_ses.region_name}-s3pathlib-test"  # <-- your bucket

# every demo of this chapter writes under this folder; start from empty
s3dir_chapter = S3Path(bucket, "projects/s3pathlib/docs/06-Copy-Move-Delete/")
s3dir_chapter.delete()
print(s3dir_chapter.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/06-Copy-Move-Delete/

1. Copy a file

copy_file(dst) copies one object and returns the new S3Path of the copy.

s3dir = s3dir_chapter / "copy-file/"
src = s3dir / "src.txt"
src.write_text("hello", metadata={"author": "alice"})

dst = s3dir / "dst.txt"
src.copy_file(dst)

S3Path('s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/06-Copy-Move-Delete/copy-file/dst.txt')

The copy has the same content and the same user metadata:

dst.read_text()

'hello'

print(f"{dst.metadata = }")

dst.metadata = {'author': 'alice'}

It never replaces an existing object unless you pass overwrite=True. On AWS the copy is conditional, so an object written by someone else at the same moment is not replaced either.

from s3pathlib.api import exc

try:
    src.copy_file(dst)
except exc.S3FileAlreadyExist as e:
    print(f"{type(e).__name__}: {e}")

S3FileAlreadyExist: S3 object 's3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/06-Copy-Move-Delete/copy-file/dst.txt' already exists!

src.copy_file(dst, overwrite=True)

S3Path('s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/06-Copy-Move-Delete/copy-file/dst.txt')

A missing source raises S3FileNotExist:

missing = s3dir / "missing.txt"
try:
    missing.copy_file(dst, overwrite=True)
except exc.S3FileNotExist as e:
    print(f"{type(e).__name__}: {e}")

S3FileNotExist: S3 object 's3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/06-Copy-Move-Delete/copy-file/missing.txt' does not exist!

The target can be in another bucket, and s3_client=... copies with another client (e.g. another account with access to both buckets). Copying works up to 5 GB per object; for a bigger one S3 raises InvalidRequest, use boto3’s managed s3_client.copy().

2. Change metadata and headers while copying

By default the copy keeps the user metadata, the tags and the headers (content_type, cache_control, …) of the source. Pass any of them to replace only that one. (S3 itself would reset the headers you did not pass; s3pathlib reads them from the source and sends them again.)

src = s3dir / "page.html"
src.write_text("<h1>Hi</h1>", content_type="text/html", metadata={"author": "alice"})

dst = s3dir / "page-copy.html"
src.copy_file(dst, metadata={"author": "bob"}, tags={"copied": "yes"})

S3Path('s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/06-Copy-Move-Delete/copy-file/page-copy.html')

The metadata and the tags are the new ones:

print(f"{dst.metadata = }")

dst.metadata = {'author': 'bob'}

version_id, tags = dst.get_tags()
tags

{'copied': 'yes'}

The content type is kept:

response = dst.head_object()
response["ContentType"]

'text/html'

S3 cannot edit the metadata of an object. Copying the object onto itself with new metadata is how you change it; the content is not uploaded again.

src.copy_file(src, metadata={"author": "carol"}, overwrite=True)
print(f"{src.metadata = }")

src.metadata = {'author': 'carol'}

src.read_text()

'<h1>Hi</h1>'

3. Copy a folder

A folder with two files and an empty sub folder (a folder marker):

s3dir = s3dir_chapter / "copy-dir/"
src_dir = s3dir / "src/"
(src_dir / "a.txt").write_text("a")
(src_dir / "sub/" / "b.txt").write_text("b")
(src_dir / "empty/").mkdir()

copy_dir(dst) copies every object under a folder, keeping the paths relative to the folder, and returns how many it copied: the two files and the marker of empty/.

dst_dir = s3dir / "dst/"
src_dir.copy_dir(dst_dir)

3

for p in dst_dir.iter_objects():
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/06-Copy-Move-Delete/copy-dir/dst/a.txt
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/06-Copy-Move-Delete/copy-dir/dst/sub/b.txt

The empty folder is copied too (iter_objects does not list folder markers, but the folder exists):

(dst_dir / "empty/").exists()

True

With overwrite=False (the default) it checks every target first, and copies nothing if one exists:

try:
    src_dir.copy_dir(dst_dir)
except exc.S3FileAlreadyExist as e:
    print(f"{type(e).__name__}: {e}")

S3FileAlreadyExist: S3 object 's3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/06-Copy-Move-Delete/copy-dir/dst/a.txt' already exists!

The two folders must not overlap: copying a folder into itself would copy objects that are being copied.

try:
    src_dir.copy_dir(src_dir / "backup/")
except ValueError as e:
    print(f"{type(e).__name__}: {e}")

ValueError: 's3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/06-Copy-Move-Delete/copy-dir/src/' and 's3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/06-Copy-Move-Delete/copy-dir/src/backup/' overlap: copying would overwrite source objects before they are copied (and moving would delete the copies); copy to another folder first

copy_to does either: copy_file for a file, copy_dir for a folder, and returns the number of objects copied.

(src_dir / "a.txt").copy_to(s3dir / "a-copy.txt")

1

src_dir.copy_to(s3dir / "dst-2/")

3

4. Move

move_to copies, then deletes exactly the source keys it copied, and returns how many it moved.

s3dir = s3dir_chapter / "move/"
(s3dir / "inbox/" / "1.txt").write_text("1")
(s3dir / "inbox/" / "2.txt").write_text("2")

S3Path('s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/06-Copy-Move-Delete/move/inbox/2.txt')

(s3dir / "inbox/").move_to(s3dir / "archive/")

2

for p in s3dir.iter_objects():
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/06-Copy-Move-Delete/move/archive/1.txt
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/06-Copy-Move-Delete/move/archive/2.txt

Move a single file:

(s3dir / "archive/" / "1.txt").move_to(s3dir / "1.txt")

1

for p in s3dir.iter_objects():
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/06-Copy-Move-Delete/move/1.txt
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/06-Copy-Move-Delete/move/archive/2.txt

Moving is copy plus delete, so it is not atomic: if it fails half way, some objects are in both places.

5. Delete

delete() deletes a file:

s3dir = s3dir_chapter / "delete/"
(s3dir / "a.txt").write_text("a")
(s3dir / "sub/" / "b.txt").write_text("b")
(s3dir / "sub/" / "c.txt").write_text("c")

S3Path('s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/06-Copy-Move-Delete/delete/sub/c.txt')

(s3dir / "a.txt").delete()

S3Path('s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/06-Copy-Move-Delete/delete/a.txt')

for p in s3dir.iter_objects():
    print(p.uri)

s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/06-Copy-Move-Delete/delete/sub/b.txt
s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/06-Copy-Move-Delete/delete/sub/c.txt

Or every object under a folder, folder markers included:

(s3dir / "sub/").delete()
s3dir.exists()

False

Deleting what does not exist is not an error:

(s3dir / "missing.txt").delete()

S3Path('s3://123456789012-us-east-1-s3pathlib-test/projects/s3pathlib/docs/06-Copy-Move-Delete/delete/missing.txt')

delete() returns a new S3Path; in a bucket with versioning, that path is the new delete marker, see Versioning.

Deleting a bucket path deletes everything in the bucket, so it first asks you to type YES (pass skip_prompt=True in scripts). Deleting the bucket itself is delete_bucket(), see Buckets.

Common pitfalls

• Overwrite is off by default. copy_file, copy_dir, copy_to, move_to and upload_* raise S3FileAlreadyExist instead of replacing; pass overwrite=True.

• Moving is copy plus delete. It is not atomic, and on a large folder it takes as long as a copy.

• Objects over 5 GB cannot be copied with these methods; use boto3’s s3_client.copy().

• Delete without versioning is final. Turn on versioning in the bucket if you need to undo deletes.

See also

• Metadata and Tags: what a copied path knows about its object.

• Versioning: copy an older version (version_id), delete markers, permanent deletes.

• Sync: copy only what changed between two folders.