How to save S3 object to a file using boto3

There is a customization that went into Boto3 recently which helps with this (among other things). It is currently exposed on the low-level S3 client, and can be used like this:

s3_client = boto3.client('s3')open('hello.txt').write('Hello, world!')# Upload the file to S3s3_client.upload_file('hello.txt', 'MyBucket', 'hello-remote.txt')# Download the file from S3s3_client.download_file('MyBucket', 'hello-remote.txt', 'hello2.txt')print(open('hello2.txt').read())

These functions will automatically handle reading/writing files as well as doing multipart uploads in parallel for large files.

Note that s3_client.download_file won't create a directory. It can be created as pathlib.Path('/path/to/file.txt').parent.mkdir(parents=True, exist_ok=True).

boto3 now has a nicer interface than the client:

resource = boto3.resource('s3')my_bucket = resource.Bucket('MyBucket')my_bucket.download_file(key, local_filename)

This by itself isn't tremendously better than the client in the accepted answer (although the docs say that it does a better job retrying uploads and downloads on failure) but considering that resources are generally more ergonomic (for example, the s3 bucket and object resources are nicer than the client methods) this does allow you to stay at the resource layer without having to drop down.

Resources generally can be created in the same way as clients, and they take all or most of the same arguments and just forward them to their internal clients.

For those of you who would like to simulate the set_contents_from_string like boto2 methods, you can try

import boto3from cStringIO import StringIOs3c = boto3.client('s3')contents = 'My string to save to S3 object'target_bucket = 'hello-world.by.vor'target_file = 'data/hello.txt'fake_handle = StringIO(contents)# notice if you do fake_handle.read() it reads like a file handles3c.put_object(Bucket=target_bucket, Key=target_file, Body=fake_handle.read())

For Python3:

In python3 both StringIO and cStringIO are gone. Use the StringIO import like:

from io import StringIO

To support both version:

try:   from StringIO import StringIOexcept ImportError:   from io import StringIO