Writing a pickle file to an s3 bucket in AWS
Further to you answer, you don't need to convert to csv. pickle.dumps method returns a byte obj. see here: https://docs.python.org/3/library/pickle.html
import boto3import picklebucket='your_bucket_name'key='your_pickle_filename.pkl'pickle_byte_obj = pickle.dumps([var1, var2, ..., varn]) s3_resource = boto3.resource('s3')s3_resource.Object(bucket,key).put(Body=pickle_byte_obj)
I've found the solution, need to call BytesIO into the buffer for pickle files instead of StringIO (which are for CSV files).
import ioimport boto3pickle_buffer = io.BytesIO()s3_resource = boto3.resource('s3')new_df.to_pickle(pickle_buffer)s3_resource.Object(bucket, key).put(Body=pickle_buffer.getvalue())
this worked for me with pandas 0.23.4 and boto3 1.7.80 :
bucket='your_bucket_name'key='your_pickle_filename.pkl'new_df.to_pickle(key)s3_resource.Object(bucket, key).put(Body=open(key, 'rb'))