How to make file creation an atomic operation?

Write data to a temporary file and when data has been successfully written, rename the file to the correct destination file e.g

f = open(tmpFile, 'w')f.write(text)# make sure that all data is on disk# see http://stackoverflow.com/questions/7433057/is-rename-without-fsync-safef.flush()os.fsync(f.fileno()) f.close()os.rename(tmpFile, myFile)

According to doc http://docs.python.org/library/os.html#os.rename

If successful, the renaming will be an atomic operation (this is a POSIX requirement). On Windows, if dst already exists, OSError will be raised even if it is a file; there may be no way to implement an atomic rename when dst names an existing file

also

The operation may fail on some Unix flavors if src and dst are on different filesystems.

Note:

• It may not be atomic operation if src and dest locations are not on same filesystem

• os.fsync step may be skipped if performance/responsiveness is more important than the data integrity in cases like power failure, system crash etc

A simple snippet that implements atomic writing using Python tempfile.

with open_atomic('test.txt', 'w') as f:    f.write("huzza")

or even reading and writing to and from the same file:

with open('test.txt', 'r') as src:    with open_atomic('test.txt', 'w') as dst:        for line in src:            dst.write(line)

using two simple context managers

import osimport tempfile as tmpfrom contextlib import contextmanager@contextmanagerdef tempfile(suffix='', dir=None):    """ Context for temporary file.    Will find a free temporary filename upon entering    and will try to delete the file on leaving, even in case of an exception.    Parameters    ----------    suffix : string        optional file suffix    dir : string        optional directory to save temporary file in    """    tf = tmp.NamedTemporaryFile(delete=False, suffix=suffix, dir=dir)    tf.file.close()    try:        yield tf.name    finally:        try:            os.remove(tf.name)        except OSError as e:            if e.errno == 2:                pass            else:                raise@contextmanagerdef open_atomic(filepath, *args, **kwargs):    """ Open temporary file object that atomically moves to destination upon    exiting.    Allows reading and writing to and from the same filename.    The file will not be moved to destination in case of an exception.    Parameters    ----------    filepath : string        the file path to be opened    fsync : bool        whether to force write the file to disk    *args : mixed        Any valid arguments for :code:`open`    **kwargs : mixed        Any valid keyword arguments for :code:`open`    """    fsync = kwargs.get('fsync', False)    with tempfile(dir=os.path.dirname(os.path.abspath(filepath))) as tmppath:        with open(tmppath, *args, **kwargs) as file:            try:                yield file            finally:                if fsync:                    file.flush()                    os.fsync(file.fileno())        os.rename(tmppath, filepath)

Since it is very easy to mess up with the details, I recommend using a tiny library for that. The advantage of a library is that it takes care all these nitty-gritty details, and is being reviewed and improved by a community.

One such library is python-atomicwrites by untitaker which even has proper Windows support:

• https://github.com/untitaker/python-atomicwrites
• https://pypi.org/project/atomicwrites/

From the README:

from atomicwrites import atomic_writewith atomic_write('foo.txt', overwrite=True) as f:    f.write('Hello world.')    # "foo.txt" doesn't exist yet.# Now it does.

Installation via PIP:

pip install atomicwrites