An appendable compressed archive

I'm using zip -Zb for that (appending text logs incrementally to compressed archive):

• fast append (index is at the end of archive, efficient to update)
• -Zb uses bzip2 compression method instead of deflate. In 2018 this seems safe to use (you'll need a reasonably modern unzip -- note some tools do assume deflate when they see a zip file, so YMMV)
• 7z was a good candidate: compression ratio is vastly better than zip when you compress all files in the same operation. But when you append files one by one to the archive (incremental appending), compression ratio is only marginally better than standard zip, and similar to zip -Zb. So for now I'm sticking with zip -Zb.

To clarify what happens and why having the index at the end is useful for "appendable" archive format, with entries compressed individually:

Before:############## ########### ################# #[foo1.png    ] [foo2.png ] [foo3.png       ] ^                                             |                                         indexAfter:############## ########### ################# ########### #[foo1.png    ] [foo2.png ] [foo3.png       ] [foo4.png ] ^                                                         |                                                 new index

So this is not fopen in append mode, but presumably fopen in write mode, then fseek, then write (that's my mental model of it, someone let me know if this is wrong). I'm not 100% certain that it would be so simple in reality, it might depend on OS and file system (e.g. a file system with snapshots might have a very different opinion about how to deal with small writes at the end of a file… huge "YMMV" here 🤷🏻‍♂️)

It's rather easy to have an appendable archive of compressed files (not same as appendable compressed archive, though).

tar has an option to append files to the end of an archive (Assuming that you have GNU tar)

 -r, --append       append files to the end of an archive

You can gzip the log files before adding to the archive and can continue to update (append) the archive with newer files.

$ ls -lfoo-20130101.logfoo-20130102.logfoo-20130103.log$ gzip foo*$ ls -lfoo-20130101.log.gzfoo-20130102.log.gzfoo-20130103.log.gz$ tar cvf backup.tar foo*gz

Now you have another log file to add to the archive:

$ ls -lfoo-20130104.log$ gzip foo-20130104.log$ tar rvf backup.tar foo-20130104.log$ tar tf backup.tarfoo-20130101.log.gzfoo-20130102.log.gzfoo-20130103.log.gzfoo-20130104.log.gz

If you don't need to use tar, I suggest 7-Zip. It has an 'add' command, which I believe does what you want.

See related SO question: Is there a way to add a folder to existing 7za archive?

Also, the 7-Zip documentation: https://sevenzip.osdn.jp/chm/cmdline/commands/add.htm