Handling log and configuration files when load balancing apache

I suggest not using NFS for logging as it can be a real performance killer. Instead use rsyslog with remote logging enabled. In you apache2.conf you can setup a LogFormat that includes the VirtualHost name and then pipe the log to rsyslog telling it to write the output to a remote host.

In apache2.conf:

LogFormat "%v %{X-FORWARDED-FOR}i %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combinedCustomLog "|/usr/bin/logger -t apache2 -p local7.info" vhost_combined

In rsyslog.conf on the webserver:

local7.* @<remote host ip>

In rsyslog.conf on the remote host:

local7.*    /var/log/webfrontends.log;precise

As for the Apache configuration files, we use NFS.
apache2.conf is a link to a remote file (different files for different machines if needed) and in apache2.conf we use an Include directive to read specific site configurations (different dirs for different machines if needed)

on the NFS server the NFS exported dir /NFS_EXPORT/etc/apache2/ contains:

 - webserver1_apache2.conf - webserver2_apache2.conf - webserver1_vhosts (dir) - webserver2_vhosts (dir)

Both webserver1_apache2.conf and webserver2_apache2.conf contain Include "/etc/apache2/vhosts"

on WebServer 1

ln -s /NFS_EXPORT/etc/apache2/webserver1_apache2.conf /etc/apache2/apache2.confln -s /NFS_EXPORT/etc/apache2/webserver1_vhosts/ /etc/apache2/vhosts

on WebServer 2

ln -s /NFS_EXPORT/etc/apache2/webserver2_apache2.conf /etc/apache2/apache2.confln -s /NFS_EXPORT/etc/apache2/webserver2_vhosts/ /etc/apache2/vhosts

If all your webservers are the same in terms of hardware specs and serve the same sites/applications then there is no need to differentiate the configs.

Of course you will need a script or some other mechanism to restart apache on all your server once you modify a configuration.Also, upgrading your apache2 software can be tricky unless you have root access to your NFS exports beacause typically your package management system will complain about not being able to modify some configuration file.

NFS will not help you with log files, for exactly the reasons you describe above. You should use syslogd (or some other solution like Splunk) to centralize the logging. It's trivial to include information about what host the log entry comes from, so you can still winnow down to per-host data when troubleshooting.

Configuration files: you need to either centralize them (a "master" copy), or have a way of distributing changes made on any server to all the others. I recommend centralization as the simpler approach. NFS will do the job here, or, as you suggest, a repository from which all hosts are periodically updated. There are a lot of options here, running all the way up to version control (SVN, git, etc) or even configuration servers (Chef, etc).

Please note that moving from a single server to a cluster has many implications. In both cases above (logging, config files), there is potential to introduce single points of failure if done naively. Since you have that already (one server), you're not worse off, but you should try to be aware of and plan for the failure scenarios you may need to respond to.

Use the tool built for the job - Puppet is designed for managing configuration files on multiple servers.. There is an open source tool or you can get their Enterprise version..

puppetlabs.com