I've decided to add my own answer with the best solution I've found so far. I'll update this if I find better options.

Best solution so far

The best solution I've found for setting up a productive development environment with Docker on OS X is: Boot2Docker + Rsync. With rsync, build times in a Docker container are on par with running the build directly on OSX! Moreover, the file watcher code does not need polling (inotify works since rsync uses normal folders), so hot reload is almost as fast.

There are two ways to set it up: an automated install and a manual install.

Automated install

I've packaged all the steps for setting up Boot2Docker with Rsync into an open source project called docker-osx-dev. The code is a bit rough, but I've been successfully using it for several weeks to easily switch between 3 projects with 3 different tech stacks. Try it out, report bugs, and submit some PRs! Also, see my blog post, A productive development environment with Docker on OS X for more info.

Manual setup

1. Install Boot2Docker: brew install boot2docker.
2. Run Boot2Docker, but with VirtualBox shared folders disabled: boot2docker init && boot2docker start --vbox-share=disable.
3. Run boot2docker shellinit and copy the environment variables it prints out into your ~/.bash_profile file.
4. Install rsync on the Boot2Docker VM: boot2docker ssh "tce-load -wi rsync".
5. Create the base folders you need on the Boot2Docker VM and set permissions correctly for them. For example, if you'll be syncing the /foo/bar folder from OS X, you need to create /foo/bar on the Boot2Docker VM: boot2docker ssh "mkdir -p /foo/bar && chown -R docker /foo/bar".
6. Run rsync to sync the files to the Boot2Docker VM: rsync --archive --rsh="ssh -i $HOME/.ssh/id_boot2docker -o StrictHostKeyChecking=no" /foo/bar docker@dockerhost:/foo. Check the rsync docs for various settings you may want to enable, such as using --exclude .git to exclude the .git folder when syncing.
7. Use a file watcher to keep files in sync. For example, you could use fswatch (brew install fswatch) piped into rsync.
8. At this point, you should be able to use docker run to fire up your Docker container and use the -v flag to mount the folder you're syncing: docker run -v /foo/bar:/src some-docker-image.
9. Update the code on OS X as usual. Changes should propagate very quickly using rsync, the normal file watcher code should pick up the changes as usual (ie, using inotify), and the build should run fast because all the files are "local" to the container.
10. If you need to test a running website, run the boot2docker ip command to find out what IP it's on.

Update: Now that docker for mac is in beta with non-hack functionality, going that route may be a lot more reasonable for local development without a essay's worth of hacks and workarounds.

Don't. I know that's not the answer you are probably hoping for, but take an honest evaluation of the cost/benefit of trying to get local source code + dockerized execution vs just doing local development on OSX.

At some point all the issues, setup effort, and operational pain points MAY be resolved well enough, but as of right now my take on this is it's a net loss.

Issue #1: Mounted volumes on Virtual Box (which use vboxfs) are extremely slow

Wait a while and this will almost certainly improve.

Issue #2: File watching is broken

I'm not sure a fix for this is in the near future. If this type of functionality is key to your development workflow, I would consider this a dealbreaker. It's not worth a major R&D effort when compared to just using rbenv/bundler to manage your jekyll/ruby installs and running them locally on OSX like folks have been doing successfully for the past decade+.

Just like "the cloud" has zero involvement in my local development setup, at the moment, docker is a win for testing/staging/deployment and for running databases and other third party components, but the applications I'm actually coding get run straight on OSX.

Docker for Mac and Windows shall be the definitive way of developing with Docker on OS X (and Windows). A Docker product, the software is an “integrated, easy-to-deploy environment for building, assembling, and shipping applications from Mac or Windows.” It purports to be able to solve the issues presented by the OP. From its March 24, 2016 announcement:

• Faster and more reliable: no more VirtualBox! The Docker engine is running in an Alpine Linux distribution on top of an xhyve Virtual Machine on Mac OS X or on a Hyper-V VM on Windows, and that VM is managed by the Docker application. You don’t need docker-machine to run Docker for Mac and Windows.
• Tools integration: Docker for Mac is a Mac application and Docker for Windows is a Windows application, including a native user interface and auto-update capability. The Docker tool set comes bundled with it: Docker command line, Docker Compose, and Docker Notary command line.
• Volume mounting for your code and data: volume data access works correctly, including file change notifications (on Mac inotify now works seamlessly inside containers for volume mounted directories). This enables edit/test cycles for “in container” development.
• Easy access to running containers on the local host network: Docker for Mac and Windows include a DNS server for containers, and are integrated with the Mac OS X and Windows networking system. On a Mac, Docker can be used even when connected to a very restrictive corporate VPN.
• Docker for Mac was architected from scratch to be able to fit the OS X sandbox security model and we are working closely with Apple to achieve this.