R nightmare: Yosemite, R, RStudio, and Homebrew

How do I go about this without blowing up the Mac again?

Don't try to delete your swap file while the computer is running! If you start deleting random system files you're going to have a bad day.

How to install R and RStudio via Homebrew?

You can install R with brew install r. You can install a binary RStudio using homebrew-cask, like brew install Caskroom/cask/rstudio.

Do I need to install XQuartz first?

No, this should no longer be necessary.

What would happen if I install R and RStudio unbrewed?

Nothing bad; brew doctor may report findings but these are not errors and you should not assume they represent problems, though they are helpful to maintainers like me if you seek help trying to diagnose build failures later.

What conflicts between R, RStudio and Homebrew should I recognize?

There are no important conflicts. If you install the compilers from http://r.research.att.com/libs/, which are unnecessary if you're using a brewed R, you may later have trouble installing things like gcc (which provides gfortran) from Homebrew.

Common quirks of having all of Homebrew, R, and RStudio installed?

As of August 2015 the comments below are no longer necessary; packages are now installed to /usr/local/lib/R/x.y/site-library by default and should survive minor R upgrades.

If you are using a brewed R and you would like your installed packages to survive minor R upgrades, you can run mkdir -p ~/Library/R/3.2/library and then R will install packages to that path by default (placing the packages in your home directory and not in the Cellar).

How can I make Homebrew, R, and RStudio live happily ever after without burning down the house?

See above!

Looking back at this issue, considering all the helpful comments and answers, regarding downloading R packages on a Mac with R and Homebrew already installed:

Do it in the R console within the RStudio interface, simply by typing install.packages("name of the package")

Since I had started downloading R packages directly in RStudio, I hadn't had any problems. The system is very happy.