What are the Web.Debug.config and Web.Release.Config files for?
It's the new Web.config transformation feature of Visual Studio 2010. More information here.
Edit:
Are these files used to specify debug and release specific settings, so you don't clutter up the main web.config?
It isn't limited to three files, you could (in theory) have as many files as you have environments. The "top level" Web.config provides a template of your web config. The files under it provide replacement values specific to that environment (like if you have different connection strings for local/stage/test/whatever).
Does it even make sense to place a connection string in the root web.config file if I have have a local and remote one in the debug and release web.configs respectively.
It would only make sense if it wasn't going to change between environments. Sounds like in your case it does so, in your case no, it would not make sense to leave it in the Web.config.
These are Web.config transformations files. From ASP.NET Web Deployment using Visual Studio: Web.config File Transformations:
There are two ways to automate the process of changing Web.config file settings: Web.config transformations and Web Deploy parameters. A Web.config transformation file contains XML markup that specifies how to change the Web.config file when it is deployed. You can specify different changes for specific build configurations and for specific publish profiles. The default build configurations are Debug and Release, and you can create custom build configurations. A publish profile typically corresponds to a destination environment.
In case anyone is interested, here is something I wrote up to have a dynamic connection string per environment. I wanted to deploy the code to any environment (Dev, Test, Pre-Prod, Prod...) without having to worry about changing connection strings. I couldn't really find a good way to do this with Asp.Net MVC 4, so I came up with my own way to rely on a properties file per environment.
There may be a better solution, I come from a Wicket/Java background and recently started developing with MVC 4 so, it's possible a better solution exists. But here is a link to my question and answer for a dynamic connection string: