How should I create a REST API using Rails 3.0?

I would propose that the API being in the same project as your website isn't a bad thing as long as the code is DRY*. Like you pointed out, having separate codebases is a challenge because you have to keep them in sync with every feature/bugfix you do. It's easier to maintain if they are in the same place. As long as you keep your code DRY, this method is the clear winner.

I would offer XML and JSON from the controllers with a subdomain handled by Rails's routing engine. When someone has picked up on the pattern of api.site.com/resource.xml and tries to access a resource that isn't there, it's really not a big deal. As long as your API is documented clearly and you fail/error gracefully when they try to access a resource not in your API, it should be just fine. I would try to return a message saying that resource isn't available and a url to your api documentation. This shouldn't be a runtime problem for any API consumers, as this should be part of discovering your API.

Just my $0.02.

*DRY = Don't Repeat Yourself. DRY code means you don't copy-paste or rewrite the same thing for your site and your api; you extract and call from multiple places.

I think the best solution for you is to merge your first two points.

I suggest using JSON instead of XML: the only point in favor of XML is XPath which is useless in returned data. JSON leads to better response time (and more readable data for better debugging ! :p). Plus, most languages can read JSON. For instance, PHP can natively parse JSON to array/object with json_decode, so it's a nice point. ;)

For controllers, you can namespace it but it's not an obligation, maybe it's better in certain cases to avoid fat actions with tons of conditions. With the Rails 3 router, the separation of API calls in a subdomain (api.webapp.com) is trivial).

For the model, sure you should use the same as used in the whole application.

The new rails router syntax is sugar, you will enjoy. Good luck & have fun! :)