Building a Swift framework with references to Objective-C code
It wasn't that complicated, actually... I was just doing some things wrong.
First, bridging headers are not required in that setting: the Google Maps iOS SDK is provided as a regular .framework file, so the development language has no impact on how it can be imported in Swift. Apple clearly mentions it in the documentation: https://developer.apple.com/library/ios/documentation/Swift/Conceptual/BuildingCocoaApps/MixandMatch.html, "Importing external frameworks".
It's as easy as adding the framework to the "Link binary with libraries" section of the project settings. Do not forget to also add depending libraries and frameworks (in GoogleMaps.framework's case, there are quite a few).
Then, in Swift code, the framework classes should be available simply by doing:
import GoogleMapsNo bridging header, no dealing with "non-modular header etc." errors.