When do browsers download sourcemaps?
Source Maps are downloaded and attached to the browser when you open the developer tools. Until then the browser is not aware of the sourceMap.
There is a code reference to the Chrome's Dev tools
this._sourceMapManager.setEnabled(Common.moduleSetting('jsSourceMapsEnabled').get());Short Gist of what happens in the above code is
Once Devtools is attached the modal triggers and enables the sourceMapManager. The sourceMapManager manages the sourceMap in Chrome. Once the SourceMapManager is initialized it looks for the #sourceMapUrl downloads and attaches the map to the debugger.
Update Note: It's not a practice to add sourcemap's to a minified file, unless the sourcemap is served from a private network. This is for various reasons security, obfuscation etc. But this is just my opinion,it varies depending on your actual requirement. For Example: If you have any error tracking tools which needs the code to be attached, then you will have to attach the sourcemap.