User-Agent affects Dropbox shared links download
Dropbox may offer the raw file content via a redirect like this for some User-Agents, e.g., for curl where it's unlikely the user wants the HTML of the preview page. This isn't documented or guaranteed though, so I wouldn't recommend relying on it.
There are some documented URL parameters you can use to manage the behavior of shared links though:
I ran into the same problem trying to pull a file from Dropbox using Java. It worked perfectly for curl, but failed for Java (the HTML page was returned instead). Setting the User-Agent header in my Java app to curl/7.54.0 worked, but that seemed like an awful hack.
The link in Greg's answer pointed me to the solution:
To force a browser to download the contents of a link rather than display it, you can use dl=1 as a query parameter in your URL. For example:
So I replaced the dl=0 in the URL with dl=1, and everything works perfectly -- no need to change the User-Agent header. There are still 301 and 302 redirects to follow, but that's not a problem for java.net.HttpURLConnection and most other libraries.