JavaScript XMLHttpRequest only works with inspector open (Access-Control-Allow-Origin error)
I ran into the same behavior today.
For me, the problem was that on the same web page, the image is first downloaded to display in an img tag. Then, next to the img tag there is a download button, which uses fetch to download the image data, create an object url and force the browser to download the image an a tag with a download attribute (to be able to set a custom file name).
I had set up Access-Control-Allowed-Origin correctly on the s3 bucket where the image was stored, and things work correctly in Firefox, so I was surprised to see it failing randomly in Chrome, and not with the Web inspector open, as you say.
It turns out that when fetching the image with that URL, in Chrome you hit a cached version of the response from when the image was loaded in the img tag, and this response does not include the required Access-Control-Allowed-Origin header.
Adding {cache: 'no-store'} to the fetch request ensures that the cache is not hit, and that you get a fresh Response with the appropriate headers that is then not blocked.