Download Large file from server using REST template Java Spring MVC

Here is how I do it. Based on hints from this Spring Jira issue.

RestTemplate restTemplate // = ...;// Optional Accept headerRequestCallback requestCallback = request -> request.getHeaders()        .setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));// Streams the response instead of loading it all in memoryResponseExtractor<Void> responseExtractor = response -> {    // Here I write the response to a file but do what you like    Path path = Paths.get("some/path");    Files.copy(response.getBody(), path);    return null;};restTemplate.execute(URI.create("www.something.com"), HttpMethod.GET, requestCallback, responseExtractor);

From the aforementioned Jira issue:

Note that you cannot simply return the InputStream from the extractor, because by the time the execute method returns, the underlying connection and stream are already closed.

Update for Spring 5

Spring 5 introduced the WebClient class which allows asynchronous (e.g. non-blocking) http requests. From the doc:

By comparison to the RestTemplate, the WebClient is:

• non-blocking, reactive, and supports higher concurrency with less hardware resources.
• provides a functional API that takes advantage of Java 8 lambdas.
• supports both synchronous and asynchronous scenarios.
• supports streaming up or down from a server.

To get WebClient in Spring Boot, you need this dependency:

<dependency>    <groupId>org.springframework.boot</groupId>    <artifactId>spring-boot-starter-webflux</artifactId></dependency>

For the moment, I'm sticking with RestTemplate because I don't want to pull in another dependency only to get access to WebClient.

As @bernie mentioned you can use WebClient to achieve this:

public Flux<DataBuffer> downloadFileUrl( ) throws IOException {    WebClient webClient = WebClient.create();    // Request service to get file data    return Flux<DataBuffer> fileDataStream = webClient.get()            .uri( this.fileUrl )            .accept( MediaType.APPLICATION_OCTET_STREAM )            .retrieve()            .bodyToFlux( DataBuffer.class );}@GetMapping( produces = MediaType.APPLICATION_OCTET_STREAM_VALUE )public void downloadFile( HttpServletResponse response ) throws IOException{    Flux<DataBuffer> dataStream = this.downloadFileUrl( );    // Streams the stream from response instead of loading it all in memory    DataBufferUtils.write( dataStream, response.getOutputStream() )            .map( DataBufferUtils::release )            .blockLast();}

You can still use WebClient even if you don't have Reactive Server stack - Rossen Stoyanchev (a member of Spring Framework team) explains it quite well in the Guide to "Reactive" for Spring MVC Developers presentation. During this presentation, Rossen Stoyanchev mentioned that they thought about deprecating RestTemplate, but they have decided to postpone it after all, but it may still happen in the future!

The main disadvantage of using WebClient so far it's a quite steep learning curve (reactive programming), but I think there is no way to avoid in the future, so better to take a look on it sooner than latter.

This prevents loading the entire request into memory.

SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();requestFactory.setBufferRequestBody(false);RestTemplate rest = new RestTemplate(requestFactory);

For java.lang.OutOfMemoryError: Java heap space can be solved adding more memory to the JVM:

-Xmxn Specifies the maximum size, in bytes, of the memory allocation pool. This value must a multiple of 1024 greater than 2 MB. Append the letter k or K to indicate kilobytes, or m or M to indicate megabytes. The default value is chosen at runtime based on system configuration.

For server deployments, -Xms and -Xmx are often set to the same value. See Garbage Collector Ergonomics at http://docs.oracle.com/javase/7/docs/technotes/guides/vm/gc-ergonomics.html

Examples:

-Xmx83886080
-Xmx81920k
-Xmx80m

Probably the problem you have is not strictly related to the request you are trying to execute (download large file) but the memory allocated for the process is not enough.