Efficient REST Endpoint in Spring Boot for Streaming Multiple Files to an Instant ZIP Download (Java) – Paweł Woroniecki

In the world of web development, one common task is to provide users with the ability to download multiple files as a single ZIP archive. This can be particularly useful when dealing with large datasets or when users need to download a collection of related documents. In this article, I will focus on the backend part of the process, specifically the implementation of a REST endpoint in a Spring Boot application to efficiently stream and generate ZIP archives on the fly.

Introduction

The traditional approach to generating ZIP files for download involves first collecting all the files, compressing them, and then making the ZIP file available for download. However, this can lead to long waiting times, especially when dealing with a large number of files or large file sizes. To enhance user experience, I will explore a more efficient method that starts the download process as soon as possible and does not require users to wait until all files are collected and compressed.

Implementing the REST Endpoint

Let’s dive into the implementation details of our REST endpoint. I will utilize Spring Boot to create an endpoint that streams and generates ZIP archives on the fly.

@RestController
public class LargeFilesController {

  // Links retrieved from https://testfile.org/
  private static final String LARGE_FILE_1GB_LINK = "https://bit.ly/1GB-testfile";
  private static final String LARGE_FILE_5GB_LINK = "https://bit.ly/5GB-TESTFILE-ORG";

  private static final List<String> ALL_FILES_TO_DOWNLOAD = List.of(
      LARGE_FILE_1GB_LINK,
      LARGE_FILE_5GB_LINK
  );

  @GetMapping("/download-large-files")
  public void downloadAndZipLargeFiles(HttpServletResponse response) throws IOException {
    response.setContentType("application/zip");
    response.setHeader("Content-Disposition", "attachment; filename=large-files.zip");

    try (var zipOut = new ZipOutputStream(response.getOutputStream())) {
      for (var fileLink : ALL_FILES_TO_DOWNLOAD) {
        try (var inputStream = new URL(fileLink).openStream()) {

          var filename = fileLink.substring(fileLink.lastIndexOf("/") + 1);
          zipOut.putNextEntry(new ZipEntry(filename));

          byte[] buffer = new byte[1024];
          int len;
          while ((len = inputStream.read(buffer)) > 0) {
            zipOut.write(buffer, 0, len);
          }
        }

        zipOut.closeEntry();
      }
    }
  }
}

In the code above, I define REST Controller in Spring Boot called LargeFilesController with an endpoint GET /download-large-files. When a user accesses this endpoint, it triggers the creation of a ZIP archive and begins streaming it to the user’s browser so the download process is started immediately.

Let me analyze the code step by step:

The first step in creating the REST endpoint is to configure the HTTP response. I specify that the response will contain a ZIP archive using Content-Header header and set the Content-Disposition header to prompt the user’s browser to download the file with the name large-files.zip.
In this example, I have a list of file links that represent the large files I want to include in the ZIP archive. These links can point to any online resources, and they are retrieved from external sources. In this specific case, there are two files large enough to observe the streaming process.
To create a ZIP archive on the fly, I open the ZipOutputStream. This stream allows me to add entries (files) to the ZIP file as we go, without the need to store all the files in memory or on the server.
Then I iterate through the list of file links and:
1. Create a ZipEntry for each file. The ZipEntry represents an individual file within the ZIP archive.
2. Read the content of the file from its link and write it to the ZipOutputStream. In the provided code, a buffer is used to efficiently transfer data from the input stream to the ZIP output stream. So this code reads the content of external files from the Internet and writes the content to the ZIP output stream in chunks, ensuring efficient data transfer.
3. Close the current ZipEntry to signal that the current file has been processed and fully written into the ZIP archive. This is crucial
The last thing is to closed the ZipOutputStream.

All streams are opened and closed using try-with-resources statement so there is no explicit close operation in the code but it is performed automatically.

File Size Considerations

While the implementation described in the previous section allows for efficient streaming and ZIP generation, it does have one significant drawback: it does not specify the file size to the user. This limitation arises because the Content-Length header, which provides the file size information, is not set in the response. This can lead to user dissatisfaction and confusion, as the download progress and the total file size are not visible. To address this issue, let’s look at the challenges of dynamically determining file size.

The Challenge of Dynamic File Size Determination

The primary challenge in providing the file size to the user is that, in the streaming approach, the final size of the ZIP file is not known until it is fully generated. Since the ZIP file is not generated before the download process starts, the Content-Length header cannot be set initially. This can leave users in the dark about the download’s progress.

One initial idea to overcome this challenge was to immediately run two processes in parallel once the endpoint is triggered:

one for downloading all files and creating the ZIP file on the server’s disk (the first process),
the second for streaming the ZIP file to the user (the second process).

The concept was to update the Content-Length header once the ZIP file was fully generated on the server, so users could receive information about the total file size during the streaming process.

However, here is where the challenge deepens: the Content-Length header can only be set before the streaming process begins. Once the streaming process starts and data begins flowing to the user, it becomes impossible to update the Content-Length header. Any attempts to modify or set this header during the process are ignored, and the initial header value remains intact. It is related to the fact that all HTTP headers are sent only at the beginning of the response and then only the content of the streamed file can be sent. No HTTP headers can be sent in the middle of the file content transmission.

Anyway, I was able to find two solutions that seemed to be feasible. Both of them base on the described idea of this parallel process and sending the total filesize to the user once the ZIP file is ready on the server’s side. Hence, both of them introduce complexity and the necessity to store the file on the server’s storage while the basic streaming solution described above had no such requirement.

Solution 1: Client-Side Customization

While setting the Content-Length header dynamically during the streaming process on the server side may be a challenge, there’s an alternative approach that can enhance the user experience on the client side. This approach involves utilizing custom file download processes and establishing communication between the server and the web client to provide real-time download progress updates.

Server-Sent Events (SSE) and Websockets

To address the issue of providing file size information and download progress to the user, you can leverage technologies like Server-Sent Events (SSE) or Websockets. These technologies allow the server and the client to communicate asynchronously, making it possible to share real-time information.

Server-Sent Events (SSE): SSE is a simple and efficient technology that enables the server to push data to the web client over a single HTTP connection. In the context of file downloads, the server can send periodic updates about the download progress, including the file size, to the web client. The client can then update the user interface with this information.
Websockets: Websockets provide a bidirectional communication channel between the server and the client. They are suitable for more complex scenarios where real-time interactions are needed. With Websockets, you can establish a continuous connection to send and receive data, including download progress updates, as the download happens.

Implementation Considerations

To implement this approach effectively, you’ll need to coordinate both the server and the web client. Here’s a high-level overview of the steps involved:

On the server side, implement logic to calculate and determine the total file size as the files are being processed and zipped. It can be implemented in the way described above – by creating a final ZIP file on the server’s disk and getting its size. Send this file size information once it is available to the connected web client using SSE or Websockets.
On the web client side (typically in JavaScript or a frontend framework like Angular), establish the connection to the SSE endpoint or Websockets provided by the server.
Inform the user about the download progress, including the total file size and the amount of data received so far.

By adopting this approach, you can strike a balance between efficient, immediate downloads and user-friendliness. Users can initiate the download process promptly, while still receiving real-time updates on the download’s progress, including the total file size. This approach empowers web clients to handle download customization and enhances the overall user experience.

Remember that the choice of SSE or Websockets depends on the specific needs of your application and the level of real-time interaction required between the server and the web client.

The drawback is that you need control over the client’s side to implement this approach. If the only thing you provide is the endpoint, then this solution cannot be applied or you must inform your clients about the way they can receive the filesize during the download process. Yet another drawback is that is it a far more complex solution than setting the Content-Length HTTP header and this one is not compatible with the standard download process handled by default by browsers.

Solution 2: Chunked Transfer

Yet another idea I have is to utilize Chunked Transfer. It could work in the following way:

Stream Small Chunks Initially: Start by streaming small, manageable chunks of the ZIP file to the client as they become available. This can begin as soon as the initial files are zipped. This allows the client to start receiving data without waiting for the entire ZIP file to be generated.
Parallel ZIP File Generation: While streaming is ongoing, run a parallel process on the server that continues to download all the files, create the ZIP file, and store it on the server’s disk. This process should not interfere with the ongoing streaming process for the client. It is what I already mentioned above in my initial idea on how to fix the filesize issues.
Stream the Remaining File: Once the ZIP file is fully ready on the server and all the initial chunks have been streamed, send a single large chunk to the client containing the remaining content. Set the Content-Length header to inform the client about the total file size, which is calculated based on the bytes already sent and the total file size which can be retrieved now from the generated ZIP file.
Client Receives the Final Chunk: The client receives the final chunk, and the download is complete. The client can display the file size and download progress based on the total file size provided in the Content-Length header.

This approach effectively balances streaming efficiency and user-friendliness by providing download progress information, including file size, to the user. It combines the benefits of immediate access to data with visibility into the download progress.

It is also compatible with the standard file download process handled by default by browsers.

Conclusion

By implementing a REST endpoint that efficiently streams and generates ZIP archives on the fly, you can significantly improve user experience and reduce waiting times. Users can start downloading their files as soon as the process begins, making the download experience faster and more user-friendly. This approach is particularly beneficial when dealing with large files or a substantial number of files. It’s a valuable addition to your web application that can enhance user satisfaction and productivity.

However, addressing file size-related issues can be challenging. The absence of the Content-Length header means users lack visibility into download progress.

To balance efficiency and user-friendliness, I explored leveraging client-side customization using technologies like Server-Sent Events (SSE) or Websockets as well as Chunked Transfers. Both solutions enable real-time download progress updates, including file size information but both of them introduce additional complexity and make the scalability more challenging.

In practice, the choice between streaming efficiency and immediate file size information depends on your specific use case and user expectations. Understanding the trade-offs and being prepared to explore alternative methods is essential for delivering an optimal user experience.

As always, it is also essential to handle edge cases and potential errors to ensure a smooth user experience. It was skipped in this post and code example but it is definitely a must-have in your application.

Full code

Full code (there is much more than presented in this article) can be found in my GitHub’s repository.