I’m working on a Java application that needs to handle various file types like images and PDFs. The files can be quite large, sometimes reaching 300MB in size. The app runs on a 64-bit Linux server.
Right now I’m using basic file operations but the performance isn’t great:
FileWriter docOutput = new FileWriter("uploads/document.pdf");
docOutput.write(dataBytes);
docOutput.close();
I’ve heard that NIO might be faster for this kind of work. My REST endpoint receives file metadata along with the actual file data as a byte stream. What would be the most efficient way to write these large files to disk? I’m open to using memory-mapped files or channels if they would help with performance.
I’ve dealt with 300MB files in production - FileChannel with direct byte buffers is your best bet. Switched from regular FileOutputStream and got about 40% faster write times. Use transferFrom() or transferTo() when you can since they hit zero-copy operations at the OS level. Memory-mapped files work too but they’re a pain to manage properly, especially cleanup and making sure buffers actually get unmapped. For your REST endpoint, I’d create a FileChannel from FileOutputStream and write your byte stream in chunks using ByteBuffer.allocateDirect(). Set your buffer size to 64KB or 128KB depending on your memory. Just heads up - direct buffers live outside the heap so garbage collection gets weird with them.
For 300MB files, use RandomAccessFile with getChannel() instead of FileWriter. You’ve got two problems - FileWriter’s for text data (you’re writing bytes), and it buffers everything in memory first which tanks performance on big files. I use RandomAccessFile raf = new RandomAccessFile(file, "rw"); FileChannel channel = raf.getChannel(); then write the byte stream in chunks with ByteBuffer.wrap(). Stream the data straight to disk instead of loading it all into memory. Memory-mapped files can be faster for reads, but for sequential writes like this, basic FileChannel will give you the performance boost without extra complexity.
Memory mapped files work great for this size range. Use MappedByteBuffer mbb = channel.map(FileChannel.MapMode.READ_WRITE, 0, fileSize) and put your bytes directly. Way faster than chunked writes since there’s no system calls per write. Just make sure you get the file size from your REST metadata first.
Been running file uploads in production for years - your approach really depends on server setup. Got concurrent uploads? Skip memory-mapped files entirely. They lock up too much virtual memory and you’ll hit OS limits fast. Use Files.newByteChannel() from java.nio.file instead. Way cleaner than old RandomAccessFile and handles large sequential writes great. I use StandardOpenOption.CREATE and WRITE with 256KB buffers. Here’s what most people miss: your REST framework already has those bytes in memory anyway. Focus on getting them to disk efficiently instead of trying to stream from the request. Also - write to fast storage. We moved uploads to SSD and got bigger performance gains than any NIO tweaks.