Fast stream copy using java.nio channels

Many times I was asked how to copy fast and simple an input stream to an output stream. I have found a nice solution on koders.com . A programmer named Mr. Hitchens has contributed this code.

Here are my utility method which makes the real fast copy stuff:


public final class ChannelTools {
  public static void fastChannelCopy(final ReadableByteChannel src, final WritableByteChannel dest) throws IOException {
    final ByteBuffer buffer = ByteBuffer.allocateDirect(16 * 1024);
    while (src.read(buffer) != -1) {
      // prepare the buffer to be drained
      buffer.flip();
      // write to the channel, may block
      dest.write(buffer);
      // If partial transfer, shift remainder down
      // If buffer is empty, same as doing clear()
      buffer.compact();
    }
    // EOF will leave buffer in fill state
    buffer.flip();
    // make sure the buffer is fully drained.
    while (buffer.hasRemaining()) {
      dest.write(buffer);
    }
  }
}

And how you can use this method to copy an input stream into an output stream? Here comes the answer:


// allocate the stream ... only for example
final InputStream input = new FileInputStream(inputFile);
final OutputStream output = new FileOutputStream(outputFile);
// get an channel from the stream
final ReadableByteChannel inputChannel = Channels.newChannel(input);
final WriteableByteChannel outputChannel = Channels.newChannel(output);
// copy the channels
ChannelTools.fastChannelCopy(inputChannel, outputChannel);
// closing the channels
inputChannel.close();
outputChannel.close()

14 Antworten

  1. [...] Notice: the fast channel copy routine was introduced by waffel. [...]

  2. Somehow i missed the point. Probably lost in translation :) Anyway … nice blog to visit.

    cheers, Reducible
    .

  3. i used your code but
    java.lang.OutOfMemoryError: Java heap space
    be shown,do you know how can i fixed this,
    (i want to implement download service for client and the file mabe on server hard disk or database)

  4. I don’t think that the OutOfMemory problem comes from the ChannelTools, because they don’t allocate much memory (only 16*1024 bytes).

    I don’t know why you get the exception. Can you provide an example?

  5. try to allocate more memory to your JVM ….
    its for example -Xmx512.

  6. Great way and fastest way I found so far but I have a question :

    Is it Thread Safe ?

    I mean can two or more classes use the same ChannelTools simulataneously ??

    I think it is not thread safe because of static method declaration

    please adviseand thank you very much your efforts is highly appreciated

  7. Yes, the method is thread safe.

    The static declaration has nothing to do with thread safty.

    I don’t hold any variables outside the method and the static declaration is only used to use this as a simple helper method.

    But you have to ensure that the source and destination buffers not “shared” between two threads. But this handling is out of scope of my method.

    We use this solution in big threading applications (like web application) and it works perfect with many concurrent users.

    kind regards

    - waffel

  8. java.lang.OutOfMemoryError for me too.
    at dest.write(buffer); for 15M file.

    Seems problem is in destination type. dest is Channels.newChannel(ByteArrayInputStream)

  9. Yes, if you use a ByteArrayInputStream you use your memory as the destination. And if you stream 15M into your memory you can of course run into OutOfMemory trouble.

    But this has nothing to do with the fastChannelCopy method. If you plan to use this in your code you have to write a good documentation or check the type of the destination channel.

    Regards,
    - waffel

  10. If you are dealing with FileChannels, you may also want to look at the transferTo() and transferFrom() methods.

  11. I have tested the above codes. It works fine. But the copy speed is the same as the normal copy from inputstream to outputstream (not using channels).

    I’ve increased the buffer size but the result is still the same :(

    Jukka Zitting said something about transferTo() and transferFrom() methods, but I couldn’t found it in google. I hope he can give a source link and details about it.

    I’m really curious if there is other ways to increase the copy speed.

    Thanks,
    PH

  12. Hi, thanks for sharing the solution. What’s the licence for the code above? Can I use it in a commercial product just mentioning this article’s URL in a source code comment?

  13. Good question about the licence. I have used the code idea from here: http://koders.com/java/fidF0ECB93DF37B56A0F43D98D701816F48FCB2E2E7.aspx?s=Mr.+Hitchens#L22

    But this project has any licence informations. My example can be used without restrictions (a note with a link to the article is welcome).

    Short: Yes you can use the code example in your commercial product ;-)

  14. It’s always better to ask than to get sued ;) Thanks for replying, thanks for permission and keep up a good work.

Einen Kommentar schreiben