waffel’s Weblog

Oktober 9, 2007

Fast stream copy using java.nio channels

Filed under: java,nio,software — Thomas Wabner @ 11:21 am
Tags: , , ,

Many times I was asked how to copy fast and simple an input stream to an output stream. I have found a nice solution on koders.com . A programmer named Mr. Hitchens has contributed this code.

Here are my utility method which makes the real fast copy stuff:

public final class ChannelTools {
  public static void fastChannelCopy(final ReadableByteChannel src, final WritableByteChannel dest) throws IOException {
    final ByteBuffer buffer = ByteBuffer.allocateDirect(16 * 1024);
    while (src.read(buffer) != -1) {
      // prepare the buffer to be drained
      buffer.flip();
      // write to the channel, may block
      dest.write(buffer);
      // If partial transfer, shift remainder down
      // If buffer is empty, same as doing clear()
      buffer.compact();
    }
    // EOF will leave buffer in fill state
    buffer.flip();
    // make sure the buffer is fully drained.
    while (buffer.hasRemaining()) {
      dest.write(buffer);
    }
  }
}

And how you can use this method to copy an input stream into an output stream? Here comes the answer:


// allocate the stream ... only for example
final InputStream input = new FileInputStream(inputFile);
final OutputStream output = new FileOutputStream(outputFile);
// get an channel from the stream
final ReadableByteChannel inputChannel = Channels.newChannel(input);
final WriteableByteChannel outputChannel = Channels.newChannel(output);
// copy the channels
ChannelTools.fastChannelCopy(inputChannel, outputChannel);
// closing the channels
inputChannel.close();
outputChannel.close()

18 Kommentare »

  1. […] Notice: the fast channel copy routine was introduced by waffel. […]

    Pingback von JSF, Ajax and file download « Andreas Höhmann’s Weblog — März 31, 2008 @ 3:10 pm | Antworten

  2. Somehow i missed the point. Probably lost in translation 🙂 Anyway … nice blog to visit.

    cheers, Reducible
    .

    Kommentar von Reducible — Juni 18, 2008 @ 11:42 pm | Antworten

  3. i used your code but
    java.lang.OutOfMemoryError: Java heap space
    be shown,do you know how can i fixed this,
    (i want to implement download service for client and the file mabe on server hard disk or database)

    Kommentar von shiva — Oktober 21, 2008 @ 12:28 pm | Antworten

  4. I don’t think that the OutOfMemory problem comes from the ChannelTools, because they don’t allocate much memory (only 16*1024 bytes).

    I don’t know why you get the exception. Can you provide an example?

    Kommentar von thomaswabner — Oktober 21, 2008 @ 1:35 pm | Antworten

  5. try to allocate more memory to your JVM ….
    its for example -Xmx512.

    Kommentar von Maciej Bukowski — November 17, 2008 @ 4:02 pm | Antworten

  6. Great way and fastest way I found so far but I have a question :

    Is it Thread Safe ?

    I mean can two or more classes use the same ChannelTools simulataneously ??

    I think it is not thread safe because of static method declaration

    please adviseand thank you very much your efforts is highly appreciated

    Kommentar von Ibrahim Odeh — Dezember 20, 2008 @ 7:20 pm | Antworten

  7. Yes, the method is thread safe.

    The static declaration has nothing to do with thread safty.

    I don’t hold any variables outside the method and the static declaration is only used to use this as a simple helper method.

    But you have to ensure that the source and destination buffers not „shared“ between two threads. But this handling is out of scope of my method.

    We use this solution in big threading applications (like web application) and it works perfect with many concurrent users.

    kind regards

    – waffel

    Kommentar von Thomas Wabner — Dezember 23, 2008 @ 12:02 pm | Antworten

  8. java.lang.OutOfMemoryError for me too.
    at dest.write(buffer); for 15M file.

    Seems problem is in destination type. dest is Channels.newChannel(ByteArrayInputStream)

    Kommentar von Mikhail — Dezember 25, 2008 @ 12:48 pm | Antworten

  9. Yes, if you use a ByteArrayInputStream you use your memory as the destination. And if you stream 15M into your memory you can of course run into OutOfMemory trouble.

    But this has nothing to do with the fastChannelCopy method. If you plan to use this in your code you have to write a good documentation or check the type of the destination channel.

    Regards,
    – waffel

    Kommentar von Thomas Wabner — Dezember 27, 2008 @ 12:08 pm | Antworten

  10. If you are dealing with FileChannels, you may also want to look at the transferTo() and transferFrom() methods.

    Kommentar von Jukka Zitting — Januar 15, 2009 @ 2:25 pm | Antworten

  11. I have tested the above codes. It works fine. But the copy speed is the same as the normal copy from inputstream to outputstream (not using channels).

    I’ve increased the buffer size but the result is still the same 😦

    Jukka Zitting said something about transferTo() and transferFrom() methods, but I couldn’t found it in google. I hope he can give a source link and details about it.

    I’m really curious if there is other ways to increase the copy speed.

    Thanks,
    PH

    Kommentar von PH — Januar 30, 2009 @ 3:05 pm | Antworten

  12. Hi, thanks for sharing the solution. What’s the licence for the code above? Can I use it in a commercial product just mentioning this article’s URL in a source code comment?

    Kommentar von gshegosh — Februar 3, 2009 @ 4:47 pm | Antworten

  13. Good question about the licence. I have used the code idea from here: http://koders.com/java/fidF0ECB93DF37B56A0F43D98D701816F48FCB2E2E7.aspx?s=Mr.+Hitchens#L22

    But this project has any licence informations. My example can be used without restrictions (a note with a link to the article is welcome).

    Short: Yes you can use the code example in your commercial product 😉

    Kommentar von Thomas Wabner — Februar 3, 2009 @ 5:07 pm | Antworten

  14. It’s always better to ask than to get sued 😉 Thanks for replying, thanks for permission and keep up a good work.

    Kommentar von gshegosh — Februar 3, 2009 @ 6:04 pm | Antworten

  15. There is a more simple way to do this using the transferTo() and transferFrom() methods of the FileChannel class.
    this is a code from wikipedia
    http://en.wikipedia.org/wiki/New_I/O

    // Getting file channels
    FileChannel in = new FileInputStream(source).getChannel();
    FileChannel out = new FileOutputStream(target).getChannel();

    // JavaVM does its best to do this as native I/O operations.
    in.transferTo (0, in.size(), out);

    // Closing file channels will close corresponding stream objects as well.
    out.close();
    in.close();

    I’ve used this and it works flawlessly!

    Kommentar von Tibi — August 13, 2009 @ 11:09 am | Antworten

    • You are right … if you know the size of the channel you can use the transferTo method.

      But what about InputStreams where the size is unknown (like in web environment)? In this case, I think my approach is a good solution AND works also for files.

      Kommentar von Thomas Wabner — August 13, 2009 @ 1:19 pm | Antworten

  16. Ein guter Tipp, solle man mal weiterverfolgen. Das wird in näherer Zukunft bestimmt noch mehr Aufmerksamkeit auf sich ziehen. Da habe ich doch schon vor kurzem einen Artikel bei Siegel Online drüber gefunden? Tschöö, Wanda Schöne

    Kommentar von Wanda Schöne — März 17, 2010 @ 5:02 pm | Antworten

  17. […] Fast stream copy using java.nio channels October 200717 comments 3 […]

    Pingback von 2010 in review « waffel’s Weblog — Januar 3, 2011 @ 1:04 pm | Antworten


RSS feed for comments on this post. TrackBack URI

Hinterlasse eine Antwort zu Thomas Wabner Antwort abbrechen

Erstelle kostenlos eine Website oder ein Blog auf WordPress.com.