waffel’s Weblog

Oktober 9, 2007

Fast stream copy using java.nio channels

Filed under: java,nio,software — Thomas Wabner @ 11:21 vormittags
Tags: , , ,

Many times I was asked how to copy fast and simple an input stream to an output stream. I have found a nice solution on koders.com . A programmer named Mr. Hitchens has contributed this code.

Here are my utility method which makes the real fast copy stuff:

public final class ChannelTools {
  public static void fastChannelCopy(final ReadableByteChannel src, final WritableByteChannel dest) throws IOException {
    final ByteBuffer buffer = ByteBuffer.allocateDirect(16 * 1024);
    while (src.read(buffer) != -1) {
      // prepare the buffer to be drained
      buffer.flip();
      // write to the channel, may block
      dest.write(buffer);
      // If partial transfer, shift remainder down
      // If buffer is empty, same as doing clear()
      buffer.compact();
    }
    // EOF will leave buffer in fill state
    buffer.flip();
    // make sure the buffer is fully drained.
    while (buffer.hasRemaining()) {
      dest.write(buffer);
    }
  }
}

And how you can use this method to copy an input stream into an output stream? Here comes the answer:


// allocate the stream ... only for example
final InputStream input = new FileInputStream(inputFile);
final OutputStream output = new FileOutputStream(outputFile);
// get an channel from the stream
final ReadableByteChannel inputChannel = Channels.newChannel(input);
final WriteableByteChannel outputChannel = Channels.newChannel(output);
// copy the channels
ChannelTools.fastChannelCopy(inputChannel, outputChannel);
// closing the channels
inputChannel.close();
outputChannel.close()

About these ads

20 Kommentare »

  1. [...] Notice: the fast channel copy routine was introduced by waffel. [...]

    Pingback von JSF, Ajax and file download « Andreas Höhmann’s Weblog — März 31, 2008 @ 3:10 nachmittags | Antwort

  2. Somehow i missed the point. Probably lost in translation :) Anyway … nice blog to visit.

    cheers, Reducible
    .

    Kommentar von Reducible — Juni 18, 2008 @ 11:42 nachmittags | Antwort

  3. i used your code but
    java.lang.OutOfMemoryError: Java heap space
    be shown,do you know how can i fixed this,
    (i want to implement download service for client and the file mabe on server hard disk or database)

    Kommentar von shiva — Oktober 21, 2008 @ 12:28 nachmittags | Antwort

  4. I don’t think that the OutOfMemory problem comes from the ChannelTools, because they don’t allocate much memory (only 16*1024 bytes).

    I don’t know why you get the exception. Can you provide an example?

    Kommentar von thomaswabner — Oktober 21, 2008 @ 1:35 nachmittags | Antwort

  5. try to allocate more memory to your JVM ….
    its for example -Xmx512.

    Kommentar von Maciej Bukowski — November 17, 2008 @ 4:02 nachmittags | Antwort

  6. Great way and fastest way I found so far but I have a question :

    Is it Thread Safe ?

    I mean can two or more classes use the same ChannelTools simulataneously ??

    I think it is not thread safe because of static method declaration

    please adviseand thank you very much your efforts is highly appreciated

    Kommentar von Ibrahim Odeh — Dezember 20, 2008 @ 7:20 nachmittags | Antwort

  7. Yes, the method is thread safe.

    The static declaration has nothing to do with thread safty.

    I don’t hold any variables outside the method and the static declaration is only used to use this as a simple helper method.

    But you have to ensure that the source and destination buffers not “shared” between two threads. But this handling is out of scope of my method.

    We use this solution in big threading applications (like web application) and it works perfect with many concurrent users.

    kind regards

    – waffel

    Kommentar von Thomas Wabner — Dezember 23, 2008 @ 12:02 nachmittags | Antwort

  8. java.lang.OutOfMemoryError for me too.
    at dest.write(buffer); for 15M file.

    Seems problem is in destination type. dest is Channels.newChannel(ByteArrayInputStream)

    Kommentar von Mikhail — Dezember 25, 2008 @ 12:48 nachmittags | Antwort

  9. Yes, if you use a ByteArrayInputStream you use your memory as the destination. And if you stream 15M into your memory you can of course run into OutOfMemory trouble.

    But this has nothing to do with the fastChannelCopy method. If you plan to use this in your code you have to write a good documentation or check the type of the destination channel.

    Regards,
    – waffel

    Kommentar von Thomas Wabner — Dezember 27, 2008 @ 12:08 nachmittags | Antwort

  10. If you are dealing with FileChannels, you may also want to look at the transferTo() and transferFrom() methods.

    Kommentar von Jukka Zitting — Januar 15, 2009 @ 2:25 nachmittags | Antwort

  11. I have tested the above codes. It works fine. But the copy speed is the same as the normal copy from inputstream to outputstream (not using channels).

    I’ve increased the buffer size but the result is still the same :(

    Jukka Zitting said something about transferTo() and transferFrom() methods, but I couldn’t found it in google. I hope he can give a source link and details about it.

    I’m really curious if there is other ways to increase the copy speed.

    Thanks,
    PH

    Kommentar von PH — Januar 30, 2009 @ 3:05 nachmittags | Antwort

  12. Hi, thanks for sharing the solution. What’s the licence for the code above? Can I use it in a commercial product just mentioning this article’s URL in a source code comment?

    Kommentar von gshegosh — Februar 3, 2009 @ 4:47 nachmittags | Antwort

  13. Good question about the licence. I have used the code idea from here: http://koders.com/java/fidF0ECB93DF37B56A0F43D98D701816F48FCB2E2E7.aspx?s=Mr.+Hitchens#L22

    But this project has any licence informations. My example can be used without restrictions (a note with a link to the article is welcome).

    Short: Yes you can use the code example in your commercial product ;-)

    Kommentar von Thomas Wabner — Februar 3, 2009 @ 5:07 nachmittags | Antwort

  14. It’s always better to ask than to get sued ;) Thanks for replying, thanks for permission and keep up a good work.

    Kommentar von gshegosh — Februar 3, 2009 @ 6:04 nachmittags | Antwort

  15. There is a more simple way to do this using the transferTo() and transferFrom() methods of the FileChannel class.
    this is a code from wikipedia
    http://en.wikipedia.org/wiki/New_I/O

    // Getting file channels
    FileChannel in = new FileInputStream(source).getChannel();
    FileChannel out = new FileOutputStream(target).getChannel();

    // JavaVM does its best to do this as native I/O operations.
    in.transferTo (0, in.size(), out);

    // Closing file channels will close corresponding stream objects as well.
    out.close();
    in.close();

    I’ve used this and it works flawlessly!

    Kommentar von Tibi — August 13, 2009 @ 11:09 vormittags | Antwort

    • You are right … if you know the size of the channel you can use the transferTo method.

      But what about InputStreams where the size is unknown (like in web environment)? In this case, I think my approach is a good solution AND works also for files.

      Kommentar von Thomas Wabner — August 13, 2009 @ 1:19 nachmittags | Antwort

  16. Ein guter Tipp, solle man mal weiterverfolgen. Das wird in näherer Zukunft bestimmt noch mehr Aufmerksamkeit auf sich ziehen. Da habe ich doch schon vor kurzem einen Artikel bei Siegel Online drüber gefunden? Tschöö, Wanda Schöne

    Kommentar von Wanda Schöne — März 17, 2010 @ 5:02 nachmittags | Antwort

  17. [...] Fast stream copy using java.nio channels October 200717 comments 3 [...]

    Pingback von 2010 in review « waffel’s Weblog — Januar 3, 2011 @ 1:04 nachmittags | Antwort

  18. I am wondering if there is a speceal reason for using final Inputstream and OutputSream.
    In my Application I seem to copy more then one Inputstream to one Outputstream one by one.
    It would not be very appropriate, since I don’t know how many that will be in the first place.

    Kommentar von Malte — August 30, 2012 @ 5:06 nachmittags | Antwort

  19. HI. Is it just prefer File I/O? I used this method with inputStream come from internet and the outputstream is socket output. It works. However , I notice there is lots of CG in the log. comparing with tradition way(read inputstream into byte[] buffer and write byte[] to outputstream). Is it implying that it is more copy with the tradition way when it is not File I/O?.The code is running on Android device but I don’t it matter.

    Kommentar von Yeung — Dezember 12, 2013 @ 11:09 vormittags | Antwort


RSS-Feed für Kommentare zu diesem Beitrag. TrackBack URI

Kommentar verfassen

Trage deine Daten unten ein oder klicke ein Icon um dich einzuloggen:

WordPress.com-Logo

Du kommentierst mit Deinem WordPress.com-Konto. Abmelden / Ändern )

Twitter-Bild

Du kommentierst mit Deinem Twitter-Konto. Abmelden / Ändern )

Facebook-Foto

Du kommentierst mit Deinem Facebook-Konto. Abmelden / Ändern )

Google+ photo

Du kommentierst mit Deinem Google+-Konto. Abmelden / Ändern )

Verbinde mit %s

The Rubric Theme. Bloggen Sie auf WordPress.com.

Folgen

Erhalte jeden neuen Beitrag in deinen Posteingang.

Schließe dich 28 Followern an

%d Bloggern gefällt das: