Terabyte challenge: preliminary results

We’ve got some exciting preliminary results from the work on the FileSender Terabyte Challenge! With the report only a couple of weeks away I won’t write too much for now and let the screenshots speak for themselves.   It looks like UvA System and Network Engineering students René Klomp and Edwin Schaap cracked the problem.

After confirming their hypothesis on the FileSender performance bottlenecks the students wrote a small library to demonstrate a solution.  A standard FileSender 1.5 trunk code installation was modified to use this library.  This installation was used today for a first full 1TB upload test over a 0 ms latency path.   The file took a little under 3.5 hours to transfer from laptop to server.  More testing with longer latency paths will follow.  Congratulations René and Edwin!

Update 25.1.2013: I forgot to mention that during this test the external disk wouldn’t go faster than 800Mbit/s despite it being eSATA.  One might imagine this has an impact on the transfer speed of the file.

Update 25.1.2013, #2: AARNet installed the prototype code on their standard service platform. and ran a quick test from Perth to Brisbane, a 67 ms rtt path, using a standard off the shelf employee’s laptop.  Standard filesender code:162 MB file in 220 seconds.  New webworker filesender code: 162MB file in 16 seconds.

Update 28.1.2013: we performed tests between Trondheim and Utrecht on a 40ms rtt path using the test installation set up by René and Edwin’s test installation in Utrecht and my Macbook Pro in Trondheim.  The laptop is a November 2012 model with 8GB RAM, OS-X 10.8 and the latest FireFox.  The network path has not been baseline-tested.  A 2GB file took about 68 seconds to upload when the disk cache was purged first.  If the entire file was cached, or if the file was read from a RAM disk, it took about 45 seconds to upload.  René and Edwin are considering the feasibility of a read-ahead buffer.  To compare: from the same desktop, disk cache purged,  to Xander’s stock 1.5-rc1 FileSender installation in Amsterdam, on a 37 ms rtt path, it took a 2GB file ca. 340 seconds.  The same upload repeated immediately afterwards without purging the disk cache, implying a read from disk cache, took about 330 seconds.

filesender-tb-challenge-1tbtransfer-Screenshot from 2013-01-24 11_15_01

Transfer of 1TB file starts at 11:15

1TB transfer at 83% around 14:00 (2 hours 45 minutes later)
1TB transfer at 83% around 14:00 (2 hours 45 minutes later)

1TB transfer at 83% around 14:00 (2 hours 45 minutes later)

Email received after succesful upload first 1TB file

Upload completed and email sent at 14:34.  Note the email says 14:44 due to graylisting at recipient

3 responses to “Terabyte challenge: preliminary results

  1. Pingback: Status uke 4 2013 « eCampus Norge

  2. Super! When might this be available for production?

    • We’ll be looking into the question of integrating it in the base filesender code in the weeks to come. I expect we’ll be able to answer the question better in some weeks, when our lead developer Chris Richter and code integrator Xander Jansen have had a chance to look at the code. The incentive for the team to include this quickly is definately there, I personally look forward to the opportunity to offer my users the speed increase.

      Stay tuned on the filesender-dev list 🙂

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s