FileSender 1.6 released!

We’re happy to announce the release of FileSender 1.6.  There are no code changes since 1.6-rc1.  All the documentation has been updated where needed.

Highlights for this release:

  • New: high speed upload module “terasender” tested with uploads up to 1 TB
  • New: auto-complete feature in the recipient (To:) field
  • New: configure switch to include (or not include) the recipient of a file on the download confirmation email the sender receives upon download of a file.  This reduces the amount of emails a recipient receives.
  • New: refactored MyFiles.  Less clutter and includes a counter of the number of downloads + downloaded date in the web UI.
  • New: support for optional Subject and Message in guest use voucher
  • New: support for multiple From: addresses from authentication source
  • New: support for configurable footer via language files
  • New language: Finnish
  • Other new features and several bug fixes

The upgrade notes from 1.5 to 1.6 contain the exact overview of the fixes and features.  Please read these upgrade notes when installing 1.6.  Pay particular attention to:

  • the new terasender feature is by default disabled.  Switch it on to benefit from high-speed uploads.  Several months of field-testing have not yielded any problems.
  • Change your logout URL as per the upgrade notes.  Your logout will be broken if you don’t.
  • You can now send less email to file recipients.  Be sure to set  ‘download_confirmation_to_downloader‘ to false if that’s what you want.

As usual you can get the software in two ways:

  • Manual download via the FileSender download page
  • For those of you using the Debian or RPM package repositories: the 1.6 packages are available in the testing repository.

We would like to thank you all for your patience and of course your input and contributions. A list of people and organisations that made this release candidate possible can be found at the acknowledgements page for version 1.6

Keep those patches, comments and translations and all other useful input coming!  We welcome feedback, preferably to the filesender-dev@filesender.org mailinglist.

With version 1.6 now released we will focus on the version 2.0 with multi-file support.

FileSender 1.6-release-candidate-1 released!

We’re happy to announce that FileSender 1.6-rc1 is now available for download and in the FileSender package repositories.  As per our release status and life cycle policy when a Release Candidate has been running on least two FileSender sites without error for a period of at least 1 week under meaningful use, this release candidate can be re-branded as a release.  We usually wait with this until also all documentation has been updated.  If anyone wants to volunteer to help with the documentation please drop me a line!

This release candidate contains a few fixes found during the beta1 cycle (thanks all for reporting!) including three security related fixes based on the security code review.

We encourage you to try this rc1 and most notably have a look at the ‘download pause/resume’ functionality that now should work for IE 11, FireFox and wget/curl.

Please note: the new terasender feature is now by default disabled when using the supplied config-dist.php template.

The changes and fixes since the previous 1.6-beta release (ticket numbers listed in parentheses) are:

  • Download pause/resume now possible with browsers supporting partial download (most notably Firefox, IE11, curl and wget)
  • ‘terasender’ default disabled in config-dist.php
  • Added IE10+ and Safari 6+ to supported browsers in HELP text for en, no and nl languages (#1063)
  • Security: also escape single quotes for externally supplied output (#1079)
  • Security: encode MMredirectURL in Flash detection code (#1078)
  • Security: strict type comparison in XSFR check (#1080)
  • ‘friendly name’ extraction fixed (#1068)
  • Various fixes and improvements in the partial download code (#1076)
  • chunked reading and buffering to prevent server side memory exhaustion with large range requests
  • more robust range request detection to make actual pause/resume possible

We would like to thank you all for your patience and of course your input and contributions. A list of people and organisations that made this release candidate possible can be found at the acknowledgements page for version 1.6

Keep those patches, comments and translations and all other useful input coming!  We welcome feedback, preferably to the filesender-dev@filesender.org mailinglist.

FileSender 1.6 Release Candidate 1 planning update

A quick update on the release planning for version 1.6 after today’s call with core-group members Xander, Chris, Vegard and myself.

Feedback from three sites running 1.6-beta1 field tests with relevant traffic shows it’s working smoothly with the TeraSender upload module enabled.  One small issue in the multiple-from address selection was reported and fixed by CESNET’s Lubos Kopecky.  Thanks Lubos!

Given these results of the beta-1 tests we see no need for another round of full-blown tests on the 1.6 Release Candidate 1, like we did on the 1.6-beta1.  .  The work that remains is checking all language files and updating them where needed as well as checking and finalising the documentation.  We strive to have this done in the coming two weeks.  Xander plans to prepare and release the 1.6-RC1 in week 9.

Q: When will the 1.6 release come out?

A: As per our release status and life cycle policy  “when a Release Candidate has been running on least two FileSender sites without error for a period of at least 1 week under meaningful use, this release candidate is re-branded as a release.  There are no code changes, no database changes and no config file changes between a release candidate and a release.”

FileSender 1.6-beta1 released!

This blog post is a bit after the fact but for those of you following our project via our blog here it is: 31 December 2013 our release manager Xander Jansen announced the availability of FileSender 1.6-beta1.  Please report any issues you find to the FileSender support mailinglist.  Like Xander I want to express my thanks to the FileSender team and all those who contributed to this important milestone for our FileSender services.

Version 1.6-beta1 is our first and hopefully only beta release for the upcoming 1.6 release. Although this beta1 has undergone thorough testing and has been subjected to a security code review it needs field testing before it can be considered production quality. We encourage you to try this beta1 and have a look at the new features (listed below). Please note that the new terasender feature is *enabled* for the beta cycle when using the supplied config-dist.php template.

Major changes since the previous release 1.6 are:

  • New TeraSender upload module which allows for much faster html5 uploads
  • Recipient address autocomplete based on previous recipients
  • Allow user to choose between multiple From: mail adresses if those are available in the user’s SAML2 attribute list
  • Configurable option to switch off download confirmation emails to the downloader (the sender still gets all of them)
  • New layout of My Files page, adding the number of downloads and cleaning up the general view to prepare for less emails whilst maintaining the file audit trail
  • Configurable footer
  • Support for SQLite
  • Add optional personal message and subject to guest  vouchers

The upgrade notes contain the full list of changes, including relevant ticket numbers.

By this time a number of sites are running 1.6-beta1, some already in production and no serious problems have surfaced so far.  The 1.6-beta1 has been through a code security review.  But for one minor issue there were no problems preventing production use.  The minor issue was fixed before releasing 1.6-beta1.

Please consult the Acknowledgements for v1.6 for the list of people and organisations that made this beta release possible. And last but not least, please keep the patches, comments and translations and all other useful input coming!

Status FileSender 1.6, Terasender

Time flies! Our last blog post was a while ago.  What happened since then?  Progress on version 1.6 has been slow with the project’s key QA people needing to attend to other priorities.  The 2.0 multi-file development has been progressing nicely, more about that in a separate post.

We’ve now planned a date for the 1.6-beta1 release.  We expect to start with the full rounds of integral client-side workflow testing 28th October.  This testing period takes about 2 weeks.  Assuming no major dramas (we have executed individual feature tests) the beta-1 is scheduled for 18 November.

Considering the delay to our original release schedule for 1.6 we’ve decided to not issue a beta-2 but rather focus on shipping a 1.6 and put what’s already available in your and your users hands rather than aim for perfection.   1.6 will in many ways behave the same as 1.5 or to put it in another way: if there’s anything wrong with the 1.6 we plan to release, it already was present in 1.5 😉

The functionality we wanted to put in 1.6-beta2 will either surface as a 1.6.x release or bundled with the 2.0 release, we haven’t decided yet.  This will become clearer as the 2.0 timeline materialises.

Optimistic as we always are we would ideally progress from there as follows:

  • 18 november: 1.6-beta1
  • 2 December: 1.6-rc1
  • 15 December: 1.6-release

Quoting from our Release status and software life cycle policy:Release: when a Release Candidate has been running on least two FileSender sites without error for a period of at least 1 week under meaningful use, this release candidate is re-branded as a release.  There are no code changes, no database changes and no config file changes between a release candidate and a release.”

The next status update for the 1.6 release can be expected early November when integral testing is to have started.  Meanwhile, please give the 1.6 code a shot from the SVN tree.  If you just want to experience 1.6 as a user, feel free to use the UNINETT test instance at terasender.uninett.no

Update FileSender roadmap: terasender, multi-file and file encryption

A summer update on the current roadmap for FileSender.  The next update is expected in the week of 22-25 July.

Next release: version 1.6 “TeraSender”

The next release is slated to be version 1.6 nicknamed “TeraSender”, leveraging the results of the FileSender TeraByte Challenge.  Release 1.6 will include:

  • significant upload speed improvement with new TeraSender upload module with upload speeds realistically enabling terabyte file sizes
  • improved MyFiles clearing the way for multi file transfers and changes in email flow
  • partial download support

As of this weekend code integration for the first beta is completed and a thorough work flow test cycle is now being planned.  After that test we expect to announce a release planning date for the first 1.6-beta.  You can track progress on the cardwall for tickets related to the 1.6 release.

We do expect a second beta with improved upload authentication handling and configurable CSS, tickets #905 Modify session validity check on upload to allow for very long uploads and #892 CSS Configurable.  These were postponed for the time being.  From previous releases we know it takes about 6 to 8 weeks to cycle from one beta to the next one.  Knowing we plan a 2nd beta the current dead line for a release candidate for the 1.6 release is 1 October.

Release 2.0: multi file with improved email flow options

UNINETT hired two summer students, Jack Kittridge and Vegard Polden, to work full time on FileSender for 2 months.  The project decided to use this opportunity to make a serious stab at support for multi-file transfers.  During the last month we’ve planned release 2.0, which we envision will be fully multi-file transfer capable: uploads, MyFiles, downloads and email receipts.  With email already being reworked it seems a good opportunity to include better user control of email receipts: we plan 5 email options that’ll allow a user control of email receipts, including the often asked for “stop spamming me” option.

After 3 weeks of work Jack and Vegard have made good progress prototyping all necessary functionality and most of what we envision looks to be possible.  We had a question mark with downloading multi-file transfers; there is no consensus across browser vendors for a fileWriter API like there is for the HTML5 fileReader API.  This lack of open standard support meant an alternative solution had to be found, we envisioned streaming zip download which now mostly works.    With the addition of 250 lines of code and no extra dependencies streaming all files into a single downloaded zip file is not a problem and it looks to work without a performance hit.  At the moment the resulting zip file refuses to be unzipped with the Mac archiver but this looks like a solvable problem.

The prototyping phase for multi-file is expected to last another 3 to 4 weeks after which we’ll start putting together the actual 2.0 release.  What we aim at is a rough and ready release 2.0 that we can then polish through a beta cycle during the remainder of 2013.

Close tracking of the progress of release 2.0:

2014: native browser-based file encryption and time stamping

We have functional prototypes for file encryption and for cryptographic time stamping.  Once the multi file release is sufficiently advanced we will use that as a basis to support native browser-based file encryption as well as an option to leverage FileSender as the user front-end for cryptograhpic time stamping.

The file encryption functionality will open up use cases for transfer of sensitive data: health research, HR data, sensitive research, personal data etc.  An ability to encrypt files of any size directly in a browser against zero per-user license cost is something we haven’t had before so who knows, it might help us bring encryption to the masses.

The cryptographic time stamping functionality we envision will enable a role for FileSender in protecting research data and protecting research ideas (draft publications).  A cryptographic time stamp proves a certain (collection of) file(s) existed at a certain point in time and hasn’t been changed since.  This type of proof is crucial if you want to re-use scientific data sets to validate scientific results, or to build new research on.  A cryptographic time stamp combined with the user’s identity is also very useful to prove a certain idea documented in a draft publication existed on a certain point in time.  In the world of competitive academic research such functionality ought to be good to have.

We expect to start work on integrating native browser based file encryption as well as time stamping in 2014 and will use the time until then to sharpen our vision and scope these new features.

Terabyte Challenge: report available

The report of the Terabyte Challenge assignment is available:

http://staff.science.uva.nl/~delaat/rp/2012-2013/p08/report.pdf

The students conclude it’s definately possible to speed up FileSender uploads significantly to the point where transferring a 1TB file in a reasonable amount of time is possible.  This is achieved in two ways:

  • be particular with the SSL ciphers your FileSender server supports: if both your server hardware and client hardware execute AES in hardware, there is a significant speedup to be had
  • by using web workers the upload job is parallellised, creating a nice gap-less stream of chunks

The FileSender team is currently working on integrating the students’ improved uploading code in FileSender version 1.6, the next planned release.  Check the roadmap for more details of what’s planned for version 1.6.  We target early June for either a beta release or if things go well a release candidate.  The release  schedule will be updated accordingly once first testing has been executed.