Category Archives: FileSender 2.0

Announcing Filesender 2.0 beta 1

The Filesender team is pleased to announce the release of Filesender 2.0 beta 1.  We welcome feedback from the community via github or on the Filesender mailing lists.

Version 2.0 is a new baseline release support the Filesender Roadmap, as recently announced at TNC17.  This latest work has been made possible by contributions received through the Filesender Programme in the Commons Conservancy – with specific thanks to HEAnet.

All details of the new release can be found on the Filesender github repository.

What’s New?

Much of the code base was rewritten, a new database design adopted, many configuration directives have been added and existing directives changed.

There is now support for secondary indexes in both database backends. There is also initial movement to greater resilience to 4 byte character encodings and index implementations in MySQL implementations.

Guest implementation has been tested. Specifically how the UI presents itself given various default options in the configuration and situations that are confusing or do not allow the guest to easily progress have been addressed.

The about and help text are now pages instead of dialogs in the web UI. There is also a new provision for sites to present custom content for these pages in specific languages where FileSender updates will not override that content.

Alpha, beta, release candidate and production releases

We haven’t really made a line-in-the-sand release announcement of version 2.0-alpha. This was partly because we didn’t really have a good definition of what we mean by an alpha release and how it’s different from a beta release. This post is intended to clarify this and to ensure that our community has the right expectations of our various releases. The related pages at www.filesender.org will be updated shortly. We would appreciate community feedback on this blog post: please let us know whether our definitions match your expectations via the filesender-dev mailing list.

A release goes from “a bucket of code”, through “alpha”, “beta” and “release candidate”, to production “release”.

Characteristics of a FileSender alpha:

  • Draws a line-in-the-sand from where development will move from “adding to a bucket of code” towards “getting code and documentation to a production release state”.
  • Has adequate installation documentation but may require more installation and configuration effort from a system administrator than a production release will.
  • Has not undergone structured client-side workflow testing.
  • Must be assumed to have unknown issues.
  • The feature list is mostly stable but can change.
  • Documentation is typically incomplete.
  • Has been through some basic tests by developers: usually this means automated unit tests on software components, and verification by the developer and verification on an independent installation installed from Subversion (SVN) that at least the most basic functionality (uploading and downloading of a file) proceeds as intended with at least one browser.
  • Is released through SVN. We want the community to be able to update their alpha release based installation as soon as new code is available and without too much hassle. Inserting a packaging step before releasing alpha versions would delay and further complicate the alpha release procedure.
  • Release announcement includes the SVN branch name and revision number – it is that specific SVN checkout that is considered to be the specific alpha release.
  • There can be multiple alpha releases. They will be labelled alpha-1, alpha-2 etc..

Characteristics of a FileSender beta:

  • Has undergone some structured client-side workflow testing.
  • Should be easy to install and the tested features should simply work.
  • Feature list is being stabilised. This implies a new-feature freeze when releasing the first beta. It also implies a documented understanding of the feature list by the last beta release.
  • A summary of structured client-side workflow tests conducted per beta release typically indicates which features have been tested and with which browser(s).
  • You should not encounter unknown issues in tested features. Note that not all features may be tested for a particular beta release.
  • Is at a minimum released as a tarball.
  • There can be multiple beta releases. They will be labelled beta-1, beta-2 etc..

Release candidate:

  • At some point in the release cycle, the list of all features that we wish to ship in a functional state should be documented and collectively present in one single beta release. After that particular beta has undergone structured client-side workflow testing and been released, and has also proven itself stable in field testing, this beta release can become a release candidate.
  • This is the release where we are confident that we have identified the important known issues and that there are no show-stopping issues.
  • Small bug fixes since the last beta release can be added to the release candidate.
  • Is at a minimum released as a tarball.
  • After release will be field-tested on at least two FileSender release candidate test sites:
    • Production sites with at least X transfers by X unique users per day.
    • The sites run with well-known configurations making what are considered standard features available to their user bases.

Release:

  • Once a release candidate has been running production traffic with live users for at least two weeks on at least two release candidate test sites and shown no significant issues impairing service for the users, it can become the production (major) release.
  • There are no code, database or configuration file changes between a release candidate and a release.
  • Is at a minimum released as a tarball.
  • At some point(s) prior to a production (major) release, the code has been subjected to at least one external code security audit.

 

The following pages will be updated as part of this policy clarification:

We would appreciate community feedback on this blog post: please let us know whether our definitions match your expectations via the filesender-dev mailing list.

Update status FileSender 2.0, end of March

A short update on where we are with version 2.0.

I guess the most important to mention is the 2nd security audit is done.  I haven’t seen the report yet but our lead developer Etienne has.  Three small issues were found, all without an active attack vector.  My conclusion so far is the 2.0 code is secure.  The report will find its way to my inbox in the next weeks after which we’ll publish a similar response document as we did for the 1st security audit.

We’ve also been fixing various bugs, simplifying code and in general making things more robust.  Right now Etienne is looking into making the uploads more robust without resorting to hashing for file integrity protection.  The latter is too slow given currently available functionality in browsers so we need to settle for next-best: be able to detect exactly which chunks we did and did not receive.  Sounds like TCP all over again doesn’t it 😉

With its required security audit almost out of the way the French NREN Renater plans to launch its public beta based on version 2.0 next week.  After some weeks of public beta the plan is to move their production site to version 2.0.  From the project’s point of view this means field testing will start 🙂

Before the project can release a beta we first need to get quite a number of things out of the way.  Client-side testing, documentation, defining an upgrade path from 1.6 to 2.0, robust installation etc.

Are you curious to see what’s in version 2.0?  Interested in trying it out?

Do you want the 2.0 release to progress faster?  Help out with documenting!  Send me an email on jan dot meijer at uninett dot no.

Status FileSender 2.0, March 2015

A brief update on where we are with version 2.0.

The 1st security audit has been executed and the results were good.  No structural security issues were found.  All but one of the issues that were identified have been addressed.  The remaining issue is not difficult but requires some further investigtion to get it right.  For details read the blog article about the 1st security audit

The 2nd security audit is underway and a report is expected some weeks from now.

In addition to the work on the security audit results we’ve been testing and trying which again lead to discovering and fixing a number of smaller bugs.  Documentation is not progresing as fast as we’d like and it’s that lack of documentation which is keeping us from releasing an alpha tarball.

We’re working on planning the client-side workflow testing but it’s too early to be able to say anything useful about when this is expected to start.

Meanwhile both RENATER and UNINETT are planning to offer a “FileSender 2.0 beta service” to their users based on the current 2.0 code base, in the next couple of months.  That would be the start of larger scale field testing of version 2.0.

Are you curious to see what’s in version 2.0?  Interested in trying it out?

Do you want the 2.0 release to progress faster?  Help out with documenting!  Send me an email on jan dot meijer at uninett dot no.

Results 1st security audit of FileSender 2.0

FileSender software is entrusted with user’s files and hence needs to be secure.  To ensure an adequate level of security is achieved each major release of FileSender is subject to at least one code security audit.  While we don’t expect FileSender to hold out against a determined state-funded attacker we do expect the software to follow all publicly known security best current practices and have no “oops” security holes.

Using funding provided by HEAnet the FileSender project hired Pine Digital Security to execute a code security audit of the FileSender 2.0 development code.  The audit was executed on revision 3390 of the SVN branche branches/filesender-2.0 and done in the timeframe 12 January 2015 – 3 February 2015.  Pine sent the report with its findings on 3 February 2015.  The report was discussed on 4 February in a meeting between Jan Meijer (FileSender project lead), Etienne Meleard (FileSender development lead) and in a conference call between the two aforementioned and Daan Keuper from Pine Digital Security.

Based on these discussions an assessment was made of each of the identified issues and the appropriate response from the project decided.  The general impression was that the code improved significantly compared with version 1.6.  No structural security issues were found.

A total of 10 issues were identified.  Two of these were of type “oops” and were fixed without discussion.  Five were of type “defence in depth” and have been addressed.  Two items identified as a vulnerability are considered by the project as a feature. The last item considers insufficiently secure random number generation which is an issue for download URL protection.  This has been addressed.

We’ve documented the issues found, our assessment and response as well as our follow-up including ticket numbers.  You find all details in this document:

FileSender project’s response to the 1st security audit of FileSender 2.0

As I write this, a second and more extensive security audit funded by RENATER and executed by French security firm Amossys. This audit is expected to report at the end of March.  As part of the contract, any significant findings would be reported promptly.  After 2 weeks of audit no significant findings have been reported.

Are you curious to see what’s in version 2.0?  Interested in trying it out?

Do you want the 2.0 release to progress faster?  Help out with documenting!  Send me an email on jan dot meijer at uninett dot no.

Status FileSender 2.0, end of january 2015

In the previous blog post I wrote about version 2.0, its features and where we were at.  In this post you’ll get an update on the current status of 2.0 development.

The code is now considered feature-complete.  We now work on documenting, testing and providing the upgrade path.  Basic documentation (installation, configuration directives) is shaping up, others we still have to start with.  With a code base and database layout that’s changed as much as it did there’s quite some work to be done to make for a smooth migration from 1.6 to 2.0.  We work towards 1st March to have a version ready for field testing, which I expect to be called the 2.0-alpha version.  Please note this version will not have undergone extensive client side workflow testing.

The field testing both UNINETT and RENATER plan is to offer our end users a 2.0 test instance next to a production 1.x instance.  I understood from others they’re considering the same.

I’ve written it before: it takes quite some time to document and to verify the documentation is correct.  Any help will speed up the release.  If you’re interested in specific features like the API, built-in email bounce handling etc. let us know, you can help verify the basic documentation.  Send me an email at jan.meijer at uninett.no if you can spare a couple of hours.

Try it: https://terasender.uninett.no/branches/filesender-2.0

Install it: https://www.assembla.com/spaces/file_sender/wiki/Installation_-_Linux_Source_2-0-Alpha-from-svn

More detailed status:

  • Most of the configuration directive documentation is done.  It needs about one more day of work and then a double-check;
  • The Linux source installation documentation is done and will be updated as other documentation and the sample config file progresses;
  • For any major FileSender release an external code security audit will be done.  This time we’ll even have two.  The 1st review is being executed by Pine Security and paid for by the project.  It is underway as I write this, the report is expected next week.  The 2nd review will be executed by a French security firm under responsibility of and paid by RENATER as part of its internal process for taking 2.0 in production.  The RENATER review is expected to start latest in the 3rd week of February;
  • Next on our list are an administrator guide to various aspects (authentication, language configuration, customisation), upgrade documentation,API documentation, upgrade scripts, client side workflow testing, packaging and upgrade testing.  All this will keep us occupied at least throughout February and March.

That’s it for now, happy testing!

FileSender 2.0 status

Reading through past blog posts I realised that we informed a lot of people about our work on multi-file and FileSender 2.0 through personal face-to-face conversations on various conferences but we never wrote anything on our blog!  Time to fix that.

I’ll leave details of the  2.0 development history for another post; for now I’ll just say the French NREN (National Research and Educational Network) RENATER joined the development effort by contributing with person hours of developer  Etienne Meleard.  Etienne has taken the lead on development of FileSender 2.0, the multi-file release.

Much code has been refactored and rewritten since June.  We now have a 2.0 branch in our SVN repository which is approaching alpha-release state.   You can install it; the install documentation has been written, tested and should work for you.  Please note that it currently only works on MySQL; Postgres database initialisation support is expected to be implemented in the coming weeks.   A link to my own demo server is included below.

Feature highlights:

  • full multi-file support: upload, download, MyFiles, email receipts and automatic transaction deletion
  • drag & drop support
  • generalised user option mechanism.  Which options are available to a user is something a service provider can control from the config file
  • fine-grained control over email receipts through various options, including the option to send no email receipts
  • UI templating mechanism
  • REST API

Try it: https://terasender.uninett.no/branches/filesender-2.0

Install it: https://www.assembla.com/spaces/file_sender/wiki/Installation_-_Linux_Source_2-0-Alpha-from-svn

Plan towards 2.0 release: the code and feature set is mostly stable and we’re working towards a 2.0 alpha release, with sufficient documentation to install and use it. While I can’t give an exact date I have good reason to believe it’ll be there before christmas.  We’ll use that mile stone to start polishing on a beta release which typically includes a thorough feature review and Wendy Mason’s client-side workflow testing.

I do hope several sites will install the alpha release and make it available to a group of test users.  We will need the field-testing feedback to make progress.  If you want to be a part of field testing version 2.0 or its documentation contact me at jan.meijer@uninett.no.

We’re also keen on volunteers to help with documentation.  Every hour you can spare is one hour closer towards a 2.0 release!