Replacing FUSE with a notification system

[silverdaz]

This PR replaces the FUSE program with a notification system. We also upgraded the authentication mechanism to support multiple user ids.

It has the following benefits:

• No mountpoint from /lega/<user> to /ega/inbox/<user>. In case something went wrong the uploaded file would not end up in the inbox.
• No need to use cron to clean up the mountpoints (since we chroot the user in its home directory, umount would fail).
• No FUSE library, no device mapping /dev/fuse, no FUSE python code. The file system calls do not go through the library, so we get performance.
• No Swarm limitation.

Regarding the authentication mechanism:

• No common user lega to impersonate all logged-in users. Instead, each user has its own id. Each site administration can configure in which range the user id lands (by shifting the user id). The fake
  CentralEGA is updated accordingly.
• NSS+PAM code uses SQLite 3.24 (the latest) as a cache.
• We added more verbose description of the NSS+PAM configurations.
• Better logic for expiration of the cache entries.
• Logs are sent to syslog, and progress is sent to stderr (if compiled with the debug flag).
• No JQ+JV dependency for JSON parsing. Instead, we use a lightweight embedded code.
• the auth code has been tested with Aspera, OpenSSH SFTP server and ProFTPd.

Regarding the SFTP server and the notification system:

• We use OpenSSH version 7.7p1 (the latest).
• We lock out SSH connections and only allow SFTP connections (on port 9000).
• We added a hook to send a notification to a listener, when a file is (re)uploaded. If connection is not established, it's just business as usual, nothing else happens. If connection is established, the name of the user and the uploaded file are sent to the listener. The listener forwards the message to the local broker. Technically, a stream of (username,filepath) are sent over a TCP connection. The listener is an async TCP server. We use out-of-band messages and did not use the TCP_NODELAY options.
• We bound the listener to the lo interface (no external access).
• One connection is established per SFTP connection.
• The solution does chroot the user in its home directory, but this option is configurable.

Moreover, this PR solves Issue #329, by making the postgres data persistent in its own volume. Rebooting the database should pick up the database where it left off.

Finally, the lega.yml is slightly updated in order to allow scalability of the ingest and verify workers.

Note: we could send the message directly to the local broker instead of the listener, but using pika and conf.ini in the listener was so much simpler than using a C-based AMQPS client. After rudimentary tests, it seems performant enough.

[blankdots]

we should fix postgres to 9.6 .. we even mention this in the docs: https://localega.readthedocs.io/en/latest/db.html?highlight=9.6
Also replace other mentions of postgres:latest

[silverdaz]

This PR also adds a stage (stages run in parallel according to Stefan) where we run the simple Makefile. That makefile has been adapted so that make is enough. No need to update any variable first. It creates a user, a random file, encrypts it, uploads it and triggers a fake submission. FILESIZE can be specified as an argument to the make command to change the size. The default is 10MB. I tried with 1GB too.

I ran some numbers:

Inbox	Upload speed	Memory consumption
OpenSSH	47MB/s	21.3MB to 38MB
Apache Mina	33MB/s	344MB to 346MB

I don't know how you get 1GB for Apache Mina, it seems to be stable roughly around 345MB.

The upload speed is not very good, mostly because we are using the default driver for the docker volumes (which are...on files). Normally, the speed should increase with better volume solutions. Let us know what you get in your respective deployment. OpenSSH seems 50% faster than Apache Mina.

[blankdots]

@silverdaz would be awesome if the history would not contain so many commits with "weird" (but funny) messages - the last 16 commit can and should be squashed.

This PR also adds a stage (stages run in parallel according to Stefan) where we run the simple Makefile.

It is according to: https://docs.travis-ci.com/user/build-stages#what-are-build-stages

I don't know how you get 1GB for Apache Mina, it seems to be stable roughly around 345MB.

Got the information from Openshift (also my instance has been up for 22 days and this grew over time):

Would like to monitor the inbox solutions over time > 10 days of uptime, don't know the uptime on the "Memory Consumption" numbers illustrated, and probably load tests should be implemented to stress out those solutions, but again this is not the subject of this issue and scalability is not address at this moment, as we have no indication of any usage expectations.

Moreover, this PR solves Issue #329 ..

We observed this behaviour in Kubernetes deployment as well (with attached volume, even the data is still there, the ids get overwritten), and would be more comfortable to say it is solved if we were to implement some sort of UUID-generation (some pitfalls of this explained here: https://tomharrisonjr.com/uuid-or-guid-as-primary-keys-be-careful-7b2aa3dcb439)

[blankdots]

This addresses some of the discussions we had with using different FTP servers with the inbox, related to User story: #4 - using two different types of inboxes allows us to cover a wider range of servers.

I do not believe that the #329 is addressed with this issue, as it should be investigated further.

Additionally it bumps the versions some of python dependencies and other fixes