End-to-End Encryption of data

[leadVisionary]

Given HIPAA regulations and my overall affinity for privacy, I want Vitzalizr to provide end-to-end encryption for data at rest and in motion.

In transit, obvious choices like TLS are available. Managing certificates and keychains is kind of painful, but the alternative would be to design a novel encryption approach, which obviously comes with its own design and implementation issues.

At rest is already doable, in principle, with some baked out pieces like password-based encryption, which, while not as strong or preferrable as asymmetric PKI, does have better ease of use.

The question is, how do I want to implement at-rest encryption?

One option is to have the file serialization directly interwoven with something like a WriteObjectAsEncrypted in the serialization package or implementing VitalAsEncryptedString to replace VitalAsGZipString. I kind of hate that, because it pushes knowledge of authentication --authN-- and authorization --authZ-- into the core logic package directly and adds more complexity to the overall Vitalizr codebase. On the plus side, it's fairly easy/straight-forward.

Another option is closer to where I want to go, ripping serialization as a responsibility out of Vitalizr and making another "service" to handle long-term persistence. This service would have to be a bridge between the serialization API and the vitalizr API that would periodically poll the in-memory vitals, turn them into a serializable form, and encrypt them along the way. I like that because it decouples responsibilities-- saving data to disk/a network location is different from the "business logic" of understanding what Vitals are and how to manipulate them. It allows me to vary serialization formats or storage locations fairly seamlessly, as all that gets encapsulated in an external service. The problem is, I add a process/potential network hop for something that could be a library call. That potentially introduces latency and availability problems. It also leaves an open question of how exactly I query and save.

One way is "long-polling", I can set some configurable window--like 5 minutes-- and have a little worker thread wake up, ListVitals for an Identifiable, and serialize them, encrypting along the way. That moves the encryption responsibility into the serialization service...

...but we can go even crazier. We could have the serialization service save its data and then have an encryption service that watches a FileSystem location with a Watcher. This is super decoupled and sexy, though the different service pitfalls apply, and there may also be some small gap of time where data is unencrypted between polls.

A final option is Eventing.

[leadVisionary]

What I like about Eventing is that it completely decouples serialization/long-term persistence from business logic AND moves towards the type of architecture I want in the End Game: a "service mesh" with processes devoted to logging, instrumentation, load-balancing communicating via an Event Bus/Enterprise Service Bus.

The downside is that it's easy for that work to turn into over-engineering, solving problems like "retry storms"/"back pressure"/"circuit breaking" and force program modeling towards more of a CSP style that looks and feels more like digital synthesis of hardware than traditional imperative/procedural programming, which makes it easier to run into a bug and get stuck because your brain flips back to the more basic model.