mod-disco enrollment A/B testing ( Ready for review )

[joe-getcouragenow]

Its well known that to know if something is effective with users you do A/B testing.

Google pioneered it publicly in IT.
https://support.google.com/optimize/answer/6211930?hl=en

https://www.monsterinsights.com/how-to-ab-test-signup-forms/

They are just copying from what Psychologists and Medical industry do to check on the efficacy of something.

https://en.wikipedia.org/wiki/Blinded_experiment

To know if the enrollment system works A/B testing should be done such that a % of the Orgs run with and without enrollment, and the results will be plainly evident in the Analytics.

It is also well know that long signup deter use signups

https://seo-hacker.com/user-registration-affects-business/

https://productled.com/user-adoption-strategies/

Outcomes are what matter

Funding from MOT in the EU, will be much more likely if you do this because:

• You showed what works and not. MOT is interested in the outcome towards tipping the scale of Climate.
• MOT is interested in what works and personal theories people have.
• you did things in an unbiased way. Its builds trust too.

Adoption from other Orgs will be higher if you show your are doing A/B testing and its results.
CTO's know it means a system has been done well.

The Testing

A

Same as now where we force a long signup process

B

Single page and then SignUp, once inside they have a button to do or redo their User supply / demand long Q and A.
Ironically, you need this button anyway, so that they can change their User supply / demand options anyway.

Design

The config of mod-disco feature flag called "enrollment". Its a boolean of yes or no.

If flag is off, the first page is the same and the "Next" button takes them to SignUp / SignIn.

The project metrics ( 3 metrics ) will still work.
The user inputted supply / demand data will not be captured, and so the Project Admin dashboard wont show any filters.

The metrics need to record a users feature flags so that you can see if they are in A or B. Thats vital.

Code

1. Add the flag
2. Modify the Flutter GUi to check for this config. It should be supplied over the GRPC API

• We have long spoken about the likely need for feature flags and this is the perfect example of it, so it would make sense to formalise the pattern into the system maybe.

3. Modify the DAL and tests to check that when no user inputted supply / demand data still keeps the thing working.
4. Check that the Project Admin Dashboard works when no no user inputted supply / demand data the thing keeps working.
5. Check that the Analytics backend works when no user inputted supply / demand data the thing keeps working.

Secondary aspect

Proper Feature flags system

Just like all sub systems we design, the feature flags system is needed by all Modules and so we can easily formalise into the system architecture as a DB, just like we are doing with Search, Analytics, etc. Its a type of MicroService essentially.

Its Data Schema would copy Chromes and be a Name and Boolean. Making it also model other value types is probably not needed and just makes the thing confusing, and for V1 Boolean will be enough for a long time anyway.

The GRPC API would reflect the Data Schema with a KV store like API.

• Because the data changes alot it should provide an event API also so that on a change the App reacts in real time.

The CLi would inherit the GRPC APi like all our architecture.

A Flutter GUI could easily be added. It might be worth it too. Not sure.

There are open golang Feature flag systems out there and listed here: https://featureflags.io/go-feature-flags/

https://github.com/thomaspoignant/go-feature-flag

• veery clean and matches our architectural self running needs.
• S3 APi already.
• To be real time, it uses polling. Since its S3 this is a reasonable solution.
• We should wrap it with a GRPC API though because we get so many things. Encapsulation, CLI for free, FLutter code for free, JSONNET configuration management for free, etc.
• It uses YAMl as the file format which makes sense, and so the GRPC wrapper DAL on the Server just needs to parse a YAML and convert it to the GRPC model. Easy peasy for Read and write.
• Search ?

Make it possible for us to control which Orgs have this feature flag

In order to really make it double blind, the Org Admin should not know.
If the Feature flags system is controlled by us, we can do this.

To have this aspect to the Design, an S3 storage is perfect.
The Lib that allows a S3 Store to be a GCS or a Badger Db is again the perfect solution for us ( https://github.com/creachadair/badgerstore and https://github.com/creachadair/gcsstore )

So then just use the shared/servers/s3/google and shared/servers/s3/badger lib we are already developing here: #119