New default analytics capability to replace Insights

[e0d]

Opportunity Objective
Development efforts on Insights and the analytics pipeline have stalled due to the complexity of contributions to this system, and a shift in how companies are processing big data, including EdX. EdX's analytics needs have always been quite divergent from the community analytics needs; for example, edx.org hosts courses for many partners, and most Open edX deployments have only a single organization.

The Open edX community wants an analytics solution that is quick to install, cheap to run, scales to large deployments, and which can be customized to meet their individual reporting needs.

Market Evidence
OpenCraft regularly hears from our clients that Insights does not meet their needs or is too expensive, and many end up creating their own limited in-house analytics pipeline.

The edx-analytics-pipeline is based on the ETL model (Extract>Transform>Load), and adding new reports is very difficult. But the market has moved toward ELT: Extract raw data, Load it into a data warehouse, and Transform it as needed to run reports. edX have been using an open-core ELT tool called dbt, which made their analytics better, more flexible, and much easier to code.

Proposal Specifics

• Build a new open source Open edX Data Warehouse built on dbt.
• Allow authorization via OAuth2 from Open edX, and allow access to reporting data to be controlled by organization, course/learning stream.
• Build minimal basic summary reports, and provide documentation on how to add custom reports, and deploy the system.

Success Measures
This project is a success when community operators can deploy the new analytics solution easily and affordably, and users of the system can create custom reports from their learner data.

[Source: https://openedx.atlassian.net/browse/OEROADMAP-17]

[gabrieldamours]

@e0d -- is this proposal based on @pomegranited 's submission from 2020?

[pomegranited]

@gabrieldamours Yep -- he linked to it above.

But point of order: this came from @bradenmacdonald 's writeup, I just posted it :)

[e0d]

I'll also mention this is a very common proposal across the community, so there are numerous sources of input here.

[jmakowski1123]

done!