As a start to gathering metrics for Hyperledger, we want a tool that will allow us to generate health assessment reports for each project within Hyperledger, as well as other projects that might be of interest. The reports will look at the following factors and score them on a RYG (Red-Yellow-Green) scale:
-
Level of Interest - demonstrates that people are interested in what we are doing and want to get involved
-
Conversion - demonstrates that people who are interested in getting involved are able to progress to the point of making an actual contribution
-
Retention - demonstrates that people who make one contribution are able to stay involved over time
-
Diversity - demonstrates that work is being done by a diverse set of individuals and organizations
-
Maturity - demonstrates how established a community is
In addition to the report per project, we would also like a summary report that shows the RYG state for the above factors for each of the projects. Figure 1 shows an example version of such a report (the RYG is randomized for illustration purposes).
We expect that over time the metrics that we will report on will continue to grow. As such, we may grow beyond just project reporting to a metrics portal that can be used to slice and dice the data in other ways (for example by providing a report for a Working Group or for the activities of specific Members). A metrics portal is not in the initial statement of work, but should be considered while developing the tool so that it is easy to make changes in the way the data is presented.
Also, it should be noted that the initial metrics are mostly based on source contributions. We will eventually want to gather metrics related to participation via our other tools -- mailing lists, chat, issue tracking, and wiki, for example.
Figure 1: Example of summary report 1
1RYG for illustration purposes only and not reflective of project health
The metrics surrounding level of interest demonstrate that people are interested in what we are doing and want to get involved. Initially, we will focus on how many people have expressed an interest in contributing.
- Monthly unique page views of Project’s Contributing Guide
The metrics surrounding level of interest demonstrate that people who are interested in getting involved are able to progress to the point of making an actual contribution. Initially, we will focus on showing how many people are able to make a contribution.
-
Number of new contributors in last month
-
Number of current active contributors (defined as people who have contributed within last 6 months)
-
Total number of contributors to the project over the life of the project
-
Conversion rate (new contributors / number of unique page views for contributing guide over last month)
The metrics surrounding level of interest demonstrate that people who make one contribution are able to stay involved over time.
-
Number of people who have become inactive (defined as people who made their last contribution over 6 months ago) in last month
-
Retention rate (defined as number active contributors divided by the total number of contributors)
-
Level of engagement (defined as the number of repeat contributors divided by the total number of contributors)
The metrics surrounding diversity demonstrate that work is being done by a diverse set of individuals and organizations.
-
Individual contribution diversity - are the majority of code contributions coming from a small set of individuals? Quarterly counts of contributors who are core, regular, and casual, where the classification is built by sorting contributors by their total number of commits and the total commits per project:
-
Core contributors - the individuals whose commits sum up to 80% of the total number of commits in the quarter.
-
Regular contributors - the individuals whose commits sum up to 95% of the total number of commits in the quarter.
-
Casual contributors - The individuals who are neither core nor regular committers
-
-
Organizational diversity -- are contributions coming from multiple organizations and how is participation spread across organizations?
-
Time zone diversity - are contributions coming from a single geographic region and how is participation in project calls spread across time zones?
The metrics surrounding maturity demonstrate how established a community is.
- A measure combining the above elements. For more mature projects, the level of interest is high, it is easy to begin contributing to the project, and the diversity is wide.
-
The tool must be developed as an open source tool that conforms to the intellectual property policies of the Hyperledger Charter.
-
The tool must allow new projects and their associated repositories to be added via a configuration file.
-
The tool must be able to run on an adhoc basis.
-
It must be possible for contributors who contribute under multiple aliases to link those aliases to a single entity.
-
It must be possible for contributors to specify the organization that they work for.
-
The organization specification should allow for individuals who move organizations to specify the timeframe for which they worked for a particular organization.
-
The report must contain the date that it was generated.
-
The report must contain a list of contributors who became inactive in the last month (i.e., this would be people whose last contribution was between 6 and 7 months).
-
Threshold values (as defined below) must be configurable.
-
If applicable, the report should include the date the project was accepted into incubation and the date graduated to active state.
The RYG for the level of interest factor will look at the Monthly unique page views of contributing guide and use the following ranges:
| <400 | Red |
| 401-800 | Yellow |
| >800 | Green |
The RYG for conversion factor will be based on the conversion rate calculation.
| <34% | Red |
| 34-66% | Yellow |
| >66% | Green |
The RYG for the retention factor should look at both the retention rate and the level of engagement. First we will calculate the RYG for each metric based on the 33:34:33 model.
| <34% | Red |
| 34-66% | Yellow |
| >66% | Green |
Then we need to combine these two to come up with the final RYG value. We will use the following matrix to decide:
| Retention Rate | Level of Engagement | Final RYG |
| Red | Red | Red |
| Red | Yellow | Red |
| Yellow | Red | Red |
| Red | Green | Yellow |
| Green | Red | Yellow |
| Yellow | Yellow | Yellow |
| Yellow | Green | Yellow |
| Green | Yellow | Yellow |
| Green | Green | Green |
When looking at the diversity factor, Hyperledger is first and foremost interested in the organizational diversity of our projects. As such, the initial RYG will look at the percentage of contributions that come from a single organization.
| >=50% | Red |
| >=25% | Yellow |
| <25% | Green |
When looking at the maturity factor, the RYG should be a combination of the following: the level of interest is high, it is easy to begin contributing to the project (i.e., the conversion rate is high), and the diversity is wide.