Skip to content

Commit

Permalink
cEP-0019.md: Meta-review system
Browse files Browse the repository at this point in the history 
Describes how meta-reviews could be tracked and handled.

Closes #121
  • Loading branch information
@li-boxuan
li-boxuan committed May 7, 2018
1 parent 9f4b596 commit a266680
Showing 1 changed file with 379 additions and 0 deletions.
379 changes: 379 additions & 0 deletions cEP-0019.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,379 @@
Meta-review System
==================

|Metadata| |
|--------|-----------------------------------|
|cEP |0019 |
|Version |1.0 |
|Title |Meta-review System |
|Authors |Boxuan Li <[email protected]>|
|Status |Proposed |
|Type |Process |

Abstract
--------

This cEP describes the details of the process of implementing the meta-review
system as a part of the
[GSoC'18 project](https://summerofcode.withgoogle.com/projects/#5188493739819008).

Background
----------

People including the author of pull request respond to comments by attaching
emojis. Those emojis are called
[reactions](https://developer.github.com/v4/enum/reactioncontent/). THUMBS_UP
and THUMBS_DOWN, together with other reactions, are used by people especially
the author of pull request to publicly acknowledge or reject review comments.
This provides feedback to reviewers and all other members.

There are three types of reviews on a pull request:

1. Issue Comment
[sample comment](https://github.com/coala/gh-board/pull/18#issuecomment-385878735)

2. Review Summary
[sample summary](https://github.com/coala/gh-board/pull/18#pullrequestreview-116783753)

3. Review Discussion
[sample discussion](https://github.com/coala/gh-board/pull/18#discussion_r186249423)

However, those reactions spread over all PRs and are not collected and
analyzed. Moreover, GitHub doesn't send notifications to reviewers who receive
reactions. By tracking reactions, a nice feedback loop would be created
for both senior and junior reviewers.

Introduction
------------

The meta-review project is about making use of reactions to build a
meta-review system, similar to a meta-moderation system. These responses
are to be collected, processed and displayed on
[gh-board](https://github.com/coala/gh-board), which is a nice serverless
kanban board.

The scope of this project would exclude non-public meta-reviews. That is,
there would not be any anonymous meta-reviews as that introduces too many
additional complexities. The public only version would be used for
a long time before adding support for non-public meta-reviews.

Note there is a technical limitation in that reactions can not be placed
onto the "review summary", which means general review comments should not
be put into the "review summary" text box, as it will be not able to be
meta-reviewed. Instead, put general review comment as a comment like
[this](https://github.com/coala/gh-board/pull/18#issuecomment-385925740).
Only put comments on "review summary" if it is indeed a summary of "review
discussions".

Objectives
----------

One goal of this project is to encourage people to do meta-reviews,
systematically.

Also, to encourage people to do more meta-reviews, statistics of meta-reviews
are to be collected and analysed. People who do meta-reviews will get scores
according to some metrics. A ranking list is to be displayed on gh-board.

To conclude, this project builds a meta-review system on gh-board to display
statistics (score ranking, meta-reviews in need), and encourage people to
do meta-review.

Proposed Approach
-----------------

### Collection of Meta-reviews

Meta-review info, namely reactions, can be fetched via GitHub API V4 GraphQL.
A sample query is as follows:

```GraphQL
{
repository(owner: "coala", name: "gh-board") {
pullRequest(number: 18) {
author {
login
url
}
reviews(first: 5) {
nodes {
bodyText
comments(first: 1) {
nodes {
bodyText
author {
login
url
}
reactions(first: 100) {
totalCount
nodes {
content
user {
login
}
}
}
}
}
}
}
comments(first: 1) {
nodes {
bodyText
author {
login
url
}
reactions(first: 100) {
totalCount
nodes {
content
user {
login
}
}
}
}
}
}
}
}
```

The response is in JSON format as follows:
```json
{
"data": {
"repository": {
"pullRequest": {
"author": {
"login": "li-boxuan",
"url": "https://github.com/li-boxuan"
},
"reviews": {
"nodes": [
{
"bodyText": "This is looking quite good, but closed issues/prs is empty:\nhttps://deploy-preview-18--coala-gh-board.netlify.com/#/r/coala:coala?s=closed\nmeta-review will need to be able to see closed PRs",
"comments": {
"nodes": []
}
},
{
"bodyText": "",
"comments": {
"nodes": [
{
"bodyText": "Leftover?",
"author": {
"login": "blazeu",
"url": "https://github.com/blazeu"
},
"reactions": {
"totalCount": 0,
"nodes": []
}
}
]
}
},
{
"bodyText": "",
"comments": {
"nodes": [
{
"bodyText": "Yup, thanks.",
"author": {
"login": "li-boxuan",
"url": "https://github.com/li-boxuan"
},
"reactions": {
"totalCount": 0,
"nodes": []
}
}
]
}
},
{
"bodyText": "",
"comments": {
"nodes": [
{
"bodyText": "You made the your own getTime function below.",
"author": {
"login": "blazeu",
"url": "https://github.com/blazeu"
},
"reactions": {
"totalCount": 1,
"nodes": [
{
"content": "LAUGH",
"user": {
"login": "li-boxuan"
}
}
]
}
}
]
}
},
{
"bodyText": "",
"comments": {
"nodes": [
{
"bodyText": "Can be turned into arrow function and put at the very top.",
"author": {
"login": "blazeu",
"url": "https://github.com/blazeu"
},
"reactions": {
"totalCount": 1,
"nodes": [
{
"content": "THUMBS_UP",
"user": {
"login": "li-boxuan"
}
}
]
}
}
]
}
}
]
},
"comments": {
"nodes": [
{
"bodyText": "To be honest, I don't understand why we need fetch-issues.js. From the relevant code, I can see fetch-issues.js does some pre-fetching and initial-data.js dumps it into the database. But what's the aim of doing that? It doesn't seem to save API hits. These pre-fetched issues in database are replaced later anyway.",
"author": {
"login": "li-boxuan",
"url": "https://github.com/li-boxuan"
},
"reactions": {
"totalCount": 0,
"nodes": []
}
}
]
}
}
}
}
}
```

The collection of meta-reviews includes two parts:

##### Pre-fetch meta-reviews

Currently, `fetch-issues.js` pre-fetches issues and pull requests to speed up
gh-board performance and save users' API hits. Meta-reviews are also to be
fetched during this procedure.

##### Fetch meta-reviews

When a new issue/pull request is detected, gh-board will fetch that issue and
loads it asynchronously. Meta-reviews also need to be fetched during this
procedure.

### Analysis of Meta-reviews

This section explains how meta-review information can be utilized to
generate a meaningful analysis.

As the collection of meta-reviews, this part also includes two parts:

##### Pre-process meta-reviews

After pre-fetching meta-reviews, they are to be processed, which is explained
in the following sections.

##### Process meta-reviews

As new meta-reviews are loaded, useful information is to be extracted and
current analysis of meta-review is to be updated correspondingly.

The following sections explain how meta-review information is used to
generate a meaningful analysis.

#### Ranking list

A ranking list would be displayed on a separate column on gh-board, according
to points people get. A sample formula for points would be as following:

PR = total number of positive reactions (e.g., THUMBS_UP) a person gets
for all reviews he did.
NR = total number of negative reactions (e.g., THUMBS_DOWN) a person gets
for all reviews he did.
PG = total number of positive reactions a person gives to other people for
their reviews.
NG = total number of negative reactions a person gives to other people for
their reviews.

Note PG and NG are used to encourage people to get involved in this
reactions feedback system. Also, it may not be limited to PRs. Then final
score S would be:

S = PR - NR + c1 * PG + c2 * NG

where c1 = 0.5, c2 = 0.8, are coefficients. c2 is slightly larger than c1
because people are usually reluctant to give negative reactions.

If a person has zeros for all factors, then he is regarded as an inactive
reviewer and would not be displayed on board.

#### Meta-reviews in Need

Review comments that need meta-review will be displayed on a separate column
on gh-board. There are three approaches:

1. The worst case is that the tracking stores in the json which
PR do not have meta-reviews, and gh-board shows only a list of those PRs
(and the user must then find the reviews in GitHub PR web app, and do
meta-review there).

2. An improvement is to store the identifier of the PR comments which
do not have meta-reviews and gh-board links to the PR review comment
in GitHub PR web app.

3. In the ultimate version, gh-board would show the review comments.
In the future, people could do meta-reviews directly on gh-board. This also
provides an approach to doing meta-reviews anonymously.

#### Other Features

The GSoC project provides a blueprint for future enhancement. Additional
features could be implemented to make the meta-review system more
sophisticated:

1. A basic ranking list will be built first. In the best case, we should
offer an overall ranking list as well as a recent ranking list (user
could switch to see overall/recent ranking) with sorting option.

2. Offer an option for users to do meta-reviews directly on gh-board. This
might be a bad idea - people shouldn't do meta-review unless they know the
whole PR context. This might be a good idea - in the future users can do
meta-reviews on gh-board anonymously (there is no way for people to do
anonymous meta-reviews on GitHub page).

3. Display visitor’s own meta-reviews if the visitor logs in. People may want
to see meta-reviews they have done before.

4. Display visitor's received meta-reviews if the visitor logs in.
GitHub won't send a notification to a user if he receives any meta-review.
Suppose I ask for someone's agreement and he uses 👍 to show his agreement.
But I wouldn't receive any notification. I have to check that page regularly.
Meta-review system is a nice place to see the collection of all meta-reviews
I receive.

#### Documentation including short videos

Meta-review system sounds like something of a novelty. Documentation is needed
to clarify the use of the system. One or more short videos are to be published
to help people get acquainted with the system.

0 comments on commit a266680

Please sign in to comment.