cEP-0020: Newcomer metrics and Gamification system #150
Conversation
cEP-0020.md
Outdated
| - 5 to 10 | ||
| - 10 and beyond | ||
| - Following are the types of comments that could be tracked: | ||
| - Issue comments |
There was a problem hiding this comment.
The text does not comply to the set style.
Origin: MarkdownBear, Section: markdown.
The issue can be fixed by applying the following patch:
--- a/tmp/tmpmbc69dpu/cEP-0020.md
+++ b/tmp/tmpmbc69dpu/cEP-0020.md
@@ -101,8 +101,8 @@
- 5 to 10
- 10 and beyond
- Following are the types of comments that could be tracked:
- - Issue comments
- - Review comments
+ - Issue comments
+ - Review comments
### Points
cEP-0020.md
Outdated
| - 5 to 10 | ||
| - 10 and beyond | ||
| - Following are the types of comments that could be tracked: | ||
| - Issue comments |
There was a problem hiding this comment.
The text does not comply to the set style.
Origin: MarkdownBear, Section: markdown.
The issue can be fixed by applying the following patch:
--- a/tmp/tmpmbc69dpu/cEP-0020.md
+++ b/tmp/tmpmbc69dpu/cEP-0020.md
@@ -101,8 +101,8 @@
- 5 to 10
- 10 and beyond
- Following are the types of comments that could be tracked:
- - Issue comments
- - Review comments
+ - Issue comments
+ - Review comments
### Points
cEP-0020.md
Outdated
| Let's start by defining some activities which could be tracked: | ||
|
|
||
| - Create Issues | ||
| - Labels? |
There was a problem hiding this comment.
? adds confusion.
It is important that we understand what will be tracked, and how it will be modeled, otherwise building it is going to be messy.
Below you refer to activities having 'factors'. Do activities only have 'factors', or do they have other attributes also? What datatypes would factors have? You have strings and ints here. How will factors with different data types be serialised?
If you are going to discuss activity attributes below, remove the ? here.
Another way would be to have complete activities like
- Create a Bug Report
- Create a Feature Request
Then how the system loads issues and determines that they have created a bug report or feature request is an implementation detail and might be defined later in the cEP, and can involve rules, lots of rules.
I am not sure about 'factors' as they attempt to describe what parts of an activity are relevant to gamification decision making, and I suspect that we will need lots of rules that become quite complex.
e.g. what happens if the issue labeled 'bug' is closed as a duplicate, or the label 'bug' is removed, or a repo has multiple labels which imply bug.
I am fairly confident that the process of awarding a badge atm requires a manual & meticulous check of all activities which contribute to the badge. The system might show that they have finished the necessary activities for a badge, but it isnt yet checked.
There was a problem hiding this comment.
I am fairly confident that the process of awarding a badge atm requires a manual & meticulous check of all activities which contribute to the badge. The system might show that they have finished the necessary activities for a badge, but it isnt yet checked.
Maybe give this privileges to developers?
To award badges to the newcomers
But I was thinking it to be automated.
Suppose that we a newcomer opened a bear proposal issues.
We could check the label of the issue and award him the bear hunter badge automatically.
But in case a newcomer opened a type/bug issues we may fail to check if its correct as gitmate label most of the issues as bug which are not.
There was a problem hiding this comment.
gitmate label most of the issues as a bug which is not.
I think this will be a big problem for us if we left labeling of issues on gitmate.
As we can never be sure if the label is correct or not.
Maybe we wait for giving the points until the issues is labeled by the maintainer?
But IGitt doesn't provide this functionality, i.e.: who added this label?, it just retrieve all the labels
There was a problem hiding this comment.
Maybe give this privileges to developers?
;-) beyond scope of your project :P
There was a problem hiding this comment.
It does need to be automated, in order to be scalable.
But as you can see we sometimes dont have quite enough information to be confident that an action has been correctly performed correctly.
Sometimes we do. e.g. the data coming from the meta-review system will have had verification already done.
We dont have a scalability problem for the 'coala Developer' badge -- it is worthwhile carefully checking the persons work before giving them the final 'coala Developer' badge.
I guess this can be solved by the same mechanism mentioned before : the implementation of the privileges is beyond scope, and we highlight that other systems implementing these privileges should use manual verification for any privilege which has a high risk profile.
Specifically, the addition of a person to the 'coala developers' team in GitHub, which provides them with new privileges, should not be automated based on the 'coala Developer' badge.
So the definition of a privilege should indicate whether it is
- not implemented
- manual activation
- automated activation
Ideally all privileges would go through those status in order, as the privilege will exist before it is implemented, and it should be run in manual mode first, and then automated only if there is no serious risk or there is a good risk reaction mechanism in place.
If manual, it is shown in a pending queue to be processed.
Even if manual, we still would often have the granting process here to be automated.
e.g. the coala Developer badge is automatically granted when the username appears in the 'coala developers' team in GitHub. The 'manual' step is a maintainer adds the user to the 'coala developers' team in GitHub.
cEP-0020.md
Outdated
|
|
||
| - Create Issues | ||
| - Labels? | ||
| - No. of comments? |
There was a problem hiding this comment.
Not sure I get this. You have 'Comments'->'Type?' below.
But how will number of comments be useful for issues? The number of comments doesnt tell me much, I think.
There was a problem hiding this comment.
The number of comments doesnt tell me much, I think.
Yeah, this is not a good idea, I was thinking to give some points to the newcomer who has opened the issue and gets lots of discussions on that.
Inspired by GitHub profile which shows "You opened an issue which receive n comments"
cEP-0020.md
Outdated
| - No. of comments? | ||
| - Create Pull Requests | ||
| - Label? | ||
| - CI status? |
There was a problem hiding this comment.
Thinking out loud ...
- Create a Pull Request on GitHub
- Create a Pull Request on GitLab
- Create a Pull Request which passes Travis CI
- Create a Pull Request which passes GitLab CI
- Create a Pull Request which passes Circle CI
- Create a Pull Request which passes GitMate
- Commit that fixes a bug.
- Commit that is an enhancement.
- Commit that is newcomer difficulty
- Commit that is low difficulty
- Commit that is medium difficulty
- Commit that includes a new large markdown file
cEP-0020.md
Outdated
| - 5 to 10 | ||
| - 10 and beyond | ||
| - Following are the types of comments that could be tracked: | ||
| - Issue comments |
There was a problem hiding this comment.
'any comment' doesnt feel like an activity we want to award points to, as then we need lots of negatives to prevent bad comments.
What types of issue comments do we want, and can accurately without meta-review?
However 'Comment on an issue' can be an activity without any points. It could be quite useful to clearly show which activities we give 0 points for, so people dont waste their time doing those types of activities.
In fact, (im not so sure about this) we may want to give all comments a preset negative points by default, because any good comment will also get a positive count somewhere else.
cEP-0020.md
Outdated
| - Name: Intermideate | ||
| - Points_required = 150 | ||
| - Permission to use corobo basic functionalities | ||
| - Invite newcomers to org |
There was a problem hiding this comment.
this isnt a basic corobo functionality. this is current maintainers only. We may make it developers only, or a really high number of points, but not level II.
look through corobo plugins for other privileges in the rules.
But remove all mention of corobo , and add a note that implementing these privileges is beyond the scope the gamification system. The gamification system will merely expose a list of privileges that each person has, and other systems are expected to adapt to use these privileges.
cEP-0020.md
Outdated
| - Points_required = 150 | ||
| - Permission to use corobo basic functionalities | ||
| - Invite newcomers to org | ||
| - Get Assigned to the issues |
There was a problem hiding this comment.
we have rules around being assigned to different difficulty level issues.
each could be a new permission.
cEP-0020.md
Outdated
| - Points_required = 300 | ||
| - Permission to use corobo advanced functionalities | ||
| - Assign newcomers to issues | ||
| - Mark PRs as wip and pending review |
There was a problem hiding this comment.
there is a basic privilege of "Mark your own PR as wip and pending review"
and then a more advanced one for "Mark a PR by another newcomer as wip"
and then more advanced again for "Mark a PR by a developer as wip"
and then "Mark a PR by a maintainer as wip"
cEP-0020.md
Outdated
| - 2 points per issue comments | ||
| - 3 points per review comments | ||
| - Gitter Messages | ||
| - No points for Bad messages |
There was a problem hiding this comment.
we need negative points. that is one of the primary goals of this project from a maintainer perspective -- we need to be able to describe & strongly discourage abusive behaviour, where 'abusive' means 'wastes maintainer time'.
cEP-0020.md
Outdated
| - Level-I: | ||
| - Name: Beginner | ||
| - Points_required = 50 | ||
| - Persmission to ping maintainers on Gitter |
cEP-0020.md
Outdated
| - Points_required = 50 | ||
| - Persmission to ping maintainers on Gitter | ||
| - Level-II: | ||
| - Name: Intermideate |
cEP-0020.md
Outdated
| gamification system which will allow us to track newcomer's progress and | ||
| assign some points to each of the activity they do. There will be different levels | ||
| based on points threshold and a newcomer will also get some special privileges | ||
| after the compeletion of each level. There will also be different badges on the |
cEP-0020.md
Outdated
| - Good? | ||
| - Bad? | ||
|
|
||
| The above activities will be tracked seperately based on their category. i.e.: |
cEP-0020.md
Outdated
| - Increase in 2 points with the increase in no. of comments | ||
| - Create Pull Requests | ||
| - If CI passes and There are no Gitmate errors | ||
| - 15 points for intial and then increase |
There was a problem hiding this comment.
spelling mistake (same at L121 & L123)
cEP-0020.md
Outdated
|
|
||
| - A newcomer will not be able to do fake reviews as review comments | ||
| data will be imported to community model from | ||
| [meta-reiview project](http://projects.coala.io/#/projects?project=meta-review&lang=en) which already deals with fake reviews. |
There was a problem hiding this comment.
Spelling. Also, could you explain fake reviews? Useless/trivial reviews?
cEP-0020.md
Outdated
| - All the points given for that pull request will be reset | ||
| - Review comments on the pull request | ||
| - 2 points per good review comments | ||
| - -2 points per bad review comments |
There was a problem hiding this comment.
The text does not comply to the set style.
Origin: MarkdownBear, Section: markdown.
The issue can be fixed by applying the following patch:
--- a/tmp/tmpitpwcw2p/cEP-0020.md
+++ b/tmp/tmpitpwcw2p/cEP-0020.md
@@ -121,9 +121,9 @@
- All the points given for that pull request will be reset
- Review comments on the pull request
- 2 points per good review comments
- - -2 points per bad review comments
+ - \-2 points per bad review comments
- Gitter Messages
- - -1 per Bad messages
+ - \-1 per Bad messages
- 1 points per Good message
- Introduce `.coafile` in other projects
- 10 points
cEP-0020.md
Outdated
| def add_points() | ||
| a_method_to_update_points_and_level() | ||
| a_method_for_the_log_of_the_acitities() | ||
|
|
There was a problem hiding this comment.
The text does not comply to the set style.
Origin: MarkdownBear, Section: markdown.
The issue can be fixed by applying the following patch:
--- a/tmp/tmpitpwcw2p/cEP-0020.md
+++ b/tmp/tmpitpwcw2p/cEP-0020.md
@@ -331,7 +331,6 @@
def add_points()
a_method_to_update_points_and_level()
a_method_for_the_log_of_the_acitities()
-Creating the Badges model
cEP-0020.md
Outdated
| - All the points given for that pull request will be reset | ||
| - Review comments on the pull request | ||
| - 2 points per good review comments | ||
| - -2 points per bad review comments |
There was a problem hiding this comment.
The text does not comply to the set style.
Origin: MarkdownBear, Section: markdown.
The issue can be fixed by applying the following patch:
--- a/tmp/tmpitpwcw2p/cEP-0020.md
+++ b/tmp/tmpitpwcw2p/cEP-0020.md
@@ -121,9 +121,9 @@
- All the points given for that pull request will be reset
- Review comments on the pull request
- 2 points per good review comments
- - -2 points per bad review comments
+ - \-2 points per bad review comments
- Gitter Messages
- - -1 per Bad messages
+ - \-1 per Bad messages
- 1 points per Good message
- Introduce `.coafile` in other projects
- 10 points
cEP-0020.md
Outdated
| def add_points() | ||
| a_method_to_update_points_and_level() | ||
| a_method_for_the_log_of_the_acitities() | ||
|
|
There was a problem hiding this comment.
The text does not comply to the set style.
Origin: MarkdownBear, Section: markdown.
The issue can be fixed by applying the following patch:
--- a/tmp/tmpitpwcw2p/cEP-0020.md
+++ b/tmp/tmpitpwcw2p/cEP-0020.md
@@ -331,7 +331,6 @@
def add_points()
a_method_to_update_points_and_level()
a_method_for_the_log_of_the_acitities()
-Creating the Badges model
cEP-0020.md
Outdated
| - All the points given for that pull request will be reset | ||
| - Review comments on the pull request | ||
| - 2 points per good review comments | ||
| - -2 points per bad review comments |
There was a problem hiding this comment.
The text does not comply to the set style.
Origin: MarkdownBear, Section: markdown.
The issue can be fixed by applying the following patch:
--- a/tmp/tmpitpwcw2p/cEP-0020.md
+++ b/tmp/tmpitpwcw2p/cEP-0020.md
@@ -121,9 +121,9 @@
- All the points given for that pull request will be reset
- Review comments on the pull request
- 2 points per good review comments
- - -2 points per bad review comments
+ - \-2 points per bad review comments
- Gitter Messages
- - -1 per Bad messages
+ - \-1 per Bad messages
- 1 points per Good message
- Introduce `.coafile` in other projects
- 10 points
cEP-0020.md
Outdated
| def add_points() | ||
| a_method_to_update_points_and_level() | ||
| a_method_for_the_log_of_the_acitities() | ||
|
|
There was a problem hiding this comment.
The text does not comply to the set style.
Origin: MarkdownBear, Section: markdown.
The issue can be fixed by applying the following patch:
--- a/tmp/tmpitpwcw2p/cEP-0020.md
+++ b/tmp/tmpitpwcw2p/cEP-0020.md
@@ -331,7 +331,6 @@
def add_points()
a_method_to_update_points_and_level()
a_method_for_the_log_of_the_acitities()
-Creating the Badges model
cEP-0020.md
Outdated
| **The problems we have:** | ||
|
|
||
| - How to attract more newcomers | ||
| - How to cope with the increasing amounts of newcomers |
There was a problem hiding this comment.
Is the relationship between these two problems parallel or progressive?
There was a problem hiding this comment.
I think its parallel; as the number of newcomers increases, we have to find out an easy way to cope with them.
cEP-0020.md
Outdated
| we can keep working on coala at a high pace. | ||
| - How to improve newcomer -> dev/maintainer ratio | ||
| - How to increase amount of feedback we get | ||
| - How to use feedback to improve coala and processes |
There was a problem hiding this comment.
Same here. Could be just me, but indentation here looks a bit weird.
There was a problem hiding this comment.
nvm; I will revised this section 👍
cEP-0020.md
Outdated
| - Invite newcomers to org | ||
| - Assign newcomers to difficulty medium issue | ||
|
|
||
| #### Note |
There was a problem hiding this comment.
I think this title is unnecessary.
cEP-0020.md
Outdated
| - Name: The coala Developer | ||
| - Details: The ones who are in the coala developers team | ||
| - Activities: | ||
| - Introduce `.coafile` in other projects |
There was a problem hiding this comment.
I see that http://api.coala.io/en/latest/Developers/Newcomers_Guide.html says run coala on a project of yours. Is there any reason for the change here?
Also Introduce .coafile sounds a bit informal/not quite clear.
cEP-0020.md
Outdated
| is not able to become a developer at coala, he/she could take away the other | ||
| badges. | ||
|
|
||
| #### Note |
There was a problem hiding this comment.
I think the title here is unnecessary.
cEP-0020.md
Outdated
| A mockup of a basic `Activity` Django model would look something like: | ||
|
|
||
| ```python | ||
| class Actvity(models.Model): |
There was a problem hiding this comment.
What if someone has done multiple activities of the same type? For example, how do you represent "Bob has created two newcomer issues"?
There was a problem hiding this comment.
Nice pick 👍
I think I need to have a field named activities in the User model which will have a ManyToMany relationship with Activity model(which will also contains a field for number_of_times_that_activity_has_been_done and default is set to 0) and a method named count_similar_activity which will be a counter to check if that activity has been done previously by the same user, if yes I will increase number_of_times_that_activity_has_been_done with one .
cEP-0020.md
Outdated
| >>>privilege = Privilege.objects.create(name='Get assigned to a newcomer issue') | ||
| >>>level1 = Level.obejects.create(name='Newbie', | ||
| points_required=5, | ||
| privileges = privilege |
There was a problem hiding this comment.
Redundant spaces before and after =
cEP-0020.md
Outdated
| These messages can be analysed and categorized with the use of | ||
| [basic sentiment analysis](http://fjavieralba.com/basic-sentiment-analysis-with-python.html) | ||
|
|
||
| ### A System for Tracking Activities, Points and Levels |
There was a problem hiding this comment.
How does this system work in general? Does it fetch data periodically (cron job maybe), process the data and display them somewhere (dashboard?)?
There was a problem hiding this comment.
Yeah, There will be Django management commands which will be triggered on every build on netlify.
display them somewhere (dashboard?)?
Yeah, there will also be a user dashboard which will show stats and metrics of each user.
cEP-0020.md
Outdated
| #### Note | ||
|
|
||
| Names and metrics of tracked activities will also be shown on the user | ||
| dashboard along with the number of times they have been done. That will |
There was a problem hiding this comment.
What's user dashboard? Maybe you need another section/paragraph to explain it a bit.
|
Ping @prnvdixit, @andrewda, please review :) |
cEP-0020.md
Outdated
| and how to cope with the increasing amounts of newcomers | ||
| - How to teach and help newcomers to learn faster so | ||
| we can keep working on coala at a high pace. | ||
| - How to improve newcomer -> dev/maintainer ratio |
There was a problem hiding this comment.
The way this specific bullet is written isnt appropriate for a cEP.
What does it mean?
Use prose.
cEP-0020.md
Outdated
| **The problems we have:** | ||
|
|
||
| - How to attract more newcomers | ||
| and how to cope with the increasing amounts of newcomers |
There was a problem hiding this comment.
why "and" here? why not use a new bullet point?
why all these 'how' questions that end with a full stop '.' ?
cEP-0020.md
Outdated
| - Messages on Gitter channels | ||
| - Good messages | ||
| - Bad messages | ||
| - Introduce `.coafile` in other projects |
There was a problem hiding this comment.
this can be split into several activities
- "introduce .coafile in the users own repository on GitHub"
- "introduce .coafile in the users own repository on GitLab"
- "introduce .coafile in an organisation with 50 or more stars for its repositories"
- "introduce CI in the users own repository to run coala using Travis CI"
- "introduce CI in the users own repository to run coala using ..."
- etc
cEP-0020.md
Outdated
| - Gitmate errors on pull request | ||
| - CI status of the pull request | ||
| - Review comments on pull request | ||
| - Messages on Gitter channels |
There was a problem hiding this comment.
all of the above are complete defined activities, but here you group them, and group them very broadly into good / bad.
At least get rid of this grouping, and use "Send a good message to coala community on Gitter" and the same for bad message.
But I think it would be better to break this down into discrete useful activities, like 'Say "hello world" on Gitter' , "asking to be assigned to an issue on gitter" , using coala gitter rooms other than the main "coala/coala".
You can also add "use corobo on Gitter (currently disabled due to ....)"
cEP-0020.md
Outdated
| to each of the tracked activities mentioned above: | ||
|
|
||
| - Create Issue | ||
| - Points will only be given when the issue has an extra label `process/verified` |
There was a problem hiding this comment.
how are you going to know that a maintainer added label process/verified -- does IGitt support that?
fwiw, this cEP will not be accepted while it creates more work for maintainers.
you need to work harder to find better ways of determining whether the issue is good enough to be counted.
There was a problem hiding this comment.
How about tracking Thumbs Up reactions on the issue.
Making a rule like after this number of thumbs up the issue is good enough to be counted.
This way developers can also participate in the issues and that will minimize the work for maintainers.
So, a newcomer has to ask developers to verify the issue by giving them the thumbs up, or it will be developers job to verify new opening issues.
There was a problem hiding this comment.
Normally, if an issue has been open for several days, without labels like question, needs info, we could consider that as a "verified" issue IMO.
There was a problem hiding this comment.
The main problem is when a new issue is opened, gitmate label it, and it may be wrong and it is wrong in most of the cases.
And I am giving points based on labels like difficulty/low issue will earn the more number of points than difficulty/newcomer issue.
So, with verifying the validity of the issues we also have to verify if the labels on the issues are correct or not.
There was a problem hiding this comment.
Maintainers will update the label if it was inappropriate. Maybe your system needs to detect that (periodically) and update the database accordingly.
There was a problem hiding this comment.
Why should we give points until we are not sure if it is varified or not? And Is it good to first give someone points then deduce it, for our bot mistakes? That would not be a very good gamification design IMO.
Maintainers will update the label if it was inappropriate
That will again create more works for maintainers
fwiw, this cEP will not be accepted while it creates more work for maintainers
@li-boxuan, what do you think about (#150 (comment)) ?
Let the first developers do their work. And if the levels seems inappropriate to them, they can ask maintainers to label it, And only then they give the thumbs up.
This way a maintainer will not have to look at every new opening issue.
cEP-0020.md
Outdated
| and that label can only be added by a maintainer. | ||
| - 20 points for type/bug | ||
| - 20 points for bear proposal | ||
| - 10 points for newcomer difficulty and then increase of 10 points with the |
There was a problem hiding this comment.
I do not understand this bullet. if you are talking about different points for different issues, you should have different bullet points for each.
dont be afraid of being repetitive , if it makes it more clear.
cEP-0020.md
Outdated
| - If the pull request is closed without merged | ||
| - All the points given for that pull request will be reset | ||
| - Review comments on the pull request | ||
| - 2 points per good review comments |
There was a problem hiding this comment.
you will not be responsible for defining the scoring of review comments.
That is clearly the responsibility of the meta-review project.
This should be clearly explained in your cEP that it is not in-scope for your project.
There was a problem hiding this comment.
The meta-review project will provide necessary data (reactions to the review comment) in community repo, but that project itself doesn't define scoring of review comments. I think this project should use that data (reactions) to calculate scoring.
There was a problem hiding this comment.
I have thought a bit and I now think that should be within the scope of the meta-review project. I am going to add score attribute to Review model, though I didn't mention that in my cEP. Needs your confirmation. @jayvdb
cEP-0020.md
Outdated
| - Details: The one who create issues about new bears | ||
| - Activities: | ||
| - Create at least one bear proposal issue that also had label | ||
| `process/varified` |
There was a problem hiding this comment.
refer to process/verified earlier (and note you spelt it wrong there)
cEP-0020.md
Outdated
| >>> privilege = Privilege.objects.create(name='Get assigned to a newcomer issue') | ||
| >>> level1 = Level.obejects.create(name='Newbie', | ||
| points_required=5, | ||
| privileges = privilege |
There was a problem hiding this comment.
argh this hurts my eyes.
please use correct style on your example code.
cEP-0020.md
Outdated
| - At least 20 answers given on coala Gitter rooms | ||
| - Badge-V: | ||
| - Name: The coala Developer | ||
| - Details: The ones who are in the coala developers team |
There was a problem hiding this comment.
no no no. you have a paragraph below which contradicts this :/
badges are based on activities, not based on who is in a GitHub team.
being in the GitHub team is a privilege.
You could describe this as "Has met the prerequisites to apply for the Developer role".
cEP-0020.md
Outdated
| - Activities: | ||
| - At least 20 answers given on coala Gitter rooms | ||
| - Badge-V: | ||
| - Name: The coala Developer |
There was a problem hiding this comment.
this name is causing confusion.
rename to "coala all-rounder", or something like that
cEP-0020.md
Outdated
| The `coala Developer` badge doesn't mean that a newcomer is | ||
| in the coala developer team. If a newcomer wants to be the | ||
| part of the developer team after earning the developer badges, | ||
| he/she will have to ask a maintainer on Gitter to verify all |
There was a problem hiding this comment.
what happens if we stop using Gitter. :P
don't specify stuff which is out of scope for your cEP.
he/she will have to follow the process for that role.
cEP-0020.md
Outdated
| have been done. That will help maintainers to check if an | ||
| automatically awarded badge is correct or not. So, in case | ||
| of any doubt on the newcomer, a maintainer can do the | ||
| manual check and not promote them to developers team. |
There was a problem hiding this comment.
So, in case ..
You talk about the developers team in the next paragraph -- dont mention it here. move anything in this sentence to the next paragraph.
cEP-0020.md
Outdated
| manual check and not promote them to developers team. | ||
|
|
||
| The `coala Developer` badge doesn't mean that a newcomer is | ||
| in the coala developer team. If a newcomer wants to be the |
There was a problem hiding this comment.
replace team with role everywhere, and rephase where appropriate.
cEP-0020.md
Outdated
|
|
||
| ## Background | ||
|
|
||
| ### Newcomer Experience |
There was a problem hiding this comment.
Currently there is no background on the existing newcomer process.
Include a very short summary of it here, and a link to http://coala.io/newcomer for readers to see the current version, and a link to the github latest commit for the document, to archive what it looked like when the cEP was created.
Then the problems with it (already below) will make more sense to the average reader.
Here would be a good spot to explain that the coala Developer role is implemented as a GitHub organisation team. (and that should be the only time 'team' is mentioned in the cEP.)
cEP-0020.md
Outdated
|
|
||
| ```python | ||
| >>> privilege = Privilege.objects.create( | ||
| name='Get assigned to a newcomer issue') |
cEP-0020.md
Outdated
| >>> privilege = Privilege.objects.create( | ||
| name='Get assigned to a newcomer issue') | ||
| >>> level1 = Level.obejects.create(name='Newbie', | ||
| points_required=5, |
cEP-0020.md
Outdated
| contributing to the community we need to find a way to teach and | ||
| help them so that we can keep working on coala at a fast pace. | ||
|
|
||
| We also need to find a way to get more effective feedback from |
cEP-0020.md
Outdated
| highest number of points and the newcomer will be awarded | ||
| with those points. | ||
|
|
||
| Also, note that both the activities will be counted while awarding |
cEP-0020.md
Outdated
| And a query to create a level would be: | ||
|
|
||
| ```python | ||
| >>> privilege = Privilege.objects.create( |
cEP-0020.md
Outdated
| ## Conclusion | ||
|
|
||
| This project addresses the most of the problems a | ||
| newcomer/maintainer at coala face during the newcomer to |
cEP-0020.md
Outdated
| - 10 points for difficulty low | ||
| - 15 points for difficulty medium | ||
| - If Gitmate errors are b/w 10 and beyond | ||
| - No points are given |
There was a problem hiding this comment.
Tbh I think merges are actually worth way more points. Filing an issue is much quicker than solving it^^
There was a problem hiding this comment.
Unfortunately we havent done any testing of these points with sample people. We definitely need to, before the end of the project, in order to validate the points allocated, and tweak accordingly.
@sks444 , can you create a milestone task in phase 2 to build a dataset for 10 of the 2018 gsoc students as of Jan 2018, and a dataset as of March 2018 & a task for phase 3 to validate the points allocated against those two datasets. We need to verify that 2018 gsoc students as of Jan 2018, and as of March 2018, would have been given points & levels which feel sensible, and tweak the configuration to as required to get a better outcome.
Filing a half-decent non-duplicate bug report (not a feature) is usually much harder than solving it, unless it is of difficulty/medium or higher. issues do need to be incentivised, because without issues there are no tasks to allocate, and we have way too many people turning up at a hackathon and doing a pretty crappy MR and then disappearing, which is a net negative on the community, costing us review time without getting a commit for the effort. If they instead used coala, and found a bug, they can disappear after the hackathon and we have a net positive value.
Anyway, there is definitely room to improve these points allocations, and we'll be able to (and will need to) adjust these after go-live to try to find the best balance. I've created a few comments above to have the allocations revised.
specifically, regarding a MR with lots of gitmate errors, it should still have some a low points allocated if it is merged.
Otherwise we encourage people to give up after 10 gitmate errors, or close their PR and create a new one hoping they will get more points on the new one.
And this raises a problem that has somehow escaped this cEP until now. negative points. (we have discussed it, but they dont appear in the cEP yet)
Creation of duplicate PRs and duplicate issues is a great example of an activity which must be actively decentivised, otherwise the points system described here will fall apart.
| Names and metrics of tracked activities will also be shown | ||
| on the leaderboard along with the number of times they | ||
| have been done. That will help maintainers to varify if an | ||
| automatically awarded badge is correct or not. |
There was a problem hiding this comment.
A newcomer can earn points by performing any kind of activity.
And the badges are given here to show a newcomer's skill in a particular type of activity.
If we give badges based on points then the activities will not have much more meaning. :)
There was a problem hiding this comment.
We're keeping badges distinct from the points/levels , as we will be revising our points/levels after go-live , and that will mean people's points may change, and they may even drop down a level , but a badge shouldnt be revoked once it has been given.
If a badge "should" be changed, that would need to be done by versioning the badge, and maybe even renaming the old badge to show it wasnt well designed.
@sks444 I dont think we have anything about this distinction yet. Probably belongs up much higher as a guiding principle.
cEP-0020.md
Outdated
| - Privileges: | ||
| - Mark a PR by another newcomer as wip | ||
| - Get assigned to difficulty medium issue | ||
| - Level-VII: |
| Names and metrics of tracked activities will also be shown | ||
| on the leaderboard along with the number of times they | ||
| have been done. That will help maintainers to varify if an | ||
| automatically awarded badge is correct or not. |
There was a problem hiding this comment.
I can also think of "levelled" badges, like a badge can have up to X levels. For example a badge could be "Number of PRs reviewed". If you have 5, you get the Level 1 badge. If you have reviewed 20, you get the Level 2 badge and so on and so on...
There was a problem hiding this comment.
All the things related to reviewing is being done in meta-review system
But yeah, we will have more badges while the implementation of this system, these are just a few ideas I have taken to give a better understanding of the design.
There was a problem hiding this comment.
Agree that 'levels' within a badge is a very common theme in badge design.
Please add a reviewer badge, and a super reviewer badge, to show how the design can be used to provide that type of functionality.
cEP-0020.md
Outdated
|
|
||
| Points are a simple tool to reward positive contributions | ||
| to coala, and they provide an excellent base upon which to | ||
| build a gamified design. Now let's assign points to some |
There was a problem hiding this comment.
let's
This is hiding an "us", which is like "we".
There is another one below.
cEP-0020.md
Outdated
|
|
||
| - Create Issue | ||
| - 20 points for type/bug | ||
| - 20 points for bear proposal |
There was a problem hiding this comment.
bear proposals should be much lower. those issues are usually just a link to a linter. very easy to create, and there are hundreds of external linters that coala dont support, and many of them are not better than the bears we already have.
Also , we can split them into
- external linter bear proposal, which are wrapping an existing tool = 5 points
- native bear proposal = 7 points
- generic bear proposal = 10 points
cEP-0020.md
Outdated
| - Create Issue | ||
| - 20 points for type/bug | ||
| - 20 points for bear proposal | ||
| - 10 points for difficulty newcomer |
There was a problem hiding this comment.
To address Makman's point at https://github.com/coala/cEPs/pull/150/files#r190873892 , these points for difficulty level should be decreased, considerably. creating lots of valid issues shouldnt be an easy way to level up. creating lots of valid issues should result in a lovely badge.
newcomer issues are usually very tiny problems and very easy to explain because they are so trivial.
low are also very plentiful and are easy to find and write up.
medium issues usually require quite a bit more effort to write up.
maybe 5, 8, 12 ?
also add difficulty high at 15
However, difficulty shouldnt be the way to get a high score for an issue. importance of an issue is more relevant. Please create additional activities for importance/* labels, and medium importance would be 15, and high would be 20 (i.e. type/bug is usually high importance). low maybe 10 and trivial = 5 .. ?
There was a problem hiding this comment.
Please create additional activities for importance/* .....
Yep, assigning points based on importance is great.
However, one thing to note that we should also check the difficulty of the issue with the importance.
E.g.: An importance/high issue could be difficulty newcomer. And assigning 20 points for this would not be very good?
(As we are assgining only 5 points for difficulty newcomer issues)
There was a problem hiding this comment.
Probably I should add another note saying that:
There will be a method to check what is the other labels on importance high issues or on other importance related issues, and points will be assigned accordingly. E.g.: If an issue is importance high and has difficulty newcomer the suitable points would be 10.
(points assigned to the important issues are just for showing an example that in these type of situations the importance label will be preferred.)
Or maybe calculate the average in any case when there are multiple activities and a single action, and assign that average points to that action.
E.g.: If an action involve type/bug(10), importance/high(20) and difficulty low(9) issue.
The resultant points would be (10+20+9) /3=13?
cEP-0020.md
Outdated
| - 10 points for difficulty low | ||
| - 15 points for difficulty medium | ||
| - If Gitmate errors are b/w 10 and beyond | ||
| - No points are given |
There was a problem hiding this comment.
Unfortunately we havent done any testing of these points with sample people. We definitely need to, before the end of the project, in order to validate the points allocated, and tweak accordingly.
@sks444 , can you create a milestone task in phase 2 to build a dataset for 10 of the 2018 gsoc students as of Jan 2018, and a dataset as of March 2018 & a task for phase 3 to validate the points allocated against those two datasets. We need to verify that 2018 gsoc students as of Jan 2018, and as of March 2018, would have been given points & levels which feel sensible, and tweak the configuration to as required to get a better outcome.
Filing a half-decent non-duplicate bug report (not a feature) is usually much harder than solving it, unless it is of difficulty/medium or higher. issues do need to be incentivised, because without issues there are no tasks to allocate, and we have way too many people turning up at a hackathon and doing a pretty crappy MR and then disappearing, which is a net negative on the community, costing us review time without getting a commit for the effort. If they instead used coala, and found a bug, they can disappear after the hackathon and we have a net positive value.
Anyway, there is definitely room to improve these points allocations, and we'll be able to (and will need to) adjust these after go-live to try to find the best balance. I've created a few comments above to have the allocations revised.
specifically, regarding a MR with lots of gitmate errors, it should still have some a low points allocated if it is merged.
Otherwise we encourage people to give up after 10 gitmate errors, or close their PR and create a new one hoping they will get more points on the new one.
And this raises a problem that has somehow escaped this cEP until now. negative points. (we have discussed it, but they dont appear in the cEP yet)
Creation of duplicate PRs and duplicate issues is a great example of an activity which must be actively decentivised, otherwise the points system described here will fall apart.
cEP-0020.md
Outdated
| - The helper: | ||
| - Details: The one who answers every newcomer questions on Gitter | ||
| - Activities: | ||
| - At least 20 answers given on coala Gitter rooms |
There was a problem hiding this comment.
Please add another activity for this badge.
I've repeatedly asked for this.
| Names and metrics of tracked activities will also be shown | ||
| on the leaderboard along with the number of times they | ||
| have been done. That will help maintainers to varify if an | ||
| automatically awarded badge is correct or not. |
There was a problem hiding this comment.
We're keeping badges distinct from the points/levels , as we will be revising our points/levels after go-live , and that will mean people's points may change, and they may even drop down a level , but a badge shouldnt be revoked once it has been given.
If a badge "should" be changed, that would need to be done by versioning the badge, and maybe even renaming the old badge to show it wasnt well designed.
@sks444 I dont think we have anything about this distinction yet. Probably belongs up much higher as a guiding principle.
| Names and metrics of tracked activities will also be shown | ||
| on the leaderboard along with the number of times they | ||
| have been done. That will help maintainers to varify if an | ||
| automatically awarded badge is correct or not. |
There was a problem hiding this comment.
Agree that 'levels' within a badge is a very common theme in badge design.
Please add a reviewer badge, and a super reviewer badge, to show how the design can be used to provide that type of functionality.
cEP-0020.md
Outdated
| have been done. That will help maintainers to varify if an | ||
| automatically awarded badge is correct or not. | ||
|
|
||
| Note that in case if a issue or a pull request has two labels which |
cEP-0020.md
Outdated
|
|
||
| Note that in case if a issue or a pull request has two labels which | ||
| are counted in this system, the performed activities by the user will | ||
| be two and both the activities will be counted while awarding badges. |
There was a problem hiding this comment.
Just a quick question. How do we plan to display these data? how will the newcomers know about the badges, points ? I think it would be cool if we had some kind of profile for each user where we could display data like points earned, badges earned, points to next level and recent activity. Fedora seem to have implemented something similar:
https://badges.fedoraproject.org/user/orangeutan
Here they have also displayed stuff like weekly leaders, etc:
https://badges.fedoraproject.org/
We could implement something similar in the community site?
Would love to hear opinions on this :)
cEP-0020.md
Outdated
| to apply for the developers role at coala. | ||
|
|
||
| [Here](https://github.com/coala/coala/blob/4f2b3c101b8d67a17078dfb2603083c9c00537a6/docs/Developers/Newcomers_Guide.rst) | ||
| is a permanent link of the newcomer guides we have at the moment. |
There was a problem hiding this comment.
"we"
newcomer guides -> Newcomers Guide
cEP-0020.md
Outdated
| track the progress of newcomers and assign some points to | ||
| each of the activities they do. There will be different | ||
| levels based on point thresholds and a newcomer will also | ||
| have some special privileges after the completion of each |
There was a problem hiding this comment.
have some special privileges
'special' is over-selling, especially when 'have' isnt true
use "be eligible for specific privileges"
cEP-0020.md
Outdated
| A newcomer at coala have to follow the | ||
| [newcomer guides](https://api.coala.io/en/latest/Developers/Newcomers_Guide.html) | ||
| and perform certain tasks mentioned there in order to be eligible | ||
| to apply for the developers role at coala. |
cEP-0020.md
Outdated
| [Here](https://github.com/coala/coala/blob/4f2b3c101b8d67a17078dfb2603083c9c00537a6/docs/Developers/Newcomers_Guide.rst) | ||
| is a permanent link of the newcomer guides we have at the moment. | ||
|
|
||
| A developers role at coala means that the newcomer is now a part |
There was a problem hiding this comment.
A developers -> The "Developers"
newcomer -> recipient
cEP-0020.md
Outdated
| points to some of the tracked activities above: | ||
|
|
||
| - Create Issue | ||
| - 10 points for type/bug |
There was a problem hiding this comment.
reorder these sub-lists into increasing points order
cEP-0020.md
Outdated
| `number_of_times` field store the value of number of times | ||
| an activity has been done by the same user. | ||
|
|
||
| The `User` model will have a ManyToMany relationship with |
cEP-0020.md
Outdated
|
|
||
| ```python | ||
| privilege = Privilege.objects.create( | ||
| name='Get assigned to a newcomer issue' |
cEP-0020.md
Outdated
| level1 = Level.obejects.create( | ||
| name='Newbie', | ||
| points_required=5, | ||
| privileges=privilege |
cEP-0020.md
Outdated
| netlify and show the useful updated user metrics and stats | ||
| on the leaderboard. | ||
|
|
||
| The leaderboard will have the following column: |
cEP-0020.md
Outdated
|
|
||
| The leaderboard will have the following column: | ||
|
|
||
| | Username | Important Activity Metrics | Points | Level and Privileges |
There was a problem hiding this comment.
shorten headers to get onto a single line.
splitting onto two lines is invalid syntax
at least "Level and Privileges" -> "Levels"
as Privileges are implicitly the same thing.
|
ack 1ec5b92 |
|
@gitmate-bot ff |
|
Hey! I'm GitMate.io! This pull request is being fastforwarded automatically. Please DO NOT push while fastforward is in progress or your changes would be lost permanently |
|
Automated fastforward with GitMate.io was successful! 🎉 |
Closes #133
(continues from #131)