More structured recommendations for review issues?

[sneakers-the-rat]

We recommend that people raise issues on reviewed repos with comments/questions rather than in the review themselves, but could we also give a bit more structure to those?

One obvious suggestion would be to have a sort of standard taxonomy of things that a given issue is about:

• Docs
• Tests
• Installation
• CI
• Bugs
• Performance

etc. We could make shortcodes for the various items in the review checklist, so people could just use like [docs] in the issue title and we know it corresponds to that checklist item.

Another is to indicate the "status" of an issue, specifically I have noticed some lack of clarity for both reviewers and authors re: whether an issue is a blocker for the review, just a suggestion, or a question. Reviewers aren't sure if they are able to 'go beyond' the checklist and say something specifically "I need this to be documented, I need tests for this, etc." and they are also unsure if they should raise issues just for asking questions (the ones i have spoken with were concerned that would be interpreted as a blocking requirement when it was just a question). On the other side, authors are uncertain if they need to wontfix something that's just a comment, unsure if they can close issues, etc.

I think having a recommended set of terms would make this clearer - obviously people can go beyond them/not use them, but merely having them I think gives a clearer indication of the kind of things that should be raised as issues.

• Question
• Optional Suggestion
• Required/Blocking (whatever we want to call this)

Some examples from me on a recent review where i was trying to signpost this more clearly: https://github.com/neuroanatomy/thresholdmann/issues?q=author%3Asneakers-the-rat

[jedbrown]

I like the intent, but note that many projects (especially the large/more mature ones) have their own taxonomy, labeling conventions, and issue templates. It feels out of our lane to prescribe that they use our convention or write/document new conventions solely for the JOSS review. (That is, the rest of their project documentation may apply to thousands of people with a time scale of many years, the JOSS review is relevant to 2-3 people over a timespan of weeks to months.)

[sneakers-the-rat]

many projects (especially the large/more mature ones) have their own taxonomy, labeling conventions, and issue templates.

Oh shoot I wasnt clear enough - I meant this not as a "requirement" but as like a set of prompting examples. If there isnt more structure/the authors dont have preferences, give more of a prompt to reviewers so they know what they can do

[danielskatz]

It seems like this is combining two things: 1. tagging types of discussions/issues, and 2. something related to intent/seriousness. I like the second in particular, but think the first would also be good. I'm unsure how to best do this, however, maybe just by prefixing new issues or comments in the review thread with [x][y] for the two?

[sneakers-the-rat]

combining two things

Yes! Two sets of tags. Really we would probably parse any tags for the purposes of display, but these would be the ones that had some meaning understood by e.g. editorialbot or the website.

prefixing new issues or comments in the review thread with [x][y] for the two?

Yes. Open to whether or not we also parse comments in the review thread too! on opened issues/pr yes I think those bracket tag prefixes (rather than e.g. github tags) are the most common convention ive seen