Skip to content

Commit

Permalink
clarification on reviewer workload
Browse files Browse the repository at this point in the history
  • Loading branch information
annargrs committed Feb 20, 2025
1 parent 9228966 commit 3f374c4
Show file tree
Hide file tree
Showing 3 changed files with 4 additions and 2 deletions.
2 changes: 2 additions & 0 deletions _posts/2025-02-10-reviewing-workload-adjustment.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ As we continue to fine-tune our review process, we would like to implement the f

(1) If a person is nominated as the reviewing volunteer for multiple submissions, we now require this volunteer reviewer to provide **4 reviews per submission**, with a cap of 16 reviews. For example, if a volunteer reviewer is nominated for 3 submissions, then we expect this reviewer to provide a minimum of 12 reviews; if a volunteer reviewer is nominated for 4 or more submissions, then the reviewer is expected to provide 16 reviews. We hope that by introducing this change, authors submitting multiple papers can share a larger reviewing load, which we deem a fair practice for all authors.

**Clarification:** If the same senior author has more than 4 submissions, then authors must coordinate so that the team overall contributes 4 reviews for each of their papers, but no one reviewer is responsible for more than 4 papers. If a qualified author already serves as an ARR AC in this cycle, this accounts for up to 4 papers they submit as authors. Serving as a SAC accounts for up to 6 papers as an author, and serving as a PC - up to 10.

(2) In the past there have been submissions with no authors qualified as reviewers based on our criteria. We are happy to announce that we will now relax the requirement to allow authors to nominate an experienced colleague who is not an author of the submission to be the reviewing volunteer of that submission. This non-author reviewing volunteer is also expected to provide 4 reviews per submission for which the volunteer is nominated for.

The above changes can also be found in the OpenReview submission form, as shown below. Please also note that we reserve the right to desk reject papers that do not contribute to reviewing effort without sufficient justification.
Expand Down
2 changes: 1 addition & 1 deletion authorchecklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Please **check all these points in advance**, ideally at least a week. Some of t
- [ ] At least one qualified author or a willing qualified colleague was [nominated to review](https://aclrollingreview.org/reviewing-workload-requirement/). For that author(s) the following has to be done (ideally prior to submission, and at most within 48 hours after submission, incl. at least one business day):
- [ ] fill in Semantic Scholar, DBLP, and ACL Anthology profiles, and [update their listed papers to reflect their current research interests](https://aclrollingreview.org/reviewerguidelines#how-to-get-assigned-to-papers-that-match-your-interests-well).
- [ ] indicate their availability in this reviewing cycle, at least 4 per submitted paper (each submission receives at least 3 reviews and a metareview, plus effort from chairs and any emergency reassignments).
- [ ] if you are a member of a large research group that submits multiple papers in the same ARR cycle: **make sure that you are nominating different senior co-authors.** If there are too few qualified people, and their nomination on multiple papers is unavoidable, they should increase their review load (at least 4 reviews for each paper, with maximum 16). They are encouraged to mentor [secondary reviewers](https://aclrollingreview.org/reviewerguidelines#q-can-i-use-a-secondary-reviewer). Again, each submitted paper receives at least 4 reviews incl. meta-review, and the system is not sustainable if some groups receive a lot more volunteer effort than they contribute.
- [ ] if you are a member of a large research group that submits multiple papers in the same ARR cycle: **make sure that you are nominating different senior co-authors.** If there are too few qualified people, and their nomination on multiple papers is unavoidable, they should increase their review load (at least 4 reviews for each paper). One reviewer can be nominated for a maximum of 4 papers; if there are more, *other* qualified volunteers should be nominated. Again, each submitted paper receives at least 4 reviews incl. meta-review, and the system is not sustainable if some groups receive a lot more volunteer effort than they contribute. Reviewers can mentor [secondary reviewers](https://aclrollingreview.org/reviewerguidelines#q-can-i-use-a-secondary-reviewer).

All author-nominated reviewers are expected to submit their reviews on time and in accordance with [ARR reviewing guidelines](./reviewerguidelines.md). ARR reserves the right to desk reject the papers the authors of which do not contribute to the reviewing effort appropriately without sufficient justification.

Expand Down
2 changes: 1 addition & 1 deletion changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@ One major benefit of ARR is that we can steadily improve the peer review process
* Updating the [workload requirement](_posts/2025-02-10-reviewing-workload-adjustment.md), in consultation with ACL'25 PCs (leads: Jing, Xiaodan)
* Corresponding updates to the guidelines and email templates (leads: Anna)


## December 2024

* Updating the submission page to reflect the updated CFP and author instructions (leads: Anna, Michael)
Expand All @@ -39,6 +38,7 @@ One major benefit of ARR is that we can steadily improve the peer review process
* Analysis of October 2024 review data (leads: Anna, Sudipta)
* Working with incoming chairs for ACL, EMNLP and AACL 2025 to discuss and coordinate proposed further changes
* Collaborating with the peer review committee on the proposal to change to 10-week cycles
* Adding a new emergency declaration form for ACs and reviewers to declare emergencies

## October 2024

Expand Down

0 comments on commit 3f374c4

Please sign in to comment.