From 3f374c4e2dfba051a657f02cc5c8fcd91f848bcf Mon Sep 17 00:00:00 2001 From: Anna Rogers Date: Thu, 20 Feb 2025 18:57:04 +0100 Subject: [PATCH] clarification on reviewer workload --- _posts/2025-02-10-reviewing-workload-adjustment.md | 2 ++ authorchecklist.md | 2 +- changelog.md | 2 +- 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/_posts/2025-02-10-reviewing-workload-adjustment.md b/_posts/2025-02-10-reviewing-workload-adjustment.md index 9deb5d280..6d8d9da2c 100644 --- a/_posts/2025-02-10-reviewing-workload-adjustment.md +++ b/_posts/2025-02-10-reviewing-workload-adjustment.md @@ -11,6 +11,8 @@ As we continue to fine-tune our review process, we would like to implement the f (1) If a person is nominated as the reviewing volunteer for multiple submissions, we now require this volunteer reviewer to provide **4 reviews per submission**, with a cap of 16 reviews. For example, if a volunteer reviewer is nominated for 3 submissions, then we expect this reviewer to provide a minimum of 12 reviews; if a volunteer reviewer is nominated for 4 or more submissions, then the reviewer is expected to provide 16 reviews. We hope that by introducing this change, authors submitting multiple papers can share a larger reviewing load, which we deem a fair practice for all authors. +**Clarification:** If the same senior author has more than 4 submissions, then authors must coordinate so that the team overall contributes 4 reviews for each of their papers, but no one reviewer is responsible for more than 4 papers. If a qualified author already serves as an ARR AC in this cycle, this accounts for up to 4 papers they submit as authors. Serving as a SAC accounts for up to 6 papers as an author, and serving as a PC - up to 10. + (2) In the past there have been submissions with no authors qualified as reviewers based on our criteria. We are happy to announce that we will now relax the requirement to allow authors to nominate an experienced colleague who is not an author of the submission to be the reviewing volunteer of that submission. This non-author reviewing volunteer is also expected to provide 4 reviews per submission for which the volunteer is nominated for. The above changes can also be found in the OpenReview submission form, as shown below. Please also note that we reserve the right to desk reject papers that do not contribute to reviewing effort without sufficient justification. diff --git a/authorchecklist.md b/authorchecklist.md index 988d2248a..8e1278dfa 100644 --- a/authorchecklist.md +++ b/authorchecklist.md @@ -19,7 +19,7 @@ Please **check all these points in advance**, ideally at least a week. Some of t - [ ] At least one qualified author or a willing qualified colleague was [nominated to review](https://aclrollingreview.org/reviewing-workload-requirement/). For that author(s) the following has to be done (ideally prior to submission, and at most within 48 hours after submission, incl. at least one business day): - [ ] fill in Semantic Scholar, DBLP, and ACL Anthology profiles, and [update their listed papers to reflect their current research interests](https://aclrollingreview.org/reviewerguidelines#how-to-get-assigned-to-papers-that-match-your-interests-well). - [ ] indicate their availability in this reviewing cycle, at least 4 per submitted paper (each submission receives at least 3 reviews and a metareview, plus effort from chairs and any emergency reassignments). - - [ ] if you are a member of a large research group that submits multiple papers in the same ARR cycle: **make sure that you are nominating different senior co-authors.** If there are too few qualified people, and their nomination on multiple papers is unavoidable, they should increase their review load (at least 4 reviews for each paper, with maximum 16). They are encouraged to mentor [secondary reviewers](https://aclrollingreview.org/reviewerguidelines#q-can-i-use-a-secondary-reviewer). Again, each submitted paper receives at least 4 reviews incl. meta-review, and the system is not sustainable if some groups receive a lot more volunteer effort than they contribute. + - [ ] if you are a member of a large research group that submits multiple papers in the same ARR cycle: **make sure that you are nominating different senior co-authors.** If there are too few qualified people, and their nomination on multiple papers is unavoidable, they should increase their review load (at least 4 reviews for each paper). One reviewer can be nominated for a maximum of 4 papers; if there are more, *other* qualified volunteers should be nominated. Again, each submitted paper receives at least 4 reviews incl. meta-review, and the system is not sustainable if some groups receive a lot more volunteer effort than they contribute. Reviewers can mentor [secondary reviewers](https://aclrollingreview.org/reviewerguidelines#q-can-i-use-a-secondary-reviewer). All author-nominated reviewers are expected to submit their reviews on time and in accordance with [ARR reviewing guidelines](./reviewerguidelines.md). ARR reserves the right to desk reject the papers the authors of which do not contribute to the reviewing effort appropriately without sufficient justification. diff --git a/changelog.md b/changelog.md index 81a4b0124..a33949b48 100644 --- a/changelog.md +++ b/changelog.md @@ -27,7 +27,6 @@ One major benefit of ARR is that we can steadily improve the peer review process * Updating the [workload requirement](_posts/2025-02-10-reviewing-workload-adjustment.md), in consultation with ACL'25 PCs (leads: Jing, Xiaodan) * Corresponding updates to the guidelines and email templates (leads: Anna) - ## December 2024 * Updating the submission page to reflect the updated CFP and author instructions (leads: Anna, Michael) @@ -39,6 +38,7 @@ One major benefit of ARR is that we can steadily improve the peer review process * Analysis of October 2024 review data (leads: Anna, Sudipta) * Working with incoming chairs for ACL, EMNLP and AACL 2025 to discuss and coordinate proposed further changes * Collaborating with the peer review committee on the proposal to change to 10-week cycles +* Adding a new emergency declaration form for ACs and reviewers to declare emergencies ## October 2024