Skip to content

Commit

Permalink
Merge pull request #2097 from PaulRosenPhD/vis2024
Browse files Browse the repository at this point in the history
fix to a couple of blog posts
  • Loading branch information
smonadjemi authored Sep 27, 2024
2 parents 7005461 + 5eb717f commit 65d9c4f
Show file tree
Hide file tree
Showing 4 changed files with 29 additions and 10 deletions.
20 changes: 13 additions & 7 deletions _posts/2024-09-01-vis-2024-openpractices-report.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ Let's have a look first at the full papers. Of the **124** accepted full papers

For VIS 2023, we asked authors about preprints. Self-reported data showed that about half of accepted authors posted their preprints on arXiv, just over 6% to OSF, 8% to another source, and around 35% not at all. How does this year compare?

![Preprint repositories used](/assets/posts/2024-09-OP/preprint.png)
![Preprint repositories used]({{ 'assets/posts/2024-09-OP/preprint.png' | relative_url }})

This year, we see a slight improvement in the posting of preprints, up to 83 (67%) of submissions. This is a 2% improvement over last year. Furthermore, we see that all reported preprints have been posted to a free and open repository (ArXiv, OSF, or HAL). This is a step in the right direction.

Expand All @@ -39,18 +39,22 @@ Open Practices recommends uploading a preprint of your work to a free and open-a
### Supplemental material

115 (93%) of accepted submissions used at least one supplemental material field, whether PCS, external, or both PCS and through an external link.
![Made use of at least one supplemental material field in PCS](/assets/posts/2024-09-OP/suppl_upload_text.png)
![Made use of at least one supplemental material field in PCS]({{ 'assets/posts/2024-09-OP/suppl_upload_text.png)

While not all papers necessarily have relevant supplemental material (for instance, position papers), for nearly all paper contributions, having supplemental material (like experimental data, code bases, or even just additional proofs and analyses) helps the work to be scrutinizable, so we view this high percentage as a step in the right direction. We do, however, recommend uploads to **both** (1) PCS for review as well as (2) a free, reliable, and long-term archive. Furthermore, our recommendations for alignment with VIS's long-term Open Science goals are uploading to **both** PCS for review and a free, reliable, and long-term archive. This redundancy allows both a static copy of record associated with the entry in the IEEE digital library, as well as an open and more easily scrutinized or re-analyzed version of the work.

Besides PCS, where did accepted submissions upload their supplementary material?
![External (to PCS) supplemental material links](/assets/posts/2024-09-OP/suppl_ext_text.png)

![External (to PCS) supplemental material links]({{ 'assets/posts/2024-09-OP/suppl_ext_text.png' | relative_url }})


We see that 80 (around 65%) of accepted submissions uploaded their supplementary materials to an external source in addition to or instead of PCS, with the majority using OSF. For further guidance on where supplementary material should be upload, please see our [FAQ: Where should I upload supplementary material?](../content/info/open-practices/supplemental-material-faq#where-should-i-upload-supplemental-material)

### Preregistration
We also asked about study preregistration, another new question for this year. Only eight (12.5%) submissions out of the 124 full papers reported preregistration of their studies, seven of which did so on [OSF Registries](https://osf.io/registries) and one on [AsPredicted](https://aspredicted.org/).
![Preregistered study](/assets/posts/2024-09-OP/pre_reg.png)

![Preregistered study]({{ 'assets/posts/2024-09-OP/pre_reg.png' | relative_url }})


This is a clear area where our community has room to improve.
VIS Open Practices still needs formal recommendations for study preregistration, and we acknowledge that this may not be less relevant for certain contributions. However, for work involving empirical studies, preregistering an analysis plan helps us as researchers to clearly separate hypothesis generation (postdiction) from hypothesis confirmation (prediction), thus leveraging the benefits and maintaining awareness of the limitations of statistical inference while improving the study's reproducibility and replicability [[1]](https://www.pnas.org/doi/10.1073/pnas.1708274114). OSF provides an [excellent set of resources](https://help.osf.io/article/158-create-a-preregistration) for preregistering your project.
Expand All @@ -66,15 +70,17 @@ Now, how does the short papers track look? Of the 66 accepted short papers, auth

Of those submissions that posted preprints, most used ArXiv, a few OSF, and a few used custom domains, mainly their university hosting services.

![Preprint repositories used](/assets/posts/2024-09-OP/SP_preprint.png)
![Preprint repositories used]({{ 'assets/posts/2024-09-OP/SP_preprint.png' | relative_url }})

### Supplementary material

Of the 66 accepted short papers, 54 (around 82%) included supplementary material in some form in their submission, whether through PCS directly and/or through upload to an external repository. The majority of authors, as we can see, did both.
![Supplementary material provided](/assets/posts/2024-09-OP/SP_suppl_upload_text.png)

![Supplementary material provided]({{ 'assets/posts/2024-09-OP/SP_suppl_upload_text.png' | relative_url }})

And where did authors upload their supplementary material externally?
![External (to PCS) supplemental material links](/assets/posts/2024-09-OP/SP_suppl_ext_text.png)

![External (to PCS) supplemental material links]({{ 'assets/posts/2024-09-OP/SP_suppl_ext_text.png' | relative_url }})

Here, we see more variation than for the full paper submission. GitHub and OSF are similarly used, and some authors also used Kaggle, Google Drive, and Observable to host their supplementary material. As for the full papers, we recommend archiving supplementary materials to a free, reliable, long-term archive—GitHub links can change or disappear.

Expand Down
19 changes: 16 additions & 3 deletions _posts/2024-09-25-vis-2024-OPC-blog.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,27 @@ permalink: /blog/vis-2024-OPC-blog-Reviewing
---

The road to VIS 2024 is coming to an end and our work as OPCs is almost done. In a few short weeks, we will be opening the paper program at the conference and we will be saying farewell to outgoing OPC Tamara Munzner, who served the conference both last year as well as this one. Together with stalwart OPC assistant Petra Specht, Holger Theisel and Niklas Elmqvist will continue on for next year, and will be joined by Melanie Tory from Northeastern University as the third OPC.

In other words, this is as good a time as any to reflect on the road to VIS 2024: what went wrong, what went well, and what will happen in the future.

This is the tenth "Road to VIS 2024" post that we are posting to the IEEE VIS website. Overall, we count this blog post series as a positive achievement for VIS 2024. We hope that these posts have helped to shed some light on the inside workings of the conference for both early-career and senior researchers alike. We have valued the chance to speak to the community directly in this way.

If you cast your mind back to the early days of the blog, you may remember our second post on the ["Call for Papers"](https://ieeevis.org/year/2024/blog/vis-2024-OPC-blog-call-for-papers), where we talked about the four changes we were making to the conference for 2024. Earlier this summer, we distributed a survey to our program committee to collect their feedback on these changes and the overall review process this year. Out of a total of 142 PC members and 12 Area Papers Chairs, we received 75 responses—thank you!—which we think is a pretty good rate, and certainly good enough to base our decision-making on.

Our first change, to reduce the number of reviewers per paper from four to three (primary, secondary, and one external reviewer) yielded 47% in favor, 18% undecided, and 35% negative (see below). There were many free-form comments in favor and against; we read them all. Nevertheless, we think that this response warrants continuing the experiment for at least another year. We are awaiting the VSC's final decision on this proposal for VIS 2025.
[statistic 1](./images/2024-09-25/Bild_1.png)

![statistic 1]({{ 'assets/posts/2024-09-25/Bild_1.png' | relative_url }})


Our second change, to increase the load for each PC member from six to eight—and thereby reducing the size of the program committee to free up external reviewers—was much less popular. As can be seen in the pie chart below, 74% were against keeping this load and wanted to go back to 6 per PC member, 11% were undecided, and only 15% felt that the 8 papers per PC member load was acceptable. The comments were rather scathing, with some respondents saying they would decline the PC next year if the higher load was maintained for VIS 2025. While we note that 8 used to be the norm for the VIS conference in the past, we have heard this feedback loud and clear. We are awaiting the VSC's approval on going back to 6 papers per PC member for VIS 2025.
[statistic 2](./images/2024-09-25/Bild_2.png)

![statistic 2]({{ 'assets/posts/2024-09-25/Bild_2.png' | relative_url }})


We also collected general feedback on the review process and read each of the responses. Overall, PC members provided many helpful tips on managing the reviewing load, finding external reviewers, increasing the reviewer pool, improving the transparency of the process, and desk rejecting more liberally. We thank everyone who took the time to offer their thoughts on the process and we will be doing our best to incorporate their feedback, directly or indirectly, for next year.

This brings us to the topic of the future: VIS 2025. While we have not yet even held VIS 2024, it is never too early to prepare for what's next. Beyond the changes discussed above—keeping three reviewers and going back to six papers per PC member—we have some bold ideas of where VIS should go. Some of these deal with improvements to PCS to reduce the impact of unethical reviewing; see our post on ["Decisions"](https://ieeevis.org/year/2024/blog/vis-2024-OPC-blog-decisions_2), for more details on this. However, some ideas that we are considering are sufficiently bold or experimental that we would first like to hear what the community has to say about them. For this reason, the VIS 2024 and the incoming 2025 OPCs plan to discuss them at the VIS town hall during the conference. To give you a flavor of some of the things we are considering, we're thinking about (1) extending the deadline a week for only supplemental material, (2) giving authors the option to publish anonymized reviews with their accepted papers, and (3) introducing a student reviewer program where primary reviewers optionally get to invite a Ph.D. or masters student as an advisory "student reviewer" for each paper they manage.

We hope that you will join us at the VIS 2024 town hall so that you can weigh in on these ideas, and offer any other ideas that you may have for improving VIS in the future.
Anyway, our work here is almost done. This has been quite a ride and we're looking forward to a vibrant finale at the conference. We can't wait to let you see the exciting scientific program that the community has assembled. See you in St. Pete's Beach!

Anyway, our work here is almost done. This has been quite a ride and we're looking forward to a vibrant finale at the conference. We can't wait to let you see the exciting scientific program that the community has assembled. See you in St. Pete Beach!
File renamed without changes
File renamed without changes

0 comments on commit 65d9c4f

Please sign in to comment.