Enable NDSS by default as one of the top 4 security conferences

[gannimo]

As the upcoming PC co-chair for NDSS (for 2023 and 2024), I would like to revisit this issue and reopen the discussion.
Even according to the survey @emeryberger did ( https://github.com/emeryberger/CSrankings/blob/gh-pages/docs/CSrankings%20survey%20results.pdf ) NDSS is clearly among the top tier conferences. Compared to other areas, security shows a clear split among the top 4 and all other second tier conferences. NDSS is considered top tier by all [system] security faculty and was actually started in the same year as CCS (1993).

As CSRankings is becoming the de-facto standard to explore/evaluate other fields, the selection in security needs to reflect the status quo of field experts. It's no longer just students evaluating areas and potential advisers but also hiring committees evaluating candidates. The security community univocally refers to the security conferences as the "top 4": https://www.s3.eurecom.fr/~balzarot/notes/top4_2020/ or https://nebelwelt.net/pubstats/top-authors-sys_sec.html and CSRankings should equally reflect this reality.

For reference, the suggestion to include NDSS was brought up earlier, e.g., #263 or #859 Interestingly, the survey actually demonstrated the extremely clear gap between the top 4 and all other security conferences.

If we would like to distinguish, Oakland has been around for the longest and CCS provides a much broader view (i.e., CCS could similarly be (co-)listed in Crypto or Verification). NDSS and Usenix are the two systems focused security conferences. As mentioned in issue 263, if we must enforce a hard cap, we could split this into "Security" with CCS and Oakland and "Applied Security" or "System Security" for Usenix and NDSS. The best option IMO would be though to include NDSS among the top 4 security conferences to correspond to the de-facto state in security.

I would like to hear what the continued arguments are to NOT include NDSS in CSRankings (by default) and what is necessary to change it!

[imranur-rahman]

Just to point out, NDSS retains A* rating in CORE2021, CORE2020, CORE2021 ranking and the associated field of research in CORE2020 and CORE2021 for NDSS is "Cybersecurity and Privacy": http://portal.core.edu.au/conf-ranks/1840/

[arthurgervais]

csrankings has currently between 2-3 venues per "area". Out of fairness to other non-security areas (in which I typically do not publish) it may make sense to focus on the top-3 security conferences then, right? Afaik there is no normalization among areas with differing number of top tier venues.

Per OP's cited survey NDSS is number 4, and certainly an absolutely excellent venue.

[gannimo]

Great remark @arthurgervais. Before moving to multiple submission deadlines per year for each conference, the top 4 were lined up so that authors could receive a response from the previous conference, incorporate remarks, and submit to the next conference. The 4 conferences were (and remain) treated as de-facto equal. Submitting to one is as good as submitting to another.

Restricting CSRankings to 2-3 conferences per "area" is an arbitrary cutoff. The number of papers at AI/ML conferences are an order of magnitude larger. The paper length at conferences in other areas is half of what NDSS/SP/CSS/Usenix have. CSRankings does not restrict the maximum number of papers at a conference per year and does not normalize papers by page length or number of words. Introducing arbitrary cut-offs is exactly what CSRankings tried to eliminate by introducing observable metrics.

We should ask the members of the community what the top tier conferences are and not impose arbitrary bounds. The quoted statistics are from a poll Emery conducted (linked in the top post) and show a clear cut after the four top tier conferences.
One possible approach would be to re-run these polls across all areas with reviewers identifying their core area(s) to get more and better data.

Let's evolve community standards and not impose arbitrary bounds!

[LordAmit]

I was looking at the FAQ: https://csrankings.org/faq.html, and could not find the cut-off limit of 3 venues per field.

However, I found these statements:

The conferences listed were developed in consultation with faculty across a range of institutions, including via community surveys.

Additional conferences are not listed when they are not roughly equivalent to the rest. This selection is generally uncontroversial but in some communities has been established by surveying recent program committees from their indisputably top conference(s).

On the contrary, the community strongly indicated that NDSS is one of the top-tier conferences in the surveys shared by @gannimo and it is considered equivalent to the rest.

If NDSS can not be included because of the limit of 3 top conferences only, IMHO at least the CSRankings FAQ should reflect this and state that it has an upper limit of 3 venues only per field.

[balzarot]

Let me chime in to support @gannimo comment: the four conferences were perfectly aligned (intentionally) so that the notification of one preceded by just few days the submission deadline of the other.
I have been monitoring the top system security conferences for many years now and I never met anyone who would skip a deadline not to submit to NDSS. The fact that a system paper is published at oakland or NDSS only depended on when the paper was completed and/or how many times it was rejected.
Saying that Oakland is better than NDSS is like saying that Usenix is a better venue for system paper than CCS (it is, and imho by far, but that does not mean that the two should be ranked differently)

[arthurgervais]

Excellent comments, and I wholeheartedly agree that imposing an arbitrary limit on the number of top conferences per csranking area isn't particularly scientific. We could work to promote consistency across csranking areas, which might subsequently improve our success in integrating NDSS.
As @gannimo points out, for example, AI/ML conferences accept many more papers than other conferences, and these papers are not currently normalized (e.g., number of words, etc) when compared to papers from other fields. Any normalization metrics may have drawbacks, but at least we would try to retain fairness across areas that csrankings compares.

[gannimo]

Thanks for the comments so far. Let me try to summarize the different discussions.

As CSRankings grows, we'll need some process to adapt the set of included conferences as areas become more/less active over time. In my opinion, a per-area board would make most sense, potentially augmented with periodic polls.

• [NDSS] There is broad agreement that, without a limit to 3 conferences per area, NDSS belongs to the top tier category.
• [NDSS] The poll from a couple of years ago shows a clear separation
• [CSRankings] There is confusion regarding the artificial limit to 3 conferences per area which seems to be enforced but not encoded in the specification. Either the specification should be adjusted along with a justification or this "rule" should be sunsetted.

To make the choice of conferences per area less arbitrary, we could either have a committee that decides on what to include (with a justifying report) or repeated polls where, e.g., PC members declare their areas of expertise and the set of perceived top tier conferences.

Thoughts? Comments?

[LordAmit]

Thanks for the summary!

[NDSS] There is broad agreement that, without a limit to 3 conferences per area, NDSS belongs to the top tier category.

I am afraid this can be interpreted as,

if there is a limit to 3 conferences per area, NDSS may not belong to the top-tier category.

I don't think this is the case. Top-tiers is a class and conferences in this class are equivalent. They should not be ranked in-between.

Other than that, I agree with the approach of having a per-area board consisting of PC members with periodic polls.

[gannimo]

Thanks all for the discussion and for showing support so far.

I've reached out to Emery several times but have not yet heard back. @emeryberger it would be great if you could jump into the discussion as well.

[ivanolive]

I want to add my support here as well. The security community refers to their top conferences as the "big 4" (4 for a reason). As CSRankings is increasingly used to evaluate universities and consequently faculty (even by some hiring committees, I've been told), it would be a shame to have NDSS status artificially lowered because it isn't listed along with the other 3 on CSRankings by default.

Imposing a conference limit of 3 seems arbitrary. For example, a single top ML conference accepts 3-5 times more papers than all "big 4" security conferences combined (2021 data).

I have no NDSS papers (so I think I'm somewhat unbiased here :-) ), but I've never considered NDSS less prestigious or decided not to submit a paper to NDSS because there would be a better ranked upcoming conference. My perception for the consensus in the system and network security community is that the 4 are on the same level.

[Ben5000]

A general comment: these internal-field discussions about "adding another really-top conference", should be made in the context of all other fields. If you look at the field of ML for instance on this website, you see a repeat demand to include ICLR, with similar arguments. Then, people from "CS Education" chime in, and ask to add their conference, etc. In general, it does not seem fair to include four conferences in an area which is already huge in total number of accepted papers. This would clearly undermine other fields that are more selective (not only in absolute terms, but selective per their acceptance rate) in my opinion.

[balzarot]

Ben.. can you actually make examples of those fields that are more "selective"?
I checked dozen of top conferences in other fields (ML, AI, Crypto, measurements, .. ), and their acceptance rate is almost always higher than the security ones.
If instead you want to compare the number of accepted papers, ICML alone accepted 1184 papers in one year (over 20% acceptance rate btw). That is two times more than ALL the four top security conferences combined! (the largest security conference in 2021 accepted 246 !!)

Plus, the issue here is that all four in security are equally important (I dont know anyone who ever skipped a deadline of NDSS just because he wanted to submit to Oakland).

[gannimo]

@Ben5000 @balzarot good points from both of you.

There are several challenges here that should be addressed, so let's tease them apart:
a) selecting the top conferences per field
b) normalizing across fields
c) normalizing across author lists

For a), each field should decide on the top conferences according to that field. Having a single person decide just calls for imprecision or injustice. Members of each field likely know best what the top tier venues are. As @balzarot mentioned, I have never seen someone in systems security pass over NDSS just to wait for the next Oakland deadline.

For b), we should normalize the number of papers per active members for each field. I don't have a good intuition on how to do that except for statistically assessing the number of active authors as a proxy for community size to then indicate how many papers/points should be awarded to that field. This is an unsolved problem and not the topic of this issue.

For c), we should likely count a fraction of the point for each active senior author, otherwise collaboration with junior members of our community such as undergrads is discouraged. This is also a challenging problem but also not the topic of this issue.

[arthurgervais]

Cross-field/venue/authorlist normalizations would be indeed great to try to minimize bias. @balzarot's comments are spot on, the numbers are as of now simply not comparable.

Adding more empirical data, there is a difference between the top-4 conferences. Given that data, we should work to strengthen NDSS to bring it up to par with the top-3.

Non-normalized Top-100 Security (https://www.mlsec.org/topnotch/sec_top100.html)
CCS: 35
S&P: 31
Usenix: 18
NDSS: 7

Normalized Top-100 Security (https://www.mlsec.org/topnotch/sec_ntop100.html)
S&P: 42
CCS: 25
Usenix: 20
NDSS: 5

[balzarot]

Interesting analysis @arthurgervais
However, I believe there is some bias due to the topic.
Among the most cited papers, a large number are crypto- and ML-related. For the first category, certainly NDSS is NOT
a top venue (and also Usenix is so-so).

[smokhov]

I think the "top 3" limit is mentioned in this comment here:

CSrankings/util/csrankings.py

Line 84 in c7d9aeb

# Max three most selective venues per area for now.

[earlence-uwm]

Thanks for all these analyses, but at the end of the day, what matters is the community opinion. NDSS is clearly in the same equivalence class as the other three conference, and so, it should be included by default.

As to the issue of different number of conferences for different research areas, this is just natural! Just like the sky is blue. Let those communities decide! if they have fewer than 3, fine. If they have more than 3, fine; as long as the community agrees. Why impose an artificial bound on anything? Nothing is magically going to become equitable by imposing a limit of 3 per area! It is simply a fact of life and we have to deal with it.

RE: polling for community opinion: Do a poll of the PC members. You'd essentially see a large overlap between the top-4 security conferences.