GNN/R-GAT Inference Submission disallows sorting of the nodes. Need a mechanism to prevent and detect sorting.

[ukurkure]

GNN/R-GAT Inference task force agreed that the sorting of the nodes is disallowed. Sorting of the nodes reduces number of unique nodes and artificially increases the throughput up to 6x. We need a mechanism to prevent and detect sorting.

[arjunsuresh]

@ukurkure Adding TEST04 compliance can solve this. But it is not straightforward to do this and will need time. Probably this can be targeted for 5.1. The idea of the test is as follows:

A run is done with limited number of unique inputs say 1000 and repeated 10 times.
A run is done with 10000 unique inputs
Run times are compared between the two and the difference should be within 10%.

[ukurkure]

The section 8.2. Model Equivalence needs to be updated https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc#82-model-equivalence.
The proposed addition to the disallowed techniques:

The following techniques are disallowed:

• Techniques that only improve performance when there are identical samples in a query. For example, sorting samples in SSD.
• The Techniques that reduce the number of unique samples in a query. For example, sorting samples in Graph Neural Networks' RGAT.

[arjunsuresh]

@ukurkure The below line already covers the case for R-GAT too right?

Techniques that only improve performance when there are identical samples in a query. For example, sorting samples in SSD.

[ukurkure]

If the working group agrees that "Techniques that only improve performance when there are identical samples in a query. For example, sorting samples in SSD." covers RGAT, we do not have to add the RGAT case explicitly. We could make it explicit by adding the following: Techniques that only improve performance when there are identical samples in a query. For example, sorting samples in SSD and RGAT.

…

On Tue, Nov 26, 2024 at 6:09 PM Arjun Suresh ***@***.***> wrote: @ukurkure <https://github.com/ukurkure> The below line already covers the case for R-GAT too right? Techniques that only improve performance when there are identical samples in a query. For example, sorting samples in SSD. — Reply to this email directly, view it on GitHub <#1950 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE7XMAXQPL3GUTYZUNGF7XT2CUSVVAVCNFSM6AAAAABSKHQRN6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBSGUYDMMJXHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>