Review results

[rcannood]

There is still a run in progress, will update when these results come in.

---

The HVG overlap metric isn't giving meaningful results -- possibly related to #46. Until this is fixed, we should remove the HVG overlap

---

We could have three plots in three different tabs:

• All methods but only graph metrics
• Embed + feature methods, graph + embed metrics
• Only feature methods, all metrics

Or we could have one plot but with three different summary metrics:

• A label indicating whether the model is able to output a feature matrix, an embedding, or graph metrics
• Average score across graph metrics
• Average score across embedding metrics
• Average score across feature metrics

---

Check PCR metric: according to the normalization plot, the feature-type methods seem to be getting a near perfect score. It might be comparing something it shouldn't be comparing.

[DarioS]

Also, the algorithms evaluated are pre-COVID era. Why are only outdated methods being evaluated?

"method_name": "Combat (full/scaled)",
"paper_year": 2007,
    ...        ...
"method_name": "FastMNN feature (full/scaled)",
"paper_year": 2019,
    ...        ...
"method_name": "Scanorama gene output (full/scaled)",
"paper_year": 2019,

[rcannood]

We've been working on a revision for this task for a while now. A preview of these results is available at https://foundation-models-results--openproblems.netlify.app/results/batch_integration/. They will be released any day now.

We're also working on a new website, which will allow us to render pre-release results on the website itself: https://openproblems-bio.deno.dev/.

If there are batch integration methods which you believe should be added to this task, feel free to create a separate github issue for each of those methods, or follow the instructions on how to add a method.

[DarioS]

Ah, good coincidence! I see lots of contemporary methods are incorporated.