GH-45159: [CI][Integration] Remove substrait consumer-testing integration job

[raulcd]

Rationale for this change

The job has been failing for the last two months due to upstream refactoring. I opened an issue upstream but we didn't got any response:

• Apache Arrow integration with substrait consumer-testing job is failing and not maintained - Removal proposal substrait-io/consumer-testing#196

Based on the following substrait consumer-testing PR there was a big refactor which removed the files that we were invoking on our tests:

• feat: factor out result generation and remove old test_*.py substrait-io/consumer-testing#162

I asked on the corresponding issue ~3 weeks ago for feedback on the removal and as there's no opposition I think we should remove this unmaintained job.

What changes are included in this PR?

Remove extremely coupled substrait-io/consumer-testing job.

Are these changes tested?

No

Are there any user-facing changes?

No

• GitHub Issue: [Python][CI] Substrait nightly job is failing due to wrong invocation of substrait consumer-testing #45159

[github-actions]

⚠️ GitHub issue #45159 has been automatically assigned in GitHub to PR creator.

[ingomueller-net]

LGTM!

Thanks for letting us know. I wasn't aware that there were any downstream users when I did the refactoring.

In the mid or long term, what would be requirements/wishes for a Substrait test suite for Arrow to consume?

[raulcd]

Hi @ingomueller-net thanks for answering. From my understanding this was testing possible regressions for pyarrow-dev acero's consumer on the previous test_boolean_functions.py mainly on our nightly jobs. I am unsure whether more tests could be run or not. There is value on validating that new developments continue passing the integration validation. The problem arises on the lack of expertise on the consumer-testing on our repository.

[ingomueller-net]

OK, thanks for the input. To paraphrase and add my own interpretation: I imagine you would like to have a suite of Substrait plans that you can run against the Arrow consumer as end-to-end tests that tell you whether the consumer respects the spec, right? Ideally, it would allow you to skip tests that you know the consumer currently (1) doesn't support or (2) produces wrong results (such that you can, at least, ensure that there are no regression on the others). You might even want the test suite to contain invalid Substrait plans that allow you to test whether your consumer correctly rejects them. Is that so?

[raulcd]

That sounds reasonable to me but I am unsure on what is the current substrait status on Arrow. Probably @westonpace or @amol- know more about it.

[amoeba]

LGTM, thanks for doing this. If the consumer/producer integration testing could live entirely under the Substrait project I think that would be best for both projects.

[EpsilonPrime]

We discussed this in the Substrait community meeting and agreed that this can be removed.

[amol-]

That sounds reasonable to me but I am unsure on what is the current substrait status on Arrow. Probably @westonpace or @amol- know more about it.

We do have some e2e tests here: https://github.com/apache/arrow/blob/main/python/pyarrow/tests/test_substrait.py
They test against substrait plans in binary and text format.
The testsuite is incomplete, but I think that it covers at least the capability to parse and run substrait itself.

[amol-]

Sounds good to me, I think that the other e2e tests that we have are a good trade-off

[raulcd]

I am merging this.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 251758d.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 5 possible false positives for unstable benchmarks that are known to sometimes produce them.