[Python] Run acceptance tests against the latest snapshot

[kanterov]

Changes

Run acceptance tests against the latest snapshot. We add matrix tests, including both the latest snapshot and the latest release, so we don't break compatibility.

We prebuild a wheel with the latest code and reuse it across acceptance tests. The approach of installing from the source directory doesn't work:

1. uv doesn't have a build tool, so it takes 10 seconds for setuptools
2. we can't reuse the same directory and cache, because setuptools can't be used concurrently

[denik]

uv doesn't have a build tool, so it takes 10 seconds for setuptools

Would it make sense / is it possible to switch backend to something more lightweight? e.g. flir_core is 50KB vs 1.4MB setuptools.

Can we vendor build backend in acceptance folder specifically so that we don't need to download anything?

10s is a lot of waiting time for a case when I want to run specific test cases that has nothing to do with python mutator. This is something I do many times per day.

Related to this, can we only build it if we see that there are tests that are going to be using it? We know parse the config before starting a test, so we can have RequiresPythonBuild option to signal to test runner that we need to build it beforehand.

we can't reuse the same directory and cache, because setuptools can't be used concurrently

when are we using it concurrently? C

[kanterov]

@denik

when are we using it concurrently?
Because we run tests concurrently

I've experimented with flit:

• with setuptools build takes ~1 second (cold cache)
• with flit build takes ~100ms (cold cache)

There are slight differences in metadata:

1. databricks/__init__.py wasn't included with setuptools, it's a bug
2. flit doesn't set License field in METADATA, only License-File. Setuptools inlines license text into License field in addition to specifying License-File
3. flit doesn't create top_level.txt with "databricks" in it. From what I know, it's very intentional because top_level.txt is only used by legacy tools.

Do you think we can switch to flit and resolve the problem this way? I'm not a big fan of setuptools overall

[kanterov]

@denik can you please take a look?

[denik]

Seems fast enough: "took 513.060481ms"

If it stays under 1s, then we can keep on critical path, otherwise let's think about optimizations I mentioned.

Should we run "uv build" with "-q" or is the output below something you want to see every time?

      acceptance_test.go:782: [uv build --wheel --out-dir /home/runner/work/cli/cli/acceptance/build/linux_amd64] took 513.060481ms
      acceptance_test.go:786: [uv build --wheel --out-dir /home/runner/work/cli/cli/acceptance/build/linux_amd64] output: Building wheel...
          Version number normalised: '0.243.0-dev' -> '0.243.0.dev0' (see PEP 440)
          Version number normalised: '0.243.0-dev' -> '0.243.0.dev0' (see PEP 440)
          Successfully built /home/runner/work/cli/cli/acceptance/build/linux_amd64/databricks_bundles-0.243.0.dev0-py3-none-any.whl
      acceptance_test.go:115: Writing coverage to /home/runner/work/cli/cli/acceptance/build/cover
      acceptance_test.go:782: [go build -mod vendor -o /home/runner/work/cli/cli/acceptance/build/linux_amd64/databricks -cover] took 25.826919651s
      acceptance_test.go:136: $TMPHOME=/tmp/TestAccept4161202475/001
      acceptance_test.go:237: Summary: 126/133/133 run/selected/total, 7 skipped

[github-actions]

An authorized user can trigger integration tests manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

• PR number: 2523
• Commit SHA: fade5f451b589d9a115cef9dffb5daf03bfae7c3

Checks will be approved automatically on success.

[kanterov]

flit doesn't like symlink we have added for LICENSE file to avoid duplication

[kanterov]

@pietern do you think it's ok to change like that?