Add tests that access AWS #208
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Test authentication via AssumeRole - Test `streamChanges: true` both with and without `skipInitialSnapshotTransfer`
…redentials - Get temporary credentials from the GitHub workflow, and forward those credentials to the Spark nodes - Remove AssumeRoleTest, which cannot be executed anymore
…ided on the source table
…state, otherwise it never terminates, and properly clean-up the handlers.
This was referenced Aug 30, 2024
tarzanek
reviewed
Aug 31, 2024
| - uses: actions/setup-java@v4 | ||
| with: | ||
| distribution: temurin | ||
| java-version: 8 |
There was a problem hiding this comment.
I thought we compile with jdk 11
There was a problem hiding this comment.
wait, ok, it was blocked due to spark 2.x
ok, so maybe we can switch to jdk 11 with spark 3.x
There was a problem hiding this comment.
This is not really important to be support Java 8, but if we can, why not do it? Using Java 8 in the CI ensures that we at least support Java 8. On the other hand, that could also prevent us from detecting possible incompatibilities with higher versions of the JVM…
But anyway, that should be a separate issue/PR :)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Alternative to #205. Fixes #113.
The first commit (4945055) is taken from #205.
The second commit (127429d) changes the authentication workflow to rely on GitHub as an OpenID Connect provider (instead of using AssumeRole with static AWS credentials, which can not expose in the CI, even as GitHub secrets), and removes the AssumeRole authentication test.
The remaining commits fix issues that I discovered while working on this, and could be extracted out to separate PRs if needed:
streamChangesis enabled but no static credentials were provided. We used to fail with aNullPointerException. We now fallback to the default AWS credentials provider.