Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG][AUDIT][SPARK-50545][CORE][SQL] AccessControlException should be thrown even if ignoreCorruptFiles is enabled #11933

Open
revans2 opened this issue Jan 7, 2025 · 0 comments
Labels
audit_4.0.0 Audit related tasks for 4.0.0 bug Something isn't working

Comments

@revans2
Copy link
Collaborator

revans2 commented Jan 7, 2025

Describe the bug
We have code to deal with ignoreCurruptFiles. This is changing some of that functionality so that if you do not have permission to access a file, then the job still fails instead of skipping files that bad in that way.

We should add in some way of testing that our current code does the same thing that older versions of spark did, and update our code to do the right thing for Spark 4.0.0

apache/spark@3bb9a72c869

This also went into 3.5.4, but there is no audit label for that yet.

The PR also says that this is similar to another issue/PR that we have an audit issue for. #11925 we might want to do them together, but the other one might be a bit harder so I am not sure.

@revans2 revans2 added ? - Needs Triage Need team to review and classify audit_4.0.0 Audit related tasks for 4.0.0 bug Something isn't working labels Jan 7, 2025
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Jan 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
audit_4.0.0 Audit related tasks for 4.0.0 bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants