You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
We have code to deal with ignoreCurruptFiles. This is changing some of that functionality so that if you do not have permission to access a file, then the job still fails instead of skipping files that bad in that way.
We should add in some way of testing that our current code does the same thing that older versions of spark did, and update our code to do the right thing for Spark 4.0.0
This also went into 3.5.4, but there is no audit label for that yet.
The PR also says that this is similar to another issue/PR that we have an audit issue for. #11925 we might want to do them together, but the other one might be a bit harder so I am not sure.
The text was updated successfully, but these errors were encountered:
Describe the bug
We have code to deal with
ignoreCurruptFiles
. This is changing some of that functionality so that if you do not have permission to access a file, then the job still fails instead of skipping files that bad in that way.We should add in some way of testing that our current code does the same thing that older versions of spark did, and update our code to do the right thing for Spark 4.0.0
apache/spark@3bb9a72c869
This also went into 3.5.4, but there is no audit label for that yet.
The PR also says that this is similar to another issue/PR that we have an audit issue for. #11925 we might want to do them together, but the other one might be a bit harder so I am not sure.
The text was updated successfully, but these errors were encountered: