You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
We removed the origin checking for some customer-specific Spark:
abstractclassGpuShuffleMetaBase {
...
overridedeftagPlanForGpu():Unit= {
// Remove the check of shuffleOrigin to adapt customer-specific origin:// `REPARTITION_FOR_BUCKET`//// if (!ShuffleOriginUtil.isSupported(shuffle.shuffleOrigin)) {// willNotWorkOnGpu(s"${shuffle.shuffleOrigin} not supported on GPU")// }
}
Add a config to skip the above checking for customer.
Describe the solution you'd like
if (conf.isIncompatEnabled && conf.incompatShuffleOrigin) {
// For customized Spark version:// e.g.: REPARTITION_FOR_BUCKET origin// skip the checking
} else {
// For vanilla Sparkif (!ShuffleOriginUtil.isSupported(shuffle.shuffleOrigin)) {
willNotWorkOnGpu(s"${shuffle.shuffleOrigin} not supported on GPU")
}
}
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
We removed the origin checking for some customer-specific Spark:
Add a config to skip the above checking for customer.
Describe the solution you'd like
The text was updated successfully, but these errors were encountered: