[SPARK-47323][K8S] Support custom executor log urls

[EnricoMi]

What changes were proposed in this pull request?

Make Kubernetes resource manager support existing config spark.ui.custom.executor.log.url.

Allow for

spark.ui.custom.executor.log.url="https://my.custom.url/logs?app={{APP_ID}}&executor={{EXECUTOR_ID}}"

Supports these variables:

• APP_ID: The unique application id
• EXECUTOR_ID: The executor id (a positive integer larger than zero)
• HOSTNAME: The name of the host where the executor runs
• KUBERNETES_NAMESPACE: The namespace where the executor pods run
• KUBERNETES_POD_NAME: The name of the pod that contains the executor
• FILE_NAME: The name of the log, which is always "log"

Why are the changes needed?

Running Spark on Kubernetes requires persisting the logs elsewhere. Having the Spark UI link to those logs is very useful. This is currently only supported by YARN.

Does this PR introduce any user-facing change?

Spark UI provides links to logs when run on Kubernetes.

How was this patch tested?

Unit test and manually tested on minikube K8S cluster.

Was this patch authored or co-authored using generative AI tooling?

No

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!