Add option to kubernetes docs

G-Research · Mar 11, 2024 · b9ea61b · b9ea61b
1 parent a67f88e
commit b9ea61b
Show file tree

Hide file tree

Showing 3 changed files with 25 additions and 2 deletions.
diff --git a/core/src/main/scala/org/apache/spark/internal/config/UI.scala b/core/src/main/scala/org/apache/spark/internal/config/UI.scala
@@ -221,7 +221,7 @@ private[spark] object UI {
     .createOptional
 
   val CUSTOM_EXECUTOR_LOG_URL = ConfigBuilder("spark.ui.custom.executor.log.url")
-    .doc("Specifies custom spark executor log url for supporting external log service instead of " +
+    .doc("Specifies custom Spark executor log url for supporting external log service instead of " +
       "using cluster managers' application log urls in the Spark UI. Spark will support " +
       "some path variables via patterns which can vary on cluster manager. Please check the " +
       "documentation for your cluster manager to see which patterns are supported, if any. " +

diff --git a/docs/configuration.md b/docs/configuration.md
@@ -1627,7 +1627,7 @@ Apart from these, the following properties are also available, and may be useful
   <td><code>spark.ui.custom.executor.log.url</code></td>
   <td>(none)</td>
   <td>
-    Specifies custom spark executor log URL for supporting external log service instead of using cluster
+    Specifies custom Spark executor log URL for supporting external log service instead of using cluster
     managers' application log URLs in Spark UI. Spark will support some path variables via patterns
     which can vary on cluster manager. Please check the documentation for your cluster manager to
     see which patterns are supported, if any. <p/>

diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
@@ -1650,6 +1650,29 @@ See the [configuration page](configuration.html) for information on Spark config
   </td>
   <td>3.3.0</td>
 </tr>
+<tr>
+  <td><code>spark.ui.custom.executor.log.url</code></td>
+  <td>(none)</td>
+  <td>
+    Specifies custom Spark executor log URL for supporting external log service instead of using cluster
+    managers' application log URLs in Spark UI. Spark will support some path variables via patterns
+    which can vary on cluster manager. Spark Kubernetes cluster manager supports the following path variables:
+    <ul>
+      <li><code>APP_ID</code>: The unique application id</li>
+      <li><code>EXECUTOR_ID</code>: The executor id (a positive integer larger than zero)</li>
+      <li><code>HOSTNAME</code>: The name of the host where the executor runs</li>
+      <li><code>POD_NAME</code>: The name of the pod that contains the executor</li>
+      <li><code>FILE_NAME</code>: The name of the log, which is always <code>"log"</code></li>
+    </ul>
+    Please note that this configuration also replaces original log urls in event log,
+    which will be also effective when accessing the application on history server. The new log urls must be
+    permanent, otherwise you might have dead link for executor log urls. <p/>
+    Example: Config value <code>"https://my.custom.url/logs?app={{APP_ID}}&amp;executor={{EXECUTOR_ID}}"</code>
+    adds for application <code>"app-example-123"</code> and executor 1 this link to the Spark UI:
+    <code>https://my.custom.url/logs?app=app-example-123&amp;executor=1</code>
+  </td>
+  <td>4.0.0</td>
+</tr>
 </table>
 
 #### Pod template properties