Skip to content

Commit

Permalink
Merge pull request #4 from opendatahub-io-contrib/pre-migration
Browse files Browse the repository at this point in the history
post-repository transfer adjustments
  • Loading branch information
guimou authored Jul 11, 2022
2 parents 890bf94 + ef08607 commit 4438928
Show file tree
Hide file tree
Showing 6 changed files with 6 additions and 13 deletions.
2 changes: 1 addition & 1 deletion README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Instructions and tools for deploying and using the https://github.com/GoogleClou

We include in this project custom Spark images with the Hadoop S3a connector to connect to S3-based object storage. Those images are based on https://catalog.redhat.com/software/containers/ubi8/openjdk-8/5dd6a48dbed8bd164a09589a[ubi8/openjdk-8], and are updated accordingly.

Pre-built images can be found here, https://quay.io/repository/guimou/spark-odh and https://quay.io/repository/guimou/pyspark-odh, but you can also choose to build your own.
Pre-built images can be found here, https://quay.io/repository/opendatahub-contrib/spark and https://quay.io/repository/opendatahub-contrib/pyspark, but you can also choose to build your own.

Available images:

Expand Down
9 changes: 1 addition & 8 deletions spark-history-server/spark-hs-deployment.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
### Adapted from Helm Chart: https://artifacthub.io/packages/helm/spot/spark-history-server
# Source: spark-history-server/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
Expand All @@ -8,21 +7,18 @@ metadata:
app.kubernetes.io/name: spark-history-server
app.kubernetes.io/instance: spark-history-server
---
# Source: spark-history-server/templates/rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: spark-history-server-cr
labels:
app.kubernetes.io/name: spark-history-server
app.kubernetes.io/instance: spark-history-server
app.kubernetes.io/managed-by: Helm
rules:
- apiGroups: [""]
resources: ["deployments", "pods"]
verbs: ["*"]
---
# Source: spark-history-server/templates/rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
Expand All @@ -36,14 +32,12 @@ roleRef:
name: spark-history-server-cr
apiGroup: rbac.authorization.k8s.io
---
# Source: spark-history-server/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: spark-history-server
labels:
app.kubernetes.io/name: spark-history-server
helm.sh/chart: spark-history-server-1.5.0
app.kubernetes.io/instance: spark-history-server
spec:
type: ClusterIP
Expand Down Expand Up @@ -82,7 +76,6 @@ metadata:
name: spark-history-server
labels:
app.kubernetes.io/name: spark-history-server
helm.sh/chart: spark-history-server-1.5.0
app.kubernetes.io/instance: spark-history-server
spec:
replicas: 1
Expand All @@ -99,7 +92,7 @@ spec:
serviceAccountName: spark-history-server
containers:
- name: spark-history-server
image: "quay.io/guimou/spark-odh:s3.0.1-h3.3.0_v0.0.2"
image: "quay.io/opendatahub-contrib/spark:s3.3.0-h3.3.3_latest"
imagePullPolicy: IfNotPresent
serviceAccount: 'spark-operator-spark'
env:
Expand Down
2 changes: 1 addition & 1 deletion test/spark_app_shakespeare_2.4.6.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ spec:
"spark.metrics.conf.*.source.jvm.class": "org.apache.spark.metrics.source.JvmSource"
"spark.metrics.appStatusSource.enabled": "true"
mainApplicationFile: 'local:///home/wordcount.py'
image: "quay.io/guimou/pyspark-odh:s2.4.6-h3.3.0_latest"
image: "quay.io/opendatahub-contrib/pyspark:s2.4.6-h3.3.0_latest"
imagePullPolicy: Always
volumes:
- name: wordcount
Expand Down
2 changes: 1 addition & 1 deletion test/spark_app_shakespeare_3.0.1.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ spec:
"spark.metrics.conf.*.source.jvm.class": "org.apache.spark.metrics.source.JvmSource"
"spark.metrics.appStatusSource.enabled": "true"
mainApplicationFile: 'local:///home/wordcount.py'
image: "quay.io/guimou/pyspark-odh:s3.0.1-h3.3.0_latest"
image: "quay.io/opendatahub-contrib/pyspark:s3.0.1-h3.3.0_latest"
imagePullPolicy: Always
volumes:
- name: wordcount
Expand Down
2 changes: 1 addition & 1 deletion test/spark_app_shakespeare_3.3.0.history_server.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ spec:
"spark.hadoop.fs.s3a.bucket.YOUR_BUCKET.path.style.access": "true"
"spark.hadoop.fs.s3a.bucket.YOUR_BUCKET.connection.ssl.enabled": "false"
mainApplicationFile: 'local:///home/wordcount.py'
image: "quay.io/guimou/pyspark-odh:s3.3.0-h3.3.3_latest"
image: "quay.io/opendatahub-contrib/pyspark:s3.3.0-h3.3.3_latest"
imagePullPolicy: Always
volumes:
- name: wordcount
Expand Down
2 changes: 1 addition & 1 deletion test/spark_app_shakespeare_3.3.0.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ spec:
"spark.metrics.conf.*.source.jvm.class": "org.apache.spark.metrics.source.JvmSource"
"spark.metrics.appStatusSource.enabled": "true"
mainApplicationFile: 'local:///home/wordcount.py'
image: "quay.io/guimou/pyspark-odh:s3.3.0-h3.3.3_latest"
image: "quay.io/opendatahub-contrib/pyspark:s3.3.0-h3.3.3_latest"
imagePullPolicy: Always
volumes:
- name: wordcount
Expand Down

0 comments on commit 4438928

Please sign in to comment.