Feature: Enhance security of the aissemble-spark baseline image

[jocobtt]

Description

The current aissemble-spark Docker image contains approximately 16 critical vulnerabilities as identified by security scans. These vulnerabilities are causing the image to be flagged in cloud environments, necessitating downstream projects to undertake additional remediation efforts.

The base image, apache/spark-py, utilized in aissemble-spark is outdated. To mitigate the identified vulnerabilities and improve overall security posture, we propose migrating to a more actively maintained PySpark base image, such as bitnami/spark.

DOD

• aissemble-spark image is patched to remove resulting critical CVE vulnerabilities before it is released to downstream projects

Test Strategy/Script

• Build the aissemble-spark image and ensure there aren't critical CVE vulnerabilities reported when scanned by Trivy image scanner or any similar container scanning tools.

References/Additional Context

N/A