[CSPL-2699]: adding Azure and GCP bucket access using sdk (#1340)

* adding gcp bucket access using sdk Signed-off-by: Vivek Reddy <[email protected]> * adding gcp skd Signed-off-by: Vivek Reddy <[email protected]> * working code cod Signed-off-by: Vivek Reddy <[email protected]> * working code Signed-off-by: Vivek Reddy <[email protected]> * azure blob sdk support Signed-off-by: Vivek Reddy <[email protected]> * azure blob sdk support Signed-off-by: Vivek Reddy <[email protected]> * logging fixed Signed-off-by: Vivek Reddy <[email protected]> * logging fixed Signed-off-by: Vivek Reddy <[email protected]> * azure code support for AzureAD Signed-off-by: Vivek Reddy <[email protected]> * azure code support for AzureAD Signed-off-by: Vivek Reddy <[email protected]> * azure code support for AzureAD Signed-off-by: Vivek Reddy <[email protected]> * adding validation Signed-off-by: Vivek Reddy <[email protected]> * go updates Signed-off-by: Vivek Reddy <[email protected]> * pipeline test for azure Signed-off-by: Vivek Reddy <[email protected]> * adding gcp pipeline Signed-off-by: Vivek Reddy <[email protected]> * fixed buildx Signed-off-by: Vivek Reddy <[email protected]> * using buildx Signed-off-by: Vivek Reddy <[email protected]> * using buildx Signed-off-by: Vivek Reddy <[email protected]> * fixed dotenv version Signed-off-by: Vivek Reddy <[email protected]> * fixed dotenv version Signed-off-by: Vivek Reddy <[email protected]> * fixed docker login Signed-off-by: Vivek Reddy <[email protected]> * fixed docker login Signed-off-by: Vivek Reddy <[email protected]> * fixed docker login Signed-off-by: Vivek Reddy <[email protected]> * fixed cluster name Signed-off-by: Vivek Reddy <[email protected]> * fixed gke cloud auth plugin issue Signed-off-by: Vivek Reddy <[email protected]> * fixed gke cloud auth plugin issue Signed-off-by: Vivek Reddy <[email protected]> * gcp variable in script Signed-off-by: Vivek Reddy <[email protected]> * gcp variable in workflow Signed-off-by: Vivek Reddy <[email protected]> * adding gcp test cases for appframework Signed-off-by: Vivek Reddy <[email protected]> * adding gcp test cases for appframework Signed-off-by: Vivek Reddy <[email protected]> * modification to support gcp in test Signed-off-by: Vivek Reddy <[email protected]> * modification to support gcp in test Signed-off-by: Vivek Reddy <[email protected]> * modification to support gcp in test Signed-off-by: Vivek Reddy <[email protected]> * adding gcp tag in test case Signed-off-by: Vivek Reddy <[email protected]> * adding gcp tag in test case Signed-off-by: Vivek Reddy <[email protected]> * adding gcp workload management Signed-off-by: Vivek Reddy <[email protected]> * testing something here * reverting the change * adding back service account * adding only smoke test Signed-off-by: Vivek Reddy <[email protected]> * adding gcloud * adding only smoke test Signed-off-by: Vivek Reddy <[email protected]> * adding glcoud cli * adding bucket * debugging * debugging * debugging upload * adding index bucket * adding secret logic test Signed-off-by: Vivek Reddy <[email protected]> * adding secret logic test Signed-off-by: Vivek Reddy <[email protected]> * adding secret logic test Signed-off-by: Vivek Reddy <[email protected]> * added script to create gcp cluster * added using credentials for gcp * changing glcoud to gcp as key * changing gcp service account key field * updates test cases * adding logic to use base64 encode Signed-off-by: Vivek Reddy <[email protected]> * adding logic to use base64 encode Signed-off-by: Vivek Reddy <[email protected]> * use base64 key * fixed code for gcp Signed-off-by: Vivek Reddy <[email protected]> * fixed apps listing * initial doc changes Signed-off-by: Vivek Reddy <[email protected]> * adding test to create mutliple clusters Signed-off-by: Vivek Reddy <[email protected]> * fixed test case to be less than 40 characters * fixed cluster name * testing cluster name * fixed cluster name * fixed bug in workflow * adding enterprise license location * doc changes * log message changed for GCP * setting license path * commenting all the test cases except one * adding gcp_sanity Signed-off-by: Vivek Reddy <[email protected]> * doc changes and comment test Signed-off-by: Vivek Reddy <[email protected]> * workflow changes Signed-off-by: Vivek Reddy <[email protected]> * doc changes Signed-off-by: Vivek Reddy <[email protected]> * adding gcp_sanity Signed-off-by: Vivek Reddy <[email protected]> * crd changes * test script changes and formats * adding back C3 test cases for gcp * adding back m4 test cases * only one C3 test case to run * added storage type as gcs * adding only 3 test cases * azure s1 only 1 test case enabled * changed path in azure * removed startAfter as its same prefix * changing storageType to gcs * adding back azure test cases * adding 2 test in c3 for gcp * adding in this c3 gcp_sanity test cases * adding only c3 test * removed unused test cases * rerunning test cases * disabling upgrade check * changed the upgrade flow * fixed indexer in upgrade flow * document changes to support azure and gcp bucket using native sdk * doc changes for gcp * fixed azure documentation * fixed license path * just increasing timeout for now * gcp test fixed * added copyright * group test cases based on SVA * fixed test focus for cluster name * removed commented code * removed commented code * removed mock client for http azure * fixed instance type Signed-off-by: Vivek Reddy <[email protected]> * adding disk size * adding cluster-up and cluster-down * updated gke scripts * remove cluster-down * removed k8s version * adding only s1 Signed-off-by: Vivek Reddy <[email protected]> * only m4 manager Signed-off-by: Vivek Reddy <[email protected]> * only m4 Signed-off-by: Vivek Reddy <[email protected]> * only c3 manager Signed-off-by: Vivek Reddy <[email protected]> * only c3 Signed-off-by: Vivek Reddy <[email protected]> * only c3 manager Signed-off-by: Vivek Reddy <[email protected]> * only m4 Signed-off-by: Vivek Reddy <[email protected]> * only s1 test cases * adding go mod changes * restoring timeout * comments added * fixed run test configuration * fixing the workflow config for now * adding default storage class gcp * running all the test cases * removed commented code from wf * commented cleanup * review comments * increasing timeout * fixed timeout * ignore mc pod reset detection * only adding 4 test cases * commenting 2 test cases * adding specific time for test case * formatting * removed smoke from gcp test cases --------- Signed-off-by: Vivek Reddy <[email protected]> Co-authored-by: Vivek Reddy <[email protected]>
splunk · Nov 21, 2024 · 07714c7 · 07714c7
1 parent 02c0cf3
commit 07714c7
Show file tree

Hide file tree

Showing 67 changed files with 15,501 additions and 1,912 deletions.
diff --git a/.github/workflows/int-test-azure-workflow.yml b/.github/workflows/int-test-azure-workflow.yml
@@ -38,11 +38,7 @@ jobs:
         password: ${{ secrets.AZURE_ACR_DOCKER_PASSWORD }}
     - name: Make Splunk Operator Image
       run: |
-        make docker-build IMG=${{ secrets.AZURE_ACR_LOGIN_SERVER }}/${{ env.SPLUNK_OPERATOR_IMAGE_NAME }}:$GITHUB_SHA
-    - name: Push Splunk Operator Image to the Container Registry
-      run: |
-        echo "Uploading Image to the Container Registry :: ${{ env.SPLUNK_OPERATOR_IMAGE_NAME }}:$GITHUB_SHA"
-        make docker-push  IMG=${{ secrets.AZURE_ACR_LOGIN_SERVER }}/${{ env.SPLUNK_OPERATOR_IMAGE_NAME }}:$GITHUB_SHA
+        make docker-buildx IMG=${{ secrets.AZURE_ACR_LOGIN_SERVER }}/${{ env.SPLUNK_OPERATOR_IMAGE_NAME }}:$GITHUB_SHA
   setup-aks-cluster:
     runs-on: ubuntu-latest
     needs: build-operator-image

diff --git a/.github/workflows/int-test-gcp-workflow.yml b/.github/workflows/int-test-gcp-workflow.yml
@@ -0,0 +1,277 @@
+name: Integration Test on GCP Workflow
+
+on:
+  push:
+    branches:
+      - develop
+      - main
+
+jobs:
+  build-operator-image:
+    runs-on: ubuntu-latest
+    env:
+      SPLUNK_ENTERPRISE_IMAGE: ${{ secrets.SPLUNK_ENTERPRISE_IMAGE }}
+      SPLUNK_OPERATOR_IMAGE_NAME: splunk/splunk-operator
+      ARTIFACT_REGISTRY: ${{ secrets.GCP_ARTIFACT_REGISTRY }}  # Updated for Artifact Registry
+    steps:
+      - name: Checkout Code
+        uses: actions/checkout@v2
+
+      - name: Load Environment Variables
+        id: dotenv
+        uses: falti/dotenv-action@d4d12eaa0e1dd06d5bdc3d7af3bf4c8c93cb5359
+        with:
+          path: .env  # Adjust the path if your dotenv file is located elsewhere
+
+      - name: Setup Go
+        uses: actions/setup-go@v2
+        with:
+          go-version: ${{ steps.dotenv.outputs.GO_VERSION }}
+
+      - name: Set up Docker Buildx
+        uses: docker/[email protected]
+
+      - name: Install Operator SDK
+        run: |
+          ARCH=$(case $(uname -m) in
+            x86_64) echo -n amd64 ;;
+            aarch64) echo -n arm64 ;;
+            *) echo -n $(uname -m) ;;
+          esac)
+          OS=$(uname | awk '{print tolower($0)}')
+          OPERATOR_SDK_DL_URL=https://github.com/operator-framework/operator-sdk/releases/download/${{ steps.dotenv.outputs.OPERATOR_SDK_VERSION }}
+          curl -LO ${OPERATOR_SDK_DL_URL}/operator-sdk_${OS}_${ARCH}
+          chmod +x operator-sdk_${OS}_${ARCH}
+          sudo mv operator-sdk_${OS}_${ARCH} /usr/local/bin/operator-sdk
+
+      - name: Authenticate to GCP
+        uses: google-github-actions/auth@v1
+        with:
+          credentials_json: ${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}
+
+      - name: Login to GCR
+        uses: docker/login-action@v3
+        with:
+          registry: ${{ secrets.GCP_ARTIFACT_REGISTRY }}
+          username: _json_key
+          password: ${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}
+
+      - name: Build Splunk Operator Image
+        run: |
+          make docker-buildx IMG=${{ secrets.GCP_ARTIFACT_REGISTRY }}/${{ env.SPLUNK_OPERATOR_IMAGE_NAME }}:$GITHUB_SHA
+
+  create-cluster-and-run-tests:
+    strategy:
+      matrix:
+        test_focus: 
+          - { order: 1, name: "c3_gcp_sanity" }
+          - { order: 2, name: "c3_mgr_gcp_sanity" }
+          - { order: 3, name: "m4_gcp_sanity" }
+          - { order: 4, name: "m4_mgr_gcp_sanity" }
+          - { order: 5, name: "s1_gcp_sanity" }
+    runs-on: ubuntu-latest
+    needs: build-operator-image
+    env:
+      CLUSTER_WORKERS: 5
+      TEST_CLUSTER_PLATFORM: gcp
+      CLUSTER_PROVIDER: gcp
+      ARTIFACT_REGISTRY: ${{ secrets.GCP_ARTIFACT_REGISTRY }}
+      GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
+      GCP_REGION: ${{ secrets.GCP_REGION }}
+      AWS_S3_REGION: ${{ secrets.GCP_REGION }}
+      GCP_ZONE: ${{ secrets.GCP_ZONE }}
+      GCP_NETWORK: default  # Adjust if using a custom network
+      GCP_SUBNETWORK: default  # Adjust if using a custom subnetwork
+      TEST_FOCUS: ${{ matrix.test_focus.name }}
+      CLUSTER_NODES: 2
+      SPLUNK_ENTERPRISE_IMAGE: ${{ secrets.SPLUNK_ENTERPRISE_IMAGE }}
+      SPLUNK_ENTERPRISE_RELEASE_IMAGE: ${{ secrets.SPLUNK_ENTERPRISE_RELEASE_IMAGE }}
+      SPLUNK_OPERATOR_IMAGE_NAME: splunk/splunk-operator
+      SPLUNK_OPERATOR_IMAGE_FILENAME: splunk-operator
+      TEST_TO_SKIP:  "^(?:[^s]+|s(?:$|[^m]|m(?:$|[^o]|o(?:$|[^k]|k(?:$|[^e])))))*$"
+      TEST_BUCKET: ${{ secrets.TEST_BUCKET }}
+      TEST_S3_BUCKET: ${{ secrets.TEST_BUCKET }}
+      TEST_INDEXES_S3_BUCKET: ${{ secrets.TEST_INDEXES_S3_BUCKET }}
+      INDEXES_S3_BUCKET: ${{ secrets.TEST_INDEXES_S3_BUCKET }}
+      GCP_ENTERPRISE_LICENSE_LOCATION: "test_licenses"
+      ENTERPRISE_LICENSE_LOCATION: "test_licenses"
+      ENTERPRISE_LICENSE_S3_PATH: "test_licenses"
+      REGISTRY_REPOSITORY: ${{ secrets.GCP_ARTIFACT_REGISTRY }}
+      CLUSTER_WIDE: "true"
+      GCP_SERVICE_ACCOUNT_ENABLED: "false"
+      PRIVATE_REGISTRY: ${{ secrets.GCP_ARTIFACT_REGISTRY }}
+      GCP_STORAGE_ACCOUNT: ${{ secrets.GCP_STORAGE_ACCOUNT }}
+      GCP_STORAGE_ACCOUNT_KEY: ${{ secrets.GCP_STORAGE_ACCOUNT_KEY }}
+      GCP_TEST_CONTAINER: ${{ secrets.GCP_TEST_CONTAINER}}
+      GCP_INDEXES_CONTAINER: ${{ secrets.GCP_INDEXES_CONTAINER}}
+      ECR_REPOSITORY: ${{ secrets.GCP_ARTIFACT_REGISTRY }}
+      GCP_CONTAINER_REGISTRY_LOGIN_SERVER: ${{ secrets.AZURE_ACR_LOGIN_SERVER }}
+    steps:
+      - name: Set Test Cluster Name
+        run: |
+          echo "CLUSTER_NAME=gke-${{ matrix.test_focus.order }}-$GITHUB_RUN_ID" >> $GITHUB_ENV
+          echo "TEST_CLUSTER_NAME=gke-${{ matrix.test_focus.order }}-$GITHUB_RUN_ID" >> $GITHUB_ENV
+      - name: Checkout Code
+        uses: actions/checkout@v2
+
+      - name: Load Environment Variables
+        id: dotenv
+        uses: falti/dotenv-action@d4d12eaa0e1dd06d5bdc3d7af3bf4c8c93cb5359
+        with:
+          path: .env
+
+      - name: Authenticate to GCP
+        uses: google-github-actions/auth@v1
+        with:
+          credentials_json: ${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}
+
+      - name: Set up Cloud SDK
+        uses: google-github-actions/setup-gcloud@v1
+        with:
+          project_id: ${{ secrets.GCP_PROJECT_ID }}
+          install_components: 'kubectl'
+
+      - name: Set GCP Project
+        run: |
+          gcloud config set project ${{ env.GCP_PROJECT_ID }}
+
+      - name: Create GKE Cluster
+        run: |
+          export EKS_CLUSTER_K8_VERSION=${{ steps.dotenv.outputs.EKS_CLUSTER_K8_VERSION }}
+          export GKE_CLUSTER_K8_VERSION=${{ steps.dotenv.outputs.EKS_CLUSTER_K8_VERSION }}
+          make cluster-up
+
+      - name: Get Kubernetes Credentials
+        run: |
+          gcloud container clusters get-credentials ${{ env.CLUSTER_NAME }} --zone ${{ env.GCP_ZONE }} --project ${{ env.GCP_PROJECT_ID }}
+
+      - name: Allow Pulling from Artifact Registry
+        run: |
+          gcloud auth configure-docker ${{ secrets.GCP_ARTIFACT_REGISTRY }}
+
+      - name: Set up Cloud SDK
+        uses: google-github-actions/setup-gcloud@v1
+        with:
+          project_id: ${{ secrets.GCP_PROJECT_ID }}
+          install_components: 'kubectl'
+
+      - name: Change Splunk Enterprise Image on Main Branches
+        if: github.ref == 'refs/heads/main'
+        run: |
+          echo "SPLUNK_ENTERPRISE_IMAGE=${{ steps.dotenv.outputs.SPLUNK_ENTERPRISE_RELEASE_IMAGE }}" >> $GITHUB_ENV
+
+      - name: Authenticate to GCP
+        uses: google-github-actions/auth@v1
+        with:
+          credentials_json: ${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}
+
+      - name: Set GCP Project
+        run: |
+          gcloud config set project ${{ env.GCP_PROJECT_ID }}
+
+      - name: Install Kubectl
+        uses: azure/setup-kubectl@v3
+        with:
+          version: ${{ steps.dotenv.outputs.KUBECTL_VERSION }}
+
+      - name: Install Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: '3.x'  # Specify the Python version if needed
+
+      - name: Setup Go
+        uses: actions/setup-go@v2
+        with:
+          go-version: ${{ steps.dotenv.outputs.GO_VERSION }}
+
+      - name: Install Go Lint
+        run: |
+          go version
+          go install golang.org/x/lint/golint@latest
+
+      - name: Install Ginkgo
+        run: |
+          make setup/ginkgo
+
+      - name: Set up Docker Buildx
+        uses: docker/[email protected]
+
+      - name: Login to GCR
+        uses: docker/login-action@v3
+        with:
+          registry: ${{ secrets.GCP_ARTIFACT_REGISTRY }}
+          username: _json_key
+          password: ${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}
+
+      - name: Pull Splunk Enterprise Image 
+        run: docker pull ${{ env.SPLUNK_ENTERPRISE_IMAGE }}
+
+      - name: Pull Splunk Operator Image Locally
+        run: |
+          docker pull ${{ secrets.GCP_ARTIFACT_REGISTRY }}/${{ env.SPLUNK_OPERATOR_IMAGE_NAME }}:$GITHUB_SHA
+          docker tag ${{ secrets.GCP_ARTIFACT_REGISTRY }}/${{ env.SPLUNK_OPERATOR_IMAGE_NAME }}:$GITHUB_SHA ${{ env.SPLUNK_OPERATOR_IMAGE_NAME }}:$GITHUB_SHA
+          echo "SPLUNK_OPERATOR_IMAGE=${{ env.SPLUNK_OPERATOR_IMAGE_NAME }}:$GITHUB_SHA" >> $GITHUB_ENV
+
+      - name: Tag and Push Splunk Enterprise Image to Artifact Registry
+        run: |
+          docker tag ${{ env.SPLUNK_ENTERPRISE_IMAGE }} ${{ secrets.GCP_ARTIFACT_REGISTRY }}/${{ env.SPLUNK_ENTERPRISE_IMAGE }}
+          docker push ${{ secrets.GCP_ARTIFACT_REGISTRY }}/${{ env.SPLUNK_ENTERPRISE_IMAGE }}
+
+      - name: Get Kubernetes Credentials
+        run: |
+          gcloud container clusters get-credentials ${{ env.CLUSTER_NAME }} --zone ${{ env.GCP_ZONE }} --project ${{ env.GCP_PROJECT_ID }}
+
+      - name: Get GKE Credentials
+        uses: google-github-actions/get-gke-credentials@v1
+        with:
+          cluster_name: ${{ env.CLUSTER_NAME }}
+          location: ${{ env.GCP_ZONE }}
+
+      - name: Install Metrics Server
+        run: |
+          kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
+
+      - name: Install Kubernetes Dashboard
+        run: |
+          kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.5/aio/deploy/recommended.yaml
+
+      - name: Setup Kustomize
+        run: |
+          curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash
+          sudo mv kustomize /usr/local/bin/
+
+      - name: Verify kubectl Configuration
+        run: |
+          kubectl config current-context
+      
+      - name: Apply StorageClass
+        run: |
+          kubectl apply -f test/gcp-storageclass.yaml
+
+      - name: Run Integration Tests
+        run: |
+          export GCP_SERVICE_ACCOUNT_KEY=${{ secrets.GCP_SERVICE_ACCOUNT_KEY_BASE64 }}
+          make int-test
+
+      - name: Collect Test Logs
+        if: ${{ always() }}
+        run: |
+          mkdir -p /tmp/pod_logs
+          find ./test -name "*.log" -exec cp {} /tmp/pod_logs \;
+
+      - name: Archive Pod Logs
+        if: ${{ always() }}
+        uses: actions/upload-artifact@v4
+        with:
+          name: splunk-pods-logs-artifacts-${{ matrix.test_focus.name }}
+          path: /tmp/pod_logs/**
+      - name: Cleanup Test Case Artifacts
+        if: ${{ always() }}
+        run: |
+          export EKS_CLUSTER_K8_VERSION=${{ steps.dotenv.outputs.EKS_CLUSTER_K8_VERSION }}
+          export GKE_CLUSTER_K8_VERSION=${{ steps.dotenv.outputs.EKS_CLUSTER_K8_VERSION }}
+          tools/cleanup.sh
+      - name: Cleanup up EKS cluster
+        if: ${{ always() }}
+        run: |
+          make cluster-down
diff --git a/Makefile b/Makefile
@@ -156,7 +156,7 @@ docker-buildx: test ## Build and push docker image for the manager for cross-pla
 	sed -e '1 s/\(^FROM\)/FROM --platform=\$$\{BUILDPLATFORM\}/; t' -e ' 1,// s//FROM --platform=\$$\{BUILDPLATFORM\}/' Dockerfile > Dockerfile.cross
 	- docker buildx create --name project-v3-builder
 	docker buildx use project-v3-builder
-	- docker buildx build --push --platform=$(PLATFORMS) --tag ${IMG} -f Dockerfile.cross
+	- docker buildx build --push --platform=$(PLATFORMS) --tag ${IMG} -f Dockerfile.cross .
 	- docker buildx rm project-v3-builder
 	rm Dockerfile.cross
 

diff --git a/api/v4/common_types.go b/api/v4/common_types.go
@@ -308,10 +308,10 @@ type VolumeSpec struct {
 	// Secret object name
 	SecretRef string `json:"secretRef"`
 
-	// Remote Storage type. Supported values: s3, blob. s3 works with aws or minio providers, whereas blob works with azure provider.
+	// Remote Storage type. Supported values: s3, blob, gcs. s3 works with aws or minio providers, whereas blob works with azure provider, gcs works for gcp.
 	Type string `json:"storageType"`
 
-	// App Package Remote Store provider. Supported values: aws, minio, azure.
+	// App Package Remote Store provider. Supported values: aws, minio, azure, gcp.
 	Provider string `json:"provider"`
 
 	// Region of the remote storage volume where apps reside. Used for aws, if provided. Not used for minio and azure.