BUG: Configuration Store: supplementalVolumes.readOnly = true is causing the Out of Sync status in ArgoCD

[csun-cpointe]

Description

When using argocd to deploy and sync the configuration-store app, the sync status will remain Out of Sync. When clicking into the pod, it will show the live manifest and desired manifest are different. The difference is supplementalVolumes.readyOnly is true. We need to investigate the root cause of the issue and make the fix.

Steps to Reproduce

Clear, specific, and detailed steps taken to enable reproduction of the bug for investigation.

• Create a downstream project with baseline version 1.9.0-SNAPSHOT

mvn archetype:generate -U -DarchetypeGroupId=com.boozallen.aissemble \
  -DarchetypeArtifactId=foundation-archetype \
  -DarchetypeVersion=1.9.0-SNAPSHOT \
  -DgroupId=com.test \
  -DartifactId=test-defect \
  -DprojectGitUrl=test.url \
  -DprojectName=test-defect \
  && cd test-defect

• Add the SparkPipeline.json file to the test-defect-pipeline-models/src/main/resources/pipelines directory
• Add to the fermenter-mda plugin executions in test-defect-deploy/pom.xml

<execution>
    <id>configuration-store</id>
    <phase>generate-sources</phase>
    <goals>
        <goal>generate-sources</goal>
    </goals>
    <configuration>
        <basePackage>com.boozallen.aissemble.test</basePackage>
        <profile>configuration-store-deploy-v2</profile>
        <!-- The property variables below are passed to the Generation Context and utilized
             to customize the deployment artifacts. -->
        <propertyVariables>
            <appName>configuration-store</appName>
        </propertyVariables>
    </configuration>
</execution>

• Run mvn clean install until all the manual actions are complete
• Once the manual actions are complete, run mvn clean install -Dmaven.build.cache.skipCache=true once to get any remaining manual actions
• Download and unzip the attached 283-helper.zip and copy the src directory to the test-defect root directory
• Modify the helm templates for Argocd deployment:
  • In the test-defect-deploy/src/main/resources/values.yaml file, update the targetRevision and repo to reflect the correct value. e.g.:

   targetRevision: main
   repo: https://github.com/abc/test-defect
  

  • In the -deploy/src/main/resources/apps/configuration-store/ directory copy the content of the values-dev.yaml to the values-ci.yaml file
    • remove service and service.type from the values-ci.yaml file

        service:
            type: LoadBalancer
    

    • Mac User: In the -deploy/src/main/resources/apps/configuration-store/values-ci.yaml file update the volumePathOnNode to be /<pathToProject>/test-defect/src/main/resources/configurations
    • Window User: In the -deploy/src/main/resources/values-ci.yaml file update the volumePathOnNode to be /mnt/c/Users/YOUR_USER/PATH/TO/283-helper/apps/configuration-store/src/main/resources/configurations
• Create a repo for the project created
  • create a new test-defect repository from the github website
  • at the test-defect root directory run below commands
  • git init
  • git add .
  • `git commit -m "init ttest-defect base'
  • git branch -M main
  • git remote add origin https://github.com/<username>/<repo_name>.git
  • git push -u origina main
• Start a local argocd server
  • helm install aissemble-infrastructure oci://ghcr.io/boozallen/aissemble-infrastructure-chart --version 1.9.0-SNAPSHOT --set jenkins.enabled=false --set ingress-nginx.enabled=false
  • Rancher Deskip -> open main window -> Port Forwarding => forward argocd-server to port 8081
  • argocd admin initial-password -n argocd (keep the password to login to argocd ui and cli console)
  • Open the argocd UI http://localhost:8081/ and login as (admin/previous generated password)
  • In the test-defect root directory run argocd login localhost:8081 (Use username: admin/password: previous generated password to login)
  • Run below command to deploy the test-defect app

        argocd app create test-defect \
         --dest-namespace argocd \
         --dest-server https://kubernetes.default.svc \
         --repo https://github.com/csun-cpointe/test-defect \
         --path test-defect-deploy/src/main/resources \
         --revision main \
         --values values.yaml \
         --values values-ci.yaml
  

• On the ArgoCD UI, verify that the test-defect app is shown
• Sync test-defect app and wait the configuration-store app is healthy. The sync status is still Out of Sync

   argocd app sync test-defect --revision main
  

Expected Behavior

A clear and concise description of the outcome you expected from the actions taken.
The configuration-store app should be deployed and sync successfully.

Actual Behavior

A clear and concise description of the outcome you experienced from the actions taken.
The configuration-store app is out of sync after deployment.

Additional Context

• Log output
• Screenshots (if applicable)
• Solution Baseline Version
• Environment details (local, cloud, Azure, AWS, etc.)