Sidecar: avp discover step is racy

[ptimofee]

Describe the bug
Currently there is documentation describing how to setup discover step in ConfigManagementPlugin for a sidecar, so that if the dir contains a helm chart, a kustomize file or just simple yaml manifests auto-detection would work.
The problem appears when you enable all sidecar plugins at once: avp, avp-helm and avp-kustomize.
It's racy how avp determines if it's a simple dir with plain yaml manifests. It will detect kustomize.yaml and Chart.yaml as well. So it's just a matter of race condition when avp takes over avp-helm or avp-kustomize.

...
data:
  avp.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
...
      discover:
        find:
          command:
            - sh
            - "-c"
            - "find . -name '*.yaml' | xargs -I {} grep \"<path\\|avp\\.kubernetes\\.io\" {} | grep ."
...

I know helm vs kustomize have similar problem. So in general you should not mix Chart.yaml and kustomize.yaml in one folder. But even if your dir is clearly a helm chart or a kustomize the avp plugin still can prevail if it wins the race:

Discovery (Auto-selection of Tool)
- The plugin discovery will run in the main repo-server container.
- Argo CD repo-server lists the shared plugins directory and runs discover command from the specification file, whichever plugin provides a positive response first will be selected.

taken from https://argo-cd.readthedocs.io/en/stable/proposals/config-management-plugin-v2/

Expected behavior
avp plugin should have a better way to determine if the dir contains only yaml manifests, but not Chart.yaml or kustomize.yaml

[ptimofee]

Ok, seems like the official ArgoCD documentation is misleading.
What I observe is:

1. repo-server lists socket files in /home/argocd/cmp-server/plugins/ created by sidecar containers, i. e. plugins
2. repo-server sends the repo content to those sockets in the order which was returned by golang's os.ReadDir, i. e. same order as you run ls
3. the actual discover step is run in a sidecar container, not in main repo-server container
4. if discover passes, then repo-server has found the right plugin. If not it takes the next socket returned by os.ReadDir and wait for discover step to pass

If I'm not mistaken (please correct me if I'm wrong) it's a matter of plugin (its socket) naming which plugin will be chosen, i. e. metadata.name is a config yaml in cmp-plugin configMap.
Again, if I'm right this detail has to be documented.