[BUG] salt.exceptions.FileserverConfigError: Failed to load git_pillar when git_pillar.update run from saltutil.runner

[ggiesen]

Description
The git_pillar.update runner, when called from saltutil.runner, returns a FileServerConfigError exception.

Setup

ext_pillar:
  - git:
     - master ssh://[email protected]/org/git_salt_pillar.git:
         - pubkey: /etc/salt/pki/master/git_id_rsa.pub
         - privkey: /etc/salt/pki/master/git_id_rsa
         - name: git_salt_pillar
         - root: pillar

Please be as specific as possible and give set-up details.

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior

# salt salt01 saltutil.runner git_pillar.update
salt01:
    Exception occurred in runner git_pillar.update: Traceback (most recent call last):
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/gitfs.py", line 513, in __init__
        self.new = self.init_remote()
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/gitfs.py", line 2016, in init_remote
        self.repo = pygit2.Repository(self._cachedir)
      File "/opt/saltstack/salt/extras-3.10/pygit2/repository.py", line 1606, in __init__
        path_backend = init_file_backend(path, flags)
    _pygit2.GitError: Repository not found at /var/cache/salt/master/git_pillar/git_salt_pillar/_

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/mixins.py", line 388, in low
        data["return"] = func(*args, **kwargs)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
        ret = self.loader.run(run_func, *args, **kwargs)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
        return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
        ret = _func_or_method(*args, **kwargs)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/runners/git_pillar.py", line 70, in update
        pillar = salt.utils.gitfs.GitPillar(
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/gitfs.py", line 2557, in __init__
        self.init_remotes(
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/gitfs.py", line 2616, in init_remotes
        repo_obj = self.git_providers[self.provider](
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/gitfs.py", line 1740, in __init__
        super().__init__(
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/gitfs.py", line 521, in __init__
        failhard(self.role)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/gitfs.py", line 229, in failhard
        raise FileserverConfigError(f"Failed to load {role}")
    salt.exceptions.FileserverConfigError: Failed to load git_pillar

It works fine if the runner is called directly:

# salt-run git_pillar.update
master ssh://[email protected]/org/git_salt_pillar.git:
    None

Expected behavior

# salt salt01 saltutil.runner git_pillar.update
salt01:
    ----------
    master ssh://[email protected]/org/git_salt_pillar.git:
        None

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

salt --versions-report
Salt Version:
              Salt: 3006.9

Python Version:
            Python: 3.10.14 (main, Jun 26 2024, 11:44:37) [GCC 11.2.0]

Dependency Versions:
              cffi: 1.14.6
          cherrypy: unknown
      cryptography: 42.0.5
          dateutil: 2.8.1
         docker-py: Not Installed
             gitdb: Not Installed
         gitpython: Not Installed
            Jinja2: 3.1.4
           libgit2: 1.5.0
      looseversion: 1.0.2
          M2Crypto: Not Installed
              Mako: Not Installed
           msgpack: 1.0.2
      msgpack-pure: Not Installed
      mysql-python: Not Installed
         packaging: 22.0
         pycparser: 2.21
          pycrypto: Not Installed
      pycryptodome: 3.19.1
            pygit2: 1.11.1
      python-gnupg: 0.4.8
            PyYAML: 6.0.1
             PyZMQ: 23.2.0
            relenv: 0.17.0
             smmap: Not Installed
           timelib: 0.2.4
           Tornado: 4.5.3
               ZMQ: 4.3.4

Salt Extensions:
 saltext.bitwarden: 0.0.1b15

System Versions:
              dist: almalinux 8.10 Cerulean Leopard
            locale: utf-8
           machine: x86_64
           release: 4.18.0-553.30.1.el8_10.x86_64
            system: Linux
           version: AlmaLinux 8.10 Cerulean Leopard

Additional context
This is used in an orch to update my Salt masters when they receive an event (via webhook) from the hosted git provider that the repo has been updated. Previously worked (although not entirely sure which version broke).

[ggiesen]

# cd /var/cache/salt/master/git_pillar/git_salt_pillar/_ && sudo -u salt git status
On branch master
nothing to commit, working tree clean

Although if I run it as root:

# cd /var/cache/salt/master/git_pillar/git_salt_pillar/_ && git status
fatal: detected dubious ownership in repository at '/var/cache/salt/master/git_pillar/git_salt_pillar/_'
To add an exception for this directory, call:

        git config --global --add safe.directory /var/cache/salt/master/git_pillar/git_salt_pillar/_

I wonder if it could be a result of the change to running master as salt, while the minion is running as root.

[ggiesen]

Indeed it is:

cd /var/cache/salt/master/git_pillar/git_salt_pillar/_ && git config --global --add safe.directory /var/cache/salt/master/git_pillar/git_salt_pillar/_
# salt salt01 saltutil.runner git_pillar.update
salt01:
    ----------
    master ssh://[email protected]/org/git_salt_pillar.git:
        None

[ggiesen]

Unless I misunderstand how the saltutil.runner execution module is supposed to work, shouldn't this run as the salt user rather than root?

[ggiesen]

Indeed it is:

cd /var/cache/salt/master/git_pillar/git_salt_pillar/_ && git config --global --add safe.directory /var/cache/salt/master/git_pillar/git_salt_pillar/_
# salt salt01 saltutil.runner git_pillar.update
salt01:
    ----------
    master ssh://[email protected]/org/git_salt_pillar.git:
        None

This isn't a fix as you'll end up with a bunch of files in .git owned by root which then prevents the regular runner from running.