error: Error in SSL_CTX use_certificate_chain_file crypto error:8000000D:system library::Permission denied

[kingii98]

Describe the bug
The remote-control feature fails to start due to permission errors accessing the .pem files. The logs show repeated “Permission denied” errors when loading the certificate chain file, causing Unbound to fail initialization.

Dec 17 11:30:26 router unbound[105138]: Dec 17 11:30:26 unbound[105138:0] error: Error in SSL_CTX use_certificate_chain_file crypto error:8000000D:system library::Permission denied
Dec 17 11:30:26 router unbound[105138]: Dec 17 11:30:26 unbound[105138:0] error: and additionally crypto error:10080002:BIO routines::system lib
Dec 17 11:30:26 router unbound[105138]: Dec 17 11:30:26 unbound[105138:0] error: and additionally crypto error:0A080002:SSL routines::system lib
Dec 17 11:30:26 router unbound[105138]: Dec 17 11:30:26 unbound[105138:0] fatal error: could not set up remote-control

Steps to reproduce the behavior:

1. Install Unbound
2. Executing sudo -u unbound unbound-control-setup
3. Enable remote-control in unbound.conf:

remote-control:
    control-enable: yes
    server-key-file: "/etc/unbound/unbound_server.key"
    server-cert-file: "/etc/unbound/unbound_server.pem"
    control-key-file: "/etc/unbound/unbound_control.key"
    control-cert-file: "/etc/unbound/unbound_control.pem"

4. Restart Unbound with sudo systemctl restart unbound

Expected behavior
Unbound should start successfully and allow remote-control commands without permission errors.

System:

• Unbound version: 1.17.1
• OS: Debian 12
• unbound -V output:

Version 1.17.1

Configure line: --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-option-checking --disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --with-pythonmodule --with-pyunbound --enable-subnet --enable-dnstap --enable-systemd --with-libnghttp2 --with-chroot-dir= --with-dnstap-socket-path=/run/dnstap.sock --disable-rpath --with-pidfile=/run/unbound.pid --with-libevent --enable-tfo-client --with-rootkey-file=/usr/share/dns/root.key --enable-tfo-server
Linked libs: libevent 2.1.12-stable (it uses epoll), OpenSSL 3.0.14 4 Jun 2024
Linked modules: dns64 python subnetcache respip validator iterator
TCP Fastopen feature available

BSD licensed, see LICENSE in source package for details.
Report bugs to [email protected] or https://github.com/NLnetLabs/unbound/issues

Additional information

Permissions

 root@router:/etc/unbound# ls -l
 total 24
 -rw-r--r-- 1 unbound unbound  679 Dec 17 13:32 unbound.conf
 drwxr-xr-x 2 unbound unbound 4096 Dec 17 11:43 unbound.conf.d
 -rw------- 1 unbound unbound 2488 Dec 17 13:53 unbound_control.key
 -rw-r----- 1 unbound unbound 1501 Dec 17 13:53 unbound_control.pem
 -rw------- 1 unbound unbound 2484 Dec 17 13:52 unbound_server.key
 -rw-r----- 1 unbound unbound 1549 Dec 17 13:52 unbound_server.pem

Config

server:
  #------------------------
  # Basic Server Settings
  #------------------------
  verbosity: 1
  interface: 127.0.0.1
  port: 5335
  do-ip4: yes
  do-ip6: yes
  do-udp: yes
  do-tcp: yes
  username: "unbound"
  directory: "/etc/unbound"
  
  #------------------------
  # Threading & Performance
  #------------------------
  num-threads: 4
  num-queries-per-thread: 4096
  outgoing-range: 4096
  jostle-timeout: 200
  msg-cache-slabs: 4
  rrset-cache-slabs: 4
  so-reuseport: yes
  
  #------------------------
  # Memory & Cache Settings
  #------------------------
  msg-cache-size: 256M
  rrset-cache-size: 512M
  neg-cache-size: 32M
  key-cache-size: 32M
  cache-min-ttl: 300
  cache-max-ttl: 86400
  infra-cache-numhosts: 5000
  
  #------------------------
  # Buffer Settings
  #------------------------
  msg-buffer-size: 65552
  stream-wait-size: 2M
  edns-buffer-size: 1232
  so-rcvbuf: 2m
  so-sndbuf: 2m
  
  #------------------------
  # Client Limits
  #------------------------
  tcp-mss: 1220
  tcp-idle-timeout: 30000
  incoming-num-tcp: 50
  
  #------------------------
  # Security Settings
  #------------------------
  hide-identity: yes
  hide-version: yes
  harden-glue: yes
  harden-dnssec-stripped: yes
  harden-referral-path: yes
  harden-algo-downgrade: yes
  use-caps-for-id: yes
  qname-minimisation: yes
  aggressive-nsec: yes
  val-clean-additional: yes
  harden-large-queries: yes
  harden-short-bufsize: yes
  
  #------------------------
  # DNSSEC
  #------------------------
  auto-trust-anchor-file: "/var/lib/unbound/root.key"
  root-hints: "/var/lib/unbound/root.hints"
  
  #------------------------
  # Prefetch Settings
  #------------------------
  prefetch: yes
  prefetch-key: yes
  serve-expired: yes
  serve-expired-ttl: 3600
  serve-expired-reply-ttl: 30
  
  #------------------------
  # Private Addresses
  #------------------------
  private-address: 10.0.0.0/8
  private-address: 172.16.0.0/12
  private-address: 192.168.0.0/16
  private-address: 169.254.0.0/16
  private-address: fd00::/8
  private-address: fe80::/10
  private-domain: "local"
  private-domain: "lan"
  
  #------------------------
  # Logging
  #------------------------
#    logfile: "/var/log/unbound/unbound.log"
#    log-time-ascii: yes
#    log-queries: no       
#    log-replies: no        
#    log-local-actions: yes
#    log-servfail: yes
  
  #------------------------
  # Statistics
  #------------------------
  statistics-interval: 3600
  statistics-cumulative: yes
  extended-statistics: yes
  shm-enable: yes

  #------------------------
  # TLS Settings
  #------------------------
  tls-cert-bundle: /etc/ssl/certs/ca-certificates.crt
  tls-win-cert: no
  tls-upstream: yes

  #------------------------
  # Cache Persistence
  #------------------------
  msg-cache-slabs: 4     
  rrset-cache-slabs: 4   
  infra-cache-slabs: 4   

  #------------------------
  # Rate Limiting
  #------------------------
  ratelimit: 1000           
  ratelimit-size: 4m        
  ratelimit-slabs: 4         

  #------------------------
  # TCP Optimizations
  #------------------------
  tcp-reuse-timeout: 15000    
  max-reuse-tcp-queries: 200  
  tcp-auth-query-timeout: 3000 

  #------------------------
  # Infrastructure Cache
  #------------------------
  infra-cache-min-rtt: 50    
  infra-cache-max-rtt: 120000 
  infra-keep-probing: yes    

  #------------------------
  # Query Management
  #------------------------
  unwanted-reply-threshold: 10000000  
  minimal-responses: yes              
  qname-minimisation-strict: no     
  rrset-roundrobin: yes             

#------------------------
# Remote Control
#------------------------
remote-control:
 control-enable: yes
 control-interface: 127.0.0.1
 control-port: 8953
 server-key-file: "/etc/unbound/unbound_server.key"
 server-cert-file: "/etc/unbound/unbound_server.pem"
 control-key-file: "/etc/unbound/unbound_control.key"
 control-cert-file: "/etc/unbound/unbound_control.pem"

#------------------------
# Forward Zones
#------------------------
forward-zone:
  name: "."
  forward-tls-upstream: yes        # Enforce DNS-over-TLS
  forward-no-cache: no             # Enable caching
  
  # Cloudflare DNS
  forward-addr: 1.1.1.1@853#cloudflare-dns.com
  forward-addr: 1.0.0.1@853#cloudflare-dns.com
  
  # Quad9 (filtered, DNSSEC enabled)
  forward-addr: 9.9.9.9@853#dns.quad9.net
  forward-addr: 149.112.112.112@853#dns.quad9.net
  
  # Google DNS
  forward-addr: 8.8.8.8@853#dns.google
  forward-addr: 8.8.4.4@853#dns.google
  
  # AdGuard DNS (filtered)
  forward-addr: 94.140.14.14@853#dns.adguard.com
  forward-addr: 94.140.15.15@853#dns.adguard.com
  
  # OpenDNS
  forward-addr: 146.112.41.2@853#dns.opendns.com
  forward-addr: 146.112.41.3@853#dns.opendns.com

  # Cleanbrowsing (family filter)
  forward-addr: 185.228.168.168@853#security-filter-dns.cleanbrowsing.org
  forward-addr: 185.228.169.168@853#security-filter-dns.cleanbrowsing.org

Unit

[Unit]
Description=Unbound DNS server
Documentation=man:unbound(8)
After=network.target
Before=nss-lookup.target
Wants=nss-lookup.target
[Service]
Type=notify
Restart=on-failure
EnvironmentFile=-/etc/default/unbound
ExecStartPre=-/usr/libexec/unbound-helper chroot_setup
ExecStartPre=-/usr/libexec/unbound-helper root_trust_anchor_update
ExecStart=/usr/sbin/unbound -d -p $DAEMON_OPTS
ExecStopPost=-/usr/libexec/unbound-helper chroot_teardown
ExecReload=+/bin/kill -HUP $MAINPID
ExecStartPre=/bin/sleep 10
[Install]
WantedBy=multi-user.target

/etc/apparmor.d/usr.sbin.unbound

# Author: Simon Deziel
# vim:syntax=apparmor
#include <tunables/global>

profile unbound /usr/sbin/unbound flags=(attach_disconnected) {
#include <abstractions/base>
#include <abstractions/nameservice>
#include <abstractions/openssl>

# chown (chgrp) the Unix control socket
capability chown,
# chmod the Unix control socket
capability fowner,
capability fsetid,

# added to abstractions/nameservices in Apparmor 2.12
/var/lib/sss/mc/initgroups r,

capability net_bind_service,
capability setgid,
capability setuid,
capability sys_chroot,
capability sys_resource,

# root hints from dns-data-root
/usr/share/dns/root.* r,

# non-chrooted paths
/etc/unbound/** r,
owner /etc/unbound/*.key* rw,
owner /etc/unbound/*.pem r,
# explicitly deny (and audit) attempts to write to the key files
# this should be unnecessary after switch to /run/unbound.ctl control socket
# (here and below)
#  audit deny /etc/unbound/unbound_control.{key,pem} rw,
#  audit deny /etc/unbound/unbound_server.key w,
/etc/unbound/unbound_control.key r,
/etc/unbound/unbound_control.pem r,
/etc/unbound/*.pem r,
/etc/unbound/*.key r,

# chrooted paths
# unbound can be chrooted into /etc/unbound (upstream default) with
#  /var/lib/unbound/ bind-mounted to /etc/unbound/var/lib/unbound/,
# or it can be chrooted into /var/lib/unbound/ with /etc/unbound/ copied
# into there (previous debian package default).
/{,etc/unbound/}var/lib/unbound/** r,
owner /{,etc/unbound/}var/lib/unbound/** rw,
audit deny /{,etc/unbound/}var/lib/unbound/**/unbound_control.{key,pem} rw,
audit deny /{,etc/unbound/}var/lib/unbound/**/unbound_server.key w,

/usr/sbin/unbound mr,

/run/systemd/notify w,
/run/unbound.pid rw,

# Unix control socket
/run/unbound.ctl rw,

#include <local/usr.sbin.unbound>
}

[gthess]

Hi there, it seems you are modifying a system configured Unbound (which comes with its own systemctl scripts and apparmor profile). I cannot help you with the apparmor side of things, but turning it off to see if that would solve the permission errors would be my first action.

If you only care about the remote control working, this is enabled by default in the debian system Unbound as you can see in /etc/unbound/unbound.conf.d/remote-control.conf. It communicates over unix domain sockets and the system apparmor profile is tailored to that. (I see that you edited the system apparmor profile btw).

[kingii98]

Hi there!

Yes, I modified apparmor profile to explicitly allow reading files I had problems with

owner /etc/unbound/*.pem r,
/etc/unbound/*.pem r,
/etc/unbound/*.key r,

The fact I modified it is the reason I attached it.
However, the permission issue was before those modifications and it persists after.

[gthess]

It makes sense to have permissions errors before the modifications because access to those files is explicitly denied in the default configuration AFAICS. I don't have any experience with apparmor to identify the issue but try to disable apparmor for Unbound completely and see if it solves the permission problem. If that is the case, you can then focus on finetuning the profile.

[kingii98]

I had to add net_admin and dac_read_search. Now it works.

[gthess]

Thanks for reporting back. However I believe in your case dac_read_search solved the permission problem by ignoring the persmissions set by apparmor. At least this is how I read the following entry from https://www.man7.org/linux/man-pages/man7/capabilities.7.html:

       CAP_DAC_READ_SEARCH
              •  Bypass file read permission checks and directory read
                 and execute permission checks;