Sven Anderson; Martin Lucina; Hannes Mehnert; Stefanie Schirmer; Jan Suhr
Nitrokey GmbH, 28.11.2023
NetHSM is a networked Hardware Security Module (HSM) which can securely store a large number of cryptographic keys, has high computing performance, and offers key management functions via a modern REST-based API.
For further introductory and user guidance, see the user documentation as a prerequisite to understand this document. This document describes the System Design and Technical Architecture of the NetHSM, which includes:
- core functionality (business logic),
- encryption architecture and local persistent storage,
- integration with Muen separation kernel, including any customizations for board support and implementation of missing functionality required for the system,
- development of a minimized Linux for Linux-based Subjects,
- integration with platform Firmware (Coreboot),
- secure software updates.
A separate document describes the REST-based API for consumers.
NetHSM's hardware is in a 19" rack chassis and consists of an Intel x86 CPU, RAM, SSD, a discrete TPM, and a discrete TRNG.
The Muen separation kernel is an open source microkernel that has been formally proven to contain no runtime errors at the source code level. Muen runs on the Intel x86-64 architecture, and uses the VT-x and VT-d features to provide isolation for multiple subjects. Muen-based systems are entirely static and configured only at build time.
We use Muen as the lowest layer, i.e. operating system that runs on the NetHSM hardware.
MirageOS is a library operating system that constructs unikernels for secure applications. Applications can be developed on a normal OS such as Linux or macOS and then retargeted to run as a unikernel on Muen without any modifications to the source code.
Authentication Store: A data store containing encrypted user credentials (user name, passphrase) and mapping to permissions resp. Roles.
Backup Key: A symmetric key used for outer layer encryption of backup files. Derived from the Backup Passphrase once configured, and stored in the Configuration Store.
Backup Passphrase: The Backup Passphrase is required to gain access to a backup file.
Boot Mode:The available Boot Modes are Attended Boot and Unattended Boot. See the user documentation for more information.
Configuration Store: An unencrypted data store containing system configuration, including: network configuration, TLS endpoint configuration (certificates and private key), logging and metrics configuration.
Device Key: A device-dependent key that is securely stored by the TPM and only accessible if signed and unmodified NetHSM software is running. The Device Key is used to encrypt the data in the Domain Key Store.
Domain Key: A cryptographic key used to encrypt data in the Authentication Store and Key Store.
Domain Key Store: A data store containing encrypted Domain Keys.
Firmware: The platform firmware (a.k.a. "BIOS", "UEFI") used by the NetHSM hardware. Refer to System Firmware for details.
Key Store: A data store containing encrypted assets (keys).
Role: Each user account configured on the NetHSM has a single Role assigned to it. Roles are referred to as R-Name throughout this document. See the user documentation for a detailed list.
Subject: An independent piece of software separated by Muen from others. Subjects are referred to as S-Name throughout this document. See Security Design for details.
System Software: The NetHSM System Software, i.e. Muen and all subjects without the Firmware.
Unlock Key: An Unlock Key is an ephemeral key required to decrypt the Domain Key, gaining access to the encrypted Authentication Store and Key Store.
Unlock Passphrase: The Unlock Passphrase is used to derive an Unlock Key and must be provided by the user at boot time, unless Unattended Boot has been configured.
User Data: Used collectively to refer to all User Data persistently stored on the NetHSM, i.e. the Domain Key Store, Configuration Store, Authentication Store and Key Store. See Data Model for details.
Verified Boot: The process used by the Firmware to ensure that only System Software cryptographically signed with a trusted key can be booted on the NetHSM hardware. See Verified Boot for details.
Locked: A Provisioned NetHSM initially boots up in a Locked state, with no access to the encrypted Authentication Store or Key Store. See Encryption Architecture for details of how a Locked NetHSM can transition into an Operational state.
Operational: An Operational NetHSM is ready to process requests, and all functionality is available. This implies that in this state the system is neither Locked nor Unprovisioned.
Provisioned: The opposite of Unprovisioned.
Unprovisioned: An Unprovisioned NetHSM has not been configured by the user and does not contain any User Data, i.e. all data stores are empty or non-existent. This implies that only the limited subset of functionality needed for Initial Provisioning is available.
Note: Other than a NetHSM hardware appliance, resetting a NetHSM test container results in a Locked state.
All User Data is stored in a number of persistent data stores on the NetHSM; see Figure 1: Data Model. Each data store is a key-value store, additionally containing a version number (denoted as "Store Version" in the diagram). This version number is incremented whenever the underlying data model (values) of a data store changes. It determines if data migration code needs to be run for that data store.
For each data store, the diagram shows, as a high-level overview, which types of keys (as in key-value) are associated with which types of values. See the REST API documentation for a detailed definition of the data types, including:
- valid input and output data types for each HTTPS endpoint, which correspond to the values stored in each data store,
- constraints for each data type, e.g. "a KeyID needs to be a string of alphanumeric characters, which is between 1 and 128 characters long".
The Authentication Store and Key Store are persisted to disk and their contents are encrypted and authenticated using the so-called Domain Key. Only the subject S-Keyfender has decrypted access to these stores. Note that, for the avoidance of doubt, contents in this context refers to only the values of each key-value store, not the keys.
The Domain Key is stored in the Domain Key Store, and encrypted using AES256-GCM (i.e. AEAD) with the Device Key.
The Domain Key Store contains two "slots" for encrypted Domain Keys; which slot is used depends on whether or not the NetHSM is configured for Unattended Boot. Specifically, a Provisioned NetHSM (via S-Keyfender) performs the following steps during boot to transition from the initial Locked state into an Operational state:
| State = Locked
| Domain Key = Decrypt_AES256GCM(Device Key, Slot 1)
| IF decryption was successful: # In case of Unattended Boot
| GOTO UNLOCKED
| ELSE LOOP: # In case of Attended Boot
| Unlock Passphrase = Await API call to /unlock endpoint
| Unlock Key = KDF_SCRYPT(Unlock Passphrase)
| locked Domain Key = Decrypt_AES256GCM(Device Key, Slot 0)
| Domain Key = Decrypt_AES256GCM(Unlock Key, locked Domain Key)
| IF decryption was successful:
| Return Success to API caller
| GOTO UNLOCKED
| ELSE:
| Return Failure to API caller
|
| UNLOCKED:
| State = Operational
A random Device Key (unique per device) is generated during the first boot of the system. The TPM is used to seal it with its Storage Root Key (SRK) and the PCR measurement of the Firmware. During subsequent boots the sealed Device Key is unsealed with the TPM. This prevents access of unauthorized Firmware and Software to the Device Key.
During Unattended Boot, (see Figure 2: Encryption Architecture, right hand side), the Device Key is used to decrypt the encrypted Domain Key stored in slot 1. If the decryption step fails the NetHSM automatically falls back to Attended Boot.
During Attended Boot, (see Figure 2: Encryption Architecture, left and right hand side), the NetHSM waits for the user (via the /unlock endpoint of the REST API) to provide an Unlock Passphrase. The Unlock Key is calculated by applying the key derivation function (KDF) to the Unlock Passphrase. The Device Key is used to decrypt the encrypted locked Domain Key stored in slot 0. The Unlock Key is then used to decrypt the Domain Key from the locked Domain Key. If any decryption step fails (due to the user providing an incorrect Unlock Passphrase or invalid Device Key), the NetHSM remains in the Locked state, and continues to await an Unlock Passphrase.
For the avoidance of doubt: The act of Enabling Unattended Boot causes S-Keyfender to populate slot 1 with a Domain Key encrypted with only the Device Key. Conversely, Disabling Unattended Boot causes S-Keyfender to erase (overwrite) the contents of slot 1. At no point does the NetHSM persistently store either the Unlock Passphrase or an Unlock Key.
Among others, the described hardware binding and encryption prevents that (a copy of) the disk of one device can be decrypted in a different device.
NetHSM consists of three Muen subjects, as shown in Figure 3: Muen Subjects. Bidirectional communication channels are drawn as arrows.
The S-Net-External subject is a minimized Linux which bridges Ethernet frames between the physical Ethernet device, which is passed to this subject, and the virtual network interface connected to S-Keyfender. Apart from the Ethernet device driver only a minimal amount of Linux "userland" needs to be present in S-Net-External, e.g. brctl and ip, to enable bridging between the two interfaces. Specifically, there is no need for a configured IP address for this subject.
The S-Platform subject is a minimized Linux which provides access to hardware components and system services. The physical disk device (SSD), the TPM and the discrete TRNG are passed to this subject. The S-Platform subject provides a key-value store for persistently storing all User Data. It provides services to update the System Software, securely erase all User Data, manage the Device Key and other data from the TPM, and shutdown and reboot the device. It utilizes the discrete TRNG to create external entropy and periodically sends datagrams to S-Keyfender containing output of the discrete TRNG.
The S-Keyfender subject is a MirageOS unikernel which provides a HTTPS endpoint for the REST API that handles requests directly or by delegating it to a different subject. S-Keyfender is the only subject with decrypted access to the Authentication Store and Key Store. This is the only subject exposed to the public network.
S-Keyfender is the core subject of NetHSM, implemented entirely in OCaml as a MirageOS unikernel. It provides a HTTPS endpoint, serving the REST API providing the core NetHSM functionality to consumers.
Additional functionality provided by S-Keyfender:
- logging of system events to a remote syslog host (as defined by RFC 3164),
- an API endpoint for retrieving metrics (e.g. network traffic, memory usage, storage usage, key operations, uptime).
For the avoidance of doubt, a DNS resolver or DHCP client is specifically not provided by S-Keyfender. This implies that a NetHSM can only be configured to use a static IPv4 address, and that any external services (such as syslog) need to be configured with an IPv4 address too.
In order to provide core system functionality, including but not limited to:
- Correct responses in HTTP headers (e.g.
Last-Modified,Expires,Date) - Meaningful timestamps in log messages
S-Keyfender requires access to "wall clock" time. Muen systems have a built-in time subject which has exclusive access to the RTC and provides system-wide "wall clock" time to all subjects. However, there is currently no support in Muen for setting the system-wide "wall clock" and persisting it to the RTC.
Therefore, a REST endpoint allows an R-Administrator to set the "wall clock" time as seen by S-Keyfender. S-Keyfender will persist the offset between the RTC and "wall clock" time to the Configuration Store, allowing S-Keyfender to maintain "wall clock" time across reboots. This implies that, in the mean time, other subjects such as S-Platform will not share the same view of "wall clock" time as S-Keyfender.
Additionally there might be the possibility to set the system RTC from outside, for example from the BMC, depending on the hardware platform. In this case the offset can be left at 0 by not providing any "wall clock" time.
Note: There will be no support for timezones. Any time values used in REST API endpoints will use Coordinated Universal Time (UTC) only, i.e. an RFC 3339 time-offset of Z. It is the responsibility of the client to translate to local time, if this is desired.
Every backup is encrypted with a Backup Key, which is computed from the Backup Passphrase using the key derivation function. When a Backup Passphrase is configured or changed by an R-Administrator, the resulting Backup Key is stored in the unencrypted Configuration Store. The backup HTTPS endpoint is only enabled after the Backup Passphrase has been configured by an R-Administrator and a Backup Key exists.
The backup is double-encrypted: The outer encryption layer uses the Backup Key. The data contained is the unencrypted Configuration Store, the Domain Key Store, which is encrypted with the Device Key, and the Authentication Store and Key Store, which both are encrypted with the Domain Key as an inner encryption layer.
In order to allow restoring the backup to another device, the locked Domain Key, that is the Domain Key only encrypted with the Unlock Key and not encrypted with the Device Key is additionally stored in the backup.
This implies that:
- To gain access to all data contained in a backup (notably the encrypted contents of the Authentication Store and Key Store), both the Backup Key and an Unlock Key are required.
- Storing the Backup Key in the unencrypted Configuration Store is not a security risk (does not change the threat model), as the only data additionally protected by the Backup Key is that contained in the same unencrypted Configuration Store. An attacker that has physical access to the NetHSM can already gain access to this data.
The backup mechanism is designed this way to support automated backups: Once the Backup Passphrase has been configured by an R-Administrator, an external client can periodically fetch the backup endpoint and store the backup file. This backup client only requires credentials for an R-Backup user account, and is not able to decrypt the backup, neither the outer nor the inner layer.
When a backup is initiated, S-Keyfender prepares a backup, encrypts it with the stored Backup Key and sends it to the requesting client.
During system restore, the backup is decrypted by S-Keyfender using an ephemeral Backup Key computed from the Backup Passphrase provided by the user and, if the decryption is successful, all User Data is restored. In order to be able to operate on the restored data, the NetHSM also requires an Unlock Key. In practice this means that either the corresponding Unlock Passphrase current at the time of the backup must be known to the person restoring the backup, or the backup must have been restored on the same hardware unit which has created it, and Unattended Boot must have been enabled. Restoring a backup of a device with Unattended Boot on a different device results in an Attended Boot which requires the Unlock Passphrase. For details refer to Encryption Architecture.
Note (Operational): When restoring from an operational state, only Authentication Store and Key Store are restored. Data in Configuration Store and Domain Key Store is ignored.
Note (Unprovisioned): For the avoidance of doubt, partial restore is not provided when restoring from an unprovisioned state. This implies that restoring a backup may change the network and TLS endpoint configuration of the NetHSM and also the Unlock Passphrase and Boot Mode, to those current at the time of the backup.
When performing backups, S-Keyfender serializes (but not decrypts) the contents of each data store (i.e. User Data) into a binary format, and encrypts the result with the Backup Key. This will only backup a snapshot of each data store without history. During system restore, the reverse process is performed; the content of each data store is de-serialized from the binary format and inserted into the store.
The subject S-Keyfender uses a communication channel with S-Platform to conduct operations that need low-level platform functionality: reboot, shutdown, reset, update, and retrieving the TPM data including the Device Key. The protocol is line-based over a TCP stream, where S-Platform is listening on a socket for client connections - only a single client connection at any time is supported.
The client, after establishing the TCP stream, sends the command (all uppercase) followed by a single newline character. The server replies with either "OK" or "ERROR", optionally followed by a whitespace character and printable ASCII characters, terminated with a newline character.
Some sample sessions:
| C->S: PLATFORM-DATA\n | S->C: OK abcdef\n
| C->S: SHUTDOWN\n | S->C: OK\n
| C->S: FOO\n | S->C: ERROR Unknown command\n
All Linux-based Muen subjects that are integrated in the system will be built on a common base. We use the term "minimized Linux" to refer to this base, which is a collection of:
- An LTS release of the Linux kernel, with Muen SK patches and a minimal, hardened, configuration applied.
- A custom, minimal, componentized "userland", built with:
- U-Root, which provides Go implementations for the bulk of the "userland" and tooling to easily build "run from initramfs" based Linux systems.
- A custom
uinitfor U-root, implemented in Go, which manages the Linux subject's lifecycle.
- The following additional "userland" software components, all of which are cross-compiled as static binaries using a musl libc toolchain, and only the required subset of each is installed into the initramfs:
- etcd
- e2fsprogs (specifically,
mke2fs)
The System Software is a binary image, signed and distributed by Nitrokey. The versioning schema is major.minor. In each major release series, upgrades and downgrades within the same major release are allowed ("eligible" below). The data layout will not change within a major release series, but may change between release major.x and major+1.y. Downgrades to an older major release are not possible.
Updates must be applied manually by the R-Administrator. Over the air (OTA) updates are not supported.
The act of updating the System Software is performed by S-Platform. When S-Keyfender is processing an update file from the REST API, the following actions happen in parallel:
- S-Keyfender performs any decompression required and streams the binary image to S-Platform, which writes it to an inactive system partition.
- at the same time, S-Keyfender verifies the update signature with the public Update Key (protecting the integrity of the binary image, version number, release notes) and update eligibility.
If and only if verification of the update signature and eligibility are successful, S-Keyfender enables the REST API endpoint allowing the user to "commit" the software update. Only after this endpoint is requested, S-Keyfender will instruct S-Platform to perform the actions necessary (e.g. modifying the GPT or interacting with the Firmware) to boot from the new system partition once on next boot.
During boot, the System Software's signature is verified with the public Boot Key by the Firmware as part of Verified Boot. If the new System Software does not boot successfully or the Verified Boot verification fails, the system will automatically reboot into the old System Software as a fallback.
If and only if both Verified Boot verification succeeds and the new System Software boots successfully, the new System Software will perform the actions necessary to permanently configure the system to boot into the new System Software (e.g. by modifying the GPT or interacting with the Firmware). The new System Software may now start data migration, if required.
The actual implementation details of S-Platform will be based on the design used by both Chromium OS and CoreOS Container Linux, and are not described in more detail in this document. Please refer to the Chromium OS Disk Format and File System/Autoupdate design documents.
Coreboot is used as the Firmware of the NetHSM, with GRUB 2 as the Coreboot payload (boot loader). It provides the following functionality:
- GPT support for System Software Update
- Muen uses the Signed Block Stream (SBS) protocol for Verified Boot. The SBS implementation from Codelabs has been integrated to GRUB 2.
- Support for VT-d and correct bring-up of the IOMMU, required for Muen.
The process of ensuring that the NetHSM hardware will only boot System Software that has been cryptographically signed is denoted as Verified Boot. This is implemented by a custom GRUB 2 payload for Coreboot, integrating support for Signed Block Stream from Codelabs and modified to unconditionally enable signature verification of all loaded components.
Build-time tooling provides:
- Embed a trusted public key (RSA-4096) into CBFS in the Firmware image to be flashed to NetHSM hardware. This is considered the root of trust, and cannot be modified without reflashing the NetHSM hardware. (Note: Not to be confused with the protection of the Device Key by another root of trust, namely the ACM in the chipset.)
- Sign the components of a System Software image with the matching private key (referred to as an "inner signature" in System Software Update).
- Build a signed "Live USB" system intended for use as an "installer" to install stock NetHSM hardware with an initial System Software image.
Internally, a System Software image consists of the following components:
- The SBS image containing the Muen separation kernel, all subjects and their associated userland. This image is internally signed with the Firmware-trusted public key.
- A GRUB configuration file.
- A detached signature for the GRUB configuration file, signed with the Firmware-trusted public key.
These components are wrapped in a plain CPIO image, which is written to the system partition.
During system boot, Coreboot will launch the GRUB 2 payload, which will initially execute the commands defined by a trusted GRUB configuration embedded in CBFS. This configuration will attempt to "chain" into a correctly signed GRUB configuration file found in any System Software image present on either the internal SSD or a USB key (for use by a Live USB installer, updater, or other component). If signature verification of the configuration file succeeds, then the system will proceed to boot from that device.
NetHSM runs on commodity hardware using a minimal amount of high-assurance software to protect private keys stored in the Key Store. A TPM is used to secure a Device Key, which is used to encrypt the Domain Key, and prevents unauthorized modifications of the Firmware.
NetHSM security design is based on software with a minimal trusted computing base (TCB): The Muen separation kernel is used for managing the physical resources of the system and isolating the subjects from each other. Device drivers for physical devices (Ethernet, USB, Storage) are provided by minimized Linux subjects, which are accessible only from the Muen subjects. The core subject is S-Keyfender, a MirageOS based unikernel, developed in OCaml.
In case a subject is exploited (e.g. by exploiting the Ethernet driver of the S-Net-External subject), it does not lead to a compromise of the Key Store assets. Muen ensures that only the S-Keyfender subject, if it is Unlocked, has access to the decrypted Key Store. The Domain Key used for encrypting data in the Key Store is only kept in memory, and never written unencrypted to persistent storage.
Each subject has only the minimum of privileges absolutely necessary for its operation. The TCB is kept small. It consists of the Muen separation kernel and S-Keyfender. The other subjects do not have access to the unencrypted private key material. A compromised subject (other than S-Keyfender) cannot access stored private key material.
Muen is implemented in Ada, a memory-safe programming language, and its isolation properties are formally verified in the Spark language. S-Keyfender is a MirageOS unikernel implemented in OCaml, a functional memory-safe language. MirageOS has been developed over 10 years to remove dependencies written in unsafe languages.
NetHSM uses etcd to store all User Data. This comes with an log for each data modification, which adds accountability and rollback points for the data (currently not used).
NetHSM has the following entropy sources:
- discrete TRNG builtar by Nitrokey (hardware, firmware)
- TPM RNG
- Intel CPU's RDRAND
- Event loop from network and IO
Entropy is consumed in S-Platform and S-Keyfender.
S-Platform reads random data from both the discrete TRNG and the TPM and feeds them both to S-Keyfender and to /dev/random of the Linux kernel. If one of the sources fail, errors are logged.
During boot of the system S-Keyfender requests a Device Key from S-Platform. At very first boot of the system a Device Key doesn't exist yet. The possible generation of a new Device Key is delayed until random data from both discrete TRNG and TPM has been fed to /dev/random at least once. The entropy used to generate the Device Key is retrieved from the Go library crypto/rand.Read() which uses getrandom(2). If any of the two entropy sources fail, no Device Key is generated, an error message is logged at the serial console, the system doesn't proceed booting and ends in an non-operational state.
The S-Keyfender subject uses as cryptographically secure random number generator, an implementation of Fortuna with 32 pools. Because S-Keyfender does not have direct access to real hardware, it is crucial to collect non-predictable entropy.
The entropy sources used during startup are the CPU instruction RDRAND (executed three times, returning 64 bit each), and once the Whirlwind bootstrap algorithm based on timing of instructions that take a variable number of cycles (with l=100 lmax=1024).
During normal operation, every second the output of an RDRAND execution is fed into each pool. When an event (wakeup of a sleeper, network read) is executed, a cycle counter (using RDTSC) is taken, and the lower four bytes are fed into the entropy pool. The first event cycle counter is fed into the first entropy pool, the second into the second, etc. Additionally S-Keyfender listens on an UDP port for packages with random data from S-Platform. These packages contain entropy from both discrete TRNG and TPM TRNG and are fed into all 32 pools.
In case during the operation either discrete TRNG or TPM RNG fails, an error message is logged repeatedly in Syslog and the system keeps running. If both of these entropy sources fail, the system stops and ends in a non-operational state.
The cryptography implementations are as follows:
RSA: part of the mirage-crypto-pk package, which uses the OCaml Zarith library for big integers (backed by gmplib). Modes (PKCS/PSS) are implemented in OCaml.
EC: part of the mirage-crypto-ec package, uses C code generated by fiat-crypto for modular arithmetic, group operations are in OCaml and in C (from BoringSSL). The C code includes YY_32.h and YY_64.h, which are generated by fiat-crypto, the files curve25519_stubs.c / curve25519_tables.h / point_operations.h are code from BoringSSL, the remaining C boilerplate to call out to the C function (in YY_stubs.c) are hand-written. The OCaml code provides the high-level operations (ECDSA/ECDH).
The key generated for the HTTPS endpoint is an EC key (secp256r1).
The key derivation function in keyfender/crypto.ml is scrypt with the following chosen parameters: b=16384, r=8, p=16, salt length 16 byte
The data stored on disk is encrypted with AES256-GCM (32 byte key, nonce size is 12).
To limit brute-forcing of passphrases, S-Keyfender rate limits logins. The rate for the Unlock Passphrase is one failed access per second per IP address. The rate limit for all endpoints requiring authentication is 1 failed authentication per second per IP address and username.