-
Notifications
You must be signed in to change notification settings - Fork 34
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MG-153 - Add Vault scripts as git subtree #157
base: master
Are you sure you want to change the base?
Changes from 21 commits
8cab8df
aedfbae
beba5e0
81356c9
22f2f60
25db402
cffc3bb
eb6f4fc
f4df847
cf5b45d
0b8ad42
ffd46e3
d7314d9
04a24ae
9c7145a
cfc0100
8fb2d05
4f98a62
fdfd93f
d05d2a8
fa8adb6
483526c
6bb155f
7cb578d
4d0aa48
1006b1c
fc151d6
e60cc9f
e0b0ce3
35e12b4
f4f15f1
42ab894
3623b16
ad4361b
4e470fa
b98267d
1564491
ab3f879
fad3ac7
36238f0
70b83ae
ea26149
58a7f37
dd608db
33c16cf
735284e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -57,6 +57,20 @@ git push origin <your-branch> | |
|
||
Replace `<your-branch>` with the branch you are working on. | ||
|
||
### Running Vault Setup Scripts with `--env-file` | ||
|
||
To run a Vault setup script, use the `--env-file` option to specify the path to your `.env` file: | ||
|
||
```bash | ||
./<script-name>.sh --env-file <path-to-your-env-file> | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Move this part to scripts/vault/Readme.md and add words like refer magistrala vault document (and its link) |
||
``` | ||
|
||
For example: | ||
|
||
```bash | ||
scripts/vault/scripts/vault_init.sh --env-file scripts/vault/.env | ||
``` | ||
|
||
## License | ||
|
||
This project is licensed under the [Apache-2.0](LICENSE). |
This file was deleted.
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -40,4 +40,4 @@ MG_VAULT_PKI_INT_CA_ST='PARIS' | |
MG_VAULT_PKI_INT_CA_ADDR='5 Av. Anatole' | ||
MG_VAULT_PKI_INT_CA_PO='75007' | ||
MG_VAULT_PKI_INT_CLUSTER_PATH=http://localhost | ||
MG_VAULT_PKI_INT_CLUSTER_AIA_PATH=http://localhost | ||
MG_VAULT_PKI_INT_CLUSTER_AIA_PATH=http://localhost | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please add new line at end of file |
Original file line number | Diff line number | Diff line change | ||
---|---|---|---|---|
|
@@ -45,44 +45,24 @@ When the Vault service is started, some initialization steps need to be done to | |||
|
||||
## Setup | ||||
|
||||
The following scripts are provided, which work on the running Vault service in Docker. | ||||
The following scripts are provided, which work on the running Vault service from within the `docker/addons/vault/scripts` directory. | ||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This doc is relative to magistrala, may be simply we should point MG vault docs for usage There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Yes, the |
||||
|
||||
### 1. `vault_init.sh` | ||||
|
||||
Calls `vault operator init` to perform the initial vault initialization and generates a `docker/addons/vault/data/secrets` file which contains the Vault unseal keys and root tokens. | ||||
Calls `vault operator init` to perform the initial vault initialization and generates a `docker/addons/vault/scripts/data/secrets` file which contains the Vault unseal keys and root tokens. | ||||
|
||||
Example contents for `data/secrets`: | ||||
|
||||
```bash | ||||
Unseal Key 1: Ay0YZecYJ2HVtNtXfPootXK5LtF+JZoDmBb7IbbYdLBI | ||||
Unseal Key 2: P6hb7x2cglv0p61jdLyNE3+d44cJUOFaDt9jHFDfr8Df | ||||
Unseal Key 3: zSBfDHzUiWoOzXKY1pnnBqKO8UD2MDLuy8DNTxNtEBFy | ||||
Unseal Key 4: 5oJuDDuMI0I8snaw/n4VLNpvndvvKi6JlkgOxuWXqMSz | ||||
Unseal Key 5: ZhsUkk2tXBYEcWgz4WUCHH9rocoW6qZoiARWlkE5Epi5 | ||||
|
||||
Initial Root Token: s.V2hdd00P4bHtUQnoWZK2hSaS | ||||
### 2. `vault_copy_env.sh` | ||||
|
||||
Vault initialized with 5 key shares and a key threshold of 3. Please securely | ||||
distribute the key shares printed above. When the Vault is re-sealed, | ||||
restarted, or stopped, you must supply at least 3 of these keys to unseal it | ||||
before it can start servicing requests. | ||||
After the initial setup, the Vault-related environment variables (`MG_VAULT_TOKEN`, `MG_VAULT_UNSEAL_KEY_1`, `MG_VAULT_UNSEAL_KEY_2`, `MG_VAULT_UNSEAL_KEY_3`) need to be updated in the `.env` file. | ||||
|
||||
Vault does not store the generated master key. Without at least 3 key to | ||||
reconstruct the master key, Vault will remain permanently sealed! | ||||
The `vault_copy_env.sh` script automatically retrieves these values from the `docker/addons/vault/scripts/data/secrets` file and updates the corresponding environment variables in your `.env` file. | ||||
|
||||
It is possible to generate new unseal keys, provided you have a quorum of | ||||
existing unseal keys shares. See "vault operator rekey" for more information. | ||||
bash-4.4 | ||||
Example: | ||||
|
||||
Use 3 out of five keys presented and put it into .env file and than start the composition again Vault should be in unsealed state ( take a note that this is not recommended in terms of security, this is deployment for development) A real production deployment can use Vault auto unseal mode where vault gets unseal keys from some 3rd party KMS ( on AWS for example) | ||||
```sh | ||||
Vault environment variables have been successfully set in ~/magistrala/docker/.env | ||||
``` | ||||
|
||||
### 2. `vault_copy_env.sh` | ||||
|
||||
After first step, the corresponding Vault environment variables (`MG_VAULT_TOKEN`, `MG_VAULT_UNSEAL_KEY_1`, `MG_VAULT_UNSEAL_KEY_2`, `MG_VAULT_UNSEAL_KEY_3`) should be updated in `.env` file. | ||||
|
||||
`vault_copy_env.sh` scripts copies values from `docker/addons/vault/data/secrets` file and update environmental variables `MG_VAULT_TOKEN`, `MG_VAULT_UNSEAL_KEY_1`, `MG_VAULT_UNSEAL_KEY_2`, `MG_VAULT_UNSEAL_KEY_3` present in `.env` file. | ||||
|
||||
### 3. `vault_unseal.sh` | ||||
|
||||
This can be run after the initialization to unseal Vault, which is necessary for it to be used to store and/or get secrets. | ||||
|
@@ -93,29 +73,169 @@ The unseal environment variables need to be set in `.env` for the script to work | |||
|
||||
This script should not be necessary to run after the initial setup, since the Vault service unseals itself when starting the container. | ||||
|
||||
### 4. `vault_set_pki.sh` | ||||
Example output: | ||||
|
||||
This script is used to generate the root certificate, intermediate certificate and HTTPS server certificate. | ||||
All generate certificates, keys and CSR by `vault_set_pki.sh` will be present at `docker/addons/vault/data`. | ||||
```bash | ||||
Key Value | ||||
--- ----- | ||||
Seal Type shamir | ||||
Initialized true | ||||
Sealed true | ||||
Total Shares 5 | ||||
Threshold 3 | ||||
Unseal Progress 1/3 | ||||
Unseal Nonce 4c248cc8-e9f5-055e-319b-00ee06f998a0 | ||||
Version 1.15.4 | ||||
Build Date 2023-12-04T17:45:28Z | ||||
Storage Type file | ||||
HA Enabled false | ||||
Key Value | ||||
--- ----- | ||||
Seal Type shamir | ||||
Initialized true | ||||
Sealed true | ||||
Total Shares 5 | ||||
Threshold 3 | ||||
Unseal Progress 2/3 | ||||
Unseal Nonce 4c248cc8-e9f5-055e-319b-00ee06f998a0 | ||||
Version 1.15.4 | ||||
Build Date 2023-12-04T17:45:28Z | ||||
Storage Type file | ||||
HA Enabled false | ||||
Key Value | ||||
--- ----- | ||||
Seal Type shamir | ||||
Initialized true | ||||
Sealed false | ||||
Total Shares 5 | ||||
Threshold 3 | ||||
Unseal Progress 3/3 | ||||
Unseal Nonce 4c248cc8-e9f5-055e-319b-00ee06f998a0 | ||||
Version 1.15.4 | ||||
Build Date 2023-12-04T17:45:28Z | ||||
Storage Type file | ||||
HA Enabled false | ||||
``` | ||||
|
||||
### 4. vault_set_pki.sh | ||||
|
||||
The `vault_set_pki.sh` script is responsible for generating the root certificate, intermediate certificate, and HTTPS server certificate. All generated certificates, keys, and CSR files are stored in the `docker/addons/vault/scripts/data` directory. | ||||
|
||||
The script pulls necessary parameters for certificate generation from environment variables, which are, by default, loaded from `docker/.env`. | ||||
|
||||
The parameters required for generating certificate are obtained from the environment variables which are loaded from `docker/.env`. | ||||
- Environment variables prefixed with `MG_VAULT_PKI` in the `docker/.env` file are used for generating the root CA. | ||||
- Environment variables prefixed with `MG_VAULT_PKI_INT` are used for generating the intermediate CA. | ||||
|
||||
Environmental variables starting with `MG_VAULT_PKI` in `docker/.env` file are used by `vault_set_pki.sh` to generate root CA. | ||||
Environmental variables starting with`MG_VAULT_PKI_INT` in `docker/.env` file are used by `vault_set_pki.sh` to generate intermediate CA. | ||||
To skip generating the server certificate and key, you can pass the `--skip-server-cert` option to the script: | ||||
|
||||
Passing command line args `--skip-server-cert` to `vault_set_pki.sh` will skip server certificate role & process of generation of server certificate & key. | ||||
```sh | ||||
./vault_set_pki.sh --skip-server-cert | ||||
``` | ||||
|
||||
#### Troubleshooting: | ||||
|
||||
### 5. `vault_create_approle.sh` | ||||
If you encounter the following error: | ||||
|
||||
This script is used to enable app role authorization in Vault. Certs service used the approle credentials to issue, revoke things certificate from vault intermedate CA. | ||||
```sh | ||||
jq command could not be found, please install it and try again. | ||||
``` | ||||
|
||||
`vault_create_approle.sh` script by default tries to enable auth approle. | ||||
If approle is already enabled in vault, then use args `--skip-enable-approle` to skip enable auth approle step. | ||||
To skip enable auth approle step use the following `vault_create_approle.sh --skip-enable-approle` | ||||
Install `jq` using: | ||||
|
||||
```sh | ||||
sudo apt-get update && sudo apt-get install -y jq | ||||
``` | ||||
|
||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||
After installing `jq`, rerun the script. | ||||
|
||||
### 5. `vault_create_approle.sh` | ||||
|
||||
This script enables AppRole authorization in Vault. The certs service uses these AppRole credentials to issue and revoke certificates from the Vault intermediate CA. | ||||
|
||||
Example output: | ||||
|
||||
```sh | ||||
Success! You are now authenticated. The token information displayed below | ||||
is already stored in the token helper. You do NOT need to run "vault login" | ||||
again. Future Vault requests will automatically use this token. | ||||
|
||||
Key Value | ||||
--- ----- | ||||
token <token_value> | ||||
token_accessor i6YVeKh4wQ4e0Aj0ONiyGw1Z | ||||
token_duration ∞ | ||||
token_renewable false | ||||
token_policies ["root"] | ||||
identity_policies [] | ||||
policies ["root"] | ||||
Creating new policy for AppRole | ||||
Successfully copied 2.56kB to magistrala-vault:/vault/magistrala_things_certs_issue.hcl | ||||
Success! Uploaded policy: magistrala_things_certs_issue | ||||
Enabling AppRole | ||||
Success! Enabled approle auth method at: approle/ | ||||
Deleting old AppRole | ||||
Success! Data deleted (if it existed) at: auth/approle/role/magistrala_things_certs_issuer | ||||
Creating new AppRole | ||||
Success! Data written to: auth/approle/role/magistrala_things_certs_issuer | ||||
Writing custom role ID | ||||
Key Value | ||||
--- ----- | ||||
role_id f23942b3-62b9-7456-784f-220ca3f703b9 | ||||
Success! Data written to: auth/approle/role/magistrala_things_certs_issuer/role-id | ||||
Writing custom secret | ||||
Key Value | ||||
--- ----- | ||||
secret_id 61d5a30f-634c-6027-f5b6-4934e6fc49b2 | ||||
secret_id_accessor 1d744f6e-e0c2-5431-a87a-2b23fde584a7 | ||||
secret_id_num_uses 0 | ||||
secret_id_ttl 0s | ||||
Testing custom role ID and secret by logging in | ||||
Key Value | ||||
--- ----- | ||||
token <token_value> | ||||
token_accessor 9cuwS4mrLHKhJQMv0pl9Bbg9 | ||||
token_duration 1h | ||||
token_renewable true | ||||
token_policies ["default" "magistrala_things_certs_issue"] | ||||
identity_policies [] | ||||
policies ["default" "magistrala_things_certs_issue"] | ||||
token_meta_role_name magistrala_things_certs_issuer | ||||
``` | ||||
|
||||
By default, the `vault_create_approle.sh` script tries to enable the AppRole authentication method. Certs service uses the approle credentials to issue and revoke things certificate from vault intermedate CA. If AppRole is already enabled, you can skip this step by passing the `--skip-enable-approle` argument: | ||||
|
||||
```sh | ||||
./vault_create_approle.sh --skip-enable-approle | ||||
``` | ||||
|
||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||
### 6. `vault_copy_certs.sh` | ||||
|
||||
This scripts copies the necessary certificates and keys from `docker/addons/vault/data` to the `docker/ssl/certs` folder. | ||||
This script copies the required certificates and keys from `docker/addons/vault/scripts/data` to the `docker/ssl/certs` folder. | ||||
|
||||
Example output: | ||||
|
||||
```bash | ||||
Copying certificate files | ||||
'data/localhost.crt' -> '~/Documents/magistrala/docker/ssl/certs/magistrala-server.crt' | ||||
'data/localhost.key' -> '~/Documents/magistrala/docker/ssl/certs/magistrala-server.key' | ||||
'data/mg_int.key' -> '~/Documents/magistrala/docker/ssl/certs/ca.key' | ||||
'data/mg_int_bundle.crt' -> '~/Documents/magistrala/docker/ssl/certs/ca.crt' | ||||
``` | ||||
|
||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||
## Custom `.env` Path Support | ||||
|
||||
Vault scripts support specifying a custom `.env` file path using the `--env-file` argument. If this argument is not provided, the scripts will use the default `.env` file located at `docker/.env`. | ||||
|
||||
To use a different `.env` file, include the `--env-file` argument followed by the path to your `.env` file when running the Vault scripts. Below are examples of how to execute each script with a custom `.env` file path: | ||||
|
||||
```bash | ||||
./vault_init.sh --env-file /custom/path/.env | ||||
./vault_copy_env.sh --env-file /custom/path/.env | ||||
./vault_unseal.sh --env-file /custom/path/.env | ||||
./vault_set_pki.sh --env-file /custom/path/.env | ||||
./vault_create_approle.sh --env-file /custom/path/.env | ||||
./vault_copy_certs.sh --env-file /custom/path/.env | ||||
``` | ||||
|
||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||
## Hashicorp Cloud Platform (HCP) Vault | ||||
|
||||
|
@@ -124,7 +244,7 @@ Requirement: [VAULT CLI](https://developer.hashicorp.com/vault/tutorials/getting | |||
|
||||
- Replace the environmental variable `MG_VAULT_ADDR` in `docker/.env` with HCP Vault address. | ||||
- Replace the environmental variable `MG_VAULT_TOKEN` in `docker/.env` with HCP Vault Admin token. | ||||
- Run script `vault_set_pki.sh` and `vault_create_approle.sh`. | ||||
- Run script `vault_set_pki.sh` and `vault_create_approle.sh`. | ||||
- Optional step, run script `vault_copy_certs.sh` to copy certificates to magistrala default path. | ||||
|
||||
## Vault CLI | ||||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -3,4 +3,3 @@ | |
|
||
data | ||
magistrala_things_certs_issue.hcl | ||
.env |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
#!/usr/bin/bash | ||
# Copyright (c) Abstract Machines | ||
# SPDX-License-Identifier: Apache-2.0 | ||
|
||
vault() { | ||
if is_container_running "magistrala-vault"; then | ||
docker exec -it magistrala-vault vault "$@" | ||
else | ||
if which vault &> /dev/null; then | ||
$(which vault) "$@" | ||
else | ||
echo "magistrala-vault container or vault command not found. Please refer to the documentation: https://github.com/absmach/magistrala/blob/main/docker/addons/vault/README.md" | ||
fi | ||
fi | ||
} | ||
|
||
is_container_running() { | ||
local container_name="$1" | ||
if [ "$(docker inspect --format '{{.State.Running}}' "$container_name" 2>/dev/null)" = "true" ]; then | ||
return 0 | ||
else | ||
return 1 | ||
fi | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,86 @@ | ||
#!/usr/bin/bash | ||
# Copyright (c) Abstract Machines | ||
# SPDX-License-Identifier: Apache-2.0 | ||
|
||
set -euo pipefail | ||
|
||
scriptdir="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)" | ||
|
||
# default env file path | ||
env_file="docker/.env" | ||
|
||
# default certs copy path | ||
certs_copy_path="docker/ssl/certs/" | ||
|
||
while [[ "$#" -gt 0 ]]; do | ||
case $1 in | ||
--env-file) | ||
if [[ -z "${2:-}" ]]; then | ||
echo "Error: --env-file requires a non-empty option argument." | ||
exit 1 | ||
fi | ||
env_file="$2" | ||
if [[ ! -f "$env_file" ]]; then | ||
echo "Error: .env file not found at $env_file" | ||
exit 1 | ||
fi | ||
shift | ||
;; | ||
--certs-copy-path) | ||
if [[ -z "${2:-}" ]]; then | ||
echo "Error: --certs-copy-path requires a non-empty option argument." | ||
exit 1 | ||
fi | ||
certs_copy_path="$2" | ||
shift | ||
;; | ||
*) | ||
echo "Error: Unknown parameter passed: $1" | ||
exit 1 | ||
;; | ||
esac | ||
shift | ||
done | ||
|
||
readDotEnv() { | ||
set -o allexport | ||
source "$env_file" | ||
set +o allexport | ||
} | ||
|
||
readDotEnv | ||
|
||
server_name="localhost" | ||
|
||
# Check if MG_NGINX_SERVER_NAME is set or not empty | ||
if [ -n "${MG_NGINX_SERVER_NAME:-}" ]; then | ||
server_name="$MG_NGINX_SERVER_NAME" | ||
fi | ||
|
||
echo "Copying certificate files to ${certs_copy_path}" | ||
|
||
if [ -e "$scriptdir/data/${server_name}.crt" ]; then | ||
cp -v "$scriptdir/data/${server_name}.crt" "${certs_copy_path}magistrala-server.crt" | ||
else | ||
echo "${server_name}.crt file not available" | ||
fi | ||
|
||
if [ -e "$scriptdir/data/${server_name}.key" ]; then | ||
cp -v "$scriptdir/data/${server_name}.key" "${certs_copy_path}magistrala-server.key" | ||
else | ||
echo "${server_name}.key file not available" | ||
fi | ||
|
||
if [ -e "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}.key" ]; then | ||
cp -v "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}.key" "${certs_copy_path}ca.key" | ||
else | ||
echo "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}.key file not available" | ||
fi | ||
|
||
if [ -e "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}_bundle.crt" ]; then | ||
cp -v "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}_bundle.crt" "${certs_copy_path}ca.crt" | ||
else | ||
echo "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}_bundle.crt file not available" | ||
fi | ||
|
||
exit 0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@JeffMboya This part is moved
scripts/vault/Readme.md
, so i think it is no more needed hereWhat do you think ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it should be removed.