Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MG-153 - Add Vault scripts as git subtree #157

Open
wants to merge 46 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
8cab8df
Squashed 'docker/addons/vault/' content from commit 54a134e7
JeffMboya Nov 18, 2024
aedfbae
Merge commit '8cab8df5a1cfd26191442818472cfc8ad0680da6' as 'docker/ad…
JeffMboya Nov 18, 2024
beba5e0
Remove docker/addons/vault directory
JeffMboya Nov 18, 2024
81356c9
Remove scripts/vault directory
JeffMboya Nov 18, 2024
22f2f60
Squashed 'vault/' content from commit a32634a1e
JeffMboya Nov 18, 2024
25db402
Merge commit '22f2f6083f30faec72f8fb72f9e7e8107739accf' as 'vault'
JeffMboya Nov 18, 2024
cffc3bb
Remove scripts/vault directory
JeffMboya Nov 18, 2024
eb6f4fc
Squashed 'vault/' content from commit a32634a1e
JeffMboya Nov 18, 2024
f4df847
Merge commit 'eb6f4fc1b399c3db293741619638ac7db06b2d29' as 'vault'
JeffMboya Nov 18, 2024
cf5b45d
Squashed 'scripts/vault/' content from commit a32634a1e
JeffMboya Nov 18, 2024
0b8ad42
Merge commit 'cf5b45de649d9b7df1f8a3711bc527a6bf87158d' as 'scripts/v…
JeffMboya Nov 18, 2024
ffd46e3
Add vault/scripts directory
JeffMboya Nov 18, 2024
d7314d9
Delete vault directory
JeffMboya Nov 18, 2024
04a24ae
Squashed 'scripts/vault/' content from commit a32634a1e
JeffMboya Nov 18, 2024
9c7145a
Merge commit '04a24aea9c420f527d7db15cb8a1b863328ce53a' as 'scripts/v…
JeffMboya Nov 18, 2024
cfc0100
Add scripts/vault directory
JeffMboya Nov 18, 2024
8fb2d05
Add .env, vault.md, and efk.sh
JeffMboya Nov 18, 2024
4f98a62
Add empty line
JeffMboya Nov 18, 2024
fdfd93f
Update vault docs
JeffMboya Nov 19, 2024
d05d2a8
Remove git pull command
JeffMboya Nov 19, 2024
fa8adb6
Address comments
JeffMboya Nov 19, 2024
483526c
Add empty line
JeffMboya Nov 19, 2024
6bb155f
move .env file
JeffMboya Nov 19, 2024
7cb578d
Squashed 'scripts/vault/' content from commit a32634a1e
JeffMboya Nov 19, 2024
4d0aa48
Merge commit '7cb578dbedee70ca0eac10a8c5caed7d8ac53d5c' as 'scripts/v…
JeffMboya Nov 19, 2024
1006b1c
Squashed 'vault/' content from commit a32634a1e
JeffMboya Nov 19, 2024
fc151d6
Merge commit '1006b1c66b57497784783f6b7414ba136dcf3d11' as 'vault'
JeffMboya Nov 19, 2024
e60cc9f
move .env file
JeffMboya Nov 19, 2024
e0b0ce3
Squashed 'vault/' content from commit a32634a1e
JeffMboya Nov 19, 2024
35e12b4
Merge commit 'e0b0ce33ba0600d7316f7bc8f2fa451c7b6da9a8' as 'vault'
JeffMboya Nov 19, 2024
f4f15f1
move .env file
JeffMboya Nov 19, 2024
42ab894
Squashed 'scripts/vault/' content from commit a32634a1e
JeffMboya Nov 19, 2024
3623b16
Merge commit '42ab894250fa1bf49f3d8a6a9e89fa3116c56ab8' as 'scripts/v…
JeffMboya Nov 19, 2024
ad4361b
move .env file
JeffMboya Nov 19, 2024
4e470fa
Add efk.sh
JeffMboya Nov 19, 2024
b98267d
Squashed 'docker/addons/vault/scripts/' content from commit d9f3bca47
JeffMboya Nov 19, 2024
1564491
Merge commit 'b98267df3d82f58f4d30ec7f34fb177f202fe030' as 'docker/ad…
JeffMboya Nov 19, 2024
ab3f879
Remove unnecessary files
JeffMboya Nov 19, 2024
fad3ac7
Address comments
JeffMboya Nov 19, 2024
36238f0
Remove duplicate instructions
JeffMboya Nov 21, 2024
70b83ae
Update README
JeffMboya Nov 28, 2024
ea26149
Fix users authentication error
JeffMboya Nov 28, 2024
58a7f37
Add AMA certs
JeffMboya Nov 28, 2024
dd608db
Bump chart version
JeffMboya Nov 28, 2024
33c16cf
Remove ref to magistrala
JeffMboya Nov 28, 2024
735284e
Remove vault config files
JeffMboya Nov 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,20 @@ git push origin <your-branch>

Replace `<your-branch>` with the branch you are working on.

### Running Vault Setup Scripts with `--env-file`

To run a Vault setup script, use the `--env-file` option to specify the path to your `.env` file:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JeffMboya This part is moved scripts/vault/Readme.md , so i think it is no more needed here
What do you think ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JeffMboya This part is moved scripts/vault/Readme.md , so i think it is no more needed here What do you think ?

Yes, it should be removed.


```bash
./<script-name>.sh --env-file <path-to-your-env-file>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this part to scripts/vault/Readme.md and add words like refer magistrala vault document (and its link)

```

For example:

```bash
scripts/vault/scripts/vault_init.sh --env-file scripts/vault/.env
```

## License

This project is licensed under the [Apache-2.0](LICENSE).
1 change: 0 additions & 1 deletion scripts/.gitignore

This file was deleted.

2 changes: 1 addition & 1 deletion scripts/vault/.env
Original file line number Diff line number Diff line change
Expand Up @@ -40,4 +40,4 @@ MG_VAULT_PKI_INT_CA_ST='PARIS'
MG_VAULT_PKI_INT_CA_ADDR='5 Av. Anatole'
MG_VAULT_PKI_INT_CA_PO='75007'
MG_VAULT_PKI_INT_CLUSTER_PATH=http://localhost
MG_VAULT_PKI_INT_CLUSTER_AIA_PATH=http://localhost
MG_VAULT_PKI_INT_CLUSTER_AIA_PATH=http://localhost
Copy link
Contributor

@arvindh123 arvindh123 Nov 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add new line at end of file

204 changes: 162 additions & 42 deletions scripts/vault/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,44 +45,24 @@ When the Vault service is started, some initialization steps need to be done to

## Setup

The following scripts are provided, which work on the running Vault service in Docker.
The following scripts are provided, which work on the running Vault service from within the `docker/addons/vault/scripts` directory.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doc is relative to magistrala, may be simply we should point MG vault docs for usage

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doc is relative to magistrala, may be simply we should point MG vault docs for usage

Yes, the README is relative to Magistrala because adding docker/addons/vault as a subtree in the DevOps repository imports content relative to Magistrala. The issue is that when the subtree is updated, it will continue importing files tied to Magistrala. A possible solution is to import docker/addons/vault locally, remove unnecessary files, and then push the filtered contents to GitHub. This way, the DevOps repository only includes the required files.


### 1. `vault_init.sh`

Calls `vault operator init` to perform the initial vault initialization and generates a `docker/addons/vault/data/secrets` file which contains the Vault unseal keys and root tokens.
Calls `vault operator init` to perform the initial vault initialization and generates a `docker/addons/vault/scripts/data/secrets` file which contains the Vault unseal keys and root tokens.

Example contents for `data/secrets`:

```bash
Unseal Key 1: Ay0YZecYJ2HVtNtXfPootXK5LtF+JZoDmBb7IbbYdLBI
Unseal Key 2: P6hb7x2cglv0p61jdLyNE3+d44cJUOFaDt9jHFDfr8Df
Unseal Key 3: zSBfDHzUiWoOzXKY1pnnBqKO8UD2MDLuy8DNTxNtEBFy
Unseal Key 4: 5oJuDDuMI0I8snaw/n4VLNpvndvvKi6JlkgOxuWXqMSz
Unseal Key 5: ZhsUkk2tXBYEcWgz4WUCHH9rocoW6qZoiARWlkE5Epi5

Initial Root Token: s.V2hdd00P4bHtUQnoWZK2hSaS
### 2. `vault_copy_env.sh`

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.
After the initial setup, the Vault-related environment variables (`MG_VAULT_TOKEN`, `MG_VAULT_UNSEAL_KEY_1`, `MG_VAULT_UNSEAL_KEY_2`, `MG_VAULT_UNSEAL_KEY_3`) need to be updated in the `.env` file.

Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!
The `vault_copy_env.sh` script automatically retrieves these values from the `docker/addons/vault/scripts/data/secrets` file and updates the corresponding environment variables in your `.env` file.

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
bash-4.4
Example:

Use 3 out of five keys presented and put it into .env file and than start the composition again Vault should be in unsealed state ( take a note that this is not recommended in terms of security, this is deployment for development) A real production deployment can use Vault auto unseal mode where vault gets unseal keys from some 3rd party KMS ( on AWS for example)
```sh
Vault environment variables have been successfully set in ~/magistrala/docker/.env
```

### 2. `vault_copy_env.sh`

After first step, the corresponding Vault environment variables (`MG_VAULT_TOKEN`, `MG_VAULT_UNSEAL_KEY_1`, `MG_VAULT_UNSEAL_KEY_2`, `MG_VAULT_UNSEAL_KEY_3`) should be updated in `.env` file.

`vault_copy_env.sh` scripts copies values from `docker/addons/vault/data/secrets` file and update environmental variables `MG_VAULT_TOKEN`, `MG_VAULT_UNSEAL_KEY_1`, `MG_VAULT_UNSEAL_KEY_2`, `MG_VAULT_UNSEAL_KEY_3` present in `.env` file.

### 3. `vault_unseal.sh`

This can be run after the initialization to unseal Vault, which is necessary for it to be used to store and/or get secrets.
Expand All @@ -93,29 +73,169 @@ The unseal environment variables need to be set in `.env` for the script to work

This script should not be necessary to run after the initial setup, since the Vault service unseals itself when starting the container.

### 4. `vault_set_pki.sh`
Example output:

This script is used to generate the root certificate, intermediate certificate and HTTPS server certificate.
All generate certificates, keys and CSR by `vault_set_pki.sh` will be present at `docker/addons/vault/data`.
```bash
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce 4c248cc8-e9f5-055e-319b-00ee06f998a0
Version 1.15.4
Build Date 2023-12-04T17:45:28Z
Storage Type file
HA Enabled false
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce 4c248cc8-e9f5-055e-319b-00ee06f998a0
Version 1.15.4
Build Date 2023-12-04T17:45:28Z
Storage Type file
HA Enabled false
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Unseal Progress 3/3
Unseal Nonce 4c248cc8-e9f5-055e-319b-00ee06f998a0
Version 1.15.4
Build Date 2023-12-04T17:45:28Z
Storage Type file
HA Enabled false
```

### 4. vault_set_pki.sh

The `vault_set_pki.sh` script is responsible for generating the root certificate, intermediate certificate, and HTTPS server certificate. All generated certificates, keys, and CSR files are stored in the `docker/addons/vault/scripts/data` directory.

The script pulls necessary parameters for certificate generation from environment variables, which are, by default, loaded from `docker/.env`.

The parameters required for generating certificate are obtained from the environment variables which are loaded from `docker/.env`.
- Environment variables prefixed with `MG_VAULT_PKI` in the `docker/.env` file are used for generating the root CA.
- Environment variables prefixed with `MG_VAULT_PKI_INT` are used for generating the intermediate CA.

Environmental variables starting with `MG_VAULT_PKI` in `docker/.env` file are used by `vault_set_pki.sh` to generate root CA.
Environmental variables starting with`MG_VAULT_PKI_INT` in `docker/.env` file are used by `vault_set_pki.sh` to generate intermediate CA.
To skip generating the server certificate and key, you can pass the `--skip-server-cert` option to the script:

Passing command line args `--skip-server-cert` to `vault_set_pki.sh` will skip server certificate role & process of generation of server certificate & key.
```sh
./vault_set_pki.sh --skip-server-cert
```

#### Troubleshooting:

### 5. `vault_create_approle.sh`
If you encounter the following error:

This script is used to enable app role authorization in Vault. Certs service used the approle credentials to issue, revoke things certificate from vault intermedate CA.
```sh
jq command could not be found, please install it and try again.
```

`vault_create_approle.sh` script by default tries to enable auth approle.
If approle is already enabled in vault, then use args `--skip-enable-approle` to skip enable auth approle step.
To skip enable auth approle step use the following `vault_create_approle.sh --skip-enable-approle`
Install `jq` using:

```sh
sudo apt-get update && sudo apt-get install -y jq
```

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

After installing `jq`, rerun the script.

### 5. `vault_create_approle.sh`

This script enables AppRole authorization in Vault. The certs service uses these AppRole credentials to issue and revoke certificates from the Vault intermediate CA.

Example output:

```sh
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key Value
--- -----
token <token_value>
token_accessor i6YVeKh4wQ4e0Aj0ONiyGw1Z
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]
Creating new policy for AppRole
Successfully copied 2.56kB to magistrala-vault:/vault/magistrala_things_certs_issue.hcl
Success! Uploaded policy: magistrala_things_certs_issue
Enabling AppRole
Success! Enabled approle auth method at: approle/
Deleting old AppRole
Success! Data deleted (if it existed) at: auth/approle/role/magistrala_things_certs_issuer
Creating new AppRole
Success! Data written to: auth/approle/role/magistrala_things_certs_issuer
Writing custom role ID
Key Value
--- -----
role_id f23942b3-62b9-7456-784f-220ca3f703b9
Success! Data written to: auth/approle/role/magistrala_things_certs_issuer/role-id
Writing custom secret
Key Value
--- -----
secret_id 61d5a30f-634c-6027-f5b6-4934e6fc49b2
secret_id_accessor 1d744f6e-e0c2-5431-a87a-2b23fde584a7
secret_id_num_uses 0
secret_id_ttl 0s
Testing custom role ID and secret by logging in
Key Value
--- -----
token <token_value>
token_accessor 9cuwS4mrLHKhJQMv0pl9Bbg9
token_duration 1h
token_renewable true
token_policies ["default" "magistrala_things_certs_issue"]
identity_policies []
policies ["default" "magistrala_things_certs_issue"]
token_meta_role_name magistrala_things_certs_issuer
```

By default, the `vault_create_approle.sh` script tries to enable the AppRole authentication method. Certs service uses the approle credentials to issue and revoke things certificate from vault intermedate CA. If AppRole is already enabled, you can skip this step by passing the `--skip-enable-approle` argument:

```sh
./vault_create_approle.sh --skip-enable-approle
```

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

### 6. `vault_copy_certs.sh`

This scripts copies the necessary certificates and keys from `docker/addons/vault/data` to the `docker/ssl/certs` folder.
This script copies the required certificates and keys from `docker/addons/vault/scripts/data` to the `docker/ssl/certs` folder.

Example output:

```bash
Copying certificate files
'data/localhost.crt' -> '~/Documents/magistrala/docker/ssl/certs/magistrala-server.crt'
'data/localhost.key' -> '~/Documents/magistrala/docker/ssl/certs/magistrala-server.key'
'data/mg_int.key' -> '~/Documents/magistrala/docker/ssl/certs/ca.key'
'data/mg_int_bundle.crt' -> '~/Documents/magistrala/docker/ssl/certs/ca.crt'
```

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

## Custom `.env` Path Support

Vault scripts support specifying a custom `.env` file path using the `--env-file` argument. If this argument is not provided, the scripts will use the default `.env` file located at `docker/.env`.

To use a different `.env` file, include the `--env-file` argument followed by the path to your `.env` file when running the Vault scripts. Below are examples of how to execute each script with a custom `.env` file path:

```bash
./vault_init.sh --env-file /custom/path/.env
./vault_copy_env.sh --env-file /custom/path/.env
./vault_unseal.sh --env-file /custom/path/.env
./vault_set_pki.sh --env-file /custom/path/.env
./vault_create_approle.sh --env-file /custom/path/.env
./vault_copy_certs.sh --env-file /custom/path/.env
```

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

## Hashicorp Cloud Platform (HCP) Vault

Expand All @@ -124,7 +244,7 @@ Requirement: [VAULT CLI](https://developer.hashicorp.com/vault/tutorials/getting

- Replace the environmental variable `MG_VAULT_ADDR` in `docker/.env` with HCP Vault address.
- Replace the environmental variable `MG_VAULT_TOKEN` in `docker/.env` with HCP Vault Admin token.
- Run script `vault_set_pki.sh` and `vault_create_approle.sh`.
- Run script `vault_set_pki.sh` and `vault_create_approle.sh`.
- Optional step, run script `vault_copy_certs.sh` to copy certificates to magistrala default path.

## Vault CLI
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,3 @@

data
magistrala_things_certs_issue.hcl
.env
24 changes: 24 additions & 0 deletions scripts/vault/scripts/vault_cmd.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
#!/usr/bin/bash
# Copyright (c) Abstract Machines
# SPDX-License-Identifier: Apache-2.0

vault() {
if is_container_running "magistrala-vault"; then
docker exec -it magistrala-vault vault "$@"
else
if which vault &> /dev/null; then
$(which vault) "$@"
else
echo "magistrala-vault container or vault command not found. Please refer to the documentation: https://github.com/absmach/magistrala/blob/main/docker/addons/vault/README.md"
fi
fi
}

is_container_running() {
local container_name="$1"
if [ "$(docker inspect --format '{{.State.Running}}' "$container_name" 2>/dev/null)" = "true" ]; then
return 0
else
return 1
fi
}
86 changes: 86 additions & 0 deletions scripts/vault/scripts/vault_copy_certs.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
#!/usr/bin/bash
# Copyright (c) Abstract Machines
# SPDX-License-Identifier: Apache-2.0

set -euo pipefail

scriptdir="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"

# default env file path
env_file="docker/.env"

# default certs copy path
certs_copy_path="docker/ssl/certs/"

while [[ "$#" -gt 0 ]]; do
case $1 in
--env-file)
if [[ -z "${2:-}" ]]; then
echo "Error: --env-file requires a non-empty option argument."
exit 1
fi
env_file="$2"
if [[ ! -f "$env_file" ]]; then
echo "Error: .env file not found at $env_file"
exit 1
fi
shift
;;
--certs-copy-path)
if [[ -z "${2:-}" ]]; then
echo "Error: --certs-copy-path requires a non-empty option argument."
exit 1
fi
certs_copy_path="$2"
shift
;;
*)
echo "Error: Unknown parameter passed: $1"
exit 1
;;
esac
shift
done

readDotEnv() {
set -o allexport
source "$env_file"
set +o allexport
}

readDotEnv

server_name="localhost"

# Check if MG_NGINX_SERVER_NAME is set or not empty
if [ -n "${MG_NGINX_SERVER_NAME:-}" ]; then
server_name="$MG_NGINX_SERVER_NAME"
fi

echo "Copying certificate files to ${certs_copy_path}"

if [ -e "$scriptdir/data/${server_name}.crt" ]; then
cp -v "$scriptdir/data/${server_name}.crt" "${certs_copy_path}magistrala-server.crt"
else
echo "${server_name}.crt file not available"
fi

if [ -e "$scriptdir/data/${server_name}.key" ]; then
cp -v "$scriptdir/data/${server_name}.key" "${certs_copy_path}magistrala-server.key"
else
echo "${server_name}.key file not available"
fi

if [ -e "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}.key" ]; then
cp -v "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}.key" "${certs_copy_path}ca.key"
else
echo "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}.key file not available"
fi

if [ -e "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}_bundle.crt" ]; then
cp -v "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}_bundle.crt" "${certs_copy_path}ca.crt"
else
echo "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}_bundle.crt file not available"
fi

exit 0
Loading