Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MG-153 - Add Vault scripts as git subtree #157

Open
wants to merge 46 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 39 commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
8cab8df
Squashed 'docker/addons/vault/' content from commit 54a134e7
JeffMboya Nov 18, 2024
aedfbae
Merge commit '8cab8df5a1cfd26191442818472cfc8ad0680da6' as 'docker/ad…
JeffMboya Nov 18, 2024
beba5e0
Remove docker/addons/vault directory
JeffMboya Nov 18, 2024
81356c9
Remove scripts/vault directory
JeffMboya Nov 18, 2024
22f2f60
Squashed 'vault/' content from commit a32634a1e
JeffMboya Nov 18, 2024
25db402
Merge commit '22f2f6083f30faec72f8fb72f9e7e8107739accf' as 'vault'
JeffMboya Nov 18, 2024
cffc3bb
Remove scripts/vault directory
JeffMboya Nov 18, 2024
eb6f4fc
Squashed 'vault/' content from commit a32634a1e
JeffMboya Nov 18, 2024
f4df847
Merge commit 'eb6f4fc1b399c3db293741619638ac7db06b2d29' as 'vault'
JeffMboya Nov 18, 2024
cf5b45d
Squashed 'scripts/vault/' content from commit a32634a1e
JeffMboya Nov 18, 2024
0b8ad42
Merge commit 'cf5b45de649d9b7df1f8a3711bc527a6bf87158d' as 'scripts/v…
JeffMboya Nov 18, 2024
ffd46e3
Add vault/scripts directory
JeffMboya Nov 18, 2024
d7314d9
Delete vault directory
JeffMboya Nov 18, 2024
04a24ae
Squashed 'scripts/vault/' content from commit a32634a1e
JeffMboya Nov 18, 2024
9c7145a
Merge commit '04a24aea9c420f527d7db15cb8a1b863328ce53a' as 'scripts/v…
JeffMboya Nov 18, 2024
cfc0100
Add scripts/vault directory
JeffMboya Nov 18, 2024
8fb2d05
Add .env, vault.md, and efk.sh
JeffMboya Nov 18, 2024
4f98a62
Add empty line
JeffMboya Nov 18, 2024
fdfd93f
Update vault docs
JeffMboya Nov 19, 2024
d05d2a8
Remove git pull command
JeffMboya Nov 19, 2024
fa8adb6
Address comments
JeffMboya Nov 19, 2024
483526c
Add empty line
JeffMboya Nov 19, 2024
6bb155f
move .env file
JeffMboya Nov 19, 2024
7cb578d
Squashed 'scripts/vault/' content from commit a32634a1e
JeffMboya Nov 19, 2024
4d0aa48
Merge commit '7cb578dbedee70ca0eac10a8c5caed7d8ac53d5c' as 'scripts/v…
JeffMboya Nov 19, 2024
1006b1c
Squashed 'vault/' content from commit a32634a1e
JeffMboya Nov 19, 2024
fc151d6
Merge commit '1006b1c66b57497784783f6b7414ba136dcf3d11' as 'vault'
JeffMboya Nov 19, 2024
e60cc9f
move .env file
JeffMboya Nov 19, 2024
e0b0ce3
Squashed 'vault/' content from commit a32634a1e
JeffMboya Nov 19, 2024
35e12b4
Merge commit 'e0b0ce33ba0600d7316f7bc8f2fa451c7b6da9a8' as 'vault'
JeffMboya Nov 19, 2024
f4f15f1
move .env file
JeffMboya Nov 19, 2024
42ab894
Squashed 'scripts/vault/' content from commit a32634a1e
JeffMboya Nov 19, 2024
3623b16
Merge commit '42ab894250fa1bf49f3d8a6a9e89fa3116c56ab8' as 'scripts/v…
JeffMboya Nov 19, 2024
ad4361b
move .env file
JeffMboya Nov 19, 2024
4e470fa
Add efk.sh
JeffMboya Nov 19, 2024
b98267d
Squashed 'docker/addons/vault/scripts/' content from commit d9f3bca47
JeffMboya Nov 19, 2024
1564491
Merge commit 'b98267df3d82f58f4d30ec7f34fb177f202fe030' as 'docker/ad…
JeffMboya Nov 19, 2024
ab3f879
Remove unnecessary files
JeffMboya Nov 19, 2024
fad3ac7
Address comments
JeffMboya Nov 19, 2024
36238f0
Remove duplicate instructions
JeffMboya Nov 21, 2024
70b83ae
Update README
JeffMboya Nov 28, 2024
ea26149
Fix users authentication error
JeffMboya Nov 28, 2024
58a7f37
Add AMA certs
JeffMboya Nov 28, 2024
dd608db
Bump chart version
JeffMboya Nov 28, 2024
33c16cf
Remove ref to magistrala
JeffMboya Nov 28, 2024
735284e
Remove vault config files
JeffMboya Nov 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,20 @@ git push origin <your-branch>

Replace `<your-branch>` with the branch you are working on.

### Running Vault Setup Scripts with `--env-file`

To run a Vault setup script, use the `--env-file` option to specify the path to your `.env` file:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JeffMboya This part is moved scripts/vault/Readme.md , so i think it is no more needed here
What do you think ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JeffMboya This part is moved scripts/vault/Readme.md , so i think it is no more needed here What do you think ?

Yes, it should be removed.


```bash
./<script-name>.sh --env-file <path-to-your-env-file>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this part to scripts/vault/Readme.md and add words like refer magistrala vault document (and its link)

```

For example:

```bash
scripts/vault/scripts/vault_init.sh --env-file scripts/vault/.env
```

## License

This project is licensed under the [Apache-2.0](LICENSE).
1 change: 0 additions & 1 deletion scripts/.gitignore

This file was deleted.

167 changes: 7 additions & 160 deletions scripts/vault/README.md
Original file line number Diff line number Diff line change
@@ -1,170 +1,17 @@
# Vault
# Running Vault Setup Scripts with `--env-file`

This is Vault service deployment to be used with Magistrala.

When the Vault service is started, some initialization steps need to be done to set things up.

## Configuration

| Variable | Description | Default |
| :-------------------------------------- | ----------------------------------------------------------------------------- | ------------------------------------- |
| MG_VAULT_ADDR | Vault Address | http://vault:8200 |
| MG_VAULT_UNSEAL_KEY_1 | Vault unseal key | "" |
| MG_VAULT_UNSEAL_KEY_2 | Vault unseal key | "" |
| MG_VAULT_UNSEAL_KEY_3 | Vault unseal key | "" |
| MG_VAULT_TOKEN | Vault cli access token | "" |
| MG_VAULT_PKI_PATH | Vault secrets engine path for Root CA | pki |
| MG_VAULT_PKI_ROLE_NAME | Vault Root CA role name to issue intermediate CA | magistrala_int_ca |
| MG_VAULT_PKI_FILE_NAME | Root CA Certificates name used by`vault_set_pki.sh` | mg_root |
| MG_VAULT_PKI_CA_CN | Common name used for Root CA creation by`vault_set_pki.sh` | Magistrala Root Certificate Authority |
| MG_VAULT_PKI_CA_OU | Organization unit used for Root CA creation by`vault_set_pki.sh` | Magistrala |
| MG_VAULT_PKI_CA_O | Organization used for Root CA creation by`vault_set_pki.sh` | Magistrala |
| MG_VAULT_PKI_CA_C | Country used for Root CA creation by`vault_set_pki.sh` | FRANCE |
| MG_VAULT_PKI_CA_L | Location used for Root CA creation by`vault_set_pki.sh` | PARIS |
| MG_VAULT_PKI_CA_ST | State or Provisions used for Root CA creation by`vault_set_pki.sh` | PARIS |
| MG_VAULT_PKI_CA_ADDR | Address used for Root CA creation by`vault_set_pki.sh` | 5 Av. Anatole |
| MG_VAULT_PKI_CA_PO | Postal code used for Root CA creation by`vault_set_pki.sh` | 75007 |
| MG_VAULT_PKI_CLUSTER_PATH | Vault Root CA Cluster Path | http://localhost |
| MG_VAULT_PKI_CLUSTER_AIA_PATH | Vault Root CA Cluster AIA Path | http://localhost |
| MG_VAULT_PKI_INT_PATH | Vault secrets engine path for Intermediate CA | pki_int |
| MG_VAULT_PKI_INT_SERVER_CERTS_ROLE_NAME | Vault Intermediate CA role name to issue server certificate | magistrala_server_certs |
| MG_VAULT_PKI_INT_THINGS_CERTS_ROLE_NAME | Vault Intermediate CA role name to issue Things certificates | magistrala_things_certs |
| MG_VAULT_PKI_INT_FILE_NAME | Intermediate CA Certificates name used by`vault_set_pki.sh` | mg_root |
| MG_VAULT_PKI_INT_CA_CN | Common name used for Intermediate CA creation by`vault_set_pki.sh` | Magistrala Root Certificate Authority |
| MG_VAULT_PKI_INT_CA_OU | Organization unit used for Root CA creation by`vault_set_pki.sh` | Magistrala |
| MG_VAULT_PKI_INT_CA_O | Organization used for Intermediate CA creation by`vault_set_pki.sh` | Magistrala |
| MG_VAULT_PKI_INT_CA_C | Country used for Intermediate CA creation by`vault_set_pki.sh` | FRANCE |
| MG_VAULT_PKI_INT_CA_L | Location used for Intermediate CA creation by`vault_set_pki.sh` | PARIS |
| MG_VAULT_PKI_INT_CA_ST | State or Provisions used for Intermediate CA creation by`vault_set_pki.sh` | PARIS |
| MG_VAULT_PKI_INT_CA_ADDR | Address used for Intermediate CA creation by`vault_set_pki.sh` | 5 Av. Anatole |
| MG_VAULT_PKI_INT_CA_PO | Postal code used for Intermediate CA creation by`vault_set_pki.sh` | 75007 |
| MG_VAULT_PKI_INT_CLUSTER_PATH | Vault Intermediate CA Cluster Path | http://localhost |
| MG_VAULT_PKI_INT_CLUSTER_AIA_PATH | Vault Intermediate CA Cluster AIA Path | http://localhost |
| MG_VAULT_THINGS_CERTS_ISSUER_ROLEID | Vault Intermediate CA Things Certificate issuer AppRole authentication RoleID | magistrala |
| MG_VAULT_THINGS_CERTS_ISSUER_SECRET | Vault Intermediate CA Things Certificate issuer AppRole authentication Secret | magistrala |

## Setup

The following scripts are provided, which work on the running Vault service in Docker.

### 1. `vault_init.sh`

Calls `vault operator init` to perform the initial vault initialization and generates a `docker/addons/vault/data/secrets` file which contains the Vault unseal keys and root tokens.

Example contents for `data/secrets`:
To execute a Vault setup script, use the `--env-file` option to provide the path to your `.env` file. Here's the general syntax:

```bash
Unseal Key 1: Ay0YZecYJ2HVtNtXfPootXK5LtF+JZoDmBb7IbbYdLBI
Unseal Key 2: P6hb7x2cglv0p61jdLyNE3+d44cJUOFaDt9jHFDfr8Df
Unseal Key 3: zSBfDHzUiWoOzXKY1pnnBqKO8UD2MDLuy8DNTxNtEBFy
Unseal Key 4: 5oJuDDuMI0I8snaw/n4VLNpvndvvKi6JlkgOxuWXqMSz
Unseal Key 5: ZhsUkk2tXBYEcWgz4WUCHH9rocoW6qZoiARWlkE5Epi5

Initial Root Token: s.V2hdd00P4bHtUQnoWZK2hSaS

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
bash-4.4

Use 3 out of five keys presented and put it into .env file and than start the composition again Vault should be in unsealed state ( take a note that this is not recommended in terms of security, this is deployment for development) A real production deployment can use Vault auto unseal mode where vault gets unseal keys from some 3rd party KMS ( on AWS for example)
./<script-name>.sh --env-file <path-to-your-env-file>
```

### 2. `vault_copy_env.sh`

After first step, the corresponding Vault environment variables (`MG_VAULT_TOKEN`, `MG_VAULT_UNSEAL_KEY_1`, `MG_VAULT_UNSEAL_KEY_2`, `MG_VAULT_UNSEAL_KEY_3`) should be updated in `.env` file.

`vault_copy_env.sh` scripts copies values from `docker/addons/vault/data/secrets` file and update environmental variables `MG_VAULT_TOKEN`, `MG_VAULT_UNSEAL_KEY_1`, `MG_VAULT_UNSEAL_KEY_2`, `MG_VAULT_UNSEAL_KEY_3` present in `.env` file.

### 3. `vault_unseal.sh`

This can be run after the initialization to unseal Vault, which is necessary for it to be used to store and/or get secrets.

This can be used if you don't want to restart the service.

The unseal environment variables need to be set in `.env` for the script to work (`MG_VAULT_TOKEN`,`MG_VAULT_UNSEAL_KEY_1`, `MG_VAULT_UNSEAL_KEY_2`, `MG_VAULT_UNSEAL_KEY_3`).
### Example

This script should not be necessary to run after the initial setup, since the Vault service unseals itself when starting the container.

### 4. `vault_set_pki.sh`

This script is used to generate the root certificate, intermediate certificate and HTTPS server certificate.
All generate certificates, keys and CSR by `vault_set_pki.sh` will be present at `docker/addons/vault/data`.

The parameters required for generating certificate are obtained from the environment variables which are loaded from `docker/.env`.

Environmental variables starting with `MG_VAULT_PKI` in `docker/.env` file are used by `vault_set_pki.sh` to generate root CA.
Environmental variables starting with`MG_VAULT_PKI_INT` in `docker/.env` file are used by `vault_set_pki.sh` to generate intermediate CA.

Passing command line args `--skip-server-cert` to `vault_set_pki.sh` will skip server certificate role & process of generation of server certificate & key.

### 5. `vault_create_approle.sh`

This script is used to enable app role authorization in Vault. Certs service used the approle credentials to issue, revoke things certificate from vault intermedate CA.

`vault_create_approle.sh` script by default tries to enable auth approle.
If approle is already enabled in vault, then use args `--skip-enable-approle` to skip enable auth approle step.
To skip enable auth approle step use the following `vault_create_approle.sh --skip-enable-approle`

### 6. `vault_copy_certs.sh`

This scripts copies the necessary certificates and keys from `docker/addons/vault/data` to the `docker/ssl/certs` folder.

## Hashicorp Cloud Platform (HCP) Vault

To have the same PKI setup can done in Hashicorp Cloud Platform (HCP) Vault follow the below steps:
Requirement: [VAULT CLI](https://developer.hashicorp.com/vault/tutorials/getting-started/getting-started-install)

- Replace the environmental variable `MG_VAULT_ADDR` in `docker/.env` with HCP Vault address.
- Replace the environmental variable `MG_VAULT_TOKEN` in `docker/.env` with HCP Vault Admin token.
- Run script `vault_set_pki.sh` and `vault_create_approle.sh`.
- Optional step, run script `vault_copy_certs.sh` to copy certificates to magistrala default path.

## Vault CLI

It can also be useful to run the Vault CLI for inspection and administration work.
To initialize Vault using the provided setup script, run:

```bash
Usage: vault <command> [args]

Common commands:
read Read data and retrieves secrets
write Write data, configuration, and secrets
delete Delete secrets and configuration
list List data or secrets
login Authenticate locally
agent Start a Vault agent
server Start a Vault server
status Print seal and HA status
unwrap Unwrap a wrapped secret

Other commands:
audit Interact with audit devices
auth Interact with auth methods
debug Runs the debug command
kv Interact with Vault's Key-Value storage
lease Interact with leases
monitor Stream log messages from a Vault server
namespace Interact with namespaces
operator Perform operator-specific tasks
path-help Retrieve API help for paths
plugin Interact with Vault plugins and catalog
policy Interact with policies
print Prints runtime configurations
secrets Interact with secrets engines
ssh Initiate an SSH session
token Interact with tokens
scripts/vault/scripts/vault_init.sh --env-file scripts/vault/.env
```

If the Vault is setup through `docker/addons/vault`, then Vault CLI can be run directly using the Vault image in Docker: `docker run -it magistrala/vault:latest vault`

## Vault Web UI

If the Vault is setup through `docker/addons/vault`, Then Vault Web UI is accessible by default on `http://localhost:8200/ui`.
For detailed documentation on the available scripts and their usage, visit the [Vault Addon Documentation](https://github.com/absmach/magistrala/tree/main/docker/addons/vault#readme).
Copy link
Contributor

@arvindh123 arvindh123 Nov 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add following guide

# Developer Guide: Updating Vault Scripts from Magistrala

This guide provides step-by-step instructions to update the Vault scripts in your local Magistrala DevOps repository, sync them with the `main` branch of the Magistrala repository, and create a pull request (PR) to merge the changes.

## Prerequisites

Make sure you have the following:
- A local clone of the Magistrala DevOps repository.
- Access to the Magistrala GitHub repository (`https://github.com/absmach/magistrala.git`).

## Step 1: Create new branch 

Create a new branch from `master` brach

```bash
git checkout -b <your-branch-name>
``

Replace `<your-branch-name>` with the name a new branch name.

## Step 2: Add the Magistrala Remote

If the Magistrala remote is not already added to your local repository, use the following command to add it:

```bash
git remote add -f magistrala https://github.com/absmach/magistrala.git
``

## Step 3: Add Subtree for Vault Scripts

If the subtree for the Vault scripts has not been added, execute the following command:

```bash
git subtree add --prefix=scripts/vault/scripts magistrala main --squash --prefix=docker/addons/vault/scripts
``

- `--prefix=scripts/vault/scripts`: Specifies the target directory in your local repository where the Vault scripts will be added.
- `magistrala main`: Refers to the `main` branch of the Magistrala repository.
- `--squash`: Combines all commits from the Magistrala `main` branch into a single commit when adding the subtree.

## Step 4: Update Vault Scripts to the Latest Version

To update the Vault scripts to the latest version and synchronize with the `main` branch of the Magistrala repository, use the following command:

```bash
git subtree pull --prefix=scripts/vault/scripts magistrala main --squash --prefix=docker/addons/vault/scripts
``

- This command pulls the latest changes from the `main` branch of the Magistrala repository.
- `--squash` creates a single commit for the changes, making the history simpler to manage.

## Step 5: Push Changes and Create a Pull Request

After syncing with the Magistrala `main` branch, push the changes to your working branch:

```bash
git push origin <your-branch-name>
``

Replace `<your-branch-name>` with the name of the branch you're working on.

Once the changes are pushed, go to your GitHub repository and create a pull request (PR) to merge the updates.

## Summary of Commands related to Git subtree

### Add Magistrala Remote
```bash
git remote add -f magistrala https://github.com/absmach/magistrala.git
``

### Add Subtree for Vault Scripts
```bash
git subtree add --prefix=scripts/vault/scripts magistrala main --squash --prefix=docker/addons/vault/scripts
``

### Update Vault Scripts to the Latest
```bash
git subtree pull --prefix=scripts/vault/scripts magistrala main --squash --prefix=docker/addons/vault/scripts
``

39 changes: 0 additions & 39 deletions scripts/vault/docker-compose.yml

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,3 @@

data
magistrala_things_certs_issue.hcl
.env
24 changes: 24 additions & 0 deletions scripts/vault/scripts/vault_cmd.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
#!/usr/bin/bash
# Copyright (c) Abstract Machines
# SPDX-License-Identifier: Apache-2.0

vault() {
if is_container_running "magistrala-vault"; then
docker exec -it magistrala-vault vault "$@"
else
if which vault &> /dev/null; then
$(which vault) "$@"
else
echo "magistrala-vault container or vault command not found. Please refer to the documentation: https://github.com/absmach/magistrala/blob/main/docker/addons/vault/README.md"
fi
fi
}

is_container_running() {
local container_name="$1"
if [ "$(docker inspect --format '{{.State.Running}}' "$container_name" 2>/dev/null)" = "true" ]; then
return 0
else
return 1
fi
}
86 changes: 86 additions & 0 deletions scripts/vault/scripts/vault_copy_certs.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
#!/usr/bin/bash
# Copyright (c) Abstract Machines
# SPDX-License-Identifier: Apache-2.0

set -euo pipefail

scriptdir="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"

# default env file path
env_file="docker/.env"

# default certs copy path
certs_copy_path="docker/ssl/certs/"

while [[ "$#" -gt 0 ]]; do
case $1 in
--env-file)
if [[ -z "${2:-}" ]]; then
echo "Error: --env-file requires a non-empty option argument."
exit 1
fi
env_file="$2"
if [[ ! -f "$env_file" ]]; then
echo "Error: .env file not found at $env_file"
exit 1
fi
shift
;;
--certs-copy-path)
if [[ -z "${2:-}" ]]; then
echo "Error: --certs-copy-path requires a non-empty option argument."
exit 1
fi
certs_copy_path="$2"
shift
;;
*)
echo "Error: Unknown parameter passed: $1"
exit 1
;;
esac
shift
done

readDotEnv() {
set -o allexport
source "$env_file"
set +o allexport
}

readDotEnv

server_name="localhost"

# Check if MG_NGINX_SERVER_NAME is set or not empty
if [ -n "${MG_NGINX_SERVER_NAME:-}" ]; then
server_name="$MG_NGINX_SERVER_NAME"
fi

echo "Copying certificate files to ${certs_copy_path}"

if [ -e "$scriptdir/data/${server_name}.crt" ]; then
cp -v "$scriptdir/data/${server_name}.crt" "${certs_copy_path}magistrala-server.crt"
else
echo "${server_name}.crt file not available"
fi

if [ -e "$scriptdir/data/${server_name}.key" ]; then
cp -v "$scriptdir/data/${server_name}.key" "${certs_copy_path}magistrala-server.key"
else
echo "${server_name}.key file not available"
fi

if [ -e "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}.key" ]; then
cp -v "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}.key" "${certs_copy_path}ca.key"
else
echo "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}.key file not available"
fi

if [ -e "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}_bundle.crt" ]; then
cp -v "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}_bundle.crt" "${certs_copy_path}ca.crt"
else
echo "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}_bundle.crt file not available"
fi

exit 0
Loading