-
Notifications
You must be signed in to change notification settings - Fork 34
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MG-153 - Add Vault scripts as git subtree #157
base: master
Are you sure you want to change the base?
Changes from 39 commits
8cab8df
aedfbae
beba5e0
81356c9
22f2f60
25db402
cffc3bb
eb6f4fc
f4df847
cf5b45d
0b8ad42
ffd46e3
d7314d9
04a24ae
9c7145a
cfc0100
8fb2d05
4f98a62
fdfd93f
d05d2a8
fa8adb6
483526c
6bb155f
7cb578d
4d0aa48
1006b1c
fc151d6
e60cc9f
e0b0ce3
35e12b4
f4f15f1
42ab894
3623b16
ad4361b
4e470fa
b98267d
1564491
ab3f879
fad3ac7
36238f0
70b83ae
ea26149
58a7f37
dd608db
33c16cf
735284e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -57,6 +57,20 @@ git push origin <your-branch> | |
|
||
Replace `<your-branch>` with the branch you are working on. | ||
|
||
### Running Vault Setup Scripts with `--env-file` | ||
|
||
To run a Vault setup script, use the `--env-file` option to specify the path to your `.env` file: | ||
|
||
```bash | ||
./<script-name>.sh --env-file <path-to-your-env-file> | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Move this part to scripts/vault/Readme.md and add words like refer magistrala vault document (and its link) |
||
``` | ||
|
||
For example: | ||
|
||
```bash | ||
scripts/vault/scripts/vault_init.sh --env-file scripts/vault/.env | ||
``` | ||
|
||
## License | ||
|
||
This project is licensed under the [Apache-2.0](LICENSE). |
This file was deleted.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,170 +1,17 @@ | ||
# Vault | ||
# Running Vault Setup Scripts with `--env-file` | ||
|
||
This is Vault service deployment to be used with Magistrala. | ||
|
||
When the Vault service is started, some initialization steps need to be done to set things up. | ||
|
||
## Configuration | ||
|
||
| Variable | Description | Default | | ||
| :-------------------------------------- | ----------------------------------------------------------------------------- | ------------------------------------- | | ||
| MG_VAULT_ADDR | Vault Address | http://vault:8200 | | ||
| MG_VAULT_UNSEAL_KEY_1 | Vault unseal key | "" | | ||
| MG_VAULT_UNSEAL_KEY_2 | Vault unseal key | "" | | ||
| MG_VAULT_UNSEAL_KEY_3 | Vault unseal key | "" | | ||
| MG_VAULT_TOKEN | Vault cli access token | "" | | ||
| MG_VAULT_PKI_PATH | Vault secrets engine path for Root CA | pki | | ||
| MG_VAULT_PKI_ROLE_NAME | Vault Root CA role name to issue intermediate CA | magistrala_int_ca | | ||
| MG_VAULT_PKI_FILE_NAME | Root CA Certificates name used by`vault_set_pki.sh` | mg_root | | ||
| MG_VAULT_PKI_CA_CN | Common name used for Root CA creation by`vault_set_pki.sh` | Magistrala Root Certificate Authority | | ||
| MG_VAULT_PKI_CA_OU | Organization unit used for Root CA creation by`vault_set_pki.sh` | Magistrala | | ||
| MG_VAULT_PKI_CA_O | Organization used for Root CA creation by`vault_set_pki.sh` | Magistrala | | ||
| MG_VAULT_PKI_CA_C | Country used for Root CA creation by`vault_set_pki.sh` | FRANCE | | ||
| MG_VAULT_PKI_CA_L | Location used for Root CA creation by`vault_set_pki.sh` | PARIS | | ||
| MG_VAULT_PKI_CA_ST | State or Provisions used for Root CA creation by`vault_set_pki.sh` | PARIS | | ||
| MG_VAULT_PKI_CA_ADDR | Address used for Root CA creation by`vault_set_pki.sh` | 5 Av. Anatole | | ||
| MG_VAULT_PKI_CA_PO | Postal code used for Root CA creation by`vault_set_pki.sh` | 75007 | | ||
| MG_VAULT_PKI_CLUSTER_PATH | Vault Root CA Cluster Path | http://localhost | | ||
| MG_VAULT_PKI_CLUSTER_AIA_PATH | Vault Root CA Cluster AIA Path | http://localhost | | ||
| MG_VAULT_PKI_INT_PATH | Vault secrets engine path for Intermediate CA | pki_int | | ||
| MG_VAULT_PKI_INT_SERVER_CERTS_ROLE_NAME | Vault Intermediate CA role name to issue server certificate | magistrala_server_certs | | ||
| MG_VAULT_PKI_INT_THINGS_CERTS_ROLE_NAME | Vault Intermediate CA role name to issue Things certificates | magistrala_things_certs | | ||
| MG_VAULT_PKI_INT_FILE_NAME | Intermediate CA Certificates name used by`vault_set_pki.sh` | mg_root | | ||
| MG_VAULT_PKI_INT_CA_CN | Common name used for Intermediate CA creation by`vault_set_pki.sh` | Magistrala Root Certificate Authority | | ||
| MG_VAULT_PKI_INT_CA_OU | Organization unit used for Root CA creation by`vault_set_pki.sh` | Magistrala | | ||
| MG_VAULT_PKI_INT_CA_O | Organization used for Intermediate CA creation by`vault_set_pki.sh` | Magistrala | | ||
| MG_VAULT_PKI_INT_CA_C | Country used for Intermediate CA creation by`vault_set_pki.sh` | FRANCE | | ||
| MG_VAULT_PKI_INT_CA_L | Location used for Intermediate CA creation by`vault_set_pki.sh` | PARIS | | ||
| MG_VAULT_PKI_INT_CA_ST | State or Provisions used for Intermediate CA creation by`vault_set_pki.sh` | PARIS | | ||
| MG_VAULT_PKI_INT_CA_ADDR | Address used for Intermediate CA creation by`vault_set_pki.sh` | 5 Av. Anatole | | ||
| MG_VAULT_PKI_INT_CA_PO | Postal code used for Intermediate CA creation by`vault_set_pki.sh` | 75007 | | ||
| MG_VAULT_PKI_INT_CLUSTER_PATH | Vault Intermediate CA Cluster Path | http://localhost | | ||
| MG_VAULT_PKI_INT_CLUSTER_AIA_PATH | Vault Intermediate CA Cluster AIA Path | http://localhost | | ||
| MG_VAULT_THINGS_CERTS_ISSUER_ROLEID | Vault Intermediate CA Things Certificate issuer AppRole authentication RoleID | magistrala | | ||
| MG_VAULT_THINGS_CERTS_ISSUER_SECRET | Vault Intermediate CA Things Certificate issuer AppRole authentication Secret | magistrala | | ||
|
||
## Setup | ||
|
||
The following scripts are provided, which work on the running Vault service in Docker. | ||
|
||
### 1. `vault_init.sh` | ||
|
||
Calls `vault operator init` to perform the initial vault initialization and generates a `docker/addons/vault/data/secrets` file which contains the Vault unseal keys and root tokens. | ||
|
||
Example contents for `data/secrets`: | ||
To execute a Vault setup script, use the `--env-file` option to provide the path to your `.env` file. Here's the general syntax: | ||
|
||
```bash | ||
Unseal Key 1: Ay0YZecYJ2HVtNtXfPootXK5LtF+JZoDmBb7IbbYdLBI | ||
Unseal Key 2: P6hb7x2cglv0p61jdLyNE3+d44cJUOFaDt9jHFDfr8Df | ||
Unseal Key 3: zSBfDHzUiWoOzXKY1pnnBqKO8UD2MDLuy8DNTxNtEBFy | ||
Unseal Key 4: 5oJuDDuMI0I8snaw/n4VLNpvndvvKi6JlkgOxuWXqMSz | ||
Unseal Key 5: ZhsUkk2tXBYEcWgz4WUCHH9rocoW6qZoiARWlkE5Epi5 | ||
|
||
Initial Root Token: s.V2hdd00P4bHtUQnoWZK2hSaS | ||
|
||
Vault initialized with 5 key shares and a key threshold of 3. Please securely | ||
distribute the key shares printed above. When the Vault is re-sealed, | ||
restarted, or stopped, you must supply at least 3 of these keys to unseal it | ||
before it can start servicing requests. | ||
|
||
Vault does not store the generated master key. Without at least 3 key to | ||
reconstruct the master key, Vault will remain permanently sealed! | ||
|
||
It is possible to generate new unseal keys, provided you have a quorum of | ||
existing unseal keys shares. See "vault operator rekey" for more information. | ||
bash-4.4 | ||
|
||
Use 3 out of five keys presented and put it into .env file and than start the composition again Vault should be in unsealed state ( take a note that this is not recommended in terms of security, this is deployment for development) A real production deployment can use Vault auto unseal mode where vault gets unseal keys from some 3rd party KMS ( on AWS for example) | ||
./<script-name>.sh --env-file <path-to-your-env-file> | ||
``` | ||
|
||
### 2. `vault_copy_env.sh` | ||
|
||
After first step, the corresponding Vault environment variables (`MG_VAULT_TOKEN`, `MG_VAULT_UNSEAL_KEY_1`, `MG_VAULT_UNSEAL_KEY_2`, `MG_VAULT_UNSEAL_KEY_3`) should be updated in `.env` file. | ||
|
||
`vault_copy_env.sh` scripts copies values from `docker/addons/vault/data/secrets` file and update environmental variables `MG_VAULT_TOKEN`, `MG_VAULT_UNSEAL_KEY_1`, `MG_VAULT_UNSEAL_KEY_2`, `MG_VAULT_UNSEAL_KEY_3` present in `.env` file. | ||
|
||
### 3. `vault_unseal.sh` | ||
|
||
This can be run after the initialization to unseal Vault, which is necessary for it to be used to store and/or get secrets. | ||
|
||
This can be used if you don't want to restart the service. | ||
|
||
The unseal environment variables need to be set in `.env` for the script to work (`MG_VAULT_TOKEN`,`MG_VAULT_UNSEAL_KEY_1`, `MG_VAULT_UNSEAL_KEY_2`, `MG_VAULT_UNSEAL_KEY_3`). | ||
### Example | ||
|
||
This script should not be necessary to run after the initial setup, since the Vault service unseals itself when starting the container. | ||
|
||
### 4. `vault_set_pki.sh` | ||
|
||
This script is used to generate the root certificate, intermediate certificate and HTTPS server certificate. | ||
All generate certificates, keys and CSR by `vault_set_pki.sh` will be present at `docker/addons/vault/data`. | ||
|
||
The parameters required for generating certificate are obtained from the environment variables which are loaded from `docker/.env`. | ||
|
||
Environmental variables starting with `MG_VAULT_PKI` in `docker/.env` file are used by `vault_set_pki.sh` to generate root CA. | ||
Environmental variables starting with`MG_VAULT_PKI_INT` in `docker/.env` file are used by `vault_set_pki.sh` to generate intermediate CA. | ||
|
||
Passing command line args `--skip-server-cert` to `vault_set_pki.sh` will skip server certificate role & process of generation of server certificate & key. | ||
|
||
### 5. `vault_create_approle.sh` | ||
|
||
This script is used to enable app role authorization in Vault. Certs service used the approle credentials to issue, revoke things certificate from vault intermedate CA. | ||
|
||
`vault_create_approle.sh` script by default tries to enable auth approle. | ||
If approle is already enabled in vault, then use args `--skip-enable-approle` to skip enable auth approle step. | ||
To skip enable auth approle step use the following `vault_create_approle.sh --skip-enable-approle` | ||
|
||
### 6. `vault_copy_certs.sh` | ||
|
||
This scripts copies the necessary certificates and keys from `docker/addons/vault/data` to the `docker/ssl/certs` folder. | ||
|
||
## Hashicorp Cloud Platform (HCP) Vault | ||
|
||
To have the same PKI setup can done in Hashicorp Cloud Platform (HCP) Vault follow the below steps: | ||
Requirement: [VAULT CLI](https://developer.hashicorp.com/vault/tutorials/getting-started/getting-started-install) | ||
|
||
- Replace the environmental variable `MG_VAULT_ADDR` in `docker/.env` with HCP Vault address. | ||
- Replace the environmental variable `MG_VAULT_TOKEN` in `docker/.env` with HCP Vault Admin token. | ||
- Run script `vault_set_pki.sh` and `vault_create_approle.sh`. | ||
- Optional step, run script `vault_copy_certs.sh` to copy certificates to magistrala default path. | ||
|
||
## Vault CLI | ||
|
||
It can also be useful to run the Vault CLI for inspection and administration work. | ||
To initialize Vault using the provided setup script, run: | ||
|
||
```bash | ||
Usage: vault <command> [args] | ||
|
||
Common commands: | ||
read Read data and retrieves secrets | ||
write Write data, configuration, and secrets | ||
delete Delete secrets and configuration | ||
list List data or secrets | ||
login Authenticate locally | ||
agent Start a Vault agent | ||
server Start a Vault server | ||
status Print seal and HA status | ||
unwrap Unwrap a wrapped secret | ||
|
||
Other commands: | ||
audit Interact with audit devices | ||
auth Interact with auth methods | ||
debug Runs the debug command | ||
kv Interact with Vault's Key-Value storage | ||
lease Interact with leases | ||
monitor Stream log messages from a Vault server | ||
namespace Interact with namespaces | ||
operator Perform operator-specific tasks | ||
path-help Retrieve API help for paths | ||
plugin Interact with Vault plugins and catalog | ||
policy Interact with policies | ||
print Prints runtime configurations | ||
secrets Interact with secrets engines | ||
ssh Initiate an SSH session | ||
token Interact with tokens | ||
scripts/vault/scripts/vault_init.sh --env-file scripts/vault/.env | ||
``` | ||
|
||
If the Vault is setup through `docker/addons/vault`, then Vault CLI can be run directly using the Vault image in Docker: `docker run -it magistrala/vault:latest vault` | ||
|
||
## Vault Web UI | ||
|
||
If the Vault is setup through `docker/addons/vault`, Then Vault Web UI is accessible by default on `http://localhost:8200/ui`. | ||
For detailed documentation on the available scripts and their usage, visit the [Vault Addon Documentation](https://github.com/absmach/magistrala/tree/main/docker/addons/vault#readme). | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please add following guide# Developer Guide: Updating Vault Scripts from Magistrala
This guide provides step-by-step instructions to update the Vault scripts in your local Magistrala DevOps repository, sync them with the `main` branch of the Magistrala repository, and create a pull request (PR) to merge the changes.
## Prerequisites
Make sure you have the following:
- A local clone of the Magistrala DevOps repository.
- Access to the Magistrala GitHub repository (`https://github.com/absmach/magistrala.git`).
## Step 1: Create new branch
Create a new branch from `master` brach
```bash
git checkout -b <your-branch-name>
``
Replace `<your-branch-name>` with the name a new branch name.
## Step 2: Add the Magistrala Remote
If the Magistrala remote is not already added to your local repository, use the following command to add it:
```bash
git remote add -f magistrala https://github.com/absmach/magistrala.git
``
## Step 3: Add Subtree for Vault Scripts
If the subtree for the Vault scripts has not been added, execute the following command:
```bash
git subtree add --prefix=scripts/vault/scripts magistrala main --squash --prefix=docker/addons/vault/scripts
``
- `--prefix=scripts/vault/scripts`: Specifies the target directory in your local repository where the Vault scripts will be added.
- `magistrala main`: Refers to the `main` branch of the Magistrala repository.
- `--squash`: Combines all commits from the Magistrala `main` branch into a single commit when adding the subtree.
## Step 4: Update Vault Scripts to the Latest Version
To update the Vault scripts to the latest version and synchronize with the `main` branch of the Magistrala repository, use the following command:
```bash
git subtree pull --prefix=scripts/vault/scripts magistrala main --squash --prefix=docker/addons/vault/scripts
``
- This command pulls the latest changes from the `main` branch of the Magistrala repository.
- `--squash` creates a single commit for the changes, making the history simpler to manage.
## Step 5: Push Changes and Create a Pull Request
After syncing with the Magistrala `main` branch, push the changes to your working branch:
```bash
git push origin <your-branch-name>
``
Replace `<your-branch-name>` with the name of the branch you're working on.
Once the changes are pushed, go to your GitHub repository and create a pull request (PR) to merge the updates.
## Summary of Commands related to Git subtree
### Add Magistrala Remote
```bash
git remote add -f magistrala https://github.com/absmach/magistrala.git
``
### Add Subtree for Vault Scripts
```bash
git subtree add --prefix=scripts/vault/scripts magistrala main --squash --prefix=docker/addons/vault/scripts
``
### Update Vault Scripts to the Latest
```bash
git subtree pull --prefix=scripts/vault/scripts magistrala main --squash --prefix=docker/addons/vault/scripts
`` |
This file was deleted.
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -3,4 +3,3 @@ | |
|
||
data | ||
magistrala_things_certs_issue.hcl | ||
.env |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
#!/usr/bin/bash | ||
# Copyright (c) Abstract Machines | ||
# SPDX-License-Identifier: Apache-2.0 | ||
|
||
vault() { | ||
if is_container_running "magistrala-vault"; then | ||
docker exec -it magistrala-vault vault "$@" | ||
else | ||
if which vault &> /dev/null; then | ||
$(which vault) "$@" | ||
else | ||
echo "magistrala-vault container or vault command not found. Please refer to the documentation: https://github.com/absmach/magistrala/blob/main/docker/addons/vault/README.md" | ||
fi | ||
fi | ||
} | ||
|
||
is_container_running() { | ||
local container_name="$1" | ||
if [ "$(docker inspect --format '{{.State.Running}}' "$container_name" 2>/dev/null)" = "true" ]; then | ||
return 0 | ||
else | ||
return 1 | ||
fi | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,86 @@ | ||
#!/usr/bin/bash | ||
# Copyright (c) Abstract Machines | ||
# SPDX-License-Identifier: Apache-2.0 | ||
|
||
set -euo pipefail | ||
|
||
scriptdir="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)" | ||
|
||
# default env file path | ||
env_file="docker/.env" | ||
|
||
# default certs copy path | ||
certs_copy_path="docker/ssl/certs/" | ||
|
||
while [[ "$#" -gt 0 ]]; do | ||
case $1 in | ||
--env-file) | ||
if [[ -z "${2:-}" ]]; then | ||
echo "Error: --env-file requires a non-empty option argument." | ||
exit 1 | ||
fi | ||
env_file="$2" | ||
if [[ ! -f "$env_file" ]]; then | ||
echo "Error: .env file not found at $env_file" | ||
exit 1 | ||
fi | ||
shift | ||
;; | ||
--certs-copy-path) | ||
if [[ -z "${2:-}" ]]; then | ||
echo "Error: --certs-copy-path requires a non-empty option argument." | ||
exit 1 | ||
fi | ||
certs_copy_path="$2" | ||
shift | ||
;; | ||
*) | ||
echo "Error: Unknown parameter passed: $1" | ||
exit 1 | ||
;; | ||
esac | ||
shift | ||
done | ||
|
||
readDotEnv() { | ||
set -o allexport | ||
source "$env_file" | ||
set +o allexport | ||
} | ||
|
||
readDotEnv | ||
|
||
server_name="localhost" | ||
|
||
# Check if MG_NGINX_SERVER_NAME is set or not empty | ||
if [ -n "${MG_NGINX_SERVER_NAME:-}" ]; then | ||
server_name="$MG_NGINX_SERVER_NAME" | ||
fi | ||
|
||
echo "Copying certificate files to ${certs_copy_path}" | ||
|
||
if [ -e "$scriptdir/data/${server_name}.crt" ]; then | ||
cp -v "$scriptdir/data/${server_name}.crt" "${certs_copy_path}magistrala-server.crt" | ||
else | ||
echo "${server_name}.crt file not available" | ||
fi | ||
|
||
if [ -e "$scriptdir/data/${server_name}.key" ]; then | ||
cp -v "$scriptdir/data/${server_name}.key" "${certs_copy_path}magistrala-server.key" | ||
else | ||
echo "${server_name}.key file not available" | ||
fi | ||
|
||
if [ -e "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}.key" ]; then | ||
cp -v "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}.key" "${certs_copy_path}ca.key" | ||
else | ||
echo "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}.key file not available" | ||
fi | ||
|
||
if [ -e "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}_bundle.crt" ]; then | ||
cp -v "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}_bundle.crt" "${certs_copy_path}ca.crt" | ||
else | ||
echo "$scriptdir/data/${MG_VAULT_PKI_INT_FILE_NAME}_bundle.crt file not available" | ||
fi | ||
|
||
exit 0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@JeffMboya This part is moved
scripts/vault/Readme.md
, so i think it is no more needed hereWhat do you think ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it should be removed.