Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MG-153 - Add Vault scripts as git subtree #157

Open
wants to merge 46 commits into
base: master
Choose a base branch
from

Conversation

JeffMboya
Copy link
Contributor

@JeffMboya JeffMboya commented Nov 18, 2024

What does this do?

This PR adds vault scripts as a git subtree in the devops repo

Which issue(s) does this PR fix/relate to?

List any changes that modify/break current functionality

None

Have you included tests for your changes?

No, manually tested

Did you document any new/modified functionality?

Yes

Notes

git-subtree-dir: docker/addons/vault
git-subtree-split: 54a134e72f47e51acfbced9a94c1b22e13c3e461
git-subtree-dir: vault
git-subtree-split: a32634a1e90508f08a75081b0e595a427d3cbb00
git-subtree-dir: vault
git-subtree-split: a32634a1e90508f08a75081b0e595a427d3cbb00
git-subtree-dir: scripts/vault
git-subtree-split: a32634a1e90508f08a75081b0e595a427d3cbb00
Signed-off-by: JeffMboya <[email protected]>
git-subtree-dir: scripts/vault
git-subtree-split: a32634a1e90508f08a75081b0e595a427d3cbb00
Signed-off-by: JeffMboya <[email protected]>
@JeffMboya
Copy link
Contributor Author

@arvindh123 There’s a small issue that needs your input:

  1. Adding the Vault scripts as a subdirectory in the DevOps repository under devops/scripts creates the following structure: devops/scripts/vault/scripts. This happens because, in the Magistrala (MG) repository, the Vault scripts are located under vault/scripts.

  2. In MG, the .env file is located at docker/.env. When the Vault scripts are added as a subtree in the DevOps repository, they are imported as-is from MG. As a result, the scripts expect the .env file to be located at docker/.env, which is not imported. This leads to the following error:

    scripts/vault/scripts/vault_init.sh: line 36: docker/.env: No such file or directory
    

How should we proceed?

@arvindh123
Copy link
Contributor

@arvindh123 There’s a small issue that needs your input:

  1. Adding the Vault scripts as a subdirectory in the DevOps repository under devops/scripts creates the following structure: devops/scripts/vault/scripts. This happens because, in the Magistrala (MG) repository, the Vault scripts are located under vault/scripts.
  2. In MG, the .env file is located at docker/.env. When the Vault scripts are added as a subtree in the DevOps repository, they are imported as-is from MG. As a result, the scripts expect the .env file to be located at docker/.env, which is not imported. This leads to the following error:
    scripts/vault/scripts/vault_init.sh: line 36: docker/.env: No such file or directory
    

How should we proceed?

I hope we had provision to pass env file via flage --env-file
https://github.com/absmach/magistrala/pull/2387
So please pass customized env file to vault scripts

@JeffMboya
Copy link
Contributor Author

@arvindh123 There’s a small issue that needs your input:

  1. Adding the Vault scripts as a subdirectory in the DevOps repository under devops/scripts creates the following structure: devops/scripts/vault/scripts. This happens because, in the Magistrala (MG) repository, the Vault scripts are located under vault/scripts.
  2. In MG, the .env file is located at docker/.env. When the Vault scripts are added as a subtree in the DevOps repository, they are imported as-is from MG. As a result, the scripts expect the .env file to be located at docker/.env, which is not imported. This leads to the following error:
    scripts/vault/scripts/vault_init.sh: line 36: docker/.env: No such file or directory
    

How should we proceed?

I hope we had provision to pass env file via flage --env-file absmach/magistrala#2387 So please pass customized env file to vault scripts

Yes, we have the provision

Signed-off-by: JeffMboya <[email protected]>
Signed-off-by: JeffMboya <[email protected]>
@JeffMboya JeffMboya marked this pull request as ready for review November 19, 2024 07:33
Copy link
Contributor

@arvindh123 arvindh123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.env can be kept at vault/.env,
The script folder should be exact duplicate of magistrala

README.md Outdated

Since we have added the `magistrala` Vault directory as a subtree in the `devops` repository, we only include the `docker/addons/vault` directory and its contents in `scripts/vault` directory. We do not include `docker/.env` or other unrelated directories from the `magistrala` repository.

As a result, running the Vault setup scripts from within the `scripts/vault/scripts` directory may throw the following error:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is better to mention only how to run scripts with --env-files

Signed-off-by: JeffMboya <[email protected]>
Copy link
Contributor

@arvindh123 arvindh123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need docker-compose and vault config files in scripts/vault ?

@@ -40,4 +40,4 @@ MG_VAULT_PKI_INT_CA_ST='PARIS'
MG_VAULT_PKI_INT_CA_ADDR='5 Av. Anatole'
MG_VAULT_PKI_INT_CA_PO='75007'
MG_VAULT_PKI_INT_CLUSTER_PATH=http://localhost
MG_VAULT_PKI_INT_CLUSTER_AIA_PATH=http://localhost
MG_VAULT_PKI_INT_CLUSTER_AIA_PATH=http://localhost
Copy link
Contributor

@arvindh123 arvindh123 Nov 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add new line at end of file

@@ -45,44 +45,24 @@ When the Vault service is started, some initialization steps need to be done to

## Setup

The following scripts are provided, which work on the running Vault service in Docker.
The following scripts are provided, which work on the running Vault service from within the `docker/addons/vault/scripts` directory.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doc is relative to magistrala, may be simply we should point MG vault docs for usage

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doc is relative to magistrala, may be simply we should point MG vault docs for usage

Yes, the README is relative to Magistrala because adding docker/addons/vault as a subtree in the DevOps repository imports content relative to Magistrala. The issue is that when the subtree is updated, it will continue importing files tied to Magistrala. A possible solution is to import docker/addons/vault locally, remove unnecessary files, and then push the filtered contents to GitHub. This way, the DevOps repository only includes the required files.

Signed-off-by: JeffMboya <[email protected]>
@JeffMboya
Copy link
Contributor Author

Do we need docker-compose and vault config files in scripts/vault ?

We do not need it. It's there because we have added all the contents of docker/addons/vault to also get docker/addons/vault/entrypoint.sh

@arvindh123
Copy link
Contributor

Do we need docker-compose and vault config files in scripts/vault ?

We do not need it. It's there because we have added all the contents of docker/addons/vault to also get docker/addons/vault/entrypoint.sh

Then is it possible to remove them and have only vault/scripts ?

Signed-off-by: JeffMboya <[email protected]>
git-subtree-dir: scripts/vault
git-subtree-split: a32634a1e90508f08a75081b0e595a427d3cbb00
git-subtree-dir: vault
git-subtree-split: a32634a1e90508f08a75081b0e595a427d3cbb00
Signed-off-by: JeffMboya <[email protected]>
git-subtree-dir: vault
git-subtree-split: a32634a1e90508f08a75081b0e595a427d3cbb00
Signed-off-by: JeffMboya <[email protected]>
git-subtree-dir: scripts/vault
git-subtree-split: a32634a1e90508f08a75081b0e595a427d3cbb00
Signed-off-by: JeffMboya <[email protected]>
Signed-off-by: JeffMboya <[email protected]>
git-subtree-dir: docker/addons/vault/scripts
git-subtree-split: d9f3bca47842e392383039b84e6b6e223de79e10
Signed-off-by: JeffMboya <[email protected]>
Copy link
Contributor

@arvindh123 arvindh123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if these files are no needed scripts/vault/config.hcl, scripts/vault/entrypoint.sh , please remove them.

README.md Outdated
To run a Vault setup script, use the `--env-file` option to specify the path to your `.env` file:

```bash
./<script-name>.sh --env-file <path-to-your-env-file>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this part to scripts/vault/Readme.md and add words like refer magistrala vault document (and its link)

Signed-off-by: JeffMboya <[email protected]>
README.md Outdated
@@ -57,6 +57,20 @@ git push origin <your-branch>

Replace `<your-branch>` with the branch you are working on.

### Running Vault Setup Scripts with `--env-file`

To run a Vault setup script, use the `--env-file` option to specify the path to your `.env` file:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JeffMboya This part is moved scripts/vault/Readme.md , so i think it is no more needed here
What do you think ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JeffMboya This part is moved scripts/vault/Readme.md , so i think it is no more needed here What do you think ?

Yes, it should be removed.

## Vault Web UI

If the Vault is setup through `docker/addons/vault`, Then Vault Web UI is accessible by default on `http://localhost:8200/ui`.
For detailed documentation on the available scripts and their usage, visit the [Vault Addon Documentation](https://github.com/absmach/magistrala/tree/main/docker/addons/vault#readme).
Copy link
Contributor

@arvindh123 arvindh123 Nov 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add following guide

# Developer Guide: Updating Vault Scripts from Magistrala

This guide provides step-by-step instructions to update the Vault scripts in your local Magistrala DevOps repository, sync them with the `main` branch of the Magistrala repository, and create a pull request (PR) to merge the changes.

## Prerequisites

Make sure you have the following:
- A local clone of the Magistrala DevOps repository.
- Access to the Magistrala GitHub repository (`https://github.com/absmach/magistrala.git`).

## Step 1: Create new branch 

Create a new branch from `master` brach

```bash
git checkout -b <your-branch-name>
``

Replace `<your-branch-name>` with the name a new branch name.

## Step 2: Add the Magistrala Remote

If the Magistrala remote is not already added to your local repository, use the following command to add it:

```bash
git remote add -f magistrala https://github.com/absmach/magistrala.git
``

## Step 3: Add Subtree for Vault Scripts

If the subtree for the Vault scripts has not been added, execute the following command:

```bash
git subtree add --prefix=scripts/vault/scripts magistrala main --squash --prefix=docker/addons/vault/scripts
``

- `--prefix=scripts/vault/scripts`: Specifies the target directory in your local repository where the Vault scripts will be added.
- `magistrala main`: Refers to the `main` branch of the Magistrala repository.
- `--squash`: Combines all commits from the Magistrala `main` branch into a single commit when adding the subtree.

## Step 4: Update Vault Scripts to the Latest Version

To update the Vault scripts to the latest version and synchronize with the `main` branch of the Magistrala repository, use the following command:

```bash
git subtree pull --prefix=scripts/vault/scripts magistrala main --squash --prefix=docker/addons/vault/scripts
``

- This command pulls the latest changes from the `main` branch of the Magistrala repository.
- `--squash` creates a single commit for the changes, making the history simpler to manage.

## Step 5: Push Changes and Create a Pull Request

After syncing with the Magistrala `main` branch, push the changes to your working branch:

```bash
git push origin <your-branch-name>
``

Replace `<your-branch-name>` with the name of the branch you're working on.

Once the changes are pushed, go to your GitHub repository and create a pull request (PR) to merge the updates.

## Summary of Commands related to Git subtree

### Add Magistrala Remote
```bash
git remote add -f magistrala https://github.com/absmach/magistrala.git
``

### Add Subtree for Vault Scripts
```bash
git subtree add --prefix=scripts/vault/scripts magistrala main --squash --prefix=docker/addons/vault/scripts
``

### Update Vault Scripts to the Latest
```bash
git subtree pull --prefix=scripts/vault/scripts magistrala main --squash --prefix=docker/addons/vault/scripts
``

Signed-off-by: JeffMboya <[email protected]>
Signed-off-by: JeffMboya <[email protected]>
Signed-off-by: JeffMboya <[email protected]>
Signed-off-by: JeffMboya <[email protected]>
Signed-off-by: JeffMboya <[email protected]>
Copy link
Contributor

@arvindh123 arvindh123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add following guide 


# Developer Guide: Updating Vault Scripts from Magistrala

This guide provides step-by-step instructions to update the Vault scripts in your local Magistrala DevOps repository, sync them with the `main` branch of the Magistrala repository, and create a pull request (PR) to merge the changes.

## Prerequisites

Make sure you have the following:
- A local clone of the Magistrala DevOps repository.
- Access to the Magistrala GitHub repository (`https://github.com/absmach/magistrala.git`).

## Step 1: Create new branch 

Create a new branch from `master` brach

```bash
git checkout -b <your-branch-name>

Replace <your-branch-name> with the name a new branch name.

Step 2: Add the Magistrala Remote

If the Magistrala remote is not already added to your local repository, use the following command to add it:

git remote add -f magistrala https://github.com/absmach/magistrala.git

Step 3: Add Subtree for Vault Scripts

If the subtree for the Vault scripts has not been added, execute the following command:

git subtree add --prefix=scripts/vault/scripts magistrala main --squash --prefix=docker/addons/vault/scripts
  • --prefix=scripts/vault/scripts: Specifies the target directory in your local repository where the Vault scripts will be added.
  • magistrala main: Refers to the main branch of the Magistrala repository.
  • --squash: Combines all commits from the Magistrala main branch into a single commit when adding the subtree.

Step 4: Update Vault Scripts to the Latest Version

To update the Vault scripts to the latest version and synchronize with the main branch of the Magistrala repository, use the following command:

git subtree pull --prefix=scripts/vault/scripts magistrala main --squash --prefix=docker/addons/vault/scripts
  • This command pulls the latest changes from the main branch of the Magistrala repository.
  • --squash creates a single commit for the changes, making the history simpler to manage.

Step 5: Push Changes and Create a Pull Request

After syncing with the Magistrala main branch, push the changes to your working branch:

git push origin <your-branch-name>

Replace <your-branch-name> with the name of the branch you're working on.

Once the changes are pushed, go to your GitHub repository and create a pull request (PR) to merge the updates.

Summary of Commands related to Git subtree

Add Magistrala Remote

git remote add -f magistrala https://github.com/absmach/magistrala.git

Add Subtree for Vault Scripts

git subtree add --prefix=scripts/vault/scripts magistrala main --squash --prefix=docker/addons/vault/scripts

Update Vault Scripts to the Latest

git subtree pull --prefix=scripts/vault/scripts magistrala main --squash --prefix=docker/addons/vault/scripts

Comment on lines +33 to 35
git checkout -b <your-branch-name>
``

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
git checkout -b <your-branch-name>
``
```bash
git checkout -b <your-branch-name>

### 4. `vault_set_pki.sh`
```bash
git remote add -f magistrala https://github.com/absmach/magistrala.git
``

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

Environmental variables starting with`MG_VAULT_PKI_INT` in `docker/.env` file are used by `vault_set_pki.sh` to generate intermediate CA.
```bash
git subtree add --prefix=scripts/vault/scripts magistrala main --squash --prefix=docker/addons/vault/scripts
``

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

To skip enable auth approle step use the following `vault_create_approle.sh --skip-enable-approle`
```bash
git subtree pull --prefix=scripts/vault/scripts magistrala main --squash --prefix=docker/addons/vault/scripts
``

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

Requirement: [VAULT CLI](https://developer.hashicorp.com/vault/tutorials/getting-started/getting-started-install)
```bash
git push origin <your-branch-name>
``

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

token Interact with tokens
```
git remote add -f magistrala https://github.com/absmach/magistrala.git
``

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

### Add Subtree for Vault Scripts
```bash
git subtree add --prefix=scripts/vault/scripts magistrala main --squash --prefix=docker/addons/vault/scripts
``

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature: Add Vault scripts as git subtree
2 participants