Update Project to Use Poetry #68

jdrew82 · 2024-01-19T03:48:51Z

This PR is to refactor the project to use Poetry for dependency management instead of requirements.txt and the older build system. I've made this update to address the following:

Now that we have multiple major versions of Nautobot and Apps we're seeing users experiencing difficulty in managing their dependencies and having unexpected upgrades and changes occurring. Using poetry will enable users to choose their expected Nautobot version and have the most compatible App versions recommended during the dependency resolution process.
The current structure requires utilizing multiple different files for creating the environment and also customizing it. If we change to Poetry this is simplified a lot to executing some shell commands in most cases.
We utilize Poetry and invoke in our Apps and are suggesting it's use in the cookiecutter so should be consistent.
This aligns closer to how we're deploying Nautobot with K8s.

…ment. To replace use of requirements.txt we'll use pyproject.toml with Poetry.

…zation.

…nvironments folder.

… MySQL.

…nto environment.

jvanderaa · 2024-01-20T00:43:46Z

What is the part about adding invoke onto this? Are we still able to do docker compose up to get going?

jdrew82 · 2024-01-20T01:28:54Z

What is the part about adding invoke onto this? Are we still able to do docker compose up to get going?

I still need to finish updating documentation to include that bit. I think you should still be able to do 'docker compose up' in the environments folder if you specify the project or which files. Need to test to confirm though.

jdrew82 · 2024-01-29T23:27:42Z

What is the part about adding invoke onto this? Are we still able to do docker compose up to get going?

I've confirmed that you are able to use docker compose up commands if you want, but you need to add all the additional toggles such as specifying the project name and folder. The invoke command makes this much simpler so I think it makes more sense to just tell users to use the invoke commands.

Cleaned up optional settings and RQ related settings that are no longer valid.

… folder copying.

…age.

sslhansen · 2024-02-15T10:57:27Z

Just a small comment to the readme. The creds.example.env file is referred to as creds.example.example instead of creds.example.env on line 128 in the code block :)

README.md

glennmatthews · 2024-02-29T15:17:35Z

README.md


-```
-docker compose -f docker-compose.yml -f docker-compose.mysql.yml up
+This repo is designed to provide a custom build of Nautobot to include a set of plugins which can then be used in a development environment or deployed in production.  Included in this repo is a skeleton Nautobot plugin which is designed only to provide a quick example of how a plugin could be developed.  Plugins should ultimately be built as packages, published to a PyPI style repository and added to the poetry `pyproject.toml` in this repo.  The plugin code should be hosted in their own repositories with their own CI pipelines and not included here.


This seems wrong? We're not going to add any plugins to this repo's pyproject.toml? If the idea is for users to maintain a custom fork with their own set of plugins, that should be made clearer.

(Also, nit - s/plugin/app/g :-) )

The example plugin was included as there is one in the current version. I think it might be worthwhile to just get rid of it and refer to the cookiecutter-nautobot-app cookies instead. Thoughts? @jvanderaa @glennmatthews

README.md

environments/docker-compose.local.yml

glennmatthews · 2024-02-29T15:48:13Z

environments/docker-compose.local.yml

+    healthcheck:
+      test: ["CMD", "true"]  # Due to layering, disable: true won't work. Instead, change the test


I think we should probably come up with a good healthcheck like for the nautobot service.

glennmatthews · 2024-02-29T15:49:42Z

environments/local.example.env

+MYSQL_USER=nautobot
+MYSQL_DATABASE=nautobot


I don't understand the logical split between creds.example.env and local.example.env - for example, why is MYSQL_USER in local but MYSQL_PASSWORD in creds?

I considered local.env to be environment variables that are safe to be stored in git vs secrets that shouldn't be saved should go into creds.env.

So a user would fork the repo and commit local.env to their fork? Maybe document that if so.

glennmatthews · 2024-02-29T15:50:11Z

invoke.ldap.yml

+  compose_dir: "environments/"
+  compose_files:
+    - "docker-compose.postgres.yml"
+    - "docker-compose.ldap.yml"


Why does the ldap one not include docker-compose.base.yml?

Because the LDAP compose file is identical to base but uses the LDAP Dockerfile. I guess we could just use overrides in this case but was trying to make it easier to understand.

…nd caching_sha2_password

jdrew82 added 11 commits January 18, 2024 18:32

feat: ✨ Add pyproject.toml and initial lockfile for dependency manage…

9c0461b

…ment. To replace use of requirements.txt we'll use pyproject.toml with Poetry.

feat: ✨ Add initial tasks.py for project tasks.

3523e67

refactor: 🚚 Move example plugin into plugins folder for better organi…

32c1e0d

…zation.

refactor: 🚚 Move all files related to building the environment into e…

f6d4ccb

…nvironments folder.

feat: ✨ Add invoke example files for local with PostgreSQL, LDAP, and…

b56cdf7

… MySQL.

build: 🏗️ Add jobs folder to enable quicker addition of Job scripts i…

c28e0e9

…nto environment.

feat: ✨ Add nautobot_config without LDAP config to config folder.

03987d4

feat: 🙈 Add dockerignore file

67c675e

feat: 🙈 Update gitignore file for more unnecessary files.

f5ede6e

docs: 📝 Update SSL docs to account for project structure changes.

d08461d

docs: 📝 Update README to account for project changes.

db1f5d3

jdrew82 self-assigned this Jan 19, 2024

jdrew82 requested a review from jvanderaa as a code owner January 19, 2024 03:48

jdrew82 marked this pull request as draft January 19, 2024 03:49

test: 🚨 Fix yamllint complaints

1cc6607

jdrew82 mentioned this pull request Jan 23, 2024

Adds celery beat to LDAP and plugin. #29

Closed

jdrew82 added 9 commits January 29, 2024 17:30

docs: 📝 Update README to add information about Poetry and Invoke

01a3d39

docs: 📝 Update plugins docs to use Poetry methodology.

e669516

fix: 🐛 Remove healthcheck from example plugin as it's not needed.

f18e657

fix: 🐛 Correct volume mount for nautobot_config

6b8e175

refactor: ♻️ Update LDAP nautobot_config to match non-LDAP version.

94beba5

Cleaned up optional settings and RQ related settings that are no longer valid.

fix: 🐛 Add PYTHON_VER argument to Dockerfile for python site-packages…

1331d5d

… folder copying.

fix: 🐛 Add plugins source folder so plugins are installed in final im…

37f8d91

…age.

fix: 🐛 Add LDAP environment variables to local.example.env

0c072d8

docs: 📝 Update LDAP documentation to use invoke.

c1b268c

jdrew82 marked this pull request as ready for review January 30, 2024 20:51

docs: 📝 Replace text with bash for CLI snippets

b426e56

jdrew82 mentioned this pull request Feb 23, 2024

Changes with 2.X #64

Closed

jvanderaa mentioned this pull request Feb 29, 2024

Updates to indicate support for 2.x #69

Closed

docs: 📝 Correct env file name

010af00

glennmatthews reviewed Feb 29, 2024

View reviewed changes

jdrew82 added 20 commits March 4, 2024 16:25

docs: 📝 Update docs to address comments in PR.

8d44bb2

fix: ♻️ Update base and ldap.yml docker-compose to be similar

6bbfea0

refactor: ♻️ Remove duplicate config that's in core settings.

a47bf8b

refactor: 🔥 Remove mentions of RQ

794a65a

docs: ♻️ Redo documentation on App installs

d1d77db

fix: ♻️ Remove duplicate code from compose file.

d0d6773

docs: 📝 Make section header clearer about what's being done.

2645543

refactor: Remove healthchecks from local compose

fcc857d

feat: Add healthcheck to Postgres service based on cookie

b96fc40

refactor: 🔥 Remove lockfile as expected to be generated at project init

6af8ebd

docs: 📝 Clarify use of local compos file

f84a904

docs: ♻️ Move MySQL note to under Getting Started

2728639

refactor: 🔥 Remove CACHES config from nautobot_config.py

773145a

docs: 📝 Add section on environment files and what the purpose is of each

18c2589

refactor: ♻️ Change sections to be sub-headers under Getting Started

a8598b5

docs: 📝 Tweaking docs for syntax and spelling

7be653d

fix: 🐛 Update authentication for MySQL to use authentication_policy a…

e2a7b7d

…nd caching_sha2_password

fix: 🐛 Add healthcheck to Nautobot service

7f1980a

docs: 📝 Correct MariaDB to MySQL in env file

c987cef

docs: 📝 Add comment about API token character length

3903112

jvanderaa approved these changes Apr 11, 2024

View reviewed changes

jvanderaa merged commit 40eec08 into main Apr 18, 2024
2 checks passed

jdrew82 deleted the feat-update_to_poetry branch April 18, 2024 15:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Project to Use Poetry #68

Update Project to Use Poetry #68

jdrew82 commented Jan 19, 2024

jvanderaa commented Jan 20, 2024

jdrew82 commented Jan 20, 2024

jdrew82 commented Jan 29, 2024

sslhansen commented Feb 15, 2024

glennmatthews Feb 29, 2024

jdrew82 Feb 29, 2024

glennmatthews Feb 29, 2024

jdrew82 Mar 4, 2024

glennmatthews Feb 29, 2024

jdrew82 Feb 29, 2024

glennmatthews Mar 4, 2024

glennmatthews Feb 29, 2024

jdrew82 Mar 4, 2024

		healthcheck:
		test: ["CMD", "true"] # Due to layering, disable: true won't work. Instead, change the test

		MYSQL_USER=nautobot
		MYSQL_DATABASE=nautobot

Update Project to Use Poetry #68

Update Project to Use Poetry #68

Conversation

jdrew82 commented Jan 19, 2024

jvanderaa commented Jan 20, 2024

jdrew82 commented Jan 20, 2024

jdrew82 commented Jan 29, 2024

sslhansen commented Feb 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment