Plugin development

[rizlas]

Hi,
I was reworking my netbox docker setup but I found some difficulties to understand the wiki section about plugins. In particular it only mention how to import a plugin that is already on PyPi (https://github.com/netbox-community/netbox-docker/wiki/Using-Netbox-Plugins), but what if I'm developing my own without publishing it? Currently I have a custom docker file that place plugin projects inside /opt/netbox/netbox. This works, even without running setup.py install command.

Another confusing thing is the presence of plugins settings. In wiki it is stated to use configuration.py and I'm using it, but here is also a plugins.py file. Which is the main difference here?

During plugin developing things becomes a bit messy, I looked around in other repos and people are countering in different ways (mostly building their own docker image). My current setup is a docker-compose override that mount my plugin project folder as a bind volume inside /opt/netbox/netbox and overrides user and command statement, but the main image is always the official netbox-docker.

Compose updated: 14/01/2024
This setup is also valid for reports and scripts.

Dockerfile:

FROM netboxcommunity/netbox:v3.6.8-2.7.0

COPY a_plugin_root_folder/a_plugin ./a_plugin

COPY ./externals_plugins.txt /tmp
RUN /opt/netbox/venv/bin/pip install --no-warn-script-location -r /tmp/externals_plugins.txt

# Custom install and plugin addition
# This SECRET_KEY is only used during the installation. There's no need to change it.
RUN SECRET_KEY="dummydummydummydummydummydummydummydummydummydummy" /opt/netbox/netbox/manage.py collectstatic --no-input

Directory tree:

netbox
├── Dockerfile
├── external_libraries.txt
├── external_plugins.txt
└── netbox-plugin-interface-sync
    ├── docs
    │   └── images
    │       ├── 1_device_page.png
    │       └── 2_interface_comparison.png
    ├── netbox_interface_sync
    │   ├── comparison.py
    │   ├── forms.py
    │   ├── __init__.py
    │   ├── template_content.py
    │   ├── templates
    │   │   └── netbox_interface_sync
    │   │       ├── compare_components_button.html
    │   │       ├── components_comparison.html
    │   │       └── number_of_interfaces_panel.html
    │   ├── urls.py
    │   ├── utils.py
    │   └── views.py
    ├── README.md
    ├── README_ru.md
    └── setup.py

Docker-compose:

x-netbox-common: &common
    image: custom_netbox_image
    restart: unless-stopped
    user: "0:0" # root user, without it setup.py develop will not work
    networks:
      - netbox
    tty: true
    env_file:
      - env/netbox.env
    volumes:
      - ./reports:/opt/netbox/netbox/reports:rw
      - ./scripts:/opt/netbox/netbox/scripts:rw
      - ./a_plugin_root_folder/a_plugin:/opt/netbox/netbox/a_plugin

version: "3.4"

services:
  netbox:
    <<: *common
    container_name: netbox
    command: >
      sh -c "/opt/netbox/venv/bin/python setup.py develop || true &&
      /opt/netbox/venv/bin/python manage.py makemigrations &&
      /opt/netbox/venv/bin/python manage.py migrate &&
      /opt/netbox/venv/bin/python manage.py createsuperuser --noinput --username local_admin --email [email protected] || true &&
      /opt/netbox/venv/bin/python manage.py runserver 0.0.0.0:8080"
    environment:
      - DJANGO_SUPERUSER_PASSWORD=supersecretpassword
    ports:
      - 8080:8080

  netbox-worker:
    <<: *common
    container_name: netbox-worker

  netbox-housekeeping:
    <<: *common
    container_name: netbox-housekeeping

Warning

Do not use this docker compose override in production

With this approach I achieved a fully working dev environment with auto reload and will also extract migration files that may be generated. Right now I was wondering: "What if I have to develop another plugin? Is this the right way?"

For clarification here is my directory tree inside /opt/netbox/netbox:
Atm I cannot put another setup.py from another custom plugin.

-rw-rw-r--    1 root     root            38 Dec  9 10:31 README.md (PLUGIN RELATED)
drwxr-xr-x    1 root     root          4096 Jan  7 05:57 circuits
drwxr-xr-x    1 root     root          4096 Jan  7 05:57 dcim
drwxr-xr-x    1 root     root          4096 Jan  7 05:57 extras
-rwxr-xr-x    1 root     root           277 Jan  7 05:51 generate_secret_key.py
drwxr-xr-x    1 root     root          4096 Jan  7 05:57 ipam
-rwxr-xr-x    1 root     root           249 Jan  7 05:51 manage.py
drwxrwxr-x    4 unit     root          4096 Apr 12 10:09 media
drwxr-xr-x    1 root     root          4096 Jan  7 05:57 netbox
drwxrwxr-x    7 root     root          4096 Apr 12 08:57 netbox_plugin_hardware_assignments  (PLUGIN RELATED)
drwxr-xr-x    1 root     root          4096 Jan  7 05:51 project-static
drwxr-xr-x    2 root     root          4096 Jan  7 05:51 reports
drwxr-xr-x    2 root     root          4096 Jan  7 05:51 scripts
-rw-rw-r--    1 root     root           429 Dec  9 10:31 setup.py  (PLUGIN RELATED)
drwxr-xr-x   11 root     root          4096 Jan  7 05:57 static
drwxr-xr-x   17 root     root          4096 Jan  7 05:51 templates
drwxr-xr-x    1 root     root          4096 Jan  7 05:57 tenancy
drwxr-xr-x    1 root     root          4096 Jan  7 05:57 users
drwxr-xr-x    1 root     root          4096 Jan  7 05:57 utilities
drwxr-xr-x    1 root     root          4096 Jan  7 05:57 virtualization
drwxr-xr-x    1 root     root          4096 Jan  7 05:57 wireless

[rizlas]

Bump

[ryanmerolle]

I am doing something similar for now, but with tight vscode integration using the devcontainer function. https://github.com/ryanmerolle/netbox-access-lists

[ddibenedetto]

Hi Ryan,
I'm a beginner, so I apologize in advance if the question may sounds stupid. I cloned your repo and vscode integration seems to work as expected: according to docker-compose.yml in devcontainer, 3 containers are running.
The problem is that seems there's no a running process listening at 0.0.0.0:8000, and if I try to start the web server inside the container I get a plugin related error.
Any hint?

python3 manage.py runserver 0.0.0.0:8000
🧬 loaded config '/etc/netbox/config/configuration.py'
🧬 loaded config '/etc/netbox/config/logging.py'
🧬 loaded config '/etc/netbox/config/plugins.py'
Traceback (most recent call last):
File "/opt/netbox/netbox/netbox/settings.py", line 655, in
plugin = importlib.import_module(plugin_name)
File "/usr/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'netbox_acls'

[mr1716]

Did you ever find a solution to this?

[Zorlin]

If it helps any, on the latest Netbox it's a different version of Python... I'm having a go now...

[ledgley]

Reloading the container each time we change a .py file takes a few minutes. Is there a faster/better approach than reloading the entire container?

[rizlas]

Have you tried my solution above?
With that you don't need to reload the entire container.

[Selfmade-RuLeZ]

Here's how I'm doing it:

• Create a plugins folder in the root of the netbox-docker folder.
• Clone all your development plugins in this directory.
• Create the Dockerfile-Plugins file as shown in 1
• Create the docker-compose.override.yml as shown in 2
• Configure netbox with your plugin name in configuration.py or plugins.py
• Build and start the containers using docker-compose build --no-cache and docker-compose up
• To migrate something, you have to connect to the container using docker exec -it <container id or name> /bin/bash and go into the virtual env as shown in the tutorial of the plugin development guide

I hope this helps you a little bit.

1:

FROM netboxcommunity/netbox:latest

# repeat the last line for every plugin you want to load editable and customize the path
COPY ./plugins /plugins
RUN /opt/netbox/venv/bin/pip install --editable /plugins/netbox-plugin-demo

2:

version: '3.4'
services:
  netbox:
    ports:
      - 8000:8080
    build:
      context: .
      dockerfile: Dockerfile-Plugins
    image: netbox:latest-plugins
    volumes: 
      - ./plugins:/plugins
  netbox-worker:
    image: netbox:latest-plugins
    build:
      context: .
      dockerfile: Dockerfile-Plugins
  netbox-housekeeping:
    image: netbox:latest-plugins
    build:
      context: .
      dockerfile: Dockerfile-Plugins

[Andreas259]

I tried your solution, but how can you make it auto-reload when you make a change in your template file?

[gfhoffmann]

I'm having the same issue as Andreas! Everytime that I change something in my custom plugin code I need to rebuild everything because the worker won't get code changes. When I used "normal" Netbox (without docker), I just needed to restart my worker and then it would get the code changes, but in docker this is not happening.

Anyone know what to do in this case?

[rizlas]

I just edited my open statement with my working setup.
It's only a part of it cause I'm using ansible+jinja, but contains everything you need to have hot reload (spoiler: you need to override command and use run server).

[MohammadMortezaie]

everything is great but I cannot find any configuration.py or plugins.py
can you help me where should I find it??

this step "Configure netbox with your plugin name in configuration.py or plugins.py"

[brotherdust]

@ryanmerolle A thousand thank-you's for creating a portable dev environment. I will be using it to develop my own plugin! May I make a suggestion? It would be good to create a repo that could serve as a template for the dev environment so your users don't have to weed out all the netbox-acls stuff to start using it. =) I'm happy to contribute if you need help.

[ryanmerolle]

I have not touched it in some time. I’m happy for someone to fork and improve the devcontianer portion or pr changes into the netbox-ACLs repo.

[rizlas]

I just updated my open statement with a setup that (at least for me) works very well for both script/report and plugin development (with hot reload).

Use it as a docker-compose override file.

Warning

Do not use it in production

[rizlas]

Hi, yes. I'm currently using Dockerfile inheritance from the original Netbox image (from statement) and then "installing" my custom plugin in /opt/netbox/netbox.
Once development is complete, I build my own image and use it instead of the original one.

The plugin directory is referenced in the Docker-compose file, which I've already shared (- ./a_plugin_root_folder/a_plugin:/opt/netbox/netbox/a_plugin)

This setup has proven to be the most effective for developing with Docker (at least for me). To the best of my knowledge, documentation on how to carry out development using Docker is not available.

Here's my current directory tree for Netbox:

netbox
├── Dockerfile
├── external_libraries.txt
├── external_plugins.txt
└── netbox-plugin-interface-sync
    ├── docs
    │   └── images
    │       ├── 1_device_page.png
    │       └── 2_interface_comparison.png
    ├── netbox_interface_sync
    │   ├── comparison.py
    │   ├── forms.py
    │   ├── __init__.py
    │   ├── template_content.py
    │   ├── templates
    │   │   └── netbox_interface_sync
    │   │       ├── compare_components_button.html
    │   │       ├── components_comparison.html
    │   │       └── number_of_interfaces_panel.html
    │   ├── urls.py
    │   ├── utils.py
    │   └── views.py
    ├── README.md
    ├── README_ru.md
    └── setup.py

[jirivrany]

@rizlas thank you very much for quick reply. Could you please share the content of external_plugins.txt ? I assume that it contains "a_plugin" but I'm not sure.

[rizlas]

No, external plugins contains a list of netbox plugins published on pypi and then installed with this command:

pip install -r external_plugins.txt

a_plugin is your custom plugin.

example of external_plugins content:

#### PLUGINS ####

# https://github.com/k01ek/netbox-qrcode
netbox-qrcode==0.0.11

# https://github.com/PieterL75/netbox_ipcalculator
netbox-ipcalculator==1.1

# https://github.com/mattieserver/netbox-topology-views
netbox-topology-views==3.9.0

# https://github.com/ArnesSI/netbox-inventory
netbox-inventory==1.5.2

[jirivrany]

Thank you for clarification. I'm new in Netbox plugins, so please apologize my somehow stupid questions :)

I probably still missing one point. When do you register the plugin in netbox? For this config/plugins.py must be edited right? I can do it by manually as well as the plugin install to venv. Just by mapping another volume, editing the file and restarting the container. But your command directive for docker-compose seems to do install automaticaly among the other tasks.

[rizlas]

https://docs.netbox.dev/en/stable/plugins/development/#development-installation
this is the command used (only for dev)

To activate the plugin, add your plugin name to plugins.py or configuration.py. I'm using the latter, but documentation suggest to use plugins.py.

https://github.com/netbox-community/netbox-docker/wiki/Using-Netbox-Plugins

[divad1196]

Hi, I just came across this discussion.
In my opinion, the docker-entrypoint.sh should do some more operations:

1. Scan a directory and do a pip install -e for each module it finds (or use a file with the list of files). The plugins can be mounted inside of it. (This part is failling due to permission issue. I tried to use PYTHONPATH environment variable but django cannot load the plugins then)
2. Use watchdog or any other tool to check if a change occurs to the files and restart the server. (Watchdog is apparently already in the docker image)

More precisely, we need something like the following two changes in docker-entrypoint.sh

# Load correct Python3 env
# shellcheck disable=SC1091
source /opt/netbox/venv/bin/activate

INSTALL_CMD="/opt/netbox/venv/bin/pip install"
if [[ $DEVELOPMENT = "true" ]]; then
INSTALL_CMD="$INSTALL_CMD -e"
fi

for plugin in $(find /opt/netbox/netbox/plugins/ -mindepth 1 -maxdepth 1 -type d);
do
  $INSTALL_CMD $plugin;
done

# Launch whatever is passed by docker
# (i.e. the RUN instruction in the Dockerfile)

CMD="$@"
if [[ $DEVELOPMENT = "true" ]]; then
CMD="watchmedo auto-restart -d /opt/netbox/netbox/plugins/ bash -- -c 'exec $CMD'"
fi

echo "Starting Command: $CMD"
exec "$CMD"

and the folder /opt/netbox/netbox/plugins/

Now we only need to mount our plugins in the docker image and add an environment variable.
Is that something that might be proposed? It has almost no impact on the container startup.

[kbreivik]

plugins.py is your friend, at least when it comes to netbox-docker.

Late reply, but can help others.

I'll write a summary on the appoach to netbox-docker and plugins when I get the time, try to iron out some quirks, and get more people onboard. The docker approach is in most cases the fastest, and especially for testing.
Example, from start, installing ubuntu vm in vm ware workstation 16.x, under 30 minutes from start to netbox-docker with netbox-topology-views on netbox- 3.7.x up and running. Probably more like 15min, I haven't timed it, just winged it.
Being new to docker and netbox-docker and plugins, there is some things to improve...

But to reply to original question, both configuration.py and plugins.py will work, but plugins.py is preferred, and consider configuration.py static, or at least edited by the development.
While plugins.py is yours to play with. They are both read during docker compose up -d (-d is peferred, if you aren't looking for errors, as it will return you to cmd prompt)

A tip vs pull, and redoing it after initial build.
sudo docker compose pull --ignore-buildable #will ignore alreay ready built images that isn't rebuilt due to no need for it. (no new commits) please correct me if I'm wrong, new to docker, but old to the IT part of it.

This is mostly for people new to docker and it's quirks, and if I can help just 1 person, then this is worth it. ;-)

[divad1196]

We all know that, this is not the point.

If you work on django/netbox, you have the possibility to run a development server that tracks the file changes and restart the server for you without you doing anything.

This is what is discussed here. Additionnaly, the proposal here also tries to automaticly handle the migrations

[dkraklan]

I was wanting to solve this problem as well, combined some of the examples from this thread as well as how some of the other developers are handling it for their plugins and put it together in a repo to quickly spin up developer eviroments, install the plugin, and hot reload when working on plugins.

https://github.com/dkraklan/netbox-plugin-development-env