diff --git a/README.md b/README.md index 8205a14..ba919ef 100644 --- a/README.md +++ b/README.md @@ -27,15 +27,18 @@ # DESCRIPTION -**ngbuddy** ("Netgraph Buddy") is an rc.d script for managing netgraph(4) networks in mixed vm and jail environments. **rc.conf** variables prefixed by **ngbuddy_** are used to manage "permanent" ng_bridge(4) and ng_eiface(4) devices. Additional tools assist with configuring vm-bhyve and naming their sockets for statistics and graphing. +**ngbuddy** ("Netgraph Buddy") is an rc.d script for managing netgraph(4) in mixed vm and jail environments. Netgraph provides a more flexible networking solution compared to the traditional if_bridge/epair/tap setup, offering a clearer and shorter list of virtual devices, and performance benefits for some workloads. + +**rc.conf** variables prefixed by **ngbuddy_** are used to manage ng_bridge(4) and ng_eiface(4) devices upon service start (and system boot). Additional tools assist with jail interface management, configuring vm-bhyve, naming vm-bhyve sockets, displaying basic statistics, and determining stable MAC addresses to help avoid collisions. # QUICK START EXAMPLE -The following commands will configure a system for netgraph in a way that is suitable for most common setups on systems where no netgraph configuration currently exists. +The following commands will configure a system for netgraph. **service ngbuddy enable** -: Set **rc.conf** variables and create a _public_ bridge interface associated with the host system's default route, then create a _private_ bridge linked to a virtual interface named **nghost0** suitable for a local or NAT interfaces. It will append three **rc.conf** lines similar to the following, which you can modify before starting the service: - +: Sets **rc.conf** variables to enable the ngbuddy service. If no bridge definitions are set, the following default bridge definitions will be added: \ +: _public_: A bridge interface associated with the host system's current default route, allowing guests to interact with the existing network. \ +: _private_: A bridge linked to a new virtual interface named **nghost0**, suitable for host-only or NAT network with your guests. \ ```sh ngbuddy_enable="YES" @@ -44,27 +47,25 @@ The following commands will configure a system for netgraph in a way that is sui ``` **service ngbuddy start** -: This command creates the above interfaces. +: Creates bridges and interfaces defined in **rc.conf**. **service ngbuddy vmconf** -: Add the our "public" and "private" bridges to the vm(8) configuration. - -If you'd like to use host-only or NAT interface, you must configure the newly created **nghost0** interface. For example, you may want to set up IP addresses, a DNS resolver, and a DHCP server. +: Adds our _public_ and _private_ bridges to the vm(8) configuration, as a substitute for the **vm switch ...*** commands. -Once post-configuration is to your liking, create jails or bhyve instances attached to your _public_ or _private_ bridges as you prefer. See the **jail_skel.conf** for assistance configuring jails. +To get the most out of the _private_ bridge, configure **nghost0** with an IP address and add a NAT service to allow your guests to have access to the network. See the **examples** in the **ngbuddy** repository for demo scripts. # SUBCOMMANDS -Subcommands are called using **service ngbuddy _SUBCOMMAND_**. Note that all commands rely on ngctl(8) and require root permissions. +Subcommands are called using **service ngbuddy** _SUBCOMMAND_. Note that all commands rely on ngctl(8) and require root permissions. **enable** -: Create a basic default **ngbuddy** configuration and enable the service. +: Enable the **ngbuddy** service. If no bridges are defined, a _public_ and _private_ bridge will be created. See _QUICK START EXAMPLE_ above for details. **start** -: Load the ng_bridge(4) and ng_eiface(4) options configured in **rc.conf**. See _RC.CONF VARIABLES_ below. +: Load the bridge and eiface options configured in **rc.conf**. See _RC.CONF VARIABLES_ below for a complete list of options. **stop** -: Destroy all ng_bridge(4) and ng_eiface(4) devices, regardless of whether they were created with **ngbuddy** or not. +: Destroy all ng_bridge(4) and ng_eiface(4) devices, even if they were not created with **ngbuddy**. **restart** : Stop, then start. @@ -73,22 +74,22 @@ Subcommands are called using **service ngbuddy _SUBCOMMAND_**. Note that all com : Print a list of ng_bridge(4), ng_eiface(4), and ng_socket(4) devices and basic usage statistics. **bridge** _bridge_ _interface_ -: Create a bridge and an associated **rc.conf** entry. If the _interface_ exists, _bridge_ will be associated with it. Otherwise, _interface_ will be created as a new ng_eiface(4) device. +: Create a bridge and an associated **rc.conf** entry. If the _interface_ exists, _bridge_ will be associated with it. Otherwise, _interface_ will be created as a new eiface node. **unbridge** _bridge_ -: Remove the indicated bridge from netgraph and **rc.conf**. This operation will fail if devices appear to be attached to it. +: Remove the indicated bridge from netgraph and **rc.conf**. **jail** _interface_ [_bridge_] -: Create a new ng_eiface(4) associated with the indicated _bridge_. If only one ng_bridge(4) is present, _bridge_ may be omitted. +: Create a new eiface associated with the indicated _bridge_. If only one ng_bridge(4) is present, _bridge_ may be omitted. **unjail** _interface_ [_jail_] -: Remove an ng_eiface(4) associated with the indicated _jail_. If the _interface_ matches the jail name, _jail_ may be omitted. +: Shut down the eiface associated with the indicated _jail_. If the _interface_ matches the jail name, _jail_ may be omitted. **create** _interface_ [_bridge_] -: Create a new ng_eiface(4) associated with the indicated _bridge_ and add it to **rc.conf** so it will be created on startup. If only one ng_bridge(4) is present, _bridge_ may be omitted. +: Create a new eiface associated with the indicated _bridge_ and add it to **rc.conf** so it will be created on startup. If only one bridge is configured, _bridge_ may be omitted. **destroy** _interface_ -: Remove the indicated ng_eiface(4) and remove it from **rc.conf**. +: Shut down the indicated eiface and remove it from **rc.conf**. **vmconf** : Add the bridges configured in **rc.conf** to the vm(8) configuration, e.g., **/vm/.config/system.conf**. @@ -98,7 +99,7 @@ Subcommands are called using **service ngbuddy _SUBCOMMAND_**. Note that all com # RC.CONF VARIABLES -The following variables can be manually configured Some of the above subcommands will use sysrc(8) to configure rc.conf with the following variables for persistent configuration on service restart or system reboot, which can also be edited manually. +The following variables can be manually configured. Some of the above subcommands will use sysrc(8) to configure rc.conf with the following variables for persistent configuration on service restart or system reboot. _ngbuddy_enable_ : Set to _YES_ to enable the service. @@ -107,13 +108,13 @@ _ngbuddy\_(_BRIDGE_)\_if_ : Link a new ng_bridge(4) device named _BRIDGE_ to the indicated interface, e.g., _eth0_. If the interface already exists, link it to the new bridge and disable LRO/TSO. If the interface does not exist, create it as an ng_eiface(4) device. This variable will be set with the **bridge** and **unbridge** subcommands. _ngbuddy\_(_BRIDGE_)\_list_ -: A space delimited list of additional ng_eiface(4) devices that will be attached to _BRIDGE_ at startup. This variable will be set with the **create** and **destroy** subcommands. +: A space delimited list of additional ng_eiface(4) devices that will be attached to _BRIDGE_ at startup. This variables will be set with the **create** and **destroy** subcommands. _ngbuddy_set_mac_ -: If set to _YES_, created ng_eiface hardware addresses will be determined only from a hash of the interface name; this ensures each interface's MAC address is stable between hosts. If set to another string, such as a host or domain name, add that seed to the MAC address generator. The default behavior will used FreeBSD's default MAC address generator, which is prone to MAC address collisions in large networks. +: If set to _YES_, eiface hardware addresses will be determined from a hash of the interface name, ensuring that the interfaces' MAC address are stable between hosts. If set to a string besides _YES_, that string will be added to the MAC address generator's seed. _ngbuddy_set_mac_prefix_ -: Override the default MAC address prefix of **58:9C:FC** (the OUI of the FreeBSD Foundation). For example, you can set _ngbuddy_set_mac_prefix="02"_ to minimize the risk of collisions. +: Override the default MAC address prefix of **58:9C:FC** (the OUI of the FreeBSD Foundation). For example, you can set _ngbuddy_set_mac_prefix="02"_ to minimize the risk of collisions. _ngbuddy_set_mac_ must also be enabled to use this feature. _ngbuddy_set_mac_hash_ : Override the default hash command of **sha1** with the command indicated. The command's output will receive the seed through standard input (see _ngbuddy_set_mac_) and must return enough hexadecimal characters to complete the MAC address. @@ -128,15 +129,12 @@ _ngbuddy_set_mac_hash_ **/usr/local/share/ngbuddy/ngbuddy-mmd.awk** : An alternative to **ngctl dot** that creates a Mermaid-JS color diagram of netgraph nodes. -# NOTES - -These scripts were developed to assist with new netgraph features in **vm-bhyve 1.5+**, and were inspired by the **/usr/share/examples/jails/jng** example script and additional examples by Klara Systems. - # EXAMPLES +For examples and demo scripts, see **examples** at: https://github.com/bellhyve/netgraph-buddy **Example 1: Quickly deploy a VNET jail with netgraph using jail.conf.d** -The following steps will configure a jail attached to the interface associated with your default route, likely your LAN, using DHCP. See the files in **examples** at: https://github.com/bellhyve/netgraph-buddy +The following steps will configure a jail attached to the interface associated with the host's current default route, likely your LAN, using DHCP. First, set up Netgraph Buddy: \ - **service ngbuddy enable** \ @@ -155,9 +153,9 @@ Configure the jail configuration: \ To create more jails, you can: \ - Copy **/jail/my_jail/** to **/jail/new_jail1/** \ - Copy **/etc/jail.conf.d/my_jail.conf** to **new_jail1.conf** \ -- Edit the new configuration as above, chaning the word **my_jail** to **new_jail1** \ -- Run: **service jail start new_jail1** -- And repeat as desired. +- Edit the new configuration as above and change the word **my_jail** to **new_jail1** \ +- Run: **service jail start new_jail1** \ +- And repeat as desired. \ **Example 2: An rc.conf example for a slightly more complex setup** @@ -201,6 +199,10 @@ wan ix0 (lower): RX 46.32 KB, TX 30.92 KB ``` +# NOTES + +These scripts were developed to assist with new netgraph features in **vm-bhyve 1.5+**, and were inspired by the **/usr/share/examples/jails/jng** example script and additional examples by Klara Systems. + # SEE ALSO jail(8), netgraph(4), ng_bridge(4), ngctl(8), ng_eiface(4), ng_socket(4), vm(8) diff --git a/examples/jail_skel.conf b/examples/jail_skel.conf index a4d38be..f177790 100644 --- a/examples/jail_skel.conf +++ b/examples/jail_skel.conf @@ -3,7 +3,7 @@ # An example simple jail using Netgraph Buddy. One useful strategy is to match # the jail's name with the interface name and relative path name. This jail # uses the devfs.rules example to allow DHCP, pf, and System V IPC primatives -# (for applications like Postgres and Zabbix). +# (for applications like Postgres). # # When using this as a template, change $bridge, $if_name, and the jail's name # as desired. diff --git a/examples/ten_jails.sh b/examples/ten_jails.sh new file mode 100644 index 0000000..e5778ad --- /dev/null +++ b/examples/ten_jails.sh @@ -0,0 +1,94 @@ +#!/bin/sh +# +# ten_jails.sh +# +# This script is intended to be used on a blank virtual machine to illustrate +# using Netgraph Buddy and ZFS to very quickly deploy ten jails in different +# ways. There are no requirements besides a network connection, a FreeBSD base +# package, and Netgraph Buddy. +# +# Netgraph Buddy will use jail.conf to create five "public" jails associated +# with a system's default route and 5 "private" jails which use the host as a +# DHCP server and router. +# +# This script is intended for use on an empty system with ZFS, such as the +# FreeBSD 140 ZFS VM image: +# +# https://download.freebsd.org/ftp/releases/VM-IMAGES/14.0-RELEASE/amd64/Latest/FreeBSD-14.0-RELEASE-amd64-zfs.raw.xz +# +# WARNING: Several commands below will overwrite jail-related files and network +# settings; please use this script with caution. + +JAIL_DS=zroot/jail +PRIVATE_NET=10.2.19 + +set -x +pkg install -y git-lite dhcpd + +# Set up ngbuddy +# TODO: Replace with ngbuddy FreeBSD package. +git clone https://github.com/bellhyve/ngbuddy.git +cd ngbuddy +cp ngbuddy /usr/local/etc/rc.d/ +cp ngbuddy.8 /usr/local/share/man/man8/ +cp -r share/ngbuddy/ /usr/local/share/ngbuddy/ +cp examples/jail_skel.conf /etc/jail.conf.d/ +cat examples/devfs.rules >> /etc/devfs.rules +cd +service ngbuddy enable +service ngbuddy start + +# Set up jail template +JAIL_SKEL_DS=$JAIL_DS/jail_skel +JAIL_SKEL_CONF=/etc/jail.conf.d/jail_skel.conf +zfs create -p $JAIL_SKEL_DS +zfs set mountpoint=/jail $JAIL_DS +JAIL_DIR=/jail/jail_skel +[ ! -e base.txz ] && fetch https://download.freebsd.org/ftp/releases/amd64/amd64/14.0-RELEASE/base.txz -q +[ ! -e $JAIL_DIR/etc/rc.conf ] && tar zxf base.txz -C $JAIL_DIR +sysrc -f $JAIL_DIR/etc/rc.conf ifconfig_DEFAULT=SYNCDHCP +zfs snapshot $JAIL_SKEL_DS@a + +# Set up jails +for j in $(jot 5); do + # Public + jname=pubjail$j + zfs clone $JAIL_SKEL_DS@a $JAIL_DS/$jname + sed s/jail_skel/$jname/ $JAIL_SKEL_CONF > /etc/jail.conf.d/$jname.conf + sysrc jail_list+=$jname + + # Private + jname=prijail$j + zfs clone $JAIL_SKEL_DS@a $JAIL_DS/$jname + sed -e s/jail_skel/$jname/ -e s/public/private/ $JAIL_SKEL_CONF > /etc/jail.conf.d/$jname.conf + sysrc jail_list+=$jname +done + +# Public jails can be started at this point; +# no additional configuration required + +# Set up networking for private jails +DNS_SERVER=$(grep -o '[0-9].*' /etc/resolv.conf) +EXT_IF=$(netstat -rn|awk '$1 == "default"{print $4}') +echo "nat on $EXT_IF from $PRIVATE_NET.0/24 to any -> ($EXT_IF)" > /etc/pf.conf +cat > /usr/local/etc/dhcpd.conf << EOF +option domain-name-servers $DNS_SERVER; +option subnet-mask 255.255.255.0; +subnet $PRIVATE_NET.0 netmask 255.255.255.0 { + range $PRIVATE_NET.101 $PRIVATE_NET.199; + option routers $PRIVATE_NET.1; +} +EOF +# Enable and start required network-related services for private jails +sysrc gateway_enable=YES pf_enable=YES dhcpd_enable=YES dhcpd_flags=nghost0 +sysrc jail_enable=YES jail_parallel_start=YES +sysrc ifconfig_nghost0="inet $PRIVATE_NET.1/24 up" +# The next two lines are required before the next reboot for private jails +ifconfig nghost0 inet $PRIVATE_NET.1/24 up +sysctl net.inet.ip.forwarding=1 +# Start firewall and DHCP server for private jails +service pf start +service dhcpd start + +# Start ten jails +service jail start diff --git a/ng-buddy.8 b/ng-buddy.8 deleted file mode 100644 index dd4f075..0000000 --- a/ng-buddy.8 +++ /dev/null @@ -1,327 +0,0 @@ -.\" Automatically generated by Pandoc 3.1.13 -.\" -.TH "ngbuddy" "8" "July 8, 2024" "" "System Manager\[cq]s Manual" -.SH NAME -\f[B]ngbuddy\f[R] \- Simplified netgraph(4) manager for jail(8) and -bhyve(8) -.SH SYNOPSIS -\f[B]service ngbuddy enable\f[R] -.PD 0 -.P -.PD -\f[B]service ngbuddy start\f[R] -.PD 0 -.P -.PD -\f[B]service ngbuddy stop\f[R] -.PD 0 -.P -.PD -\f[B]service ngbuddy restart\f[R] -.PD 0 -.P -.PD -\f[B]service ngbuddy status\f[R] -.PP -\f[B]service ngbuddy bridge\f[R] \f[I]bridge\f[R] \f[I]interface\f[R] -.PD 0 -.P -.PD -\f[B]service ngbuddy unbridge\f[R] \f[I]bridge\f[R] -.PP -\f[B]service ngbuddy jail\f[R] \f[I]interface\f[R] [\f[I]bridge\f[R]] -.PD 0 -.P -.PD -\f[B]service ngbuddy unjail\f[R] \f[I]interface\f[R] [\f[I]jail\f[R]] -.PD 0 -.P -.PD -\f[B]service ngbuddy create\f[R] \f[I]interface\f[R] [\f[I]bridge\f[R]] -.PD 0 -.P -.PD -\f[B]service ngbuddy destroy\f[R] \f[I]interface\f[R] -.PP -\f[B]service ngbuddy vmconf\f[R] -.PD 0 -.P -.PD -\f[B]service ngbuddy vmname\f[R] -.SH DESCRIPTION -\f[B]ngbuddy\f[R] (\[lq]Netgraph Buddy\[rq]) is an rc.d script for -managing netgraph(4) networks in mixed vm and jail environments. -\f[B]rc.conf\f[R] variables prefixed by \f[B]ngbuddy_\f[R] are used to -manage \[lq]permanent\[rq] ng_bridge(4) and ng_eiface(4) devices. -Additional tools assist with configuring vm\-bhyve and naming their -sockets for statistics and graphing. -.SH QUICK START EXAMPLE -The following commands will configure a system for netgraph in a way -that is suitable for most common setups on systems where no netgraph -configuration currently exists. -.TP -\f[B]service ngbuddy enable\f[R] -Set \f[B]rc.conf\f[R] variables and create a \f[I]public\f[R] bridge -interface associated with the host system\[cq]s default route, then -create a \f[I]private\f[R] bridge linked to a virtual interface named -\f[B]nghost0\f[R] suitable for a local or NAT interfaces. -It will append three \f[B]rc.conf\f[R] lines similar to the following, -which you can modify before starting the service: -.IP -.EX - ngbuddy_enable=\[dq]YES\[dq] - ngbuddy_public_if=\[dq]em0\[dq] - ngbuddy_private_if=\[dq]nghost0\[dq] -.EE -.TP -\f[B]service ngbuddy start\f[R] -This command creates the above interfaces. -.TP -\f[B]service ngbuddy vmconf\f[R] -Add the our \[lq]public\[rq] and \[lq]private\[rq] bridges to the vm(8) -configuration. -.PP -If you\[cq]d like to use host\-only or NAT interface, you must configure -the newly created \f[B]nghost0\f[R] interface. -For example, you may want to set up IP addresses, a DNS resolver, and a -DHCP server. -.PP -Once post\-configuration is to your liking, create jails or bhyve -instances attached to your \f[I]public\f[R] or \f[I]private\f[R] bridges -as you prefer. -See the \f[B]jail_skel.conf\f[R] for assistance configuring jails. -.SH SUBCOMMANDS -Subcommands are called using \f[B]service ngbuddy -\f[BI]SUBCOMMAND\f[B]\f[R]. -Note that all commands rely on ngctl(8) and require root permissions. -.TP -\f[B]enable\f[R] -Create a basic default \f[B]ngbuddy\f[R] configuration and enable the -service. -.TP -\f[B]start\f[R] -Load the ng_bridge(4) and ng_eiface(4) options configured in -\f[B]rc.conf\f[R]. -See \f[I]RC.CONF VARIABLES\f[R] below. -.TP -\f[B]stop\f[R] -Destroy all ng_bridge(4) and ng_eiface(4) devices, regardless of whether -they were created with \f[B]ngbuddy\f[R] or not. -.TP -\f[B]restart\f[R] -Stop, then start. -.TP -\f[B]status\f[R] -Print a list of ng_bridge(4), ng_eiface(4), and ng_socket(4) devices and -basic usage statistics. -.TP -\f[B]bridge\f[R] \f[I]bridge\f[R] \f[I]interface\f[R] -Create a bridge and an associated \f[B]rc.conf\f[R] entry. -If the \f[I]interface\f[R] exists, \f[I]bridge\f[R] will be associated -with it. -Otherwise, \f[I]interface\f[R] will be created as a new ng_eiface(4) -device. -.TP -\f[B]unbridge\f[R] \f[I]bridge\f[R] -Remove the indicated bridge from netgraph and \f[B]rc.conf\f[R]. -This operation will fail if devices appear to be attached to it. -.TP -\f[B]jail\f[R] \f[I]interface\f[R] [\f[I]bridge\f[R]] -Create a new ng_eiface(4) associated with the indicated -\f[I]bridge\f[R]. -If only one ng_bridge(4) is present, \f[I]bridge\f[R] may be omitted. -.TP -\f[B]unjail\f[R] \f[I]interface\f[R] [\f[I]jail\f[R]] -Remove an ng_eiface(4) associated with the indicated \f[I]jail\f[R]. -If the \f[I]interface\f[R] matches the jail name, \f[I]jail\f[R] may be -omitted. -.TP -\f[B]create\f[R] \f[I]interface\f[R] [\f[I]bridge\f[R]] -Create a new ng_eiface(4) associated with the indicated \f[I]bridge\f[R] -and add it to \f[B]rc.conf\f[R] so it will be created on startup. -If only one ng_bridge(4) is present, \f[I]bridge\f[R] may be omitted. -.TP -\f[B]destroy\f[R] \f[I]interface\f[R] -Remove the indicated ng_eiface(4) and remove it from \f[B]rc.conf\f[R]. -.TP -\f[B]vmconf\f[R] -Add the bridges configured in \f[B]rc.conf\f[R] to the vm(8) -configuration, e.g., \f[B]/vm/.config/system.conf\f[R]. -.TP -\f[B]vmname\f[R] -Name ng_socket(4) devices associated with bhyve instances running via -vm(8). -.SH RC.CONF VARIABLES -The following variables can be manually configured Some of the above -subcommands will use sysrc(8) to configure rc.conf with the following -variables for persistent configuration on service restart or system -reboot, which can also be edited manually. -.TP -\f[I]ngbuddy_enable\f[R] -Set to \f[I]YES\f[R] to enable the service. -.TP -\f[I]ngbuddy_(\f[R]BRIDGE\f[I])_if\f[R] -Link a new ng_bridge(4) device named \f[I]BRIDGE\f[R] to the indicated -interface, e.g., \f[I]eth0\f[R]. -If the interface already exists, link it to the new bridge and disable -LRO/TSO. -If the interface does not exist, create it as an ng_eiface(4) device. -This variable will be set with the \f[B]bridge\f[R] and -\f[B]unbridge\f[R] subcommands. -.TP -\f[I]ngbuddy_(\f[R]BRIDGE\f[I])_list\f[R] -A space delimited list of additional ng_eiface(4) devices that will be -attached to \f[I]BRIDGE\f[R] at startup. -This variable will be set with the \f[B]create\f[R] and -\f[B]destroy\f[R] subcommands. -.TP -\f[I]ngbuddy_set_mac\f[R] -If set to \f[I]YES\f[R], created ng_eiface hardware addresses will be -determined only from a hash of the interface name; this ensures each -interface\[cq]s MAC address is stable between hosts. -If set to another string, such as a host or domain name, add that seed -to the MAC address generator. -The default behavior will used FreeBSD\[cq]s default MAC address -generator, which is prone to MAC address collisions in large networks. -.TP -\f[I]ngbuddy_set_mac_prefix\f[R] -Override the default MAC address prefix of \f[B]58:9C:FC\f[R] (the OUI -of the FreeBSD Foundation). -For example, you can set \f[I]ngbuddy_set_mac_prefix=\[lq]02\[rq]\f[R] -to minimize the risk of collisions. -.TP -\f[I]ngbuddy_set_mac_hash\f[R] -Override the default hash command of \f[B]sha1\f[R] with the command -indicated. -The command\[cq]s output will receive the seed through standard input -(see \f[I]ngbuddy_set_mac\f[R]) and must return enough hexadecimal -characters to complete the MAC address. -.SH FILES -.TP -\f[B]/usr/local/etc/rc.d/ngbuddy\f[R] -The Netgraph Buddy run control script. -.TP -\f[B]/usr/local/share/ngbuddy/ngbuddy\-status.awk\f[R] -Helper for \f[B]service ngbuddy status\f[R] -.TP -\f[B]/usr/local/share/ngbuddy/ngbuddy\-mmd.awk\f[R] -An alternative to \f[B]ngctl dot\f[R] that creates a Mermaid\-JS color -diagram of netgraph nodes. -.SH NOTES -These scripts were developed to assist with new netgraph features in -\f[B]vm\-bhyve 1.5+\f[R], and were inspired by the -\f[B]/usr/share/examples/jails/jng\f[R] example script and additional -examples by Klara Systems. -.SH EXAMPLES -\f[B]Example 1: Quickly deploy a VNET jail with netgraph using -jail.conf.d\f[R] -.PP -The following steps will configure a jail attached to the interface -associated with your default route, likely your LAN, using DHCP. -See the files in \f[B]examples\f[R] at: -https://github.com/bellhyve/netgraph\-buddy -.PP -First, set up Netgraph Buddy: -.PD 0 -.P -.PD -\- \f[B]service ngbuddy enable\f[R] -.PD 0 -.P -.PD -\- \f[B]service ngbuddy start\f[R] -.PD 0 -.P -.PD -\- Append \f[B]examples/devfs.rules\f[R] to \f[B]/etc/devfs.rules\f[R] -.PD 0 -.P -.PD -.PP -Next, create a new jail: -.PD 0 -.P -.PD -\- Set up a FreeBSD base: \f[B]bsdinstall jail /jail/my_jail\f[R] -.PD 0 -.P -.PD -\- Enable DHCP in the jail: \f[B]sysrc \-f /jail/my_jail/etc/rc.conf -ifconfig_DEFAULT=SYNCDHCP\f[R] -.PD 0 -.P -.PD -.PP -Configure the jail configuration: -.PD 0 -.P -.PD -\- Copy \f[B]examples/jail_skel.conf\f[R] to -\f[B]/etc/jail.conf.d/my_jail.conf\f[R] -.PD 0 -.P -.PD -\- In \f[B]my_jail.conf\f[R] after the comments, change the word -\f[B]jail_skel\f[R] to your jail\[cq]s name, \f[B]my_jail\f[R] -.PD 0 -.P -.PD -\- Run: \f[B]service jail start my_jail\f[R] -.PD 0 -.P -.PD -.PP -To create more jails, simply copy \f[B]/jail/my_jail\f[R] and -\f[B]/etc/jail.conf.d/my_jail.conf\f[R], edit the jail configuration as -above, and start them. -.PP -\f[B]Example 2: An rc.conf example for a slightly more complex -setup\f[R] -.IP -.EX -ngbuddy_enable=\[dq]YES\[dq] -ngbuddy_lan_if=\[dq]igb0\[dq] -ngbuddy_private0_if=\[dq]ng0\[dq] -ngbuddy_private0_list=\[dq]j1p0 j2p0\[dq] -ngbuddy_private1_if=\[dq]ng1\[dq] -ngbuddy_private1_list=\[dq]j1p1 j2p1\[dq] -ngbuddy_tenant_lan_if=\[dq]igb1\[dq] -ngbuddy_tenant_wan_if=\[dq]ix1\[dq] -ngbuddy_wan_if=\[dq]ix0\[dq] -ngbuddy_set_mac=\[dq]belltower\[dq] -ngbuddy_set_mac_prefix=\[dq]02\[dq] -ngbuddy_set_mac_hash=\[dq]sha256\[dq] -.EE -.PP -\f[B]Example 3: Initial status of the above configuration\f[R] -.IP -.EX -lan - igb0 \f[B](\f[R]upper\f[B])\f[R]: RX 0B, TX 0B - igb0 \f[B](\f[R]lower\f[B])\f[R]: RX 0B, TX 0B -private0 - j2p0: RX 0B, TX 0B - j1p0: RX 0B, TX 0B - ng0: RX 0B, TX 0B -private1 - j2p1: RX 0B, TX 0B - j1p1: RX 0B, TX 0B - ng1: RX 0B, TX 0B -tenant_lan - igb1 \f[B](\f[R]upper\f[B])\f[R]: RX 0B, TX 0B - igb1 \f[B](\f[R]lower\f[B])\f[R]: RX 0B, TX 0B -tenant_wan - ix1 \f[B](\f[R]upper\f[B])\f[R]: RX 0B, TX 0B - ix1 \f[B](\f[R]lower\f[B])\f[R]: RX 0B, TX 0B -wan - ix0 \f[B](\f[R]upper\f[B])\f[R]: RX 30.69 KB, TX 46.16 KB - ix0 \f[B](\f[R]lower\f[B])\f[R]: RX 46.32 KB, TX 30.92 KB -.EE -.SH SEE ALSO -jail(8), netgraph(4), ng_bridge(4), ngctl(8), ng_eiface(4), -ng_socket(4), vm(8) -.SH HISTORY -Netgraph Buddy was originally developed as an internal tool for Bell -Tower Integration\[cq]s private cloud in August 2022. -.SH AUTHORS -Daniel J. Bell. diff --git a/ngbuddy b/ngbuddy index c1f7ec2..2eb179d 100755 --- a/ngbuddy +++ b/ngbuddy @@ -57,9 +57,10 @@ ngbuddy_linkpp() { then LNUM=$((LNUM + 1)) else + # If there are no links (x), return link0, otherwise increment (y) LNUM=$(ngctl list -l | awk ' - $1 ~ /^link/ {x = int(substr($1, 5)); if(x > y) y = x} - END { print y ? (++y) : 0 }') + $1 ~ /^link/ { x = int(substr($1, 5)); if (x > y) y = x } + END { print (x != "") ? (++y) : 0}') fi } @@ -269,7 +270,7 @@ ngbuddy_start() { ngbuddy_bridge $bridge bridge_count=$(($bridge_count + 1)) done - echo Created $LNUM links. + echo Created $(($LNUM + 1)) links. } # Return a netgraph link count @@ -280,17 +281,11 @@ ngbuddy_status() { # Blindly lay waste to all netgraph nodes ngbuddy_stop() { - ngbuddy_init - if [ "$LNUM" = "0" ] - then - echo No links found. - exit 1 - fi - ngctl list |\ - awk '{print $2}'|\ - xargs -tn1 -I% ngctl shutdown %: 2>/dev/null - echo Removed $LNUM links. - LNUM="" # In case this is a restart. + local shutdown_nodes=$( ngctl list | \ + awk '$4 ~ /eiface|bridge/ { print $2 }'| \ + xargs -tn1 -I% ngctl shutdown %: 2>&1 | \ + awk 'END { print NR }') + echo Shut down "$shutdown_nodes" nodes. } # Configure vm-bhyve with ngbuddy bridges (version 1.5+ required) diff --git a/ngbuddy.8 b/ngbuddy.8 index d76e6a7..fd18bc3 100644 --- a/ngbuddy.8 +++ b/ngbuddy.8 @@ -1,4 +1,4 @@ -.\" Automatically generated by Pandoc 3.2 +.\" Automatically generated by Pandoc 3.1.13 .\" .TH "ngbuddy" "8" "July 8, 2024" "" "System Manager\[cq]s Manual" .SH NAME @@ -50,23 +50,39 @@ bhyve(8) \f[B]service ngbuddy vmname\f[R] .SH DESCRIPTION \f[B]ngbuddy\f[R] (\[lq]Netgraph Buddy\[rq]) is an rc.d script for -managing netgraph(4) networks in mixed vm and jail environments. +managing netgraph(4) in mixed vm and jail environments. +Netgraph provides a more flexible networking solution compared to the +traditional if_bridge/epair/tap setup, offering a clearer and shorter +list of virtual devices, and performance benefits for some workloads. +.PP \f[B]rc.conf\f[R] variables prefixed by \f[B]ngbuddy_\f[R] are used to -manage \[lq]permanent\[rq] ng_bridge(4) and ng_eiface(4) devices. -Additional tools assist with configuring vm\-bhyve and naming their -sockets for statistics and graphing. +manage ng_bridge(4) and ng_eiface(4) devices upon service start (and +system boot). +Additional tools assist with jail interface management, configuring +vm\-bhyve, naming vm\-bhyve sockets, displaying basic statistics, and +determining stable MAC addresses to help avoid collisions. .SH QUICK START EXAMPLE -The following commands will configure a system for netgraph in a way -that is suitable for most common setups on systems where no netgraph -configuration currently exists. +The following commands will configure a system for netgraph. .TP \f[B]service ngbuddy enable\f[R] -Set \f[B]rc.conf\f[R] variables and create a \f[I]public\f[R] bridge -interface associated with the host system\[cq]s default route, then -create a \f[I]private\f[R] bridge linked to a virtual interface named -\f[B]nghost0\f[R] suitable for a local or NAT interfaces. -It will append three \f[B]rc.conf\f[R] lines similar to the following, -which you can modify before starting the service: +Sets \f[B]rc.conf\f[R] variables to enable the ngbuddy service. +If no bridge definitions are set, the following default bridge +definitions will be added: +.PD 0 +.P +.PD +\f[I]public\f[R]: A bridge interface associated with the host +system\[cq]s current default route, allowing guests to interact with the +existing network. +.PD 0 +.P +.PD +\f[I]private\f[R]: A bridge linked to a new virtual interface named +\f[B]nghost0\f[R], suitable for host\-only or NAT network with your +guests. +.PD 0 +.P +.PD .IP .EX ngbuddy_enable=\[dq]YES\[dq] @@ -75,38 +91,36 @@ which you can modify before starting the service: .EE .TP \f[B]service ngbuddy start\f[R] -This command creates the above interfaces. +Creates bridges and interfaces defined in \f[B]rc.conf\f[R]. .TP \f[B]service ngbuddy vmconf\f[R] -Add the our \[lq]public\[rq] and \[lq]private\[rq] bridges to the vm(8) -configuration. -.PP -If you\[cq]d like to use host\-only or NAT interface, you must configure -the newly created \f[B]nghost0\f[R] interface. -For example, you may want to set up IP addresses, a DNS resolver, and a -DHCP server. +Adds our \f[I]public\f[R] and \f[I]private\f[R] bridges to the vm(8) +configuration, as a substitute for the \f[B]vm switch \&...\f[R]* +commands. .PP -Once post\-configuration is to your liking, create jails or bhyve -instances attached to your \f[I]public\f[R] or \f[I]private\f[R] bridges -as you prefer. -See the \f[B]jail_skel.conf\f[R] for assistance configuring jails. +To get the most out of the \f[I]private\f[R] bridge, configure +\f[B]nghost0\f[R] with an IP address and add a NAT service to allow your +guests to have access to the network. +See the \f[B]examples\f[R] in the \f[B]ngbuddy\f[R] repository for demo +scripts. .SH SUBCOMMANDS -Subcommands are called using \f[B]service ngbuddy -\f[BI]SUBCOMMAND\f[B]\f[R]. +Subcommands are called using \f[B]service ngbuddy\f[R] +\f[I]SUBCOMMAND\f[R]. Note that all commands rely on ngctl(8) and require root permissions. .TP \f[B]enable\f[R] -Create a basic default \f[B]ngbuddy\f[R] configuration and enable the -service. +Enable the \f[B]ngbuddy\f[R] service. +If no bridges are defined, a \f[I]public\f[R] and \f[I]private\f[R] +bridge will be created. +See \f[I]QUICK START EXAMPLE\f[R] above for details. .TP \f[B]start\f[R] -Load the ng_bridge(4) and ng_eiface(4) options configured in -\f[B]rc.conf\f[R]. -See \f[I]RC.CONF VARIABLES\f[R] below. +Load the bridge and eiface options configured in \f[B]rc.conf\f[R]. +See \f[I]RC.CONF VARIABLES\f[R] below for a complete list of options. .TP \f[B]stop\f[R] -Destroy all ng_bridge(4) and ng_eiface(4) devices, regardless of whether -they were created with \f[B]ngbuddy\f[R] or not. +Destroy all ng_bridge(4) and ng_eiface(4) devices, even if they were not +created with \f[B]ngbuddy\f[R]. .TP \f[B]restart\f[R] Stop, then start. @@ -119,30 +133,27 @@ basic usage statistics. Create a bridge and an associated \f[B]rc.conf\f[R] entry. If the \f[I]interface\f[R] exists, \f[I]bridge\f[R] will be associated with it. -Otherwise, \f[I]interface\f[R] will be created as a new ng_eiface(4) -device. +Otherwise, \f[I]interface\f[R] will be created as a new eiface node. .TP \f[B]unbridge\f[R] \f[I]bridge\f[R] Remove the indicated bridge from netgraph and \f[B]rc.conf\f[R]. -This operation will fail if devices appear to be attached to it. .TP \f[B]jail\f[R] \f[I]interface\f[R] [\f[I]bridge\f[R]] -Create a new ng_eiface(4) associated with the indicated -\f[I]bridge\f[R]. +Create a new eiface associated with the indicated \f[I]bridge\f[R]. If only one ng_bridge(4) is present, \f[I]bridge\f[R] may be omitted. .TP \f[B]unjail\f[R] \f[I]interface\f[R] [\f[I]jail\f[R]] -Remove an ng_eiface(4) associated with the indicated \f[I]jail\f[R]. +Shut down the eiface associated with the indicated \f[I]jail\f[R]. If the \f[I]interface\f[R] matches the jail name, \f[I]jail\f[R] may be omitted. .TP \f[B]create\f[R] \f[I]interface\f[R] [\f[I]bridge\f[R]] -Create a new ng_eiface(4) associated with the indicated \f[I]bridge\f[R] -and add it to \f[B]rc.conf\f[R] so it will be created on startup. -If only one ng_bridge(4) is present, \f[I]bridge\f[R] may be omitted. +Create a new eiface associated with the indicated \f[I]bridge\f[R] and +add it to \f[B]rc.conf\f[R] so it will be created on startup. +If only one bridge is configured, \f[I]bridge\f[R] may be omitted. .TP \f[B]destroy\f[R] \f[I]interface\f[R] -Remove the indicated ng_eiface(4) and remove it from \f[B]rc.conf\f[R]. +Shut down the indicated eiface and remove it from \f[B]rc.conf\f[R]. .TP \f[B]vmconf\f[R] Add the bridges configured in \f[B]rc.conf\f[R] to the vm(8) @@ -152,10 +163,10 @@ configuration, e.g., \f[B]/vm/.config/system.conf\f[R]. Name ng_socket(4) devices associated with bhyve instances running via vm(8). .SH RC.CONF VARIABLES -The following variables can be manually configured Some of the above -subcommands will use sysrc(8) to configure rc.conf with the following -variables for persistent configuration on service restart or system -reboot, which can also be edited manually. +The following variables can be manually configured. +Some of the above subcommands will use sysrc(8) to configure rc.conf +with the following variables for persistent configuration on service +restart or system reboot. .TP \f[I]ngbuddy_enable\f[R] Set to \f[I]YES\f[R] to enable the service. @@ -172,32 +183,29 @@ This variable will be set with the \f[B]bridge\f[R] and \f[I]ngbuddy_(\f[R]BRIDGE\f[I])_list\f[R] A space delimited list of additional ng_eiface(4) devices that will be attached to \f[I]BRIDGE\f[R] at startup. -This variable will be set with the \f[B]create\f[R] and +This variables will be set with the \f[B]create\f[R] and \f[B]destroy\f[R] subcommands. .TP \f[I]ngbuddy_set_mac\f[R] -If set to \f[I]YES\f[R], created ng_eiface hardware addresses will be -determined only from a hash of the interface name; this ensures each -interface\[cq]s MAC address is stable between hosts. -If set to another string, such as a host or domain name, add that seed -to the MAC address generator. -The default behavior will used FreeBSD\[cq]s default MAC address -generator, which is prone to MAC address collisions in large networks. +If set to \f[I]YES\f[R], eiface hardware addresses will be determined +from a hash of the interface name, ensuring that the interfaces\[cq] MAC +address are stable between hosts. +If set to a string besides \f[I]YES\f[R], that string will be added to +the MAC address generator\[cq]s seed. .TP \f[I]ngbuddy_set_mac_prefix\f[R] Override the default MAC address prefix of \f[B]58:9C:FC\f[R] (the OUI of the FreeBSD Foundation). For example, you can set \f[I]ngbuddy_set_mac_prefix=\[lq]02\[rq]\f[R] to minimize the risk of collisions. +\f[I]ngbuddy_set_mac\f[R] must also be enabled to use this feature. .TP \f[I]ngbuddy_set_mac_hash\f[R] Override the default hash command of \f[B]sha1\f[R] with the command indicated. -The command\[cq]s output will receive the seed (see -\f[I]ngbuddy_set_mac\f[R]) and must return enough hexadecimal characters -to complete the MAC address after its prefix. -Note that using more complex SHA algorithms does not decrease the risk -of collisions. +The command\[cq]s output will receive the seed through standard input +(see \f[I]ngbuddy_set_mac\f[R]) and must return enough hexadecimal +characters to complete the MAC address. .SH FILES .TP \f[B]/usr/local/etc/rc.d/ngbuddy\f[R] @@ -209,37 +217,59 @@ Helper for \f[B]service ngbuddy status\f[R] \f[B]/usr/local/share/ngbuddy/ngbuddy\-mmd.awk\f[R] An alternative to \f[B]ngctl dot\f[R] that creates a Mermaid\-JS color diagram of netgraph nodes. -.SH NOTES -These scripts were developed to assist with new netgraph features in -\f[B]vm\-bhyve 1.5+\f[R], and were inspired by the -\f[B]/usr/share/examples/jails/jng\f[R] example script and additional -examples by Klara Systems. .SH EXAMPLES -\f[B]Example 1: Quickly deploy VNET jails with netgraph using +For examples and demo scripts, see \f[B]examples\f[R] at: +https://github.com/bellhyve/netgraph\-buddy +.PP +\f[B]Example 1: Quickly deploy a VNET jail with netgraph using jail.conf.d\f[R] .PP -See the files in \f[B]examples\f[R] at: -https://github.com/bellhyve/netgraph\-buddy +The following steps will configure a jail attached to the interface +associated with the host\[cq]s current default route, likely your LAN, +using DHCP. +.PP +First, set up Netgraph Buddy: +.PD 0 +.P +.PD +\- \f[B]service ngbuddy enable\f[R] +.PD 0 +.P +.PD +\- \f[B]service ngbuddy start\f[R] +.PD 0 +.P +.PD +\- Append \f[B]examples/devfs.rules\f[R] to \f[B]/etc/devfs.rules\f[R] +.PD 0 +.P +.PD .PP -After following the above \f[B]QUICK START EXAMPLE\f[R]: +Next, create a new jail: .PD 0 .P .PD -\- Append the \f[B]devfs.rules\f[R] example to -\f[B]/etc/devfs.rules\f[R] +\- Set up a FreeBSD base: \f[B]bsdinstall jail /jail/my_jail\f[R] .PD 0 .P .PD -\- Extract a FreeBSD \f[B]base.txz\f[R] in \f[B]/jail/my_jail\f[R] +\- Enable DHCP in the jail: \f[B]sysrc \-f /jail/my_jail/etc/rc.conf +ifconfig_DEFAULT=SYNCDHCP\f[R] .PD 0 .P .PD -\- Copy the \f[B]jail_skel.conf\f[R] to +.PP +Configure the jail configuration: +.PD 0 +.P +.PD +\- Copy \f[B]examples/jail_skel.conf\f[R] to \f[B]/etc/jail.conf.d/my_jail.conf\f[R] .PD 0 .P .PD -\- In \f[B]my_jail.conf\f[R], change the jail name to \f[B]my_jail\f[R] +\- In \f[B]my_jail.conf\f[R] after the comments, change the word +\f[B]jail_skel\f[R] to your jail\[cq]s name, \f[B]my_jail\f[R] .PD 0 .P .PD @@ -248,22 +278,46 @@ After following the above \f[B]QUICK START EXAMPLE\f[R]: .P .PD .PP -This provides a simple framework for cloning jails and editing a single -template line for rapid deployment of many VNET jails. +To create more jails, you can: +.PD 0 +.P +.PD +\- Copy \f[B]/jail/my_jail/\f[R] to \f[B]/jail/new_jail1/\f[R] +.PD 0 +.P +.PD +\- Copy \f[B]/etc/jail.conf.d/my_jail.conf\f[R] to +\f[B]new_jail1.conf\f[R] +.PD 0 +.P +.PD +\- Edit the new configuration as above and change the word +\f[B]my_jail\f[R] to \f[B]new_jail1\f[R] +.PD 0 +.P +.PD +\- Run: \f[B]service jail start new_jail1\f[R] +.PD 0 +.P +.PD +\- And repeat as desired. +.PD 0 +.P +.PD .PP \f[B]Example 2: An rc.conf example for a slightly more complex setup\f[R] .IP .EX ngbuddy_enable=\[dq]YES\[dq] -ngbuddy_wan_if=\[dq]ix0\[dq] -ngbuddy_tenantwan_if=\[dq]ix1\[dq] ngbuddy_lan_if=\[dq]igb0\[dq] -ngbuddy_tenantlan_if=\[dq]igb1\[dq] ngbuddy_private0_if=\[dq]ng0\[dq] -ngbuddy_private0_list=\[dq]j1p0 j2p1\[dq] +ngbuddy_private0_list=\[dq]j1p0 j2p0\[dq] ngbuddy_private1_if=\[dq]ng1\[dq] ngbuddy_private1_list=\[dq]j1p1 j2p1\[dq] +ngbuddy_tenant_lan_if=\[dq]igb1\[dq] +ngbuddy_tenant_wan_if=\[dq]ix1\[dq] +ngbuddy_wan_if=\[dq]ix0\[dq] ngbuddy_set_mac=\[dq]belltower\[dq] ngbuddy_set_mac_prefix=\[dq]02\[dq] ngbuddy_set_mac_hash=\[dq]sha256\[dq] @@ -272,29 +326,32 @@ ngbuddy_set_mac_hash=\[dq]sha256\[dq] \f[B]Example 3: Initial status of the above configuration\f[R] .IP .EX +lan + igb0 \f[B](\f[R]upper\f[B])\f[R]: RX 0B, TX 0B + igb0 \f[B](\f[R]lower\f[B])\f[R]: RX 0B, TX 0B +private0 + j2p0: RX 0B, TX 0B + j1p0: RX 0B, TX 0B + ng0: RX 0B, TX 0B private1 j2p1: RX 0B, TX 0B j1p1: RX 0B, TX 0B ng1: RX 0B, TX 0B -tenantlan +tenant_lan igb1 \f[B](\f[R]upper\f[B])\f[R]: RX 0B, TX 0B igb1 \f[B](\f[R]lower\f[B])\f[R]: RX 0B, TX 0B -tenantwan +tenant_wan ix1 \f[B](\f[R]upper\f[B])\f[R]: RX 0B, TX 0B ix1 \f[B](\f[R]lower\f[B])\f[R]: RX 0B, TX 0B wan - ix0 \f[B](\f[R]upper\f[B])\f[R]: RX 90.43 KB, TX 127.50 KB - ix0 \f[B](\f[R]lower\f[B])\f[R]: RX 127.50 KB, TX 88.70 KB -lan - hmm1: RX 127.50 KB, TX 88.70 KB - hmm0: RX 127.50 KB, TX 88.70 KB - igb0 \f[B](\f[R]upper\f[B])\f[R]: RX 127.50 KB, TX 88.70 KB - igb0 \f[B](\f[R]lower\f[B])\f[R]: RX 127.50 KB, TX 88.70 KB -private0 - j2p0: RX 127.50 KB, TX 88.70 KB - j1p0: RX 127.50 KB, TX 88.70 KB - ng0: RX 127.50 KB, TX 88.70 KB + ix0 \f[B](\f[R]upper\f[B])\f[R]: RX 30.69 KB, TX 46.16 KB + ix0 \f[B](\f[R]lower\f[B])\f[R]: RX 46.32 KB, TX 30.92 KB .EE +.SH NOTES +These scripts were developed to assist with new netgraph features in +\f[B]vm\-bhyve 1.5+\f[R], and were inspired by the +\f[B]/usr/share/examples/jails/jng\f[R] example script and additional +examples by Klara Systems. .SH SEE ALSO jail(8), netgraph(4), ng_bridge(4), ngctl(8), ng_eiface(4), ng_socket(4), vm(8)