diff --git a/website/content/docs/concepts/plugins/cni.mdx b/website/content/docs/concepts/plugins/cni.mdx index 9c0dbb461c4..74c13eb6519 100644 --- a/website/content/docs/concepts/plugins/cni.mdx +++ b/website/content/docs/concepts/plugins/cni.mdx @@ -61,4 +61,4 @@ Nomad Networking documentation](/nomad/docs/networking/cni). [cni_spec]: https://www.cni.dev/docs/spec/ [cni_spec_net_config]: https://github.com/containernetworking/cni/blob/main/SPEC.md#configuration-format [cni_spec_plugin_config]: https://github.com/containernetworking/cni/blob/main/SPEC.md#plugin-configuration-objects -[nomad_install]: /nomad/tutorials/get-started/get-started-install#post-installation-steps +[nomad_install]: /nomad/tutorials/get-started/get-started-install#linux-post-installation-steps diff --git a/website/content/docs/install/index.mdx b/website/content/docs/install/index.mdx index 8044ab877a2..ca2f3f86d13 100644 --- a/website/content/docs/install/index.mdx +++ b/website/content/docs/install/index.mdx @@ -1,78 +1,23 @@ --- layout: docs -page_title: Installing Nomad -description: Learn how to install Nomad. +page_title: Install Nomad +description: Learn how to install Nomad on Linux, Mac, and Windows. --- -# Installing Nomad +# Install Nomad Nomad is available as a pre-compiled binary or as a package for several operating systems. You can also [build Nomad from source](#from-source). --> If you are interested in trialing Nomad without installing it locally, see the -[Quickstart](/nomad/docs/install/quickstart) for options to get started with Nomad. +-> If you are interested in trialing Nomad without installing it locally, see +the [Quickstart](/nomad/docs/install/quickstart) for options to get started with +Nomad. - - -You can download a [precompiled binary](/nomad/downloads) and -run it on your machine locally. You can also verify the binary using the -available SHA-256 sums. After downloading Nomad, unzip the package. Make sure -that the `nomad` binary is available on your `PATH` before continuing with the -other guides. - - - - -[Homebrew](https://brew.sh) is a free and open source package management system -for Mac OS X. Install the official [Nomad -formula](https://github.com/hashicorp/homebrew-tap) from the terminal. - -First, install the HashiCorp tap, a repository of all of the HashiCorp Homebrew -packages. - -```shell-session -$ brew tap hashicorp/tap -``` - -Now, install Nomad with `hashicorp/tap/nomad`. - -```shell-session -$ brew install hashicorp/tap/nomad -``` - -~> **NOTE:** This installs a signed binary and is automatically updated with -every new official release. - -To update to the latest, run - -```shell-session -$ brew upgrade hashicorp/tap/nomad -``` - - - - -[Chocolatey](https://chocolatey.org/) is a free and open-source package -management system for Windows. Install the [Nomad -package](https://chocolatey.org/packages/nomad) from the command-line. - -```shell-session -$ choco install nomad -``` - -~> **NOTE:** Chocolatey and the Nomad package are **NOT** directly maintained -by HashiCorp. The latest version of Nomad is always available by manual -installation. - - - - -HashiCorp officially maintains and signs packages for the following Linux -distributions. + - + Install the required packages. @@ -84,13 +29,15 @@ $ sudo apt-get update && \ Add the HashiCorp [GPG key][gpg-key]. ```shell-session -$ wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg +$ wget -O- https://apt.releases.hashicorp.com/gpg | \ + sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg ``` Add the official HashiCorp Linux repository. ```shell-session -$ echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list +$ echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" \ +| sudo tee /etc/apt/sources.list.d/hashicorp.list ``` Update and install. @@ -100,7 +47,7 @@ $ sudo apt-get update && sudo apt-get install nomad ``` - + Install `yum-config-manager` to manage your repositories. @@ -121,7 +68,7 @@ $ sudo yum -y install nomad ``` - + Install `dnf config-manager` to manage your repositories. @@ -132,7 +79,8 @@ $ sudo dnf install -y dnf-plugins-core Use `dnf config-manager` to add the official HashiCorp Linux repository. ```shell-session -$ sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/fedora/hashicorp.repo +$ sudo dnf config-manager \ + --add-repo https://rpm.releases.hashicorp.com/fedora/hashicorp.repo ``` Install. @@ -142,7 +90,7 @@ $ sudo dnf -y install nomad ``` - + Install `yum-config-manager` to manage your repositories. @@ -153,7 +101,8 @@ $ sudo yum install -y yum-utils Use `yum-config-manager` to add the official HashiCorp Linux repository. ```shell-session -$ sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo +$ sudo yum-config-manager \ + --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo ``` Install. @@ -163,250 +112,107 @@ $ sudo yum -y install nomad ``` - + --> **TIP:** Now that you have added the HashiCorp repository, you can install -[Consul](/consul/tutorials) and -[Vault](/vault/tutorials) with the same command. +@include 'install/manual-install.mdx' -## Post-installation steps - -These steps are considered optional but can be helpful for running Nomad and to -take advantage of additional Nomad functionalities. + - - + - -

Add the Nomad binary to your system path

- -Permanently add a new location to your path by editing your shell's settings -file (usually called something like `~/.bashrc`, where the part of the filename -after the `.` and before `rc` is the name of your shell). In that file you -should see a line that starts with `export PATH=`, followed by a -colon-separated list of locations. Add the location of the Nomad binary to that -list and save the file. Then reload your shell's configuration with the command -`source ~/.bashrc`, replacing `bash` with the name of your shell. - -

Install CNI plugins

- -Nomad uses CNI plugins to configure network namespaces when using the `bridge` -network mode. All Linux Nomad client nodes using network namespaces must have -CNI plugins installed. - -

Install reference CNI plugins

+ -The following commands install the CNI reference plugins. - -```shell-session -$ export ARCH_CNI=$( [ $(uname -m) = aarch64 ] && echo arm64 || echo amd64) -$ curl -L -o cni-plugins.tgz "https://github.com/containernetworking/plugins/releases/download/v1.5.0/cni-plugins-linux-${ARCH_CNI}"-v1.5.0.tgz && \ - sudo mkdir -p /opt/cni/bin && \ - sudo tar -C /opt/cni/bin -xzf cni-plugins.tgz -``` - -

Install consul-cni CNI plugin

+[Homebrew](https://brew.sh) is a free and open source package management system +for Mac OS X. Install the official [Nomad +formula](https://github.com/hashicorp/homebrew-tap) from the terminal. -If the `transparent_proxy` block is used for Consul Service Mesh, then the -`consul-cni` CNI Plugin must be installed on each client node to redirect inbound -and outbound traffic for services to the Envoy proxy. The following commands -assume that reference CNI plugins have already been installed. +First, install the HashiCorp tap, a repository of all of the HashiCorp Homebrew +packages. ```shell-session -$ export ARCH_CNI=$( [ $(uname -m) = aarch64 ] && echo arm64 || echo amd64) -$ curl -L -o consul-cni.zip "https://releases.hashicorp.com/consul-cni/1.5.0/consul-cni_1.5.0_linux_${ARCH_CNI}".zip && \ - sudo unzip consul-cni.zip -d /opt/cni/bin -x LICENSE.txt +$ brew tap hashicorp/tap ``` -

Configure bridge network to route traffic through iptables

- -Ensure your Linux operating system distribution has been configured to allow -container traffic through the bridge network to be routed via iptables. These -tunables can be set as follows. +Now, install Nomad with `hashicorp/tap/nomad`. ```shell-session -$ echo 1 | sudo tee /proc/sys/net/bridge/bridge-nf-call-arptables && \ - echo 1 | sudo tee /proc/sys/net/bridge/bridge-nf-call-ip6tables && \ - echo 1 | sudo tee /proc/sys/net/bridge/bridge-nf-call-iptables -``` - -To preserve these settings on startup of a client node, add a file including the -following to `/etc/sysctl.d/` or remove the file your Linux distribution puts in -that directory. - - - -```ini -net.bridge.bridge-nf-call-arptables = 1 -net.bridge.bridge-nf-call-ip6tables = 1 -net.bridge.bridge-nf-call-iptables = 1 +$ brew install hashicorp/tap/nomad ``` - - -

Verify cgroup controllers

+-> This installs a signed binary and is automatically updated with +every new official release. -On Linux, Nomad uses cgroups to control resource usage of tasks. If one or more -required cgroups are unavailable, Nomad will disable resource controls that -require cgroups entirely. With cgroups v2, you can verify that you have all -required controllers as follows: +To update to the latest, run ```shell-session -$ cat /sys/fs/cgroup/cgroup.controllers -cpuset cpu io memory pids +$ brew upgrade hashicorp/tap/nomad ``` - -See the documentation on [cgroup controller requirements][] for more details. - + - -

Add the Nomad binary to your system path

+@include 'install/manual-install.mdx' -Permanently add a new location to your path by editing your shell's settings -file (usually called something like `~/.bashrc`, where the part of the filename -after the `.` and before `rc` is the name of your shell). In that file you -should see a line that starts with `export PATH=`, followed by a -colon-separated list of locations. Add the location of the Nomad binary to that -list and save the file. Then reload your shell's configuration with the command -`source ~/.bashrc`, replacing `bash` with the name of your shell. - - - - -

Add the Nomad binary to your system path

- -Add a location to your path via the GUI by navigating to `Environment -Variables` in your system settings, and looking for the variable called `PATH`. -You should see a semicolon-separated list of locations. Add the Nomad binary's -location to that list and then launch a new console window. - - - -No additional steps necessary after installing Nomad using Homebrew. - - -No additional steps necessary after installing Nomad using Chocolatey. - - -Note that if you are running Nomad on Linux, you'll need to run client agents as -root (or with `sudo`) so that cpuset accounting and network namespaces work -correctly. - -

Install CNI plugins

- -Nomad uses CNI plugins to configure network namespaces when using the `bridge` -network mode. All Linux Nomad client nodes using network namespaces must have -CNI plugins installed. - -

Install reference CNI plugins

- -The following commands install the CNI reference plugins. - -```shell-session -$ curl -L -o cni-plugins.tgz "https://github.com/containernetworking/plugins/releases/download/v1.0.0/cni-plugins-linux-$( [ $(uname -m) = aarch64 ] && echo arm64 || echo amd64)"-v1.0.0.tgz && \ - sudo mkdir -p /opt/cni/bin && \ - sudo tar -C /opt/cni/bin -xzf cni-plugins.tgz -``` - -

Install consul-cni CNI plugin

- -If the `transparent_proxy` block is used for Consul Service Mesh, then the -`consul-cni` CNI Plugin must be installed on each client node to redirect inbound -and outbound traffic for services to the Envoy proxy. The following commands -assume that the linux packages repositories have already been added using `apt` or -`yum` and that the reference CNI plugins have already been installed. + + - - -Install the `consul-cni` plugin on each client node. +[Chocolatey](https://chocolatey.org/) is a free and open-source package +management system for Windows. Install the [Nomad +package](https://chocolatey.org/packages/nomad) from the command-line. ```shell-session -$ sudo yum -y install consul-cni +$ choco install nomad ``` - - -Install the `consul-cni` plugin on each client node. - -```shell-session -$ sudo apt-get install -y consul-cni -``` +-> Chocolatey and the Nomad package are **NOT** directly maintained +by HashiCorp. The latest version of Nomad is always available by manual +installation. - - -Install the `consul-cni` plugin on each client node. + -```shell-session -$ sudo dnf -y install consul-cni -``` +@include 'install/manual-install.mdx' - - -Install the `consul-cni` plugin on each client node. + -```shell-session -$ sudo yum -y install consul-cni -``` - -

Configure bridge network to route traffic through iptables

+ -Ensure your Linux operating system distribution has been configured to allow -container traffic through the bridge network to be routed via iptables. These -tunables can be set as follows. -```shell-session -$ echo 1 | sudo tee /proc/sys/net/bridge/bridge-nf-call-arptables && \ - echo 1 | sudo tee /proc/sys/net/bridge/bridge-nf-call-ip6tables && \ - echo 1 | sudo tee /proc/sys/net/bridge/bridge-nf-call-iptables -``` +## Linux post-installation steps -To preserve these settings on startup of a client node, add a file including the -following to `/etc/sysctl.d/` or remove the file your Linux distribution puts in -that directory. +These steps are optional but can be helpful for running Nomad and to +take advantage of additional Nomad functionalities. - +-> You need to run client agents as root (or with `sudo`) so that cpuset accounting and network namespaces work correctly. -```ini -net.bridge.bridge-nf-call-arptables = 1 -net.bridge.bridge-nf-call-ip6tables = 1 -net.bridge.bridge-nf-call-iptables = 1 -``` +### Install CNI reference plugins - +@include 'install/install-cni-plugins.mdx' -

Verify cgroup controllers

+### Install consul-cni plugin -On Linux, Nomad uses cgroups to control resource usage of tasks. If one or more -required cgroups are unavailable, Nomad will disable resource controls that -require cgroups entirely. With cgroups v2, you can verify that you have all -required controllers as follows: +@include 'install/install-consul-cni-plugin.mdx' -```shell-session -$ cat /sys/fs/cgroup/cgroup.controllers -cpuset cpu io memory pids -``` +### Configure bridge network to route traffic through iptables -See the documentation on [cgroup controller requirements][] for more details. +@include 'install/bridge-iptables.mdx' - - +### Verify cgroup controllers +@include 'install/cgroup-controllers.mdx' ## Verify the Installation @@ -502,4 +308,4 @@ and ensuring `GOPATH/bin` is within your `PATH`. A copy of [gpg-key]: https://apt.releases.hashicorp.com/gpg "HashiCorp GPG key" [go-version]: https://github.com/hashicorp/nomad/blob/main/.go-version -[cgroup controller requirements]: /nomad/docs/install/production/requirements#hardening-nomad + diff --git a/website/content/docs/install/production/requirements.mdx b/website/content/docs/install/production/requirements.mdx index c6a551e75b8..559f3ab4902 100644 --- a/website/content/docs/install/production/requirements.mdx +++ b/website/content/docs/install/production/requirements.mdx @@ -103,52 +103,11 @@ $ echo "49152 65535" > /proc/sys/net/ipv4/ip_local_port_range ## Bridge Networking and `iptables` -Nomad's task group networks and Consul Connect integration use bridge networking and iptables to send traffic between containers. The Linux kernel bridge module has three "tunables" that control whether traffic crossing the bridge are processed by iptables. Some operating systems (RedHat, CentOS, and Fedora in particular) configure these tunables to optimize for VM workloads where iptables rules might not be correctly configured for guest traffic. - -These tunables can be set to allow iptables processing for the bridge network as follows: - -```shell-session -$ echo 1 > /proc/sys/net/bridge/bridge-nf-call-arptables -$ echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables -$ echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables -``` - -To preserve these settings on startup of a client node, add a file including the following to `/etc/sysctl.d/` or remove the file your Linux distribution puts in that directory. - -```text -net.bridge.bridge-nf-call-arptables = 1 -net.bridge.bridge-nf-call-ip6tables = 1 -net.bridge.bridge-nf-call-iptables = 1 -``` +@include 'install/bridge-iptables.mdx' ## Cgroup Controllers -On Linux, Nomad uses cgroups to control access to resources like CPU and -memory. Nomad can support both [cgroups v2][] and the legacy [cgroups -v1][]. When Nomad clients start, they will determine the available cgroup -controllers and include the attribute `os.cgroups.version` in their -fingerprint. - -Nomad can only use cgroups to control resources if all the required controllers -are available. If one or more required cgroups are unavailable, Nomad will -disable resource controls that require cgroups entirely. You will most often see -missing controllers on platforms used outside of datacenters, such as Raspberry -Pi or similar hobbyist computers. - -On cgroups v2, you can verify that you have all required controllers as follows: - -```shell-session -$ cat /sys/fs/cgroup/cgroup.controllers -cpuset cpu io memory pids -``` - -On legacy cgroups v1, you can look for this same list of required controllers as -directories under the directory `/sys/fs/cgroup`. - -To enable missing cgroups, add the appropriate boot command line arguments. For -example, to enable the `cpuset` cgroup, you'll need to add `cgroup_cpuset=1 -cgroup_enable=cpuset`. These arguments should be added wherever specified by -your bootloader. +@include 'install/cgroup-controllers.mdx' ## Hardening Nomad @@ -260,5 +219,3 @@ in automated pipelines for [CLI operations][docs_cli], such as [`nomad fmt`]: /nomad/docs/commands/fmt [mTLS]: /nomad/tutorials/transport-security/security-enable-tls [ephemeral disk migration]: /nomad/docs/job-specification/ephemeral_disk#migrat -[cgroups v1]: https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/cgroups.html -[cgroups v2]: https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html diff --git a/website/content/docs/integrations/consul/service-mesh.mdx b/website/content/docs/integrations/consul/service-mesh.mdx index 6bd63c2757d..21b0afd150a 100644 --- a/website/content/docs/integrations/consul/service-mesh.mdx +++ b/website/content/docs/integrations/consul/service-mesh.mdx @@ -188,7 +188,7 @@ Consul service mesh sidecar proxy. All Nomad client nodes using network namespac must have these CNI plugins [installed][cni_install]. To use [`transparent_proxy`][] mode, Nomad client nodes will also need the -[`consul-cni`][] plugin installed. See the Linux post-installation [steps](/nomad/docs/install#post-installation-steps) for more detail on how to install CNI plugins. +[`consul-cni`][] plugin installed. See the Linux post-installation [steps](/nomad/docs/install#linux-post-installation-steps) for more detail on how to install CNI plugins. ## Run the Service Mesh-enabled Services @@ -503,7 +503,7 @@ nomad node meta apply -node-id $nodeID \ [anon_token]: /consul/docs/security/acl/acl-tokens#special-purpose-tokens [consul_ports]: /consul/docs/agent/config/config-files#ports [consul_grpc_tls]: /consul/docs/upgrading/upgrade-specific#changes-to-grpc-tls-configuration -[cni_install]: /nomad/docs/install#post-installation-steps +[cni_install]: /nomad/docs/install#linux-post-installation-steps [transparent proxy]: /consul/docs/k8s/connect/transparent-proxy [go-sockaddr/template]: https://pkg.go.dev/github.com/hashicorp/go-sockaddr/template [`recursors`]: /consul/docs/agent/config/config-files#recursors diff --git a/website/content/docs/networking/cni.mdx b/website/content/docs/networking/cni.mdx index 2d8d2233ccb..0cee27400fc 100644 --- a/website/content/docs/networking/cni.mdx +++ b/website/content/docs/networking/cni.mdx @@ -1,93 +1,97 @@ --- layout: docs -page_title: CNI +page_title: CNI plugins and bridge networking description: |- - Learn about Nomad's runtime environment variables, interpolation, caveats, - and more. + Learn how to install Container Network Interface (CNI) reference plugins, configure Nomad's bridge networking, and how to use CNI networks with Nomad jobs. --- # Container Network Interface (CNI) -Nomad has built-in support for scheduling compute resources such as -CPU, memory, and networking. Nomad's network plugin support extends -this to allow scheduling tasks with purpose-created or specialty network -configurations. Network plugins are third-party plugins that conform to the -[Container Network Interface (CNI)][cni_spec] specification. +This page show you how to install the [CNI reference plugins][cni-plugin-docs] +on your Linux distribution and configure bridge networking on your Nomad +clients. You can apply this guide's workflow to installing any plugin that +complies with the [Container Network Interface (CNI) Specification][cni-spec], +but you should verify plugin compatibility with Nomad before deploying in +production. -Network plugins need to be installed and configured on each client. The [Nomad -installation instructions][nomad_install] recommend installing the [CNI -reference plugins][cni_ref] because certain Nomad networking features, like -`bridge` network mode and Consul service mesh, leverage them to provide an -operating-system agnostic interface to configure workload networking. +## Introduction -Custom networking in Nomad is accomplished with a combination of CNI plugin -binaries and CNI configuration files. +The CNI specification defines a network configuration format for administrators. +The configuration contains directives for both the orchestrator and the plugins +to consume. Nomad implements custom networking through a combination of CNI +reference plugin binaries and CNI configuration files. Networking features, like +bridge network mode and Consul service mesh, leverage the CNI reference plugins +to provide an operating-system agnostic interface to configure workload +networking. -## CNI reference plugins +## Requirements -The CNCF maintains a set of reference CNI plugins for basic network operations. -These plugins are required for Nomad's `bridge` networking mode and for integrating -with Consul service mesh. +- You are familiar with [CNI reference plugins][cni-plugin-docs]. +- You are familiar with [how Nomad uses Container Network Interface (CNI) plugins for bridge networking](/nomad/docs/networking#bridge-networking). +- You are running Nomad on Linux. -See the Linux [post-install steps][cni_install] for installing CNI reference plugins. +## CNI plugins and bridge networking workflow -To use Nomad's [`transparent_proxy`][] feature, you will also need the -[`consul-cni`][] plugin. +Perform the following on each Nomad client: -## CNI plugins +1. [Install CNI reference plugins](#install-cni-reference-plugins). +1. [Configure bridge module to route traffic through iptables](#configure-bridge-module-to-route-traffic-through-iptables). +1. [Create a bridge mode configuration](#create-a-cni-bridge-mode-configuration). +1. [Configure Nomad clients](#configure-nomad-clients). -Spec-compliant plugins should work with Nomad, however, it's possible a plugin -vendor has implemented their plugin to make non-standard API calls, or it is -otherwise non-compliant with the CNI specification. In those situations the -plugin may not function correctly in a Nomad environment. You should verify -plugin compatibility with Nomad before deploying in production. +After you configure and restart your Nomad clients, [use a CNI network with a +job](#use-a-cni-network-with-a-job). -CNI plugins are installed and configured on a per-client basis. Nomad consults -the path given in the client's [`cni_path`][] to find CNI plugin executables. +### Install CNI reference plugins -## CNI configuration files +@include 'install/install-cni-plugins.mdx' -The CNI specification defines a network configuration format for administrators. -It contains directives for both the orchestrator and the plugins to consume. -At plugin execution time, this configuration format is interpreted by Nomad -and transformed into arguments for the plugins. +### Configure bridge module to route traffic through iptables -Nomad reads the following extensions from the [`cni_config_dir`][]— -`/opt/cni/config` by default: +@include 'install/bridge-iptables.mdx' -* `.conflist` files are loaded as [network - configurations][cni_spec_net_config] that contain a list of plugin - configurations. +## Create a CNI bridge mode configuration + +Nomad itself uses CNI plugins and configuration as the underlying implementation +for the `bridge` network mode, using the loopback, [bridge][], [firewall][], and +[portmap][] CNI reference plugins configured together to create Nomad's bridge +network. -* `.conf` and `.json` files are loaded as individual [plugin - configurations][cni_spec_plugin_config] for a specific network. +[comment-source-image]: + https://www.figma.com/file/Ne2qaPUlBTmTYer9biCfK9/Networking?node-id=0%3A1&t=BepgOoQ0kb76GwIr-1 -## Using CNI networks with Nomad jobs +[![Visual example of Nomad bridge +network](/img/nomad-bridge-network.png)](/img/nomad-bridge-network.png) -To specify that a job should use a CNI network, set the task group's -network [`mode`][] attribute to the value `cni/`. -Nomad will then schedule the workload on client nodes that have fingerprinted a -CNI configuration with the given name. For example, to use the configuration -named `mynet`, you should set the task group's network mode to `cni/mynet`. -Nodes that have a network configuration defining a network named `mynet` in -their [`cni_config_dir`][] will be eligible to run the workload. +When setting up a bridge network, Nomad uses a configuration template based on +the CNI Specification's [example +configuration](https://www.cni.dev/docs/spec/#example-configuration). Refer to +the external [configuration +format](https://www.cni.dev/docs/spec/#example-configuration) for a complete +explanation of the fields. -## Nomad's `bridge` configuration +You can use this template as a basis for your own CNI-based bridge network +configuration in cases where you need access to an unsupported option in the +default configuration, like hairpin mode. -Nomad itself uses CNI plugins and configuration as the underlying implementation -for the `bridge` network mode, using the [loopback][], [bridge][], [firewall][], -and [portmap][] CNI plugins configured together to create Nomad's bridge -network. +This example uses two default values from Nomad client configuration. -The following is the configuration template Nomad uses when setting up these -networks with the template placeholders replaced with the default configuration -values for [`bridge_network_name`][], [`bridge_network_subnet`][], and an -internal constant that provides the value for `iptablesAdminChainName`. This is -a convenient jumping off point to discuss a worked example. +- The default value for + [`bridge_network_name`](/nomad/docs/configuration/client#bridge_network_name) + is the value for the bridge plugin name. +- The default value for bridge subnet [`bridge_network_subnet`]( + /nomad/docs/configuration/client#bridge_network_subnet) is the bridge plugin + subnet. + +The `NOMAD-ADMIN` internal constant provides the value for +`iptablesAdminChainName`. In your own configuration, ensure that you change the +`iptablesAdminChainName` to a unique value. + + ```json { - "cniVersion": "0.4.0", + "cniVersion": "1.0.0", "name": "nomad", "plugins": [ { @@ -128,105 +132,129 @@ a convenient jumping off point to discuss a worked example. } ``` - -[![Visual example of Nomad bridge network](/img/nomad-bridge-network.png)](/img/nomad-bridge-network.png) - -For a more thorough understanding of this configuration, consider each CNI -plugin's configuration in turn. - -### loopback - -The `loopback` plugin sets the default local interface, `lo0`, created inside -the bridge network's network namespace to **UP**. This allows workload running -inside the namespace to bind to a namespace-specific loopback interface. - -### bridge - -The `bridge` plugin creates a bridge (virtual switch) named `nomad` that resides -in the host network namespace. Because this bridge is intended to provide -network connectivity to allocations, it's configured to be a gateway by setting -`isGateway` to `true`. This tells the plugin to assign an IP address to the -bridge interface - -The bridge plugin connects allocations (on the same host) into a bridge (virtual -switch) that resides in the host network namespace. By default Nomad creates a -single bridge for each client. Since Nomad's bridge network is designed to -provide network connectivity to the allocations, it configures the bridge -interface to be a gateway for outgoing traffic by providing it with an address -using an `ipam` configuration. The default configuration creates a host-local -address for the host side of the bridge in the `172.26.64.0/20` subnet at -`172.26.64.1`. When associating allocations to the bridge, it creates addresses -for the allocations from that same subnet using the host-local plugin. The -configuration also specifies a default route for the allocations of the -host-side bridge address. - -### firewall - -The firewall plugin creates firewall rules to allow traffic to/from the -allocation's IP address via the host network. Nomad uses the iptables backend -for the firewall plugin. This configuration creates two new iptables chains, -`CNI-FORWARD` and `NOMAD-ADMIN`, in the filter table and add rules that allow -the given interface to send/receive traffic. - -The firewall creates an admin chain using the name provided in the -`iptablesAdminChainName` attribute. For this case, it's called `NOMAD-ADMIN`. -The admin chain is a user-controlled chain for custom rules that run before -rules managed by the firewall plugin. The firewall plugin does not add, delete, -or modify rules in the admin chain. - -A new chain, `CNI-FORWARD` is added to the `FORWARD` chain. `CNI-FORWARD` is -the chain where rules will be added when allocations are created and from where -rules will be removed when those allocations stop. The `CNI-FORWARD` chain -first sends all traffic to `NOMAD-ADMIN` chain. - -You can use the following command to list the iptables rules present in each -chain. - -```shell-session -$ sudo iptables -L -``` + + +This configuration uses the following CNI reference plugins: + +- loopback: The loopback plugin sets the default local interface, lo0, created + inside the bridge network's network namespace to UP. This allows workload + running inside the namespace to bind to a namespace-specific loopback + interface. +- bridge: The [bridge][] plugin creates a bridge (virtual switch) named `nomad` + that resides in the host network namespace. Because Nomad intends this bridge + to provide network connectivity to allocations, we configured it to be a + gateway by setting `isGateway` to `true`. This tells the plugin to assign an + IP address to the bridge interface. + + The bridge plugin connects allocations on the same host into a virtual switch + bridge that resides in the host network namespace. By default, Nomad creates + a single bridge for each client. + + Since Nomad's bridge network is designed to provide network connectivity to + the allocations, Nomad configures the bridge interface to be a gateway for + outgoing traffic by providing it with an address using an `ipam` + configuration. The default configuration creates a host-local address for the + host side of the bridge in the `172.26.64.0/20` subnet at `172.26.64.1`. + + When associating allocations to the bridge, Nomad creates addresses for the + allocations from that same subnet using the host-local plugin. The + configuration also specifies a default route for the allocations of the + host-side bridge address. +- firewall: The [firewall][] plugin creates firewall rules to allow traffic to + and from the allocation's IP address via the host network. Nomad uses the + iptables backend for the firewall plugin. + + This configuration creates two new iptables chains, `CNI-FORWARD` and + `NOMAD-ADMIN`, in the filter table and adds rules that allow the given + interface to send and receive traffic. + + The firewall creates an admin chain using the `iptablesAdminChainName` value, + which is `NOMAD-ADMIN` in this example. The admin chain is a user-controlled + chain for custom rules that run before rules managed by the firewall plugin. + The firewall plugin does not add, delete, or modify rules in the admin chain. -### portmap + Nomad adds a new chain called `CNI-FORWARD` to to the `FORWARD` chain. + `CNI-FORWARD` is the chain where Nomad adds rules when it creates allocations + and removes the rules when those allocations stop. The `CNI-FORWARD` chain + first sends all traffic to the `NOMAD-ADMIN` chain. -Nomad needs to be able to map specific ports from the host to tasks running in -the allocation namespace. The `portmap` plugin forwards traffic from one or more -ports on the host to the allocation using network address translation (NAT) -rules. + Use the `iptables` command to list the iptables rules present in each chain. -The plugin sets up two sequences of chains and rules: + ```shell-session + $ sudo iptables -L + ``` -- One “primary” `DNAT` (destination NAT) sequence to rewrite the destination. -- One `SNAT` (source NAT) sequence that will masquerade traffic as needed. +- portmap: Nomad needs to be able to map specific ports from the host to tasks + running in the allocation namespace. The [portmap][] plugin forwards traffic + from one or more ports on the host to the allocation using network address + translation (NAT) rules. -You can use the following command to list the iptables rules in the NAT table. + The plugin sets up two sequences of chains and rules: -```shell-session -$ sudo iptables -t nat -L + - One primary destination NAT (DNAT) sequence to rewrite the destination. + - One source NAT (SNAT) sequence to masquerade traffic as needed. + + Use the `iptables` command to list the iptables rules in the NAT table. + + ```shell-session + $ sudo iptables -t nat -L + ``` + +Save your bridge network configuration file to a Nomad-accessible directory. By +default, Nomad loads configuration files from the `/opt/cni/config` directory. +However, you may configure a different location using the +[`cni_config_dir`](/nomad/docs/configuration/client#cni_config_dir) parameter. +Refer to the [Configure Nomad clients](#configure-nomad-clients) section for an +example. + +## Configure Nomad clients + +At plugin execution time, Nomad interprets your CNI network configuration and +transforms it into arguments for the plugins. + +Nomad reads the following files from the +[`cni_config_dir`](/nomad/docs/configuration/client#cni_config_dir) parameter — +`/opt/cni/config` by default: + +- `.conflist`: Nomad loads these files as [network + configurations](https://www.cni.dev/docs/spec/#configuration-format) that + contain a list of plugin configurations. + +- `.conf` and `.json`: Nomad loads these files as individual [plugin + configurations](https://www.cni.dev/docs/spec/#plugin-configuration-objects) + for a specific network. + +Add the [`cni_path`](/nomad/docs/configuration/client#cni_path) and +[`cni_config_dir`](/nomad/docs/configuration/client#cni_config_dir) attributes +to each client's `client.hcl` file. + +This example uses the default values for both attributes. + + + +```hcl +client { + enabled = true + cni_path = "opt/cni/bin" + cni_config_dir = "opt/cni/config" +} ``` -## Create your own + -You can use this template as a basis for your own CNI-based bridge network -configuration in cases where you need access to an unsupported option in the -default configuration, like hairpin mode. When making your own bridge network -based on this template, ensure that you change the `iptablesAdminChainName` to -a unique value for your configuration. - -[3rd_party_cni]: https://www.cni.dev/docs/#3rd-party-plugins -[`bridge_network_name`]: /nomad/docs/configuration/client#bridge_network_name -[`bridge_network_subnet`]: /nomad/docs/configuration/client#bridge_network_subnet -[`cni_config_dir`]: /nomad/docs/configuration/client#cni_config_dir -[`cni_path`]: /nomad/docs/configuration/client#cni_path -[`mode`]: /nomad/docs/job-specification/network#mode +## Use a CNI network with a job + +To specify that a job should use a CNI network, set the task group's network +[`mode`](/nomad/docs/job-specification/network#mode) attribute to the value +`cni/`. Nomad then schedules the workload on client nodes +that have fingerprinted a CNI configuration with the given name. For example, to +use the configuration named `mynet`, you should set the task group's network +mode to `cni/mynet`. Nodes that have a network configuration defining a network +named `mynet` in their `cni_config_dir` are eligible to run the workload. + + +[cni-spec]: https://www.cni.dev/docs/spec/ +[cni-plugin-docs]: https://www.cni.dev/plugins/current/ [bridge]: https://www.cni.dev/plugins/current/main/bridge/ -[cni_install]: /nomad/docs/install#post-installation-steps -[cni_ref]: https://github.com/containernetworking/plugins -[cni_spec]: https://www.cni.dev/docs/spec/ -[cni_spec_net_config]: https://github.com/containernetworking/cni/blob/main/SPEC.md#configuration-format -[cni_spec_plugin_config]: https://github.com/containernetworking/cni/blob/main/SPEC.md#plugin-configuration-objects [firewall]: https://www.cni.dev/plugins/current/meta/firewall/ -[loopback]: https://github.com/containernetworking/plugins#main-interface-creating -[nomad_install]: /nomad/tutorials/get-started/get-started-install#post-installation-steps [portmap]: https://www.cni.dev/plugins/current/meta/portmap/ -[`transparent_proxy`]: /nomad/docs/job-specification/transparent_proxy -[`consul-cni`]: https://releases.hashicorp.com/consul-cni diff --git a/website/content/docs/networking/index.mdx b/website/content/docs/networking/index.mdx index 96c6e302a6d..4a5aa1c5fdd 100644 --- a/website/content/docs/networking/index.mdx +++ b/website/content/docs/networking/index.mdx @@ -8,26 +8,28 @@ description: |- # Networking -Nomad is a workload orchestrator and so it focuses on the scheduling aspects of -a deployment, touching areas such as networking as little as possible. +Nomad is a workload orchestrator and focuses on the scheduling aspects of a +deployment, touching areas such as networking as little as possible. -**Networking in Nomad is usually done via _configuration_ instead of -_infrastructure_**. This means that Nomad provides ways for you to access the +Networking in Nomad is usually done via _configuration_ instead of +_infrastructure_. This means that Nomad provides ways for you to access the information you need to connect your workloads instead of running additional components behind the scenes, such as DNS servers and load balancers. -This can be confusing at first since it is quite different from what you may -be used to from other tools. This section explains how networking works in -Nomad, some of the different patterns and configurations you are likely to find -and use, and how Nomad differs from other tools in this aspect. +Nomad networking is quite different from what you may familiar with from other +tools. This section explains how networking works in Nomad, some of the +different patterns and configurations you are likely to find and use, and how +Nomad differs from other tools in this aspect. ## Allocation networking -The base unit of scheduling in Nomad is an [allocation][], which means that all +The base unit of scheduling in Nomad is an +[allocation](/nomad/docs/concepts/architecture#allocation), which means that all tasks in the same allocation run in the same client and share common resources, such as disk and networking. Allocations can request access to network -resources, such as ports, using the [`network`][jobspec_network] block. At its -simplest configuration, a `network` block can be defined as: +resources, such as ports, using the +[`network`](/nomad/docs/job-specification/network) block. You can define a basic +`network` block as the following: ```hcl job "..." { @@ -41,20 +43,27 @@ job "..." { } ``` -Nomad reserves a random port in the client between [`min_dynamic_port`][] and -[`max_dynamic_port`][] that has not been allocated yet and creates a port -mapping from the host network interface to the allocation. +Nomad reserves a random port in the client between +[`min_dynamic_port`](/nomad/docs/configuration/client#min_dynamic_port) and +[`max_dynamic_port`](/nomad/docs/configuration/client#max_dynamic_port) that has +not been allocated yet. Nomad then creates a port mapping from the host network +interface to the allocation. - -[![Nomad Port Mapping](/img/networking/port_mapping.png)](/img/networking/port_mapping.png) +[comment-image-source]: + https://drive.google.com/file/d/1q4a2ab0TyLEPdWiO2DIianAPWuPqLqZ4/view?usp=share_link -The selected port number can be accessed by tasks using the -[`NOMAD_PORT_