Allow to configure custom network adapter labels

EVE already offers adapter labels "uplink" and "freeuplink" to match
a specific group of ports to provide external connectivity for a network
instance.

We should allow users to define and assign their own adapter labels and
support the following new network instance use-cases:

1. Switch network instance with multiple ports: user should be able to
   select a group of ports, all of which will be bridged together under
   one switch NI. EVE should then automatically run the STP protocol for
   the bridge to avoid bridge loops and broadcast storms.
2. Local network instance with multiple ports: the routing table on the
   host (EVE) side will contain routes from all selected ports and will
   be used to select output port for the given flow.
   For the default route we should:
     a) allow the user to pick the "default output interface"
     b) periodically test port connectivity and failover between default
        gateways when currently used one has broken connectivity
   a) and b) could be combined together to allow the user to select a
   subset of NI ports for the default route and then let EVE to pick one
   of them based on the probing results.
   We should also allow to limit a port-forwarding rule to only a subset
   of ports used by a local network instance. This can be done by
   defining a new ACEMatch type "adapter" where value is adapter label
   to match.

Currently, EVE supports a variant of the second use-case for hard-coded
"uplink" and "freeuplink" labels. It would make sense to unify the behaviour
between these existing labels and the user-defined ones. This means that
we would make small change to the current implementation:
when "uplink" or "freeuplink" is selected for a local network instance,
the NI routing table will have IP routes from *all* matching (mgmt)
ports and the default route will be selected by connectivity testing.
This is different to the present behaviour, where EVE installs only
routes of the port selected by connectivity testing into the NI routing
table. But lets say that there are 2 mgmt ports: eth0 and eth1, and
connectivity testing selected eth0. Then even if app is accessing hosts
which are directly inside the eth1 subnet, traffic will be routed via eth0,
which can be confusing. Also, user-defined static routes for eth1 are
not present in the routing table and therefore inactive when eth0 is
selected. The idea is to apply connectivity testing to only default
route selection.

Another change in behaviour is wrt. the inbound (port forwarding) rules.
Without the newly introduced "adapter" label specified (e.g. when EVE is
upgraded but config remains the same), the inbound rules will be applied
to all the ports used by a network instance. With "uplink" labels this
means that port forwarding will be enabled on all the management ports,
not just on the one with the default route activated. However, this new
behaviour is actually more practical for the user. Accessing applications
will be possible through any of the mgmt ports, without having to know
which port is being used by the NI for the default route. After all,
NI with the "uplink" label is configured whichever mgmt port is working,
meaning that the user should ensure that opening ports on any of them
is safe in their environment. It therefore makes no difference if port
forwarding rules are always activated on all the mgmt ports.

Signed-off-by: Milan Lenco <[email protected]>

proto/config/devmodel.proto

@@ -75,6 +75,18 @@ message SystemAdapter {
   // Load spreading will apply when multiple adapters have the same cost.
   // Higher cost adapters are only tried when none of the lower cost ones work.
   uint32 cost = 9;
+
+  // A set of user-defined labels attached to the adapter.
+  // There are no restrictions on the format of an adapter label, it can be any
+  // non-empty string.
+  // Note that EVE automatically assigns "all" label to every port, "uplink" label to every
+  // management port and "freeuplink" label to every management port with zero cost.
+  // ("automatically assigns" is meant here in imaginary sense, these labels are not supposed
+  // to be explicitly added by the controller to SystemAdapters inside EdgeDevConfig).
+  // It can be used to designate a group of network adapters to be used by a network
+  // instances for external connectivity, or to limit port-forwarding firewall rule
+  // to only a subset of network adapters.
+  repeated string labels = 10;
 }
 
 // Given additional details for EVE software to how to treat this

proto/config/fw.proto

@@ -9,7 +9,46 @@ option java_package = "org.lfedge.eve.config";
 
 
 message ACEMatch {
-  // FIXME: We should convert this to enum
+  // Supported ACE match types:
+  //   * "ip": value should be an IP address of a remote endpoint. The match is satisfied
+  //           for outbound and inbound flow if the destination and the source IP address
+  //           matches the given value, respectively. Can be combined with any other match
+  //           type to further narrow down the selection criteria.
+  //   * "host": value should be a domain name of a remote endpoint. It can be either a fully
+  //             qualified, or a partially qualified domain name (FQDN or PQDN). A packet is
+  //             matched if it is destined to or originated from an IP address that was obtained
+  //             by a DNS query for that exact domain or any of its subdomains. For example,
+  //             match of type "host" with value "domain.com" will also apply to the endpoint
+  //             "subdomain.domain.com". Can be combined with other match types except for "eidset".
+  //   * "eidset": special match type for the overlay network. Matches IPs of all applications
+  //               deployed in the same network as well as all IPs with statically configured
+  //               DNS entries (under the config field NetworkInstanceConfig.Dns). For this type,
+  //               value field is not used. Can be combined with other match types except for
+  //               "host".
+  //   * "protocol": value should specify the protocol to match. Protocol can be one of "tcp",
+  //                 "udp", "icmp", or "all", or it can be a numeric value, representing one
+  //                 of these protocols or a different one. A protocol name from /etc/protocols
+  //                 is also allowed. Protocol match can be combined with any other match type
+  //                 (often combined with port numbers).
+  //   * "lport": value should be an application local port number. For filtering actions,
+  //              this is the source port for outbound traffic and destination port for inbound
+  //              traffic. For PORTMAP action, this represents application port as exposed
+  //              to the external network (i.e., if <edge-node-ip>:2222 is mapped to <app-ip>:22,
+  //              lport refers to 2222). lport can be combined with any other match type.
+  //              It is actually required to combine "lport" and "protocol" inside the same ACE.
+  //              In other words, port without protocol is not valid.
+  //   * "fport": value should be a remote endpoint port number (foreign port). Used for filtering
+  //              actions, but not for PORTMAP (do not confuse with "lport", which is still used to
+  //              represent the forwarded port - the forwarded port is still considered as local).
+  //              "fport" can be combined with any other match type. It is actually required
+  //              to combine "fport" with "protocol" inside the same ACE. In other words, port
+  //              without protocol is not valid.
+  //  * "adapter": value should be an adapter label (SystemAdapter.Labels). It can be used for
+  //               an inbound ACE to apply the rule only to packets arriving via one of the
+  //               matched network adapters. Typically used to activate a given port-forwarding
+  //               rule (PORTMAP) for only a subset of network adapters.
+  //               Adapter label cannot be used for outbound ACE. This is because the EVE firewall
+  //               is applied before routing, and the output network adapter is not yet known.
   string type = 1;
   string value = 2;
 }
@@ -39,6 +78,7 @@ message ACE {
   //  for example
   //     1) host=www.example.com & port=http
   //     2) ip=8.8.8.8 & port=53 & proto=UDP
+  //     3) adapter=uplink && port=8080 && proto=TCP
   repeated ACEMatch matches = 1;
 
   // Expect only single action...repeated here is

proto/config/netcmn.proto

@@ -114,3 +114,27 @@ enum WiFiKeyScheme {
   WPAPSK = 1;        // WPA-PSK
   WPAEAP = 2;        // WPA-EAP or WPA2 Enterprise
 }
+
+// User-defined method to use to determine the connectivity status.
+enum CustomProbeMethod {
+  // Custom probing method is not specified and therefore will not be used.
+  CUSTOM_PROBE_METHOD_UNSPECIFIED = 0;
+  // Use ICMP ping against the probed endpoint to determine the connectivity status.
+  CUSTOM_PROBE_METHOD_ICMP = 1;
+  // Try to establish TCP connection with the probed endpoint to determine the connectivity
+  // status.
+  CUSTOM_PROBE_METHOD_TCP = 2;
+}
+
+// Configuration for user-defined connectivity-testing probe.
+message CustomProbe {
+  // Method to use to determine the connectivity status.
+  CustomProbeMethod probe_method = 1;
+  // IP or FQDN to probe using the selected probing mechanism to determine the connectivity
+  // status.
+  // For CUSTOM_PROBE_METHOD_TCP the address should include port number in one of the formats:
+  //   ipv4:port
+  //   [ipv6]:port
+  //   hostname:port
+  string probe_address = 2;
+}

proto/config/netconfig.proto

@@ -101,17 +101,23 @@ message CellularConnectivityProbe {
   // If true, then probing is disabled.
   bool disable = 1;
   // IP/FQDN to periodically probe using 3 pings per iteration to determine connectivity status.
-  // The default probing behavior (i.e. without user-defined probing endpoint), is more complex
-  // that just pinging a remote endpoint and the goal is to minimize generated traffic,
-  // probing time and false negatives (that would trigger undesired re-connect).
+  // This field is now deprecated and replaced with custom_probe.
+  // To maintain backward-compatibility, EVE will check both probe_address and custom_probe
+  // and will use whichever has non-zero value (if any), but preferring custom_probe.
+  string probe_address = 2 [deprecated=true];
+  // User-defined probe for cellular connectivity testing.
+  // Only IP address is allowed as CustomProbe.probe_address (and port if needed), not hostname.
+  // The probing algorithm is more complex that just testing reachability of a remote endpoint
+  // and the goal is to minimize generated traffic, probing time and false negatives (that would
+  // trigger undesired re-connect).
   // EVE will periodically check the modem status, accessibility of DNS server(s) and potentially
   // also proxies in the local network, and only as the last resort (if connectivity state
-  // is still unclear) it will probe the default remote endpoint "8.8.8.8" (Google public DNS)
-  // using ICMP ping.
-  // It is recommended to use custom probe_address only for private LTE networks where
-  // public DNS servers are not allowed by the firewall (and hence ping would fail even with
-  // working connectivity).
-  string probe_address = 2;
+  // is still unclear) it will probe the remote endpoint. If custom_probe is not defined,
+  // EVE will try to ping "8.8.8.8" (Google public DNS).
+  // It is recommended to use custom probe for private LTE networks where public DNS servers
+  // might not be allowed by the firewall (and hence ping would fail even with a working
+  // connectivity).
+  CustomProbe custom_probe = 3;
 }
 
 // CellularAccessPoint contains config parameters for connecting to a cellular network.

proto/config/netinst.proto

@@ -85,6 +85,57 @@ message IPRoute {
   string gateway = 2;
 }
 
+// Configuration for port connectivity probe.
+message PortProbe {
+  // EVE uses ICMP ping against the port's gateway IP to determine connectivity status.
+  // Use disable_gw_ping to switch off this probing method. This is typically needed
+  // when the gateway router is configured to drop/ignore ICMP pings and therefore
+  // this probe would return false negatives.
+  bool disable_gw_ping = 1;
+  // User-defined method to use to determine the port connectivity status.
+  // Run additionally to gateway pings (unless turned off by disable_gw_ping).
+  // If custom probe is not specified and the port is marked as management, EVE will try
+  // to run HTTP GET request against the controller's URL.
+  // For app-shared ports there is no alternative default probing mechanism other than
+  // pinging the gateway IP.
+  CustomProbe custom_probe = 3;
+}
+
+// DefaultRouteSelection defines how local network instance picks the default route
+// when it has multiple ports assigned.
+message DefaultRouteSelection {
+  // Either a single NI port referenced by its name (SystemAdapter.Name) or an adapter label
+  // matching zero or more NI ports (NI ports containing this label in SystemAdapter.Labels).
+  // Empty default_port matches all NI ports.
+  // Currently, load-balancing is not supported and thus even if default_port matches multiple
+  // NI ports, EVE will pick only one of them for the default route. The selection is based on
+  // probing (if enabled, see below) or random (preferring the currently used one if any).
+  string default_port = 1;
+  // Probe remote endpoint to determine connectivity status of each port and pick one
+  // with a working connectivity (and known gateway IP) for the default route (preferring
+  // the currently used one if any).
+  // Provides automatic fail-over between ports.
+  // Probes NI ports, potentially further filtered down by default_port label (see above).
+  // If default_port references single port (e.g. directly by its name), probing is skipped
+  // (nothing to fail-over to anyway).
+  PortProbe probe = 2;
+  // TODO: Add config for load-balancing, applied when multiple ports match the criteria
+  //       for the default route selection (matching default_port label and have working
+  //       connectivity or probing is disabled).
+  //       For now we will assume that load-balancing is disabled and EVE will always pick
+  //       one port for the default route.
+}
+
+// Configuration for the Spanning Tree Protocol (STP), which is run for switch network
+// instance when the port label matches multiple network ports.
+message SpanningTreeProtocol {
+  // Ports matched by this adapter label will have the BPDU guard enabled.
+  // It makes sense to enable the guard on ports which are expected to be leafs
+  // and therefore not participate in STP, or where the downstream bridges are not fully
+  // trusted.
+  string ports_with_bpdu_guard = 1;
+}
+
 message NetworkInstanceConfig {
   UUIDandVersion uuidandversion = 1;
   string displayname = 2;
@@ -97,14 +148,37 @@ message NetworkInstanceConfig {
   //    itself is not created on the device.
   bool activate = 5;
 
-  // port - Only a single port is supported.
-  //    This is used as the external connection for the network instance.
-  //    This can be a physical (eth0 ) or logical port (vlan 0).
-  //    The port name comes from DeviceConfig ( When it is supported in future).
-  //    If the user needs multiple physical ports, Device config should be
-  //    used to create a label for multiple physical ports.
+  // Port(s) providing external connectivity for the network instance.
+  // Either a single port referenced by its name (SystemAdapter.Name) or an adapter label
+  // matching zero or more network ports (ports containing this label in SystemAdapter.Labels).
+  // Note that apart from the user-defined adapter labels, EVE automatically assigns "all"
+  // label to every port, "uplink" label to every management port and "freeuplink" label
+  // to every management port with zero cost.
+  // Both physical (e.g. eth0) and logical (e.g. vlan0) ports are allowed.
+  // However, EVE may filter out some of the matched ports which are not compatible
+  // with the network instance type (e.g. wireless adapters cannot be used with switch NI
+  // and ports without IP addresses are not usable for local NIs).
+  // Network instance with no assigned port (empty port reference or label not matching
+  // any port) is air-gapped, i.e. completely isolated from external networks.
+  // With multiple ports assigned, the behaviour depends on the network instance type.
+  // Switch network instance will run Rapid Spanning Tree Protocol (RSTP) to avoid bridge loops
+  // and the broadcast storm that results from them.
+  // Local network instance will apply IP routes configured for the matched ports (either
+  // statically or via DHCP) to select the next hop and the output port for the given flow.
+  // Traffic not matching any link-local, statically-configured, or DHCP-received route,
+  // will be routed according to the default route, which is selected based on the
+  // defaultRouteSelection algorithm (see below).
   Adapter port = 20;
 
+  // DefaultRouteSelection defines how local network instance picks the default route
+  // when it has multiple ports assigned.
+  // Not used with switch network instance.
+  DefaultRouteSelection default_route_selection = 21;
+
+  // Configuration for the Spanning Tree Protocol (STP), which is run for switch network
+  // instance when the port label matches multiple network ports.
+  SpanningTreeProtocol stp = 22;
+
   // cfg - Used to pass some feature-specific configuration to the
   //       network instance. For Ex: Lisp, StriongSwan etc
   NetworkInstanceOpaqueConfig cfg = 30;

proto/info/info.proto

@@ -670,6 +670,7 @@ enum APICapability {
   API_CAPABILITY_VOLUME_SNAPSHOTS = 5; // Volume snapshots supported
   API_CAPABILITY_NETWORK_INSTANCE_ROUTING = 6; // routing config in NetworkInstanceConfig supported
   API_CAPABILITY_BOOT_MODE = 7; // Support different boot modes for Edge Applications (VMs)
+  API_CAPABILITY_ADAPTER_LABELS = 8; // Supports user-defined network adapter labels
   // Add new values as new EdgeDevConfig API features are implemented
 }