Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation for BGP support #441

Merged
merged 2 commits into from
Jun 27, 2024
Merged

Add documentation for BGP support #441

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
aa1b8e1
Add documentation for BGP support
wenovus Jun 27, 2024
270454a
Try to disable indenting lint
wenovus Jun 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 85 additions & 0 deletions bgp/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# BGP Reconciler

## OpenConfig BGP Paths Supported by Lemming (as of 2024-06-27)

### Session Establishment

```text
/network-instances/network-instance/protocols/protocol/bgp/global/config/as
/network-instances/network-instance/protocols/protocol/bgp/global/config/router-id
/network-instances/network-instance/protocols/protocol/bgp/neighbors/neighbor/config/peer-as
/network-instances/network-instance/protocols/protocol/bgp/neighbors/neighbor/config/neighbor-address
/network-instances/network-instance/protocols/protocol/bgp/neighbors/neighbor/config/neighbor-port
/network-instances/network-instance/protocols/protocol/bgp/neighbors/neighbor/transport/config/local-address
```

### BGP Policy

```text
/routing-policy/defined-sets/prefix-sets/prefix-set/config/name
/routing-policy/defined-sets/prefix-sets/prefix-set/config/mode
/routing-policy/defined-sets/prefix-sets/prefix-set/state/name
/routing-policy/defined-sets/prefix-sets/prefix-set/state/mode
/routing-policy/defined-sets/prefix-sets/prefix-set/prefixes/prefix/ip-prefix
/routing-policy/defined-sets/prefix-sets/prefix-set/prefixes/prefix/masklength-range
/routing-policy/defined-sets/prefix-sets/prefix-set/prefixes/prefix/config/ip-prefix
/routing-policy/defined-sets/prefix-sets/prefix-set/prefixes/prefix/config/masklength-range
/routing-policy/defined-sets/prefix-sets/prefix-set/prefixes/prefix/state/ip-prefix
/routing-policy/defined-sets/prefix-sets/prefix-set/prefixes/prefix/state/masklength-range
/routing-policy/defined-sets/bgp-defined-sets/community-sets/community-set/config/community-set-name
/routing-policy/defined-sets/bgp-defined-sets/community-sets/community-set/config/community-member
/routing-policy/defined-sets/bgp-defined-sets/community-sets/community-set/state/community-set-name
/routing-policy/defined-sets/bgp-defined-sets/community-sets/community-set/state/community-member
/routing-policy/defined-sets/bgp-defined-sets/as-path-sets/as-path-set/config/as-path-set-name
/routing-policy/defined-sets/bgp-defined-sets/as-path-sets/as-path-set/config/as-path-set-member
/routing-policy/defined-sets/bgp-defined-sets/as-path-sets/as-path-set/state/as-path-set-name
/routing-policy/defined-sets/bgp-defined-sets/as-path-sets/as-path-set/state/as-path-set-member

/routing-policy/policy-definitions/policy-definition/statements/statement/config/name
/routing-policy/policy-definitions/policy-definition/statements/statement/conditions/match-prefix-set/config/prefix-set
/routing-policy/policy-definitions/policy-definition/statements/statement/conditions/match-prefix-set/config/match-set-options
/routing-policy/policy-definitions/policy-definition/statements/statement/conditions/bgp-conditions/config/med-eq
/routing-policy/policy-definitions/policy-definition/statements/statement/conditions/bgp-conditions/config/origin-eq
/routing-policy/policy-definitions/policy-definition/statements/statement/conditions/bgp-conditions/config/local-pref-eq
/routing-policy/policy-definitions/policy-definition/statements/statement/conditions/bgp-conditions/config/route-type
/routing-policy/policy-definitions/policy-definition/statements/statement/conditions/bgp-conditions/config/community-set
/routing-policy/policy-definitions/policy-definition/statements/statement/conditions/bgp-conditions/config/next-hop-in
/routing-policy/policy-definitions/policy-definition/statements/statement/conditions/bgp-conditions/community-count/config/operator
/routing-policy/policy-definitions/policy-definition/statements/statement/conditions/bgp-conditions/community-count/config/value
/routing-policy/policy-definitions/policy-definition/statements/statement/conditions/bgp-conditions/match-as-path-set/config/as-path-set
/routing-policy/policy-definitions/policy-definition/statements/statement/conditions/bgp-conditions/match-as-path-set/config/match-set-options
/routing-policy/policy-definitions/policy-definition/statements/statement/conditions/bgp-conditions/match-community-set/config/community-set
/routing-policy/policy-definitions/policy-definition/statements/statement/conditions/bgp-conditions/match-community-set/config/match-set-options
/routing-policy/policy-definitions/policy-definition/statements/statement/actions/bgp-actions/config/set-route-origin
/routing-policy/policy-definitions/policy-definition/statements/statement/actions/bgp-actions/config/set-local-pref
/routing-policy/policy-definitions/policy-definition/statements/statement/actions/bgp-actions/config/set-next-hop
/routing-policy/policy-definitions/policy-definition/statements/statement/actions/bgp-actions/config/set-med
/routing-policy/policy-definitions/policy-definition/statements/statement/actions/bgp-actions/set-as-path-prepend/config/repeat-n
/routing-policy/policy-definitions/policy-definition/statements/statement/actions/bgp-actions/set-as-path-prepend/config/asn
/routing-policy/policy-definitions/policy-definition/statements/statement/actions/bgp-actions/set-community/config/method
/routing-policy/policy-definitions/policy-definition/statements/statement/actions/bgp-actions/set-community/config/options
/routing-policy/policy-definitions/policy-definition/statements/statement/actions/bgp-actions/set-community/inline/config/communities
/routing-policy/policy-definitions/policy-definition/statements/statement/actions/bgp-actions/set-community/reference/config/community-set-ref (deprecated)
/routing-policy/policy-definitions/policy-definition/statements/statement/actions/bgp-actions/set-community/reference/config/community-set-refs
```

### BGP RIB

```text
/network-instances/network-instance/protocols/protocol/bgp/rib/attr-sets/attr-set/index
/network-instances/network-instance/protocols/protocol/bgp/rib/attr-sets/attr-set/state/index
/network-instances/network-instance/protocols/protocol/bgp/rib/attr-sets/attr-set/state/origin
/network-instances/network-instance/protocols/protocol/bgp/rib/attr-sets/attr-set/state/med
/network-instances/network-instance/protocols/protocol/bgp/rib/attr-sets/attr-set/state/local-pref
/network-instances/network-instance/protocols/protocol/bgp/rib/attr-sets/attr-set/as-path/as-segment/index
/network-instances/network-instance/protocols/protocol/bgp/rib/attr-sets/attr-set/as-path/as-segment/state/index
/network-instances/network-instance/protocols/protocol/bgp/rib/attr-sets/attr-set/as-path/as-segment/state/type
/network-instances/network-instance/protocols/protocol/bgp/rib/attr-sets/attr-set/as-path/as-segment/state/member
/network-instances/network-instance/protocols/protocol/bgp/rib/communities/community/index
/network-instances/network-instance/protocols/protocol/bgp/rib/communities/community/state/index
/network-instances/network-instance/protocols/protocol/bgp/rib/communities/community/state/community
```

## Design

For design see [Lemming and GoBGP](docs/gobgp_integration.md).
173 changes: 173 additions & 0 deletions bgp/docs/gobgp_integration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
<!-- markdownlint-disable MD007 MD030 -->

# Lemming and GoBGP

![gobgp integration system diagram](img/gobgp_integration_system_diagram.png)

## Integration of GoBGP into Lemming

Lemming’s support for OpenConfig BGP features relies on the open source
[GoBGP](https://github.com/osrg/gobgp/) project. This choice was made to make
use of an existing production-ready BGP agent written in Go, as well as to take
advantage of its
[OpenConfig-derived configuration](https://github.com/osrg/gobgp/blob/master/docs/sources/policy.md)
to minimize glue code. To understand how lemming integrates GoBGP to enable this
support, it is helpful to break down system requirements into several focused
mechanisms and properties, which together provides lemming with the above BGP
support:

1. Translating from OpenConfig path-value pairs to GoBGP configuration (**GoBGP
reconciler**).
2. Support for declarative configuration (**GoBGP reconciler**).
3. Support for route redistribution from RIB manager to BGP agent (**zebra
server**).
4. Support for route injection from BGP agent to RIB manager (**zebra
server**).

In addition to feature support, the lemming project supports lightweight ways of
validating BGP features. This includes testing using lemming images within a KNE
cluster, as well as a lighter-weight local testing where multiple lemmings are
instantiated as separate threads within a single test process.

The following sections will explain the relevant lemming system components that
enables the above system requirements as well as the testing environment.

## Explanation of System Components

### GoBGP server

The GoBGP server lemming’s BGP speaker. The GoBGP reconciler starts the GoBGP
server as a separate set of threads (goroutines) alongside other threads within
lemming. This enables regular TCP connections to be established with other BGP
speakers. Its implementation and documentation resides in
<https://github.com/osrg/gobgp>. Notably, it uses OpenConfig-derived
configuration that can be easily mapped from the latest OpenConfig models, and
supports declarative configuration semantics upon configuration updates.

### GoBGP Reconciler

A running lemming process is organized as a set of threads communicating
information with each other via a gNMI-based central database (gNMI datastore).
This datastore also exposes a gNMI interface to the user, who is able to update
the datastore with the desired OpenConfig configuration. Threads that subscribe
to the gNMI datastore and “reconcile” the system state into the user-desired
configuration are named “reconcilers”. Therefore, the GoBGP reconciler is
responsible providing OpenConfig BGP features to lemming.

The reconciler achieves this by translating OpenConfig configuration to the
semantically-equivalent GoBGP configuration, and then configuring the GoBGP
server. The configuration is done in a full config replace manner to achieve
declarative semantics. Its other primary responsibility is to monitor the GoBGP
server and update BGP applied state in the gNMI datastore, for example BGP RIB
state.

### RIB Manager (sysrib) and Zebra Server

The RIB manager (named the `sysrib` package) manages all routes by all routing
protocols supported in lemming and carries out recursive resolution. In the
context of BGP, it is also responsible for redistributing routes to GoBGP as
well as receiving BGP routes for resolution in the RIB.

In order to achieve RIB integration with GoBGP, lemming takes advantage of the
fact that it natively supports
[Zebra API integration](https://github.com/osrg/gobgp/blob/master/docs/sources/zebra.md)
as a Zebra API client. On startup, lemming is initialized with a simple Zebra
server that syncs route updates between the RIB manager and the GoBGP server. In
particular, whenever a new resolved route appears in the RIB manager, it is sent
to the Zebra server that then redistributes it to GoBGP; in the other direction,
whenever GoBGP receives a route in its local RIB, the route gets injected into
the Zebra server, which is then received by the RIB manager, making it available
for route resolution.

## Life of a BGP Route High-level Walkthrough

1. Upon lemming’s initialization with necessary BGP parameters (AS path and
RouterID), the **GoBGP reconciler** starts the **GoBGP server**, which
registers itself as a client to the **Zebra server**.
2. Upon receiving configuration for BGP peer, the **GoBGP reconciler**
configures the peer in the **GoBGP server**, which starts its internal BGP
state machine to look for the peer. See
[GoBGP: Life of a BGP UPDATE message](life_of_a_bgp_update_message.md) for
some rough notes on how BGP update messages propagate through GoBGP itself.
3. When a BGP route is received, the **GoBGP server** internally applies
configured policies against the route, which may or may not be propagated to
the local RIB, as well as other peers via the `ADJ_OUT` RIBs.
4. If the route was accepted into the local RIB, **GoBGP server** then
internally gives the route to the **Zebra server**. Since the **RIB
manager** is also a client of the **Zebra server**, it receives the route
and injects it into the RIB patricia tree. This triggers route resolution,
which will lead to programming of the route in the dataplane (forwarding
table) if it is resolvable and lemming is initialized with a dataplane.
5. The **GoBGP reconciler** monitors the **GoBGP server**’s RIBs, and updates
the OpenConfig BGP RIB paths accordingly.

## High-level Code Walkthrough

* `bgp/gobgp.go` contains the core logic for the **GoBGP reconciler**. In
particular, the `bgpTask` struct contains state and methods necessary for
managing the GoBGP server as well as reconciling between OpenConfig
configuration changes and the server. To understand `bgpTask`, start
hierarchically from the reconciler interface methods, the most important
being `start()`.
* `updateAppliedState()` is a wrapper required for any function that may
update the BGP applied state (i.e. /state values).
* `reconcile()` demonstrates at the high-level how GoBGP is managed: its
`config.InitialConfig` and `config.UpdateConfig` are used to
declaratively change the running configuration of GoBGP via a
translation from OpenConfig to GoBGP config through `intendedToGoBGP()`.
* Notable TODO: This TODO in `bgp/gobgp.go` tracks not populating the
policy definitions being populated into the /state paths.
* The preferred way to resolve this is to create a mechanism to reject
unsupported paths in lemming. This is currently not supported.
* Another way to resolve this is to manually merge policy definition
leaves such that only supported leaves are merged.

```go
err := ygot.MergeStructInto(t.appliedBGP, intendedBGP, &ygot.MergeOverwriteExistingFields{})
// TODO(wenbli): Since policy definitions is an atomic node,
// unsupported policy leaves will be merged as well. Therefore omitting
// them from the applied state until we find a way to to prune out
// unsupported paths prior to merge.
t.appliedRoutingPolicy.PolicyDefinition = nil
```

* `bgp/config.go` contains `intendedToGoBGP()`, which converts from OpenConfig
BGP configuration to GoBGP configuration.
* Notable implementation detail:
* Because GoBGP doesn’t support per-neighbour policies, whereas
OpenConfig only supports per-neighbour policies, the OpenConfig
policies are duplicated per-neighbour in order to arrive at the
equivalent behavior.
* `bgp/ocgobgp.go` contains smaller functions for converting from OpenConfig
to GoBGP configuration used by `bgp/config.go.`
* `sysrib/server_zapi.go` (ZAPI helpers) and `sysrib/zapi.go` (**Zebra
server**) contain Zebra API (ZAPI) related functionality.

## BGP Policy Testing

Lemming’s BGP policy features can be tested in two ways: 1) spinning up multiple
instances of lemming as different threads and have them talk to each other via
loopback addresses (`bgp/tests/local_tests`), or 2) running Ondatra tests in KNE
(`integration_tests/dut_policy_tests`). The local instances of lemming are
controlled by Ondatra-like tests as defined by the
[PolicyTestCase struct](https://github.com/openconfig/lemming/blob/7aae358c380f1cbc6925905be14cae57f7f3ef43/bgp/tests/local_tests/policy_test.go#L51-L57).

**Limitation**: When testing lemming instances locally, only IPv4 routes are
allowed. This is due to IPv6 having a single localhost address (::1/128),
instead of a set of addresses for IPv4 (127.0.0.0/8), preventing multiple GoBGP
servers from being started on the same port on multiple loopback addresses. This
challenge would need to be resolved before being able to run IPv6 tests in a
single process.

For running Ondatra/KNE tests with lemming, see
<https://github.com/openconfig/lemming?tab=readme-ov-file#running-integration-tests>.

## Notes on Adding Features to GoBGP

New BGP features must be added in [GoBGP](https://github.com/osrg/gobgp/), or in
a fork.
[Example of feature addition to GoBGP](https://github.com/osrg/gobgp/pull/2817).
Since GoBGP development must happen in a fork, it is recommended to test the
feature using the fork repository before filing a PR to GoBGP to minimize
back-and-forth with the GoBGP maintainers, who may take up to many days, or even
weeks, before responding to new issues or PRs.
Binary file added bgp/docs/img/gobgp_integration_system_diagram.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading