CoreBGP - Plugging in to BGP

2020/07/05

cover

BGP

BGP is one of many protocols that powers the Internet. Chances are you have heard of it, even if you don’t work in or around the computer networking space. If you aren’t familiar, I’ll try to provide some quick background:

Expanding on that last bullet point, it’s difficult to summarize exactly how/where BGP is used due to its flexibility and extensibility. Various IETF Working Groups continue to publish BGP-related RFCs for a protocol that took shape in the early 90s. As the BGP landscape and application widens, we need software that enables us to keep up.

In this post I’ll provide some of my personal experience and history working with BGP, and introduce a new BGP library, CoreBGP, which can be used to build the next generation of BGP-enabled applications.

Peachtree Street

In October of 2010 I attended my first NANOG meeting in Atlanta, GA after accidentally falling into the position of Network Operations Engineer at work. I worked for a modest-sized hosting provider at the time, and was intrigued with BGP. Upon arriving in Atlanta, I vaguely remember some confusion after telling a cab driver that the hotel I needed to be dropped at was on Peachtree St. I later learned that there are 71 streets in Atlanta with a variant of “Peachtree” in their name, according to Wikpedia.

I got where I needed to go, eventually, and the first talk I attended was BGP techniques for Internet Service Providers by Philip Smith. Philip started with the basics before getting into the techniques used at ISPs. So many light bulbs went off for me during this talk. I have yet to see any other BGP presentation cover such a breadth of information but still do it in a way that is beginner-friendly, useful as a refresher for any expert, and just downright interesting.

Fast-forward 10 years and I’ve gained a fair share of experience operating networks that use BGP. In more recent years I’ve shifted to software engineering where I’ve had the opportunity to implement various BGP-enabled applications for network observability, data analytics, and SDN purposes.

Each time I started a new BGP-enabled app, I had to answer the following question – which existing BGP implementation should be its foundation?

The Case for Pluginized Routing Protocols

Of the handful of open source BGP implementations out there, I’ve had hands-on experience with projects making use of:

BIRD shines where a rich policy language is needed. GoBGP has a feature-rich gRPC API, and can be embedded as a library. OpenDaylight’s BGP implementation is part of a larger SDN controller solution and has extensive support for BGP-LS. Quagga can reliably produce MRT dumps and has been around a long time, though I believe FRRouting is now considered its successor.

These are all mature, established implementations. Some of them are in production at large ISPs, Cloud Providers, and Internet Exchange Points. They are purpose-built and make various tradeoffs to suit their use cases (programming language, threading model, data structures, API, etc…).

But what if we are building something that doesn’t line up with the primary use cases of these widely used implementations? We may be locked in to decisions that are ultimately burdensome if we choose to build around them. Swapping in our own data structures for routing tables, or adding a new NLRI is non-trivial. Even if an implementation is intended to be embedded as library, it can still back us into a corner with resource consumption. There’s clearly a need to plug in or hook into specific parts of the BGP FSM, without inheriting decisions that went into a full-blown BGP daemon.

At the 27th IEEE International Conference On Network Protocols (ICNP), a group from the Université catholique de Louvain presented a paper on The Case for Pluginized Routing Protocols:

Abstract—Routing protocols such as BGP and OSPF are key components of Internet Service Provider (ISP) networks. These protocols and the operator’s requirements evolve over time, but it often takes many years for network operators to convince their different router vendors and the IETF to extend routing protocols. Some network operators, notably in enterprise and datacenters have adopted Software Defined Networking (SDN) with its centralised control to be more agile. We propose a new approach to implement routing protocols that enables network operators to innovate while still using distributed routing protocols and thus keeping all their benefits compared to centralised routing approaches. We extend a routing protocol with a virtual machine that is capable of executing plugins. These plugins extend the protocol or modify its underlying algorithms through a simple API to meet the specific requirements of operators. We modify the OSPF and BGP implementations provided by FRRouting and demonstrate the applicability of our approach with several use cases.

— The Case for Pluginized Routing Protocols1

In their paper they present a method for plugging into a previously mentioned open-source BGP implementation, FRRouting. Plugins exist at a function level, either prior to invocation (PRE), as a replacement (REPLACE), or just before returning (POST). Much of their BGP plugin focus is around the reception of messages, and decisions made shortly after:

The BGP daemon is also extended similarly. We add insertion points on functions receiving BGP messages from neighbours, on filters and inside the decision process. We also expose specific functions to the plugins that are executed by the uBPF VM.

— The Case for Pluginized Routing Protocols1

They take a clever approach with plugin sandboxing by leveraging a user space eBPF VM (uBPF) linked to the FRRouting protocol implementation. Each plugin compiles to eBPF bytecode and runs inside of said VM. Plugins can be loaded and unloaded without impacting the primary protocol implementation. Using an eBPF VM also allowed them to utilise all the pre-existing Linux Kernel tooling.

I found this approach inspiring, but still not quite a match for my use cases:

This experience and research led me to create CoreBGP, a BGP library that I could re-use across my BGP-enabled applications.

CoreBGP

CoreBGP is a BGP library written in Go that implements the BGP FSM with an event-driven, pluggable model. It exposes an API that empowers the user to:

CoreBGP does not decode UPDATE messages (besides header validation), manage a routing table, or send its own UPDATE messages. These responsibilities are all passed down to the user. Therefore, the intended user is someone who wants that responsibility.

The primary building block of CoreBGP is a Plugin, defined by the following interface:

// Plugin is a BGP peer plugin.
type Plugin interface {
	// GetCapabilities is fired when a peer's FSM is in the Connect state prior
	// to sending an Open message. The returned capabilities are included in the
	// Open message sent to the peer.
	GetCapabilities(peer *PeerConfig) []*Capability

	// OnOpenMessage is fired when an Open message is received from a peer
	// during the OpenSent state. Returning a non-nil Notification will cause it
	// to be sent to the peer and the FSM will transition to the Idle state.
	//
	// Per RFC5492 a BGP speaker should only send a Notification if a required
	// capability is missing; unknown or unsupported capabilities should be
	// ignored.
	OnOpenMessage(peer *PeerConfig, capabilities []*Capability) *Notification

	// OnEstablished is fired when a peer's FSM transitions to the Established
	// state. The returned UpdateMessageHandler will be fired when an Update
	// message is received from the peer.
	//
	// The provided writer can be used to send Update messages to the peer for
	// the lifetime of the FSM's current, established state. It should be
	// discarded once OnClose() fires.
	OnEstablished(peer *PeerConfig, writer UpdateMessageWriter) UpdateMessageHandler

	// OnClose is fired when a peer's FSM transitions out of the Established
	// state.
	OnClose(peer *PeerConfig)
}

Here’s an example Plugin that logs when a peer enters/leaves an established state and when an UPDATE message is received:

type plugin struct{}

func (p *plugin) GetCapabilities(c *corebgp.PeerConfig) []*corebgp.Capability {
	caps := make([]*corebgp.Capability, 0)
	return caps
}

func (p *plugin) OnOpenMessage(peer *corebgp.PeerConfig, capabilities []*corebgp.Capability) *corebgp.Notification {
	return nil
}

func (p *plugin) OnEstablished(peer *corebgp.PeerConfig, writer corebgp.UpdateMessageWriter) corebgp.UpdateMessageHandler {
	log.Println("peer established")
	// send End-of-Rib
	writer.WriteUpdate([]byte{0, 0, 0, 0})
	return p.handleUpdate
}

func (p *plugin) OnClose(peer *corebgp.PeerConfig) {
	log.Println("peer closed")
}

func (p *plugin) handleUpdate(peer *corebgp.PeerConfig, u []byte) *corebgp.Notification {
	log.Printf("got update message of len: %d", len(u))
	return nil
}

Plugins are attached to peers when they are added to the Server, which manages their lifetime:

routerID := net.ParseIP("192.0.2.1")
srv, err := corebgp.NewServer(routerID)
if err != nil {
    log.Fatalf("error constructing server: %v", err)
}
p := &plugin{}
err = srv.AddPeer(&corebgp.PeerConfig{
    IP:       net.ParseIP("198.51.100.10"),
    LocalAS:  65001,
    RemoteAS: 65010,
}, p)
if err != nil {
    log.Fatalf("error adding peer: %v", err)
}

CoreBGP is flexible enough to function as the foundation for a full-blown BGP daemon. Conversely, it can also fit into a use case for simply logging UPDATE messages, as hinted at in the above example.

For more examples check out the examples directory and pkg.go.dev for the complete API. CoreBGP is open-source and available on GitHub. Please try it out, report bugs, and contribute!

  1. The Case for Pluginized Routing Protocols ↩︎ ↩︎