Peering and core opflex protocol support is provided by a suite of third-party libraries: libopflex, libuv, boost, and rapidjson. The major roles are played by the libopflex library, which is responsible for the opflex protocol implementation, and libuv, which is responsible for the networking connections.

The opflex protocol uses TCP or IP for communication. Packets are sent and received through the data plane by leveraging the LFTS layer which allows BINOS processes to use TCP or IP sockets.. The peer ip-address and tcp-port are set through IOS configuration commands and used to provision the connection.

For each peer, once a connection is established, policy updates are pushed to the opflex agent from the Controller through the peer. At present time there is no way to send an indication to the peer or controller about the success of handling a policy update. That is, from the controller's view, all policy updates are successful. This is a limitation imposed by libopflex.

The policies are encoded in JSON format. It is the role of libopflex to parse JSON encoded policies and call an appropriate handler to process the data. This data pertained in the policy is service-specific, that is, it is used to configure a particular networking solution and thus requires service-specific handling.

Each service provides a dedicated BINOS side parser which is used by OFA to extract the policy data, used later for configuration rendering. The invocation of these parser is the responsibility of the Domain MGR.

Different services are implemented as distinct sub-systems and can be packaged according to specific platform requirements. If a service is packaged, it is required to register with the Service MGR in OFA. During registration, the service provides several callbacks which are used by OFA to interact with the service.

The role of a service is to take the extracted policy data and apply the logic contained in it to produce the required concrete auto-generated configuration. The actual configuration rendering is performed in IOS. Similar to policy data extraction, rendering the concrete configuration requires service-specific implementation.

OFA also provides several event notifications to services. The handling of these events are within the service responsibility. The notified events are:

• Removal of a domain

• NVGEN request

• RP becoming active (following a switchover)

As the data extraction and configuration rendering take place in two different components, with the former being on the BINOS side and the latter in IOS, the policy data needs to be delivered from the service-specific parser to the corresponding OFA service in IOS.

Therefore, while maintaining the generic nature of the OFA infrastructure, the policy data is transported as payload in a generic datagram. That is, the OFA infrastructure is not made aware of the service-specific data, which allows a simple addition of supported services.

For peering information OFA uses the notion of a domain. Domain is a logical collection of one or two peers which are synchronized. That is, two peers cannot send the same policy update.

Having two peers adds to fault resistance of the particular domain. Below is an example of an Opflex domain configuration.

Opflex agent
  Domain dom1
       Identity dci-[10.20.30.40]
       Peer p1 ip-address 156.2.21.10 tcp-port 8009 src-ip-address 148.2.15.1
       Peer p2 ip-address 156.2.21.10 tcp-port 8009 src-ip-address 148.2.15.1

Domain: At most two peers per domain.

Identity: The identity is a string that uniquely identifies a domain and needs to be matched with the Opflex peer definition of a domain identity, which is governed by the Opflex library in use.

The third party library (used by OFA), libopflex, uses the format dci-[ip], where ip is the ip-address of the bgp ip loopback protocol.

Peers. The required information for each peer is the ip-address and tcp-port of the peer. The src-ip-address is the ip-address used by the OFA to send Opflex requests.

It is the responsibility of the service to support HA. To help facilitate this, OFA provides several workflows to support various HA scenarios.

1. The policy data is delivered to the service on the active

2. The policy data is handled on the active

3. The policy data is augmented by the service with auto-generated values and repackaged

4. OFA sends the augmented policy data to the standby

5. The augmented policy data is delivered to the service

• OFA requests the service for the current state in the form of a list of policy data items (which will recreate the state)

• OFA sends the list to the standby

• Each policy data item on the list is delivered to the service

• Domain is removed on the active (through CLI)

• Service is notified of a domain removal on the active

• Domain is removed on the standby (through CONFIG_SYNC)

• Service is notified of a domain removal on the standby

• Each service can choose the respective parts of the workflow

Services are used to configure particular feature sets and the one service supported by OFA is the Service VXLAN-EVPN, or in short SVE.

Details about the actual service and the business logic it provides can be found in a document describing the VXLAN-EVPN solution. Here we focus on the service functionality from the Opflex perspective, i.e. service configuration and the rendering template.

In a nutshell, SVE receives policy updates which hold tenant information and render the corresponding configuration based on a template. The policy updates are encoded in the JSON format and convey several pieces of information:

• Policy space

• Routing domain

• Local VRF name

• A list of route targets (RTs)

Following updates, SVE builds an internal Tenant DB with the following properties for each tenant:

• Unique name (based on policy space and routing domain)

• Route targets (RTs)

• VRF name

Each tenant contributes its list of RTs to the VRF it references (name received in the policy update). Note that the same VRF can be referenced by multiple tenants, while each tenant references only a single VRF.

Each policy update carries information for a single tenant. The update can be of two types:

• Tenant PUT completely replaces current information in the Tenant DB with the new one.

• Tenant DELETE removes the tenant from the Tenant DB.

The rendered configuration is an up-to-date projection of the Tenant DB into VRF, BD, BGP, and NVE features. For each referenced VRF the rendering template in Fig. 3 is used

The rendered configuration is an up-to-date projection of the Tenant DB into VRF, BD, BGP, and NVE features. For each referenced VRF, the rendering template is used.

SVE is configured as a subtree under opflex agent. It has two configurable items:

• nve-id: The interfaceis used for VNI generation

• bdi-ip: It is used for the configuration of the generated BDI

• Both the items are used in the rendering template

At times the rendered and manual configuration might have a conflict. The following set of rules outlines the current behavior of SVE in various scenarios where there is a potential collision between an auto-generated and manual config.

• SVE configuration may overwrite existing manual configuration for BDI attributes (ip address/forwarding vrf).

• Manual config is allowed to overwrite any SVE-generated configuration. Note that if done improperly service disruption is possible.

• Additional features, such as, NAT or QoS are manually configured on an SVE-generated BDI.

• SVE never deletes VRF/BD/BDI/NVE definitions (including those which are auto-generated following a tenant update)

• The only cleanup allowed by SVE is the RTs (normal or stitching) configured inside the VRF by SVE. That is, SVE will delete RTs during tenant updates. As a result we might have a rare scenario where a user configured RT is deleted by SVE due to tenant update if the tenant had a similar RT before the update.

Opflex agent
    Service vxlan-evpn
        Nve-id 1
        Bdi-ip 9.9.9.9 255.255.255.0

When a domain is removed (unconfigured), SVE deletes the tenants which were last modified through that domain. For example, in the case of 4 tenants (as shown in the table below), if domain D1 is removed, this would lead to deletion of tenants T1, and T3.

Tenant deletion does not alter any of the existing configuration, except for route targets, as indicated in Section 0). Also, recall that the generated concrete configuration is a projection of the Tenant DB. Thus RTs will remain part of a VRF as long as there at least one tenant in the DB which still holds the RT in question.

As mentioned earlier, the features are responsible for successful syncing of their respective configuration to the standby (VRF, BD, BGP, NVE).

On switchover, however, SVE attempts to re-apply the expected configuration. For each tenant, if successful, this results in no-op, otherwise, the tenant is deleted.