MGMTD Development

Overview

mgmtd (Management Daemon) is a new centralized management daemon for FRR.

Previously, vtysh was the only centralized management service provided. Internally vtysh connects to each daemon and sends CLI commands (both configuration and operational state queries) over a socket connection. This service only supports CLI which is no longer sufficient.

An important next step was made with the addition of YANG support. A YANG infrastructure was added through a new development called northbound. This northbound interface added the capability of daemons to be configured and queried using YANG models. However, this interface was per daemon and not centralized, which is not sufficient.

mgmtd harnesses this new northbound interface to provide a centralized interface for all daemons. It utilizes the daemons YANG models to interact with each daemon. mgmtd currently provides the CLI interface for each daemon that has been converted to it, but in the future RESTCONF and NETCONF servers can easily be added as front-ends to mgmtd to support those protocols as well.

Conversion Status

Fully Converted To MGMTD

• lib/affinitymap

• lib/distribute

• lib/filter

• lib/if

• lib/if_rmap

• lib/keychain

• lib/routemap

• lib/vrf

• ripd

• ripngd

• staticd

• zebra

Converted To Northbound

• bfdd

• pathd

• pbrd

• pimd

Converted To Northbound With Issues

• eigrp

• isisd

Unconverted

• babel

• bgpd

• ldpd

• lib/event

• lib/log_vty

• lib/nexthop_group

• lib/zlog_5424_cli

• nhrpd

• ospfd

• ospf6d

• pceplib

• qdb

• sharpd

• vrrpd

Converting A Daemon to MGMTD

A daemon must first be transitioned to the new Northbound API interface if that has not already been done (see Retrofitting Configuration Commands for how to do this). Once this is done a few simple steps are all that is required move the daemon over to mgmtd control.

Overview of Changes

Adding support for a northbound converted daemon involves very little work. It requires enabling frontend (CLI and YANG) and backend (YANG) support. mgmtd was designed to keep this as simple as possible.

Front-End Interface:

1. Add YANG module file to mgmtd/subdir.am (e.g., yang/frr-staticd.yang.c).

2. Add CLI handler file[s] to mgmtd/subdir.am. The subdir.am variable to use is indicated in the next 2 steps.

   1. [if needed] Exclude (#ifndef) non-configuration CLI handlers from CLI source file (e.g., inside staticd/static_vty.c) and add the file to nodist_mgmtd_libmgmt_be_nb_la_SOURCES in mgmtd/subdir.am.

   2. [otherwise] Remove CLI handler file from _SOURCES variable in the daemon subdir.am file (e.g in staticd/subdir.am) and add to mgmtd_libmgmtd_a_SOURCES in mgmtd/subdir.am.

3. In order to have mgmtd try and load existing per-daemon config files, add the daemon to the mgmt_daemons array in lib/vty.c. With the official release of the mgmtd code FRR is no longer supporting per daemon log files but it will take a while before all of the topotest is converted.

4. In the daemon’s struct frr_daemon_info (i.e., inside it’s FRR_DAEMON_INFO()) set the .flags bit FRR_NO_SPLIT_CONFIG. This will keep the daemon from trying to read it’s per-daemon config file as mgmtd will now be doing this.

5. Add the daemon’s YANG module description[s] into the array mgmt_yang_modules defined in mgmtd/mgmt_main.c (see CLI Config Write Handlers (cli_show)). Make sure that all YANG modules that the daemon uses are present in the mgmtd list. To find this list look in the daemon’s equivalent yang module array variable.

6. Initialize the CLI handlers inside mgmt_vty_init in mgmtd/mgmt_vty.c.

7. Direct vtysh to send CLI commands to mgmtd by modifying vtysh/vtysh.h. At the top of this file each daemon has a bit #define’d (e.g., #define VTYSH_STATICD 0x08000) below this there are groupings, replace all the uses of the daemons bit with VTYSH_MGMTD instead so that the CLI commands get properly routed to mgmtd rather than the daemon now.

1. Remove initialization (and installation) of library CLI routines. These will correspond with the VTYSH removals from the last step i.e.,:

   • change access_list_init() to access_list_init_new(false) and remove from VTYSH_ACL_CONFIG (leave in VTYSH_ACL_SHOW).

   • remove if_cmd_init_default() => remove from VTYSH_INTERFACE_SUBSET

   • remove if_cmd_init() => remove from VTYSH_INTERFACE_SUBSET

   • change route_map_init() to route_map_init_new(false) and remove from VTYSH_ROUTE_MAP_CONFIG (leave in VTYSH_ROUTE_MAP_SHOW).

   • remove vrf_cmd_init(NULL) => remove from VTYSH_INTERFACE_SUBSET

Back-End Interface:

1. In the daemon’s main file initialize the BE client library. You add a global struct mgmt_be_client *mgmt_be_client near the daemons event_loop *master variable. Then where the daemon used to initialize it’s CLI/VTY code replace that with the client initialization by calling mgmt_be_client_create. Likewise in the daemon’s sigint cleanup code, operational walks should be canceled with a call to nb_oper_cancel_all_walks, and then the BE client should be destroyed with a call to mgmt_be_client_destroy and to be safe NULL out the global mgmt_be_client variable.

2. In mgmtd/mgmt_be_adapter.c add xpath prefix mappings to a each of the mapping arrays (be_client_config_xpaths, be_client_oper_xpaths, and be_client_rpc_xpaths) to direct mgmtd to send config, oper-state, and RPC requests to your daemon.

   NOTE: make sure to include library supported xpaths prefixes as well (e.g., “/frr-interface:lib”). A good way to figure these paths out are to look in each of the YANG modules that the daemon uses and include each of their paths in the array.

3. In python/xref2vtysh.py add VTYSH_xxxD (for client xxx) to lib/mgmt_be_client.c entry in the daemon_falgs dictionary.

Add YANG and CLI into MGMTD

As an example here is the addition made to mgmtd/subdir.am for adding staticd support.

if STATICD
nodist_mgmtd_mgmtd_SOURCES += \
    yang/frr-staticd.yang.c \
    yang/frr-bfdd.yang.c \
    # end
nodist_mgmtd_libmgmt_be_nb_la_SOURCES += staticd/static_vty.c
endif

An here is the addition to the modules array in mgmtd/mgmt_main.c:

#ifdef HAVE_STATICD
extern const struct frr_yang_module_info frr_staticd_info;
#endif

static const struct frr_yang_module_info *const mgmt_yang_modules[] = {
       &frr_filter_info,
       ...
#ifdef HAVE_STATICD
       &frr_staticd_info,
#endif
}

CLI Config and Show Handlers

The daemon’s CLI handlers for configuration (which having been converted to the Northbound API now simply generate YANG changes) will be linked directly into mgmtd.

If the operational and debug CLI commands are kept in files separate from the daemon’s configuration CLI commands then no extra work is required. Otherwise some CPP #ifndef’s will be required.

mgmtd supports both config and operational state. However, many daemons have not had their operational state CLI commands converted over to the new YANG based methods. If that is the case and if both types of CLI handlers are present in a single file (e.g. a xxx_vty.c or xxx_cli.c file) then #ifndef will need to be used to exclude the non-config CLI handlers from mgmtd. The same goes for unconverted debug CLI handlers. For example:

DEFPY(daemon_one_config, daemon_one_config_cmd,
      "daemon one [optional-arg]"
      ...
{
      ...
}

#ifndef INCLUDE_MGMTD_CMDDEFS_ONLY
DEFPY(daemon_show_oper, daemon_show_oper_cmd,
      "show daemon oper [all]"
      ...
{
      ...
}
#endif /* ifndef INCLUDE_MGMTD_CMDDEFS_ONLY */

void daemon_vty_init(void)
{
      install_element(CONFIG_NODE, &daemon_one_config_cmd);
      ...

#ifndef INCLUDE_MGMTD_CMDDEFS_ONLY
        install_element(ENABLE_NODE, &daemon_show_oper_cmd);
#endif /* ifndef INCLUDE_MGMTD_CMDDEFS_ONLY */

}

CLI Config Write Handlers (cli_show)

To support writing out the CLI configuration file the northbound API defines a 2 callbacks (cli_show and cli_show_end). Pointers to these callbacks used to live side-by-side in a daemons struct frr_yang_module_info, with the daemons back-end configuration and operational state callbacks (normally in a file named <daemon>_nb.c).

However, these 2 functionalities need to be split up now. The frontend config writing callbacks (cli_show) should now be linked into mgmtd while the backend config and oper-state callbacks (e.g., create, modify, etc) should continue to be linked into the daemon.

So you will need to define 2 struct frr_yang_module_info arrays.

1. The existing array remains in the same place in the daemon, but with all the cli_show handlers removed.

2. The removed cli_show handlers should be added to a new struct frr_yang_module_info array. This second array should be included in the same file that includes that actual function pointed to by the the cli_show callbacks (i.e., the file is compiled into mgmtd).

   This new struct frr_yang_module_info array is the one to be included in mgmtd in mgmt_yang_modules inside mgmtd/mgmt_main.c.

Back-End Client Connection

In order for your daemon to communicate with mgmtd you need to initialize the backend client library. You normally do this where you used to initialize your CLI/VTY code.

...
struct event_loop *master;

static struct mgmt_be_client *mgmt_be_client;
...

int main(int argc, char **argv)
{
    ...
    rip_init();
    rip_if_init();
    mgmt_be_client = mgmt_be_client_create("ripd", NULL, 0, master);

Likewise the client should be cleaned up in the daemon cleanup routine.

/* SIGINT handler. */
static void sigint(void)
{
        zlog_notice("Terminating on signal");
        ...
        nb_oper_cancel_all_walks();
        mgmt_be_client_destroy(mgmt_be_client);
        mgmt_be_client = NULL;

Back-End XPATH mappings

In order for mgmtd to direct YANG modeled data to your daemon you should add some XPATH mappings to mgmtd/mgmt_be_adapter.c. These XPATHs determine which YANG modeled data (e.g., config changes) get sent over the back-end interface to your daemon. There are 4 arrays to possibly update: configuration, operational, notification, and RPC. You only need to add entries to the array that you require mapping for.

Additionally the back-end client can specify these XPATH mappings when it first connects to mgmtd using it’s initial SUBSCRIBE message.

NOTE: the notif array (be_client_notif_xpaths), is a slightly different from the other 3 types (config, oper and rpc) in that it maps xpaths the back-end client wishes to receive notifications for, not the ones it may generate. Normally a back-end client is generating notifications; however, mgmtd supports back-end clients also “subscribing” to receive these notifications as well from other back-end clients through notif_xpath maps.

Config Map Example

Below are the strings added for staticd config support:

#if HAVE_STATICD
static const char *const staticd_xpaths[] = {
        "/frr-vrf:lib",
        "/frr-interface:lib",
        "/frr-routing:routing/control-plane-protocols/control-plane-protocol/frr-staticd:staticd",
        NULL,
};
#endif

static const char *const *be_client_xpaths[MGMTD_BE_CLIENT_ID_MAX] = {
#ifdef HAVE_STATICD
        [MGMTD_BE_CLIENT_ID_STATICD] = staticd_xpaths,
#endif
};

Operational Map Example

Below are the strings added for zebra operational state support (note zebra is not conditionalized b/c it should always be present):

static const char *const zebra_oper_xpaths[] = {
        "/frr-interface:lib/interface",
        "/frr-vrf:lib/vrf/frr-zebra:zebra",
        "/frr-zebra:zebra",
        NULL,
};

static const char *const *be_client_oper_xpaths[MGMTD_BE_CLIENT_ID_MAX] = {
        [MGMTD_BE_CLIENT_ID_ZEBRA] = zebra_oper_xpaths,
};

RPC Map Example

Below is the string added for ripd RPC support:

static const char *const ripd_rpc_xpaths[] = {
        "/frr-ripd",
        NULL,
};

static const char *const *be_client_rpc_xpaths[MGMTD_BE_CLIENT_ID_MAX] = {
#ifdef HAVE_RIPD
        [MGMTD_BE_CLIENT_ID_RIPD] = ripd_rpc_xpaths,
#endif
};

Notification Map Example

There are no current back-end daemons that wish to receive other back-end notifications so the array is empty. This may change in the future, and of course any back-end daemon can utilize the connect (BeSubscribeReq) messages as well.

MGMTD Internals

This section will describe the internal functioning of mgmtd, for now a couple diagrams are included to aide in source code perusal.

The client side of a CLI configuration change

The server (mgmtd) side of a CLI configuration change

The client and server sides of oper-state query

Config datastore cleanup for non-implict commits (i.e., file reads currently)