Welcome to RTD Test Project’s Documentation!

Getting Started with Cumulus Linux

Cumulus Linux is the first full-featured Linux operating system for the networking industry. The Debian Jessie-based, networking-focused distribution runs on hardware produced by a broad partner ecosystem, ensuring unmatched customer choice regarding silicon, optics, cables, and systems.

This user guide provides in-depth documentation on the Cumulus Linux installation process, system configuration and management, network solutions, and monitoring and troubleshooting recommendations. In addition, the quick start guide provides an end-to-end setup process to get you started.

What’s New in Cumulus Linux 3.7

Cumulus Linux 3.7 contains a number of new platforms, features and improvements:

For information on bug fixes and known issues present in this release, refer to the product release notes.

Open Source Contributions

To implement various Cumulus Linux features, Cumulus Networks has forked various software projects, like CFEngine, Netdev and some Puppet Labs packages. The forked code resides in the Cumulus Networks GitHub repository.

Cumulus Networks has also developed and released new applications as open source. The list of open source projects is on the open source software page.

Hardware Compatibility List

You can find the most up-to-date hardware compatibility list (HCL) here. Use the HCL to confirm that your switch model is supported by Cumulus Networks. The HCL is updated regularly, listing products by port configuration, manufacturer, and SKU part number.

Quick Start Guide

This quick start guide provides an end-to-end setup process for installing and running Cumulus Linux, as well as a collection of example commands for getting started after installation is complete.

Prerequisites

  • Intermediate-level Linux knowledge is assumed for this guide. You should be familiar with basic text editing, Unix file permissions, and process monitoring. A variety of text editors are pre-installed, including vi and nano.
  • You must have access to a Linux or UNIX shell. If you are running Windows, use a Linux environment like Cygwin as your command line tool for interacting with Cumulus Linux.

Tip

If you are a networking engineer but are unfamiliar with Linux concepts, refer to this reference guide to compare the Cumulus Linux CLI and configuration options, and their equivalent Cisco Nexus 3000 NX-OS commands and settings. You can also watch a series of short videos introducing you to Linux and Cumulus Linux-specific concepts.

Installation

To install Cumulus Linux, you use ONIE (Open Network Install Environment), an extension to the traditional U-Boot software that allows for automatic discovery of a network installer image. This facilitates the ecosystem model of procuring switches, with a user’s own choice of operating system loaded, such as Cumulus Linux.

Note

If Cumulus Linux 3.0.0 or later is already installed on your switch and you need to upgrade the software only, you can skip to Upgrading Cumulus Linux below.

The easiest way to install Cumulus Linux with ONIE is with local HTTP discovery:

  1. If your host (like a laptop or server) is IPv6-enabled, make sure it is running a web server. If the host is IPv4-enabled, make sure it is running DHCP as well as a web server.
  2. Download the Cumulus Linux installation file to the root directory of the web server. Rename this file onie-installer.
  3. Connect your host using an Ethernet cable to the management Ethernet port of the switch.
  4. Power on the switch. The switch downloads the ONIE image installer and boots. You can watch the progress of the install in your terminal. After the installation completes, the Cumulus Linux login prompt appears in the terminal window.

Note

These steps describe a flexible unattended installation method. You do not need a console cable. A fresh install with ONIE using a local web server typically completes in less than ten minutes. You have more options for installing Cumulus Linux with ONIE. Read Installing a New Cumulus Linux Image to install Cumulus Linux using ONIE in the following ways:

  • DHCP/web server with and without DHCP options
  • Web server without DHCP
  • FTP or TFTP without a web server
  • Local file
  • USB

ONIE supports many other discovery mechanisms using USB (copy the installer to the root of the drive), DHCPv6 and DHCPv4, and image copy methods including HTTP, FTP, and TFTP. For more information on these discovery methods, refer to the ONIE documentation.

After installing Cumulus Linux, you are ready to:

  • Log in to Cumulus Linux on the switch.
  • Install the Cumulus Linux license.
  • Configure Cumulus Linux. This quick start guide provides instructions on configuring switch ports and a loopback interface.

Upgrade to the Latest Version

If you are running a Cumulus Linux version earlier than 3.0.0, you must perform a complete install, as described above. If you already have Cumulus Linux 3.0.0 or later installed on your switch, read Upgrading Cumulus Linux for considerations before starting the process.

Getting Started

When starting Cumulus Linux for the first time, the management port makes a DHCPv4 request. To determine the IP address of the switch, you can cross reference the MAC address of the switch with your DHCP server. The MAC address is typically located on the side of the switch or on the box in which the unit ships.

Login Credentials

The default installation includes one system account, root, with full system privileges, and one user account, cumulus, with sudo privileges. The root account password is set to null by default (which prohibits login), while the cumulus account is configured with this default password:

CumulusLinux!

In this quick start guide, you use the cumulus account to configure Cumulus Linux.

Warning

For optimum security, change the default password (using the passwd command) before you configure Cumulus Linux on the switch.

All accounts except root are permitted remote SSH login; you can use sudo to grant a non-root account root-level access. Commands that change the system configuration require this elevated level of access.

For more information about sudo, read Using sudo to Delegate Privileges.

Serial Console Management

You are encouraged to perform management and configuration over the network, either in band or out of band. Use of the serial console is fully supported; however, many customers prefer the convenience of network-based management.

Typically, switches ship from the manufacturer with a mating DB9 serial cable. Switches with ONIE are always set to a 115200 baud rate.

Wired Ethernet Management

Switches supported in Cumulus Linux always contain at least one dedicated Ethernet management port, which is named eth0. This interface is geared specifically for out-of-band management use. The management interface uses DHCPv4 for addressing by default. You can set a static IP address with the Network Command Line Utility (NCLU).

Example IP Configuration

Set the static IP address with the interface address and interface gateway NCLU commands:

cumulus@switch:~$ net add interface eth0 ip address 192.0.2.42/24
cumulus@switch:~$ net add interface eth0 ip gateway 192.0.2.1
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands produce the following snippet in the /etc/network/interfaces file:

auto eth0
iface eth0
    address 192.0.2.42/24
    gateway 192.0.2.1

[ui-tab title=”NCLU”]

Set the static IP address with the interface address and interface gateway NCLU commands:

cumulus@switch:~$ net add interface eth0 ip address 192.0.2.42/24
cumulus@switch:~$ net add interface eth0 ip gateway 192.0.2.1
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands produce the following snippet in the /etc/network/interfaces file:

auto eth0
iface eth0
    address 192.0.2.42/24
    gateway 192.0.2.1

[/ui-tab][ui-tab title=”Linux”]

Edit the /etc/network/interfaces file and add the following configuration:

cumulus@switch:~$ sudo nano /etc/network/interfaces

auto eth0
iface eth0
    address 192.0.2.42/24
    gateway 192.0.2.1

[/ui-tab]

Configuring the Hostname and Timezone

To change the hostname, run net add hostname, which modifies both the/etc/hostnameand /etc/hosts files with the desired hostname.

cumulus@switch:~$ net add hostname <hostname>
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

Note

The command prompt in the terminal does not reflect the new hostname until you either log out of the switch or start a new shell.

When you use this NCLU command to set the hostname, DHCP does not override the hostname when you reboot the switch. However, if you disable the hostname setting with NCLU, DHCP does override the hostname the next time you reboot the switch.

To update the timezone, use the NTP interactive mode:

  1. Run the following command in a terminal:

    sudo dpkg-reconfigure tzdata

  2. Follow the on screen menu options to select the geographic area and region.

Note

Programs that are already running (including log files) and users currently logged in, do not see timezone changes made with interactive mode. To have the timezone set for all services and daemons, a reboot is required.

Verifying the System Time

Before you install the license, verify that the date and time on the switch are correct. You must correct the date and time if they are incorrect. The wrong date and time can have impacts on the switch, such as the inability to synchronize with Puppet or return errors like this one after you restart switchd:

Warning: Unit file of switchd.service changed on disk, ‘systemctl daemon-reload’ recommended.

Installing the License

Cumulus Linux is licensed on a per-instance basis. Each network system is fully operational, enabling any capability to be utilized on the switch with the exception of forwarding on switch panel ports. Only eth0 and console ports are activated on an unlicensed instance of Cumulus Linux. Enabling front panel ports requires a license.

You receive a license key from Cumulus Networks or an authorized reseller. Here is a sample license key:

user@company.com|thequickbrownfoxjumpsoverthelazydog312

There are three ways to install the license onto the switch:

  • Copy the license from a local server. Create a text file with the license and copy it to a server accessible from the switch. On the switch, use the following command to transfer the file directly on the switch, then install the license file:

    cumulus@switch:~$ scp user@my_server:/home/user/my_license_file.txt .
cumulus@switch:~$ sudo cl-license -i my_license_file.txt

  • Copy the file to an HTTP server (not HTTPS), then reference the URL when you run cl-license: cumulus@switch:~$ sudo cl-license -i <URL>

  • Copy and paste the license key into the cl-license command: cumulus@switch:~$ sudo cl-license -i   <paste license key>   ^+d

Note

It is not necessary to reboot the switch to activate the switch ports. After you install the license, restart the switchd service. All front panel ports become active and show up as swp1, swp2, and so on.

  cumulus@switch:~$ sudo systemctl restart switchd.service

If a license is not installed on a Cumulus Linux switch, the ``switchd``
service does not start. After you install the license, start
``switchd`` as described above.

Configure Breakout Ports with Splitter Cables

If you are using 4x10G DAC or AOC cables, or want to break out 100G or 40G switch ports, configure the breakout ports. For more details, see Layer 1 and Switch Port Attributes.

Test Cable Connectivity

By default, all data plane ports (every Ethernet port except the management interface, eth0) are disabled.

To test cable connectivity, administratively enable a port:

cumulus@switch:~$ net add interface swp1
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

To administratively enable all physical ports, run the following command, where swp1-52 represents a switch with switch ports numbered from swp1 to swp52:

cumulus@switch:~$ net add interface swp1-52
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

To view link status, use the net show interface all command. The following examples show the output of ports in admin down, down, and up modes:

cumulus@switch:~$ net show interface all
       Name                      Speed    MTU    Mode           Summary
-----  ------------------------  -------  -----  -------------  --------------------------------------
UP     lo                        N/A      65536  Loopback       IP: 10.0.0.11/32, 127.0.0.1/8, ::1/128
UP     eth0                      1G       1500   Mgmt           IP: 192.168.0.11/24(DHCP)
UP     swp1 (hypervisor_port_1)  1G       1500   Access/L2      Untagged: br0
UP     swp2                      1G       1500   NotConfigured
ADMDN  swp45                     0M       1500   NotConfigured
ADMDN  swp46                     0M       1500   NotConfigured
ADMDN  swp47                     0M       1500   NotConfigured
ADMDN  swp48                     0M       1500   NotConfigured
ADMDN  swp49                     0M       1500   NotConfigured
ADMDN  swp50                     0M       1500   NotConfigured
UP     swp51                     1G       1500   BondMember     Master: bond0(DN)
UP     blue                      N/A      65536  NotConfigured
DN     bond0                     N/A      1500   Bond           Bond Members: swp51(UN)
UP     br0                       N/A      1500   Bridge/L3      IP: 172.16.1.1/24
                                                                Untagged Members: swp1
                                                                802.1q Tag: Untagged
                                                                STP: RootSwitch(32768)
UP     red                       N/A      65536  NotConfigured
ADMDN  rename13                  0M       1500   NotConfigured
ADMDN  vagrant                   0M       1500   NotConfigured

Configure Switch Ports

Layer 2 Port Configuration

Cumulus Linux does not put all ports into a bridge by default. To create a bridge and configure one or more front panel ports as members of the bridge, use the following examples as guides.

Examples

Example One

In the following configuration example, the front panel port swp1 is placed into a bridge called bridge. The NCLU commands are:

cumulus@switch:~$ net add bridge bridge ports swp1
cumulus@switch:~$ net pending !!! cumulus@switch:~$ net commit

The commands above produce the following ``/etc/network/interfaces``snippet:

auto bridge
iface bridge
    bridge-ports swp1
    bridge-vlan-aware yes

Example Two

You can add a range of ports in one command. For example, add swp1 through swp10, swp12, and swp14 through swp20 to bridge:

cumulus@switch:~$ net add bridge bridge ports swp1-10,12,14-20
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

The commands above produce the following snippet in the /etc/network/interfaces file:

auto bridge
iface bridge
     bridge-ports swp1 swp2 swp3 swp4 swp5 swp6 swp7 swp8 swp9 swp10 swp12 swp14 swp15 swp16 swp17 swp18 swp19 swp20
     bridge-vlan-aware yes

To view the changes in the kernel, use the brctl command:

cumulus@switch:~$ brctl show
bridge name     bridge id              STP enabled     interfaces
bridge          8000.443839000004      yes             swp1
                                                       swp2

Layer 3 Port Configuration

You can also use NCLU to configure a front panel port or bridge interface as a layer 3 port.

In the following configuration example, the front panel port swp1 is configured as a layer 3 access port:

cumulus@switch:~$ net add interface swp1 ip address 10.1.1.1/30
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

The commands above produce the following snippet in the /etc/network/interfaces file:

auto swp1
iface swp1
    address 10.1.1.1/30

To add an IP address to a bridge interface, you must put it into a VLAN interface:

cumulus@switch:~$ net add vlan 100 ip address 10.2.2.1/24
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

The commands above produce the following snippet in the /etc/network/interfaces file:

auto bridge
iface bridge
    bridge-vids 100
    bridge-vlan-aware yes

auto vlan100
iface vlan100
    address 192.168.10.1/24
    vlan-id 100
    vlan-raw-device bridge

To view the changes in the kernel, use the ip addr show command:

cumulus@switch:~$ ip addr show
...

4. swp1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bridge state UP group default qlen 1000
   link/ether 44:38:39:00:6e:fe brd ff:ff:ff:ff:ff:ff

...

14: bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 44:38:39:00:00:04 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::4638:39ff:fe00:4/64 scope link
      valid_lft forever preferred_lft forever
...

Configure a Loopback Interface

Cumulus Linux has a loopback preconfigured in the /etc/network/interfaces file. When the switch boots up, it has a loopback interface, called lo, which is up and assigned an IP address of 127.0.0.1.

Tip

The loopback interface lo must always be specified in the /etc/network/interfaces file and must always be up.

To see the status of the loopback interface (lo), use the net show interface lo command:

cumulus@switch:~$ net show interface lo
    Name    MAC                Speed      MTU  Mode
--  ------  -----------------  -------  -----  --------
UP  lo      00:00:00:00:00:00  N/A      65536  Loopback

IP Details
-------------------------  --------------------
IP:                        127.0.0.1/8, ::1/128
IP Neighbor(ARP) Entries:  0

Note that the loopback is up and is assigned an IP address of 127.0.0.1.

To add an IP address to a loopback interface, configure the lo interface with NCLU:

cumulus@switch:~$ net add loopback lo ip address 10.1.1.1/32
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

You can configure multiple loopback addresses by adding additional address lines:

cumulus@switch:~$ net add loopback lo ip address 172.16.2.1/24
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

The commands above produce the following snippet in the /etc/network/interfaces file:

auto lo
iface lo inet loopback
    address 10.1.1.1/32
    address 172.16.2.1/24

Network Command Line Utility - NCLU

The Network Command Line Utility (NCLU) is a command line interface for Cumulus Networks products that simplifies the networking configuration process for all users.

NCLU resides in the Linux user space and provides consistent access to networking commands directly through bash, making configuration and troubleshooting simple and easy; no need to edit files or enter modes and sub-modes. NCLU:

  • Embeds help, examples, and automatic command checking with suggestions in case you enter a typo.
  • Runs directly from and integrates with bash, while being interoperable with the regular way of accessing underlying configuration files and automation.
  • Configures dependent features automatically so that you don’t have to.
NCLU overview

The NCLU wrapper utility called net is capable of configuring layer 2 and layer 3 versions of the networking stack, installing ACLs and VXLANs, rolling back and deleting snapshots, as well as providing monitoring and troubleshooting functionality for these features. You can configure both the /etc/network/interfaces and /etc/frr/frr.conf files with net, in addition to running show and clear commands related to ifupdown2 and FRRouting.

Install NCLU

If you upgraded Cumulus Linux from a version earlier than 3.2 instead of performing a full binary install, you need to install the nclu package on your switch:

cumulus@switch:~$ sudo -E apt-get update
cumulus@switch:~$ sudo -E apt-get install nclu
cumulus@switch:~$ sudo -E apt-get upgrade

Note

The nclu package installs a new bash completion script and displays the following message:

Setting up nclu (1.0-cl3u3) ...
To enable the newly installed bash completion for nclu in this shell, execute...
source /etc/bash_completion

NCLU Basics

Use the following workflow to stage and commit changes to Cumulus Linux with NCLU:

  1. Use the net add and net del commands to stage and remove configuration changes.
  2. Use the net pending command to review staged changes.
  3. Use net commit and net abort to commit and delete staged changes.

Note

net commit applies the changes to the relevant configuration files, such as /etc/network/interfaces, then runs necessary follow on commands to enable the configuration, such as ifreload -a. If two different users try to commit a change at the same time, NCLU displays a warning but implements the change according to the first commit received. The second user will need to abort the commit.

When you have a running configuration, you can review and update the configuration with the following commands:

  • net show is a series of commands for viewing various parts of the network configuration. For example, use net show configuration to view the complete network configuration, net show commit history to view a history of commits using NCLU, and net show bgp to view BGP status.
  • net clear provides a way to clear ``net show`` counters, BGP and OSPF neighbor content, and more.
  • net rollback provides a mechanism to revert back to an earlier configuration.
  • net commit confirm requires you to press Enter to commit changes using NCLU. If you run net commit confirm but do not press Enter within 10 seconds, the commit automatically reverts and no changes are made.
  • net commit permanent retains the taken when committing the change. Otherwise, the snapshots created from NCLU commands are cleaned up periodically with a snapper cron job.
  • net commit delete deletes one or more snapshots created when committing changes with NCLU.
  • net del all deletes all configurations and stops the IEEE 802.1X service.

Note

This command does not remove configurations, as NCLU does not interact with eth0 interfaces and management VRF.

Tab Completion, Verification, and Inline Help

In addition to tab completion and partial keyword command identification, NCLU includes verification checks to ensure correct syntax is used. The examples below show the output for incorrect commands:

cumulus@switch:~$ net add bgp router-id 1.1.1.1/32
ERROR: Command not found

Did you mean one of the following?

    net add bgp router-id <ipv4>
        This command is looking for an IP address, not an IP/prefixlen

cumulus@switch:~$ net add bgp router-id 1.1.1.1
cumulus@switch:~$ net add int swp10 mtu <TAB>
    <552-9216> :
cumulus@switch:~$ net add int swp10 mtu 9300
ERROR: Command not found

Did you mean one of the following?
    net add interface <interface> mtu <552-9216>

NCLU has a comprehensive built in help system. In addition to the net man page, you can use ? and help to display available commands:

cumulus@switch:~$ net help

Usage:
    # net <COMMAND> [<ARGS>] [help]
    #
    # net is a command line utility for networking on Cumulus Linux switches.
    #
    # COMMANDS are listed below and have context specific arguments which can
    # be explored by typing "<TAB>" or "help" anytime while using net.
    #
    # Use 'man net' for a more comprehensive overview.


    net abort
    net commit [verbose] [confirm] [description <wildcard>]
    net commit delete (<number>|<number-range>)
    net help [verbose]
    net pending
    net rollback (<number>|last)
    net show commit (history|<number>|<number-range>|last)
    net show rollback (<number>|last)
    net show configuration [commands|files|acl|bgp|ospf|ospf6|interface <interface>]


Options:

    # Help commands
    help     : context sensitive information; see section below
    example  : detailed examples of common workflows


    # Configuration commands
    add      : add/modify configuration
    del      : remove configuration


    # Commit buffer commands
    abort    : abandon changes in the commit buffer
    commit   : apply the commit buffer to the system
    pending  : show changes staged in the commit buffer
    rollback : revert to a previous configuration state


    # Status commands
    show     : show command output
    clear    : clear counters, BGP neighbors, etc

cumulus@switch:~$ net help bestpath
The following commands contain keyword(s) 'bestpath'

    net (add|del) bgp bestpath as-path multipath-relax [as-set|no-as-set]
    net (add|del) bgp bestpath compare-routerid
    net (add|del) bgp bestpath med missing-as-worst
    net (add|del) bgp vrf <text> bestpath as-path multipath-relax [as-set|no-as-set]
    net (add|del) bgp vrf <text> bestpath compare-routerid
    net (add|del) bgp vrf <text> bestpath med missing-as-worst
    net add bgp debug bestpath <ip/prefixlen>
    net del bgp debug bestpath [<ip/prefixlen>]
    net show bgp (<ipv4>|<ipv4/prefixlen>) [bestpath|multipath] [json]
    net show bgp (<ipv6>|<ipv6/prefixlen>) [bestpath|multipath] [json]
    net show bgp vrf <text> (<ipv4>|<ipv4/prefixlen>) [bestpath|multipath] [json]

Note

You can configure multiple interfaces at once:

cumulus@switch:~$ net add int swp7-9,12,15-17,22 mtu 9216

Add ? (Question Mark) Ability to NCLU

While tab completion is enabled by default, you can also configure NCLU to use the ? (question mark character) to look at available commands. To enable this feature for the cumulus user, open the following file:

cumulus@leaf01:~$ sudo nano ~/.inputrc

Uncomment the very last line in the .inputrc file so that the file changes from this:

# Uncomment to use ? as an alternative to
# ?: complete

to this:

# Uncomment to use ? as an alternative to
# ?: complete

Save the file and reconnect to the switch. The ? (question mark) ability will work on all subsequent sessions on the switch.

cumulus@leaf01:~$ net
    abort     :  abandon changes in the commit buffer
    add       :  add/modify configuration
    clear     :  clear counters, BGP neighbors, etc
    commit    :  apply the commit buffer to the system
    del       :  remove configuration
    example   :  detailed examples of common workflows
    help      :  Show this screen and exit
    pending   :  show changes staged in the commit buffer
    rollback  :  revert to a previous configuration state
    show      :  show command output

Note

When the question mark is typed, NCLU autocompletes and shows all available options, but the question mark does not actually appear on the terminal. This is normal, expected behavior.

Built-In Examples

NCLU has a number of built in examples to guide users through basic configuration setup:

cumulus@switch:~$ net example
    acl              :  access-list
    bgp              :  Border Gateway Protocol
    bond             :  Bond, port-channel, etc
    bridge           :  A layer2 bridge
    clag             :  Multi-Chassis Link Aggregation
    dot1x            :  Configure, Enable, Delete or Show IEEE 802.1X EAPOL
    link-settings    :  Physical link parameters
    lnv              :  Lightweight Network Virtualization
    management-vrf   :  Management VRF
    mlag             :  Multi-Chassis Link Aggregation
    ospf             :  Open Shortest Path First (OSPFv2)
    vlan-interfaces  :  IP interfaces for VLANs

cumulus@switch:~$ net example bridge

Scenario
========
We are configuring switch1 and would like to configure the following
- configure switch1 as an L2 switch for host-11 and host-12
- enable vlans 10-20
- place host-11 in vlan 10
- place host-12 in vlan 20
- create an SVI interface for vlan 10
- create an SVI interface for vlan 20
- assign IP 10.0.0.1/24 to the SVI for vlan 10
- assign IP 20.0.0.1/24 to the SVI for vlan 20
- configure swp3 as a trunk for vlans 10, 11, 12 and 20
              swp3
     *switch1 --------- switch2
        /\
  swp1 /  \ swp2
      /    \
     /      \
 host-11   host-12

switch1 net commands
====================
- enable vlans 10-20
switch1# net add vlan 10-20
- place host-11 in vlan 10
- place host-12 in vlan 20
switch1# net add int swp1 bridge access 10
switch1# net add int swp2 bridge access 20
- create an SVI interface for vlan 10
- create an SVI interface for vlan 20
- assign IP 10.0.0.1/24 to the SVI for vlan 10
- assign IP 20.0.0.1/24 to the SVI for vlan 20
switch1# net add vlan 10 ip address 10.0.0.1/24
switch1# net add vlan 20 ip address 20.0.0.1/24
- configure swp3 as a trunk for vlans 10, 11, 12 and 20
switch1# net add int swp3 bridge trunk vlans 10-12,20
# Review and commit changes
switch1# net pending
switch1# net commit

Verification
============
switch1# net show interface
switch1# net show bridge macs

Configure User Accounts

You can configure user accounts in Cumulus Linux with read-only or edit permissions for NCLU:

  • You create user accounts with read-only permissions for NCLU by adding them to the netshow group. A user in the netshow group can run NCLU net show commands, such as net show interface or net show config, and certain general Linux commands, such as ls, cd or man, but cannot run net add, net del or net commit commands.
  • You create user accounts with edit permissions for NCLU by adding them to the netedit group. A user in the netedit group can run NCLU configuration commands, such as net add, net del or net commit in addition to NCLU net show commands.

The examples below demonstrate how to add a new user account or modify an existing user account called myuser.

To add a new user account with NCLU show permissions:

cumulus@switch:~$ sudo adduser --ingroup netshow myuser
Adding user `myuser' ...
Adding new user `myuser' (1001) with group `netshow' …

To add NCLU show permissions to a user account that already exists:

cumulus@switch:~$ sudo addgroup myuser netshow
Adding user `myuser' to group `netshow' ...
Adding user myuser to group netshow
Done

To add a new user account with NCLU edit permissions:

cumulus@switch:~$ sudo adduser --ingroup netedit myuser
Adding user `myuser' ...
Adding new user `myuser' (1001) with group `netedit' …

To add NCLU edit permissions to a user account that already exists:

cumulus@switch:~$ sudo addgroup myuser netedit
Adding user `myuser' to group `netedit' ...
Adding user myuser to group netedit
Done

Note

You can use the adduser command for local user accounts only. You can use the addgroup command for both local and remote user accounts. For a remote user account, you must use the mapping username, such as tacacs3 or radius_user, not the or account name.

If the user tries to run commands that are not allowed, the following error displays:

myuser@switch:~$ net add hostname host01
ERROR: User username does not have permission to make networking changes.

Edit the netd.conf File

Instead of using the NCLU commands described above, you can manually configure users and groups to be able to run NCLU commands.

Edit the /etc/netd.conf file to add users to the users_with_edit and users_with_show lines in the file, then save the file.

For example, if you want the user netoperator to be able to run both edit and show commands, add the user to the users_with_edit and users_with_show`` lines in the ``/etc/netd.conf file:

cumulus@switch:~$ sudo nano /etc/netd.conf

# Control which users/groups are allowed to run 'add', 'del',
# 'clear', 'net abort', 'net commit' and restart services
# to apply those changes
users_with_edit = root, cumulus, netoperator
groups_with_edit = root, cumulus


# Control which users/groups are allowed to run 'show' commands
users_with_show = root, cumulus, netoperator
groups_with_show = root, cumulus

To configure a new user group to use NCLU, add that group to the groups_with_edit and groups_with_show lines in the file.

Note

Use caution giving edit permissions to groups. For example, don’t give edit permissions to the tacacs group.

Restart the netd Service

Whenever you modify netd.conf, you must restart the netd service for the changes to take effect:

cumulus@switch:~$ sudo systemctl restart netd.service

Back Up the Configuration to a Single File

You can easily back up your NCLU configuration to a file by outputting the results of net show configuration commands to a file, then retrieving the contents of the file using the source command. You can then view the configuration at any time or copy it to other switches and use the source command to apply that configuration to those switches.

For example, to copy the configuration of a leaf switch called leaf01, run the following command:

cumulus@leaf01:~$ net show configuration commands >> leaf01.txt

With the commands all stored in a single file, you can now copy this file to another ToR switch in your network called leaf01 and apply the configuration by running:

cumulus@leaf01:~$ source leaf01.txt

Advanced Configuration

NCLU needs no initial configuration; it is ready to go in Cumulus Linux. However, if you need to modify its configuration, you must manually update the /etc/netd.conf file. You can configure this file to allow different permission levels for users to edit configurations and run show commands. It also contains a blacklist that hides less frequently used terms from the tabbed autocomplete.

Configuration Variable Default Setting Description
show_linux_command False When true, displays the Linux command running in the background.
enable_ifupdown2 True Enables net wrapping of ifupdown2 commands.
enable_frr True Enables net wrapping of FRRouting commands.
users_with_edit root, cumulus Sets the Linux users with root edit privileges.
groups_with_edit root, cumulus Sets the Linux groups with root edit privileges.
users_with_show root, cumulus Controls which users are allowed to run show commands.
groups_with_show root, cumulus Controls which groups are allowed to run show commands.
ifupdown_blacklist address-purge, bond-ad-actor-sys- prio, bond-ad-actor-syst em, bond-mode, bond-num-grat-arp, bond-num-unsol-na, bond-use-carrier, bond-xmit-hash-pol icy, bridge-bridgeprio, bridge-fd, bridge-hashel, bridge-hashmax, bridge-hello, bridge-maxage, bridge-maxwait, bridge-mclmc, bridge-mclmi, bridge-mcmi, bridge-mcqi, bridge-mcqpi, bridge-mcqri, bridge-mcrouter, bridge-mcsqc, bridge-mcsqi, bridge-pathcosts, bridge-port-pvids, bridge-port-vids, bridge-portprios, bridge-stp, bridge-waitport, broadcast, hwaddress, link-type, mstpctl-ageing, mstpctl-fdelay, mstpctl-forcevers, mstpctl-hello, mstpctl-maxage, mstpctl-maxhops, mstpctl-portp2p, mstpctl-portpathco st, mstpctl-portrestrr ole, mstpctl-portrestrt cn, mstpctl-treeportco st, mstpctl-treeportpr io, mstpctl-txholdcoun t, netmask, preferred-lifetime , scope, vxlan-ageing, vxlan-learning, up, down, bridge-ageing, bridge-gcint, bridge-mcqifaddr, bridge-mcqv4src Hides corner case command options from tab complete, to simplify and streamline output.

Hint

net provides an environment variable to set where the net output is directed. To only use stdout, set the NCLU_TAB_STDOUT environment variable to true. The value is not case sensitive.

Interface Configuration and Management

ifupdown is the network interface manager for Cumulus Linux. Cumulus Linux uses an updated version of this tool, ifupdown2.

For more information on network interfaces, see Switch Port Attributes.

Tip

By default, ifupdown is quiet; use the verbose option -v when you want to know what is going on when bringing an interface down or up.

Basic Commands

To bring up an interface or apply changes to an existing interface, run:

cumulus@switch:~$ sudo ifup

To bring down a single interface, run:

cumulus@switch:~$ sudo ifdown

Tip

ifdown always deletes logical interfaces after bringing them down. Use the --admin-state option if you only want to administratively bring the interface up or down.

To see the link and administrative state, use the ip link show command:

cumulus@switch:~$ ip link show dev swp1
3: swp1:  mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 500
    link/ether 44:38:39:00:03:c1 brd ff:ff:ff:ff:ff:ff

In this example, swp1 is administratively UP and the physical link is UP (LOWER_UP flag). More information on interface administrative state and physical state can be found in this knowledge base article.

To put an interface into an admin down state. The interface remains down after any future reboots or applying configuration changes with ifreload -a. For example:

cumulus@switch:~$ net add interface swp1 link down
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands create the following configuration in the /etc/network/interfaces file:

auto swp1
iface swp1
    link-down yes

ifupdown2 Interface Classes

ifupdown2 provides for the grouping of interfaces into separate classes, where a class is a user-defined label that groups interfaces sharing a common function (like uplink, downlink or compute). You specify classes in the /etc/network/interfaces file.

The most common class is auto, which you configure like this:

auto swp1
iface swp1

You can add other classes using the allow prefix. For example, if you have multiple interfaces used for uplinks, you can make up a class called uplinks:

auto swp1
allow-uplink swp1
iface swp1 inet static
    address 10.1.1.1/31

auto swp2
allow-uplink swp2
iface swp2 inet static
    address 10.1.1.3/31

This allows you to perform operations on only these interfaces using the --allow=uplinks option, or still use the -a options since these interfaces are also in the auto class:

cumulus@switch:~$ sudo ifup --allow=uplinks
cumulus@switch:~$ sudo ifreload -a

If you are using management VRF, you can use the special interface class called mgmt, and put the management interface into that class.

The mgmt interface class is not supported if you are configuring Cumulus Linux using Network Command Line Utility - NCLU.

allow-mgmt eth0
 iface eth0 inet dhcp
     vrf mgmt

 allow-mgmt mgmt
 iface mgmt
     address 127.0.0.1/8
     vrf-table auto

All ifupdown2 commands (ifup, ifdown, ifquery, ifreload) can take a class. Include the --allow=<class> option when you run the command. For example, to reload the configuration for the management interface described above, run:

cumulus@switch:~$ sudo ifreload --allow=mgmt

You can easily bring up or down all interfaces marked with the common auto class in /etc/network/interfaces. Use the -a option. For further details, see individual man pages for ifup(8), ifdown(8), ifreload(8).

To administratively bring up all interfaces marked auto, run:

cumulus@switch:~$ sudo ifup -a

To administratively bring down all interfaces marked auto, run:

cumulus@switch:~$ sudo ifdown -a

To reload all network interfaces marked auto, use the ifreload command, which is equivalent to running ifdown then ifup, the one difference being that ifreload skips any configurations that didn’t change):

cumulus@switch:~$ sudo ifreload -a

Tip

Some syntax checks are done by default, however it may be safer to apply the configs only if the syntax check passes, using the following compound command:

cumulus@switch:~$ sudo bash -c "ifreload -s -a && ifreload -a"

Configure a Loopback Interface

Cumulus Linux has a loopback preconfigured in /etc/network/interfaces. When the switch boots up, it has a loopback interface, called lo, which is up and assigned an IP address of 127.0.0.1.

Tip

The loopback interface lo must always be specified in /etc/network/interfaces and must always be up.

ifupdown Behavior with Child Interfaces

By default, ifupdown recognizes and uses any interface present on the system — whether a VLAN, bond or physical interface — that is listed as a dependent of an interface. You are not required to list them in the interfaces file unless they need a specific configuration, for MTU, link speed, and so forth. And if you need to delete a child interface, you should delete all references to that interface from the interfaces file.

For this example, swp1 and swp2 below do not need an entry in the interfaces file. The following stanzas defined in /etc/network/interfaces provide the exact same configuration:

With Child Interfaces Defined Without Child Interfaces Defined 
auto swp1
iface swp1

auto swp2
iface swp2

auto bridge
iface bridge
    bridge-vlan-aware yes
    bridge-ports swp1 swp2
    bridge-vids 1-100
    bridge-pvid 1
    bridge-stp on

auto bridge
iface bridge
    bridge-vlan-aware yes
    bridge-ports swp1 swp2
    bridge-vids 1-100
    bridge-pvid 1
    bridge-stp on

Bridge in Traditional Mode - Example

For this example, swp1.100 and swp2.100 below do not need an entry in the interfaces file. The following stanzas defined in /etc/network/interfaces provide the exact same configuration:

For more information on the bridge in traditional mode vs the bridge in VLAN-aware mode, please read this knowledge base article.

ifupdown2 Interface Dependencies

ifupdown2 understands interface dependency relationships. When ifup and ifdown are run with all interfaces, they always run with all interfaces in dependency order. When run with the interface list on the command line, the default behavior is to not run with dependents. But if there are any built-in dependents, they will be brought up or down.

To run with dependents when you specify the interface list, use the --with-depends option. --with-depends walks through all dependents in the dependency tree rooted at the interface you specify. Consider the following example configuration:

auto bond1
iface bond1
    address 100.0.0.2/16
    bond-slaves swp29 swp30

auto bond2
iface bond2
    address 100.0.0.5/16
    bond-slaves swp31 swp32

auto br2001
iface br2001
    address 12.0.1.3/24
    bridge-ports bond1.2001 bond2.2001
    bridge-stp on

Using ifup --with-depends br2001 brings up all dependents of br2001: bond1.2001, bond2.2001, bond1, bond2, bond1.2001, bond2.2001, swp29, swp30, swp31, swp32.

cumulus@switch:~$ sudo ifup --with-depends br2001

Similarly, specifying ifdown --with-depends br2001 brings down all dependents of br2001: bond1.2001, bond2.2001, bond1, bond2, bond1.2001, bond2.2001, swp29, swp30, swp31, swp32.

cumulus@switch:~$ sudo ifdown --with-depends br2001

Warning

As mentioned earlier, ifdown2 always deletes logical interfaces after bringing them down. Use the --admin-state option if you only want to administratively bring the interface up or down. In terms of the above example, ifdown br2001 deletes br2001.

To guide you through which interfaces will be brought down and up, use the --print-dependency option to get the list of dependents.

Use ifquery --print-dependency=list -a to get the dependency list of all interfaces:

cumulus@switch:~$ sudo ifquery --print-dependency=list -a
lo : None
eth0 : None
bond0 : ['swp25', 'swp26']
bond1 : ['swp29', 'swp30']
bond2 : ['swp31', 'swp32']
br0 : ['bond1', 'bond2']
bond1.2000 : ['bond1']
bond2.2000 : ['bond2']
br2000 : ['bond1.2000', 'bond2.2000']
bond1.2001 : ['bond1']
bond2.2001 : ['bond2']
br2001 : ['bond1.2001', 'bond2.2001']
swp40 : None
swp25 : None
swp26 : None
swp29 : None
swp30 : None
swp31 : None
swp32 : None

To print the dependency list of a single interface, use:

cumulus@switch:~$ sudo ifquery --print-dependency=list br2001
br2001 : ['bond1.2001', 'bond2.2001']
bond1.2001 : ['bond1']
bond2.2001 : ['bond2']
bond1 : ['swp29', 'swp30']
bond2 : ['swp31', 'swp32']
swp29 : None
swp30 : None
swp31 : None
swp32 : None

To print the dependency information of an interface in dot format:

cumulus@switch:~$ sudo ifquery --print-dependency=dot br2001
/* Generated by GvGen v.0.9 (http://software.inl.fr/trac/wiki/GvGen) */
digraph G {
    compound=true;
    node1 [label="br2001"];
    node2 [label="bond1.2001"];
    node3 [label="bond2.2001"];
    node4 [label="bond1"];
    node5 [label="bond2"];
    node6 [label="swp29"];
    node7 [label="swp30"];
    node8 [label="swp31"];
    node9 [label="swp32"];
    node1->node2;
    node1->node3;
    node2->node4;
    node3->node5;
    node4->node6;
    node4->node7;
    node5->node8;
    node5->node9;
}

You can use dot to render the graph on an external system where dot is installed.

../../images/interfaces.png

To print the dependency information of the entire interfaces file:

cumulus@switch:~$ sudo ifquery --print-dependency=dot -a >interfaces_all.dot
../../images/interfaces_all.png

Subinterfaces

On Linux an interface is a network device, and can be either a physical device like switch port (such as swp1), or virtual, like a VLAN (vlan100). A VLAN subinterface is a VLAN device on an interface, and the VLAN ID is appended to the parent interface using dot (.) VLAN notation. For example, a VLAN with ID 100 that is a subinterface of swp1 is named swp1.100 in Cumulus Linux. The dot VLAN notation for a VLAN device name is a standard way to specify a VLAN device on Linux. Many Linux configuration tools, most notably ifupdown2 and its predecessor ifupdown, recognize such a name as a VLAN interface name.

A VLAN subinterface only receives traffic tagged for that VLAN, so swp1.100 only receives packets tagged with VLAN 100 on switch port swp1. Similarly, any transmits from swp1.100 result in tagging the packet with VLAN 100.

For an MLAG deployment, the peerlink interface that connects the two switches in the MLAG pair has a VLAN subinterface named 4094 by default, provided you configured the subinterface with Network Command Line Utility - NCLU. The peerlink.4094 subinterface only receives traffic tagged for VLAN 4094.

ifup and Upper (Parent) Interfaces

When you run ifup on a logical interface (like a bridge, bond or VLAN interface), if the ifup resulted in the creation of the logical interface, by default it implicitly tries to execute on the interface’s upper (or parent) interfaces as well. This helps in most cases, especially when a bond is brought down and up, as in the example below. This section describes the behavior of bringing up the upper interfaces.

Consider this example configuration:

auto br100
iface br100
    bridge-ports bond1.100 bond2.100

auto bond1
iface bond1
    bond-slaves swp1 swp2

If you run ifdown bond1, ifdown deletes bond1 and the VLAN interface on bond1 (bond1.100); it also removes bond1 from the bridge br100. Next, when you run ifup bond1, it creates bond1 and the VLAN interface on bond1 (bond1.100); it also executes ifup br100 to add the bond VLAN interface (bond1.100) to the bridge br100.

As you can see above, implicitly bringing up the upper interface helps, but there can be cases where an upper interface (like br100) is not in the right state, which can result in warnings. The warnings are mostly harmless.

If you want to disable these warnings, you can disable the implicit upper interface handling by setting skip_upperifaces=1 in /etc/network/ifupdown2/ifupdown2.conf.

With skip_upperifaces=1, you will have to explicitly execute ifup on the upper interfaces. In this case, you will have to run ifup br100 after an ifup bond1 to add bond1 back to bridge br100.

Note

Although specifying a subinterface like swp1.100 and then running ifup swp1.100 will also result in the automatic creation of the swp1 interface in the kernel, Cumulus Networks recommends you specify the parent interface swp1 as well. A parent interface is one where any physical layer configuration can reside, such as link-speed 1000 or link-duplex full.

It’s important to note that if you only create swp1.100 and not swp1, then you cannot run ifup swp1 since you did not specify it.

Configure IP Addresses

IP addresses are configured with the net add interface command.

Example IP Address Configuration

The following commands configure three IP addresses for swp1: two IPv4 addresses, and one IPv6 address.

cumulus@switch:~$ net add interface swp1 ip address 12.0.0.1/30
cumulus@switch:~$ net add interface swp1 ip address 12.0.0.2/30
cumulus@switch:~$ net add interface swp1 ipv6 address 2001:DB8::1/126
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands create the following code snippet:

auto swp1
iface swp1
    address 12.0.0.1/30
    address 12.0.0.2/30
    address 2001:DB8::1/126

Note

You can specify both IPv4 and IPv6 addresses for the same interface.

For IPv6 addresses, you can create or modify the IP address for an interface using either “::” or “0:0:0” notation. Both of the following examples are valid:

cumulus@switch:~$ net add bgp neighbor 2620:149:43:c109:0:0:0:5 remote-as internal
cumulus@switch:~$
cumulus@switch:~$ net add interface swp1 ipv6 address 2001:DB8::1/126

Note

The address method and address family are added by NCLU when needed, specifically when you are creating DHCP or loopback interfaces.

auto lo
iface lo inet loopback

To show the assigned address on an interface, use ip addr show:

cumulus@switch:~$ ip addr show dev swp1
3: swp1:  mtu 1500 qdisc pfifo_fast state UP qlen 500
    link/ether 44:38:39:00:03:c1 brd ff:ff:ff:ff:ff:ff
    inet 192.0.2.1/30 scope global swp1
    inet 192.0.2.2/30 scope global swp1
    inet6 2001:DB8::1/126 scope global tentative
       valid_lft forever preferred_lft forever

Specify IP Address Scope

ifupdown2 does not honor the configured IP address scope setting in /etc/network/interfaces, treating all addresses as global. It does not report an error. Consider this example configuration:

auto swp2
iface swp2
    address 35.21.30.5/30
    address 3101:21:20::31/80
    scope link

When you run ifreload -a on this configuration, ifupdown2 considers all IP addresses as global.

cumulus@switch:~$ ip addr show swp2
5: swp2:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 74:e6:e2:f5:62:82 brd ff:ff:ff:ff:ff:ff
inet 35.21.30.5/30 scope global swp2
valid_lft forever preferred_lft forever
inet6 3101:21:20::31/80 scope global
valid_lft forever preferred_lft forever
inet6 fe80::76e6:e2ff:fef5:6282/64 scope link
valid_lft forever preferred_lft forever

To work around this issue, configure the IP address scope:

cumulus@switch:~$ net add interface swp6 post-up ip address add 71.21.21.20/32 dev swp6 scope site
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands create the following code snippet in the /etc/network/interfaces file:

auto swp6
iface swp6
    post-up ip address add 71.21.21.20/32 dev swp6 scope site

Now it has the correct scope:

cumulus@switch:~$ ip addr show swp6
9: swp6:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 74:e6:e2:f5:62:86 brd ff:ff:ff:ff:ff:ff
inet 71.21.21.20/32 scope site swp6
valid_lft forever preferred_lft forever
inet6 fe80::76e6:e2ff:fef5:6286/64 scope link
valid_lft forever preferred_lft forever

Purge Existing IP Addresses on an Interface

By default, ifupdown2 purges existing IP addresses on an interface. If you have other processes that manage IP addresses for an interface, you can disable this feature including the address-purge setting in the interface’s configuration.

cumulus@switch:~$ net add interface swp1 address-purge no
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands create the following configuration snippet in the /etc/network/interfaces file:

CDATA[auto swp1
iface swp1
    address-purge no

Note

Purging existing addresses on interfaces with multiple iface stanzas is not supported. Doing so can result in the configuration of multiple addresses for an interface after you change an interface address and reload the configuration with ifreload -a. If this happens, you must shut down and restart the interface with ifup and ifdown, or manually delete superfluous addresses with ip address delete specify.ip.address.here/mask dev DEVICE. See also the Caveats and Errata section below for some cautions about using multiple iface stanzas for the same interface.

Specify User Commands

You can specify additional user commands in the interfaces file. As shown in the example below, the interface stanzas in /etc/network/interfaces can have a command that runs at pre-up, up, post-up, pre-down, down, and post-down:

cumulus@switch:~$ net add interface swp1 post-up /sbin/foo bar
cumulus@switch:~$ net add interface ip address 12.0.0.1/30
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands create the following configuration in the /etc/network/interfaces file:

auto swp1
iface swp1
    address 12.0.0.1/30
    post-up /sbin/foo bar

Any valid command can be hooked in the sequencing of bringing an interface up or down, although commands should be limited in scope to network-related commands associated with the particular interface.

For example, it wouldn’t make sense to install some Debian package on ifup of swp1, even though that is technically possible. See man interfaces for more details.

Warning

If your post-up command also starts, restarts or reloads any systemd service, you must use the --no-block option with systemctl. Otherwise, that service or even the switch itself may hang after starting or restarting.

For example, to restart the dhcrelay service after bringing up VLAN 100, first run:

cumulus@switch:~$ net add vlan 100 post-up systemctl --no-block restart dhcrelay.service
cumulus@switch:~$ net commit

These commands create the following configuration in the /etc/network/interfaces file:

auto bridge
iface bridge
    bridge-vids 100
    bridge-vlan-aware yes

auto vlan100
iface vlan100
    post-up systemctl --no-block restart dhcrelay.service
    vlan-id 100
    vlan-raw-device bridge

Source Interface File Snippets

Sourcing interface files helps organize and manage the interfaces file. For example:

cumulus@switch:~$ cat /etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet dhcp

source /etc/network/interfaces.d/bond0

The contents of the sourced file used above are:

cumulus@switch:~$ cat /etc/network/interfaces.d/bond0
auto bond0
iface bond0
    address 14.0.0.9/30
    address 2001:ded:beef:2::1/64
    bond-slaves swp25 swp26

Use Globs for Port Lists

NCLU supports globs to define port lists (that is, a range of ports). The glob keyword is implied when you specify bridge ports and bond slaves:

cumulus@switch:~$ net add bridge bridge ports swp1-4,6,10-12
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

Tip

While you must use commas to separate different ranges of ports in the NCLU command, the /etc/network/interfaces file renders the list of ports individually, as in the example output below.

These commands produce the following snippet in the /etc/network/interfaces file:

...

auto bridge
iface bridge
    bridge-ports swp1 swp2 swp3 swp4 swp6 swp10 swp11 swp12
    bridge-vlan-aware yes
auto swp1
iface swp1

auto swp2
iface swp2

auto swp3
iface swp3

auto swp4
iface swp4

auto swp6
iface swp6

auto swp10
iface swp10

auto swp11
iface swp11

auto swp12
iface swp12

Use Templates

ifupdown2 Mako-style templates. The Mako template engine is run over the interfaces file before parsing.

Use the template to declare cookie-cutter bridges in the interfaces file:

%for v in [11,12]:
auto vlan${v}
iface vlan${v}
    address 10.20.${v}.3/24
    bridge-ports glob swp19-20.${v}
    bridge-stp on
%endfor

And use it to declare addresses in the interfaces file:

%for i in [1,12]:
auto swp${i}
iface swp${i}
    address 10.20.${i}.3/24

Note

Regarding Mako syntax, use square brackets ([1,12]) to specify a list of individual numbers (in this case, 1 and 12). Use range(1,12) to specify a range of interfaces.

Tip

You can test your template and confirm it evaluates correctly by running mako-render /etc/network/interfaces.

Tip

For more examples of configuring Mako templates, read this `knowledge base article https://support.cumulusnetworks.com/hc/en-us/articles/202868023>`_.

To comment out content in Mako templates, use double hash marks (##). For example:

## % for i in range(1, 4):
## auto swp${i}
## iface swp${i}
## % endfor
##

Run ifupdown Scripts under /etc/network/ with ifupdown2

Unlike the traditional ifupdown system, ifupdown2 does not run scripts installed in /etc/network/*/ automatically to configure network interfaces.

To enable or disable ifupdown2 scripting, edit the addon_scripts_support line in the /etc/network/ifupdown2/ifupdown2.conf file. 1 enables scripting and 2 disables scripting. The following example enables scripting.

cumulus@switch:~$ sudo nano /etc/network/ifupdown2/ifupdown2.conf
# Support executing of ifupdown style scripts.
# Note that by default python addon modules override scripts with the same name
addon_scripts_support=1

ifupdown2 sets the following environment variables when executing commands:

  • $IFACE represents the physical name of the interface being processed; for example, br0 or vxlan42. The name is obtained from the /etc/network/interfaces file.
  • $LOGICAL represents the logical name (configuration name) of the interface being processed.
  • $METHOD represents the address method; for example, loopback, DHCP, DHCP6, manual, static, and so on.
  • $ADDRFAM represents the address families associated with the interface, formatted in a comma-separated list; for example, "inet,inet6".

Add Descriptions to Interfaces

You can add descriptions to the interfaces configured in /etc/network/interfaces by using the alias keyword.

The following commands create an alias for swp1:

cumulus@switch:~$ net add interface swp1 alias hypervisor_port_1
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands create the following code snippet:

auto swp1
iface swp1
    alias hypervisor_port_1

You can query the interface description using NCLU:

cumulus@switch$ net show interface swp1
    Name   MAC                Speed     MTU   Mode
--  ----   -----------------  -------   -----  ---------
UP  swp1   44:38:39:00:00:04  1G        1500   Access/L2
Alias
-----
hypervisor_port_1

Interface descriptions also appear in the SNMP OID IF-MIB::ifAlias.

Note

Aliases are limited to 256 characters.

To show the interface description (alias) for all interfaces on the switch, run the net show interface alias command. For example:

cumulus@switch:~$ net show interface alias
State    Name            Mode              Alias
-----    -------------   -------------     ------------------
UP       bond01          LACP
UP       bond02          LACP
UP       bridge          Bridge/L2
UP       eth0            Mgmt
UP       lo              Loopback          loopback interface
UP       mgmt            Interface/L3
UP       peerlink        LACP
UP       peerlink.4094   SubInt/L3
UP       swp1            BondMember        hypervisor_port_1
UP       swp2            BondMember        to Server02
...

To show the interface description for all interfaces on the switch in JSON format, run the net show interface alias json command.

Caveats and Errata

While ifupdown2 supports the inclusion of multiple iface stanzas for the same interface, Cumulus Networks recommends you use a single iface stanza for each interface, if possible.

There are cases where you must specify more than one iface stanza for the same interface. For example, the configuration for a single interface can come from many places, like a template or a sourced file.

If you do specify multiple iface stanzas for the same interface, make sure the stanzas do not specify the same interface attributes. Otherwise, unexpected behavior can result.

For example, swp1 is configured in two places:

cumulus@switch:~$ cat /etc/network/interfaces

source /etc/network/interfaces.d/speed_settings

auto swp1
iface swp1
  address 10.0.14.2/24

As well as /etc/network/interfaces.d/speed_settings

cumulus@switch:~$ cat /etc/network/interfaces.d/speed_settings

auto swp1
iface swp1
  link-speed 1000
  link-duplex full

ifupdown2 correctly parses a configuration like this because the same attributes are not specified in multiple iface stanzas.

And, as stated in the note above, you cannot purge existing addresses on interfaces with multiple iface stanzas.

ifupdown2 and sysctl

For sysctl commands in the pre-up, up, post-up, pre-down, down, and post-down lines that use the $IFACE variable, if the interface name contains a dot (.), ifupdown2 does not change the name to work with sysctl. For example, the interface name bridge.1 is not converted to bridge/1.

Long Interface Names

The Linux kernel limits interface names to 15 characters in length and cannot have a number as the first character. Longer interface names can result in errors. To work around this issue, remove the interface from the /etc/network/interfaces file, then restart the networking.service.

cumulus@switch:~$ sudo vi /etc/network/interfaces
cumulus@switch:~$ sudo systemctl restart networking.service

switchd

switchd is the daemon at the heart of Cumulus Linux. It communicates between the switch and Cumulus Linux, and all the applications running on Cumulus Linux.

The switchd configuration is stored in /etc/cumulus/switchd.conf.

The switchd File System

switchd also exports a file system, mounted on /cumulus/switchd, that presents all the switchd configuration options as a series of files arranged in a tree structure. You can see the contents by parsing the switchd tree; run tree /cumulus/switchd. The output below is for a switch with one switch port configured:

cumulus@switch:~$ sudo tree /cumulus/switchd/
/cumulus/switchd/
|-- config
|   |-- acl
|   |   |-- non_atomic_update_mode
|   |   `-- optimize_hw
|   |-- arp
|   |   `-- next_hops
|   |-- buf_util
|   |   |-- measure_interval
|   |   `-- poll_interval
|   |-- coalesce
|   |   |-- reducer
|   |   `-- timeout
|   |-- disable_internal_restart
|   |-- ignore_non_swps
|   |-- interface
|   |   |-- swp1
|   |   |   `-- storm_control
|   |   |       |-- broadcast
|   |   |       |-- multicast
|   |   |       `-- unknown_unicast
|   |-- logging
|   |-- route
|   |   |-- host_max_percent
|   |   `-- table
|   `-- stats
|       `-- poll_interval
|-- ctrl
|   |-- acl
|   |-- hal
|   |   `-- resync
|   |-- logger
|   |-- netlink
|   |   `-- resync
|   |-- resync
|   `-- sample
|       `-- ulog_channel
|-- run
|   `-- route_info
|       |-- ecmp_nh
|       |   |-- count
|       |   |-- max
|       |   `-- max_per_route
|       |-- host
|       |   |-- count
|       |   |-- count_v4
|       |   |-- count_v6
|       |   `-- max
|       |-- mac
|       |   |-- count
|       |   `-- max
|       `-- route
|           |-- count_0
|           |-- count_1
|           |-- count_total
|           |-- count_v4
|           |-- count_v6
|           |-- mask_limit
|           |-- max_0
|           |-- max_1
|           `-- max_total
`-- version

Configure switchd Parameters

You can use cl-cfg to configure many switchd parameters at runtime (like ACLs, interfaces, and route table utilization), which minimizes disruption to your running switch. However, some options are read only and cannot be configured at runtime.

For example, to see data related to routes, run:

cumulus@switch:~$ sudo cl-cfg -a switchd | grep route
route.table = 254
route.host_max_percent = 50
cumulus@cumulus:~$

To modify the configuration, run cl-cfg -w. For example, to set the buffer utilization measurement interval to 1 minute, run:

cumulus@switch:~$ sudo cl-cfg -w switchd buf_util.measure_interval=1

To verify that the value changed, use grep:

cumulus@switch:~$ cl-cfg -a switchd | grep buf
buf_util.poll_interval = 0
buf_util.measure_interval = 1

Note

You can get some of this information by running cl-resource-query; though you cannot update the switchd configuration with it.

Restart switchd

Whenever you modify any switchd hardware configuration file (typically changing any *.conf file that requires making a change to the switching hardware, like /etc/cumulus/datapath/traffic.conf), you must restart switchd for the change to take effect:

cumulus@switch:~$ sudo systemctl restart switchd.service

Note

You do not have to restart the switchd service when you update a network interface configuration (that is, edit /etc/network/interfaces).

Warning

Restarting switchd causes all network ports to reset in addition to resetting the switch hardware configuration.

Setting Date and Time

Setting the time zone, monitoring date and time requires root privileges; use sudo.

Set the Time Zone

You can use one of two methods to set the time zone on the switch:

  • Edit the /etc/timezone file.
  • Use the guided wizard.

Edit the /etc/timezone File

To see the current time zone time zone, list the contents of /etc/timezone:

cumulus@switch:~$ cat /etc/timezone
US/Eastern

Edit the file to add your desired time zone. A list of valid time zones can be found at the following link.

Use the following command to apply the new time zone immediately.

cumulus@switch:~$ sudo dpkg-reconfigure --frontend noninteractive tzdata

Use the Guided Wizard

To set the time zone using the guided wizard, run dpkg-reconfigure tzdata as root:

cumulus@switch:~$ sudo dpkg-reconfigure tzdata

Then navigate the menus to enable the time zone you want. The following example selects the US/Pacific time zone:

cumulus@switch:~$ sudo dpkg-reconfigure tzdata

Configuring tzdata
------------------

Please select the geographic area in which you live. Subsequent configuration
questions will narrow this down by presenting a list of cities, representing
the time zones in which they are located.

  1. Africa      4. Australia  7. Atlantic  10. Pacific  13. Etc
  2. America     5. Arctic     8. Europe    11. SystemV
  3. Antarctica  6. Asia       9. Indian    12. US
Geographic area: 12

Please select the city or region corresponding to your time zone.

  1. Alaska    4. Central  7. Indiana-Starke  10. Pacific
  2. Aleutian  5. Eastern  8. Michigan        11. Pacific-New
  3. Arizona   6. Hawaii   9. Mountain        12. Samoa
Time zone: 10

Current default time zone: 'US/Pacific'
Local time is now:      Mon Jun 17 09:27:45 PDT 2013.
Universal Time is now:  Mon Jun 17 16:27:45 UTC 2013.

For more info see the Debian System Administrator’s Manual – Time.

Set the Date and Time

The switch contains a battery backed hardware clock that maintains the time while the switch is powered off and in between reboots. When the switch is running, the Cumulus Linux operating system maintains its own software clock.

During boot up, the time from the hardware clock is copied into the operating system’s software clock. The software clock is then used for all timekeeping responsibilities. During system shutdown, the software clock is copied back to the battery backed hardware clock.

You can set the date and time on the software clock using the date command. First, determine your current time zone:

cumulus@switch$ date +%Z

Note

If you need to reconfigure the current time zone, refer to the instructions above.

Then, to set the system clock according to the time zone configured:

cumulus@switch$ sudo date -s "Tue Jan 12 00:37:13 2016"

See man date(1) for more information.

You can write the current value of the system (software) clock to the hardware clock using the hwclock command:

cumulus@switch$ sudo hwclock -w

See man hwclock(8) for more information.

You can find a good overview of the software and hardware clocks in the Debian System Administrator’s Manual – Time, specifically the section Setting and showing hardware clock.

Set the Time Using NTP and NCLU

The ntpd daemon running on the switch implements the NTP protocol. It synchronizes the system time with time servers listed in /etc/ntp.conf. The ntpd daemon is started at boot by default. See man ntpd(8) for ntpd details. You can check this site for an explanation of the output.

By default, /etc/ntp.conf contains some default time servers. You can specify the NTP server or servers you want to use with NCLU; include the iburst option to increase the sync speed.

cumulus@switch:~$ net add time ntp server 4.cumulusnetworks.pool.ntp.org iburst
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands add the NTP server to the list of servers in /etc/ntp.conf:

# pool.ntp.org maps to about 1000 low-stratum NTP servers.  Your server will
# pick a different set every time it starts up.  Please consider joining the
# pool: <http://www.pool.ntp.org/join.html>
server 0.cumulusnetworks.pool.ntp.org iburst
server 1.cumulusnetworks.pool.ntp.org iburst
server 2.cumulusnetworks.pool.ntp.org iburst
server 3.cumulusnetworks.pool.ntp.org iburst
server 4.cumulusnetworks.pool.ntp.org iburst

To set the initial date and time via NTP before starting the ntpd daemon, use ntpd -q. This is the same as ntpdate, which is to be retired and no longer available. See man ntp.conf(5) for details on configuring ntpd using ntp.conf.

Note

ntpd -q can hang if the time servers are not reachable.

To verify that ntpd is running on the system:

cumulus@switch:~$ ps -ef | grep ntp
ntp       4074     1  0 Jun20 ?        00:00:33 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 101:102

To check the NTP peer status:

cumulus@switch:~$ net show time ntp servers
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+minime.fdf.net  58.180.158.150   3 u  140 1024  377   55.659    0.339   1.464
+69.195.159.158  128.138.140.44   2 u  259 1024  377   41.587    1.011   1.677
*chl.la          216.218.192.202  2 u  210 1024  377    4.008    1.277   1.628
+vps3.drown.org  17.253.2.125     2 u  743 1024  377   39.319   -0.316   1.384

To remove one or more NTP servers:

cumulus@switch:~$ net del time ntp server 0.cumulusnetworks.pool.ntp.org iburst
cumulus@switch:~$ net del time ntp server 1.cumulusnetworks.pool.ntp.org iburst
cumulus@switch:~$ net del time ntp server 2.cumulusnetworks.pool.ntp.org iburst
cumulus@switch:~$ net del time ntp server 3.cumulusnetworks.pool.ntp.org iburst
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

Specify the NTP Source Interface

You can change the source interface that NTP uses if you want to use an interface other than eth0, which is the default.

cumulus@switch:~$ net add time ntp source swp10
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands create the following configuration snippet in the ntp.conf file:

...

# Specify interfaces
interface listen swp10

...

NTP Default Configuration

The default NTP configuration comprises the following servers, which are listed in the /etc/ntpd.conf file:

The contents of the /etc/ntpd.conf file are listed below.

# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help

driftfile /var/lib/ntp/ntp.drift


# Enable this if you want statistics to be logged.
#statsdir /var/log/ntpstats/

statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable


# You do need to talk to an NTP server or two (or three).
#server ntp.your-provider.example

# pool.ntp.org maps to about 1000 low-stratum NTP servers.  Your server will
# pick a different set every time it starts up.  Please consider joining the
# pool: <http://www.pool.ntp.org/join.html>
server 0.cumulusnetworks.pool.ntp.org iburst
server 1.cumulusnetworks.pool.ntp.org iburst
server 2.cumulusnetworks.pool.ntp.org iburst
server 3.cumulusnetworks.pool.ntp.org iburst


# Access control configuration; see /usr/share/doc/ntp-doc/html/accopt.html for
# details.  The web page <http://support.ntp.org/bin/view/Support/AccessRestrictions>
# might also be helpful.
#
# Note that "restrict" applies to both servers and clients, so a configuration
# that might be intended to block requests from certain clients could also end
# up blocking replies from your own upstream servers.

# By default, exchange time with everybody, but don't allow configuration.
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery

# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1

# Clients from this (example!) subnet have unlimited access, but only if
# cryptographically authenticated.
#restrict 192.168.123.0 mask 255.255.255.0 notrust


# If you want to provide time to your local subnet, change the next line.
# (Again, the address is an example only.)
#broadcast 192.168.123.255

# If you want to listen to time broadcasts on your local subnet, de-comment the
# next lines.  Please do this only if you trust everybody on the network!
#disable auth
#broadcastclient

# Specify interfaces, don't listen on switch ports
interface listen eth0

Precision Time Protocol (PTP) Boundary Clock

With the growth of low latency and high performance applications, precision timing has become increasingly important. Precision Time Protocol (PTP) is used to synchronize clocks in a network and is capable of sub-microsecond accuracy. The clocks are organized in a master-slave hierarchy. The slaves are synchronized to their masters, which can be slaves to their own masters. The hierarchy is created and updated automatically by the best master clock (BMC) algorithm, which runs on every clock. The grandmaster clock is the top-level master and is typically synchronized by using a Global Positioning System (GPS) time source to provide a high-degree of accuracy.

A boundary clock has multiple ports; one or more master ports and one or more slave ports. The master ports provide time (the time can originate from other masters further up the hierarchy) and the slave ports receive time. The boundary clock absorbs sync messages in the slave port, uses that port to set its clock, then generates new sync messages from this clock out of all of its master ports.

Cumulus Linux includes the ptp4l package for PTP, which uses the phc2sys daemon to synchronize the PTP clock with the system clock.

Note

  • Cumulus Linux currently supports PTP on the Mellanox Spectrum ASIC only.
  • If you do not perform a binary (full image) install of Cumulus Linux 3.6 or later, you need to install the ptp4l package with the apt-get install ptp4l command.
  • PTP is supported in boundary clock mode only (the switch provides timing to downstream servers; it is a slave to a higher-level clock and a master to downstream clocks).
  • The switch uses hardware time stamping to capture timestamps from an Ethernet frame at the physical layer. This allows PTP to account for delays in message transfer and greatly improves the accuracy of time synchronization. ! * Only IPv4/UDP PTP packets are supported.
  • Only a single PTP domain per network is supported. A PTP domain is a network or a portion of a network within which all the clocks are synchronized.

In the following example, boundary clock 2 receives time from Master 1 (the grandmaster) on a PTP slave port, sets its clock and passes the time down from the PTP master port to boundary clock 1. Boundary clock 1 receives the time on a PTP slave port, sets its clock and passes the time down the hierarchy through the PTP master ports to the hosts that receive the time.

PTP example

Enable the PTP Boundary Clock on the Switch

To enable the PTP boundary clock on the switch:

  1. Open the /etc/cumulus/switchd.conf file in a text editor and add the following line:

    ptp.timestamping = TRUE

  2. Restart switchd:

    cumulus@switch:~$ sudo systemctl restart switchd.service

Configure the PTP Boundary Clock

To configure a boundary clock:

  1. Configure the interfaces on the switch that you want to use for PTP. Each interface must be configured as a layer 3 routed interface with an IP address. ! PTP is supported on BGP unnumbered interfaces. ! PTP is not supported on switched virtual interfaces (SVIs).

    cumulus@switch:~$ net add interface swp13s0 ip address 10.0.0.9/32
cumulus@switch:~$ net add interface swp13s1 ip address 10.0.0.10/32
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

  2. Configure PTP options on the switch:

    • Set the gm-capable option to no to configure the switch to be a boundary clock.
    • Set the priority, which selects the best master clock. You can set priority 1 or 2. For each priority, you can use a number between 0 and 255. The default priority is 255. For the boundary clock, use a number above 128. The lower priority is applied first.
    • Add the time-stamping parameter. The switch automatically enables hardware time-stamping to capture timestamps from an Ethernet frame at the physical layer. If you are testing PTP in a virtual environment, hardware time-stamping is not available; however the time-stamping parameter is still required.
    • Add the PTP master and slave interfaces. You do not specify which is a master interface and which is a slave interface; this is determined by the PTP packet received.

    The following commands provide an example configuration:

    cumulus@switch:~$ net add ptp global gm-capable no
cumulus@switch:~$ net add ptp global priority2 254
cumulus@switch:~$ net add ptp global priority1 254
cumulus@switch:~$ net add ptp global time-stamping
cumulus@switch:~$ net add ptp interface swp13s0
cumulus@switch:~$ net add ptp interface swp13s1
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

    The ptp4l man page describes all the configuration parameters.

  3. Restart the ptp4l and phc2sys daemons:

    cumulus@switch:~$ sudo systemctl restart ptp41.service phc2sys.service

    The configuration is saved in the /etc/ptp4l.conf file.

  4. Enable the services to start at boot time:

    cumulus@switch:~$ sudo systemctl enable ptp41.service phc2sys.service

Example Configuration

In the following example, the boundary clock on the switch receives time from Master 1 (the grandmaster) on PTP slave port swp3s0, sets its clock and passes the time down through PTP master ports swp3s1, swp3s2, and swp3s3 to the hosts that receive the time.

PTP configuration

The configuration for the above example is shown below. The example assumes that you have already configured the layer 3 routed interfaces (swp3s0, swp3s1, swp3s2, and swp3s3) you want to use for PTP.

cumulus@switch:~$ net add ptp global gm-capable no
cumulus@switch:~$ net add ptp global priority2 254
cumulus@switch:~$ net add ptp global priority1 254
cumulus@switch:~$ net add ptp global time-stamping
cumulus@switch:~$ net add ptp interface swp3s0
cumulus@switch:~$ net add ptp interface swp3s1
cumulus@switch:~$ net add ptp interface swp3s2
cumulus@switch:~$ net add ptp interface swp3s3
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

Verify PTP Boundary Clock Configuration

To view a summary of the PTP configuration on the switch, run the net show configuration ptp command:

cumulus@switch:~$ net show configuration ptp

ptp
  global

    slaveOnly
      0

    priority1
      255

    priority2
      255

    domainNumber
      0

    logging_level
      5

    path_trace_enabled
      0

    use_syslog
      1

    verbose
      0

    summary_interval
      0

    time_stamping
      hardware

    gmCapable
      0
  swp15s0
  swp15s1
...

View PTP Status Information

To view PTP status information, run the net show ptp parent_data_set command:

cumulus@switch:~$ net show ptp parent_data_set
parent_data_set
===============
parentPortIdentity                    000200.fffe.000001-1
parentStats                           0
observedParentOffsetScaledLogVariance 0xffff
observedParentClockPhaseChangeRate    0x7fffffff
grandmasterPriority1                  127
gm.ClockClass                         248
gm.ClockAccuracy                      0xfe
gm.OffsetScaledLogVariance            0xffff
grandmasterPriority2                  127
grandmasterIdentity                   000200.fffe.000001

To view the additional PTP status information, including the delta in nanoseconds from the master clock, run the sudo pmc -u -b 0 'GET TIME_STATUS_NP' command:

cumulus@switch:~$ sudo pmc -u -b 0 'GET TIME_STATUS_NP'
sending: GET TIME_STATUS_NP
    7cfe90.fffe.f56dfc-0 seq 0 RESPONSE MANAGEMENT TIME_STATUS_NP
        master_offset              12610
        ingress_time               1525717806521177336
        cumulativeScaledRateOffset +0.000000000
        scaledLastGmPhaseChange    0
        gmTimeBaseIndicator        0
        lastGmPhaseChange          0x0000'0000000000000000.0000
        gmPresent                  true
        gmIdentity                 000200.fffe.000005
    000200.fffe.000005-1 seq 0 RESPONSE MANAGEMENT TIME_STATUS_NP
        master_offset              0
        ingress_time               0
        cumulativeScaledRateOffset +0.000000000
        scaledLastGmPhaseChange    0
        gmTimeBaseIndicator        0
        lastGmPhaseChange          0x0000'0000000000000000.0000
        gmPresent                  false
        gmIdentity                 000200.fffe.000005
    000200.fffe.000006-1 seq 0 RESPONSE MANAGEMENT TIME_STATUS_NP
        master_offset              5544033534
        ingress_time               1525717812106811842
        cumulativeScaledRateOffset +0.000000000
        scaledLastGmPhaseChange    0
        gmTimeBaseIndicator        0
        lastGmPhaseChange          0x0000'0000000000000000.0000
        gmPresent                  true
        gmIdentity                 000200.fffe.000005

Delete PTP Boundary Clock Configuration

To delete the PTP configuration, delete the PTP master and slave interfaces. The following example commands delete the PTP interfaces swp3s0, swp3s1, and swp3s2.

cumulus@switch:~$ net del ptp interface swp3s0
cumulus@switch:~$ net del ptp interface swp3s1
cumulus@switch:~$ net del ptp interface swp3s2
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

Use NTP in a DHCP Environment

If you use DHCP and want to specify your NTP servers, you must specify an alternate configuration file for NTP.

Before you create the file, ensure that the DHCP-generated configuration file exists. In Cumulus Linux 3.6.1 and later (which uses NTP 1:4.2.8), the DHCP-generated file is named /run/ntp.conf.dhcp while in Cumulus Linux 3.6.0 and earlier (which uses NTP 1:4.2.6) the file is named /var/lib/ntp/ntp.conf.dhcp. This file is generated by the /etc/dhcp/dhclient-exit-hooks.d/ntp script and is a copy of the default /etc/ntp.conf with a modified server list from the DHCP server. If this file does not exist and you plan on using DHCP in the future, you can copy your current /etc/ntp.conf file to the location of the DHCP file.

To use an alternate configuration file that persists across upgrades of Cumulus Linux, create a systemd unit override file called /etc/systemd/system/ntp.service.d/config.conf and add the following content:

cumulus@switch:~$ sudo echo '
[Service]
ExecStart=
ExecStart=/usr/sbin/ntpd -n -u ntp:ntp -g -c /run/ntp.conf.dhcp
' > ~/over
sudo mkdir -p /etc/systemd/system/ntp.service.d
sudo mv ~/over/etc/systemd/system/ntp.service.d/dhcp.conf
sudo chown root:root/etc/systemd/system/ntp.service.d/dhcp.conf

To validate that your configuration, run these commands:

cumulus@switch:~$ sudo systemctl daemon-reload
cumulus@switch:~$ sudo systemctl restart ntp
cumulus@switch:~$ sudo systemctl status -n0 ntp.service

If the state is not Active, or the alternate configuration file does not appear in the ntp command line — for example:

cumulus@switch:~$ /usr/sbin/ntpd -n -u ntp:ntp -g -c /run/ntp.conf.dhcp

then it is likely that a mistake was made. In this case, correct the mistake and rerun the three commands above to verify.

Note

With this unit file override present, changing NTP settings using NCLU do not take effect until the DHCP script regenerates the alternate NTP configuration file.

Quick Start Guide MD

This quick start guide provides an end-to-end setup process for installing and running Cumulus Linux, as well as a collection of example commands for getting started after installation is complete.

Prerequisites

  • Intermediate-level Linux knowledge is assumed for this guide. You should be familiar with basic text editing, Unix file permissions, and process monitoring. A variety of text editors are pre-installed, including vi and nano.
  • You must have access to a Linux or UNIX shell. If you are running Windows, use a Linux environment like Cygwin as your command line tool for interacting with Cumulus Linux.
.. tip:: If you are a networking engineer but are unfamiliar with Linux concepts, refer to [this reference guide](https://support.cumulusnetworks.com/hc/en-us/articles/201787636) to compare the Cumulus Linux CLI and configuration options, and their equivalent Cisco Nexus 3000 NX-OS commands and settings. You can also [watch a series of short videos](http://cumulusnetworks.com/technical-videos/) introducing you to Linux and Cumulus Linux-specific concepts.

Installation

To install Cumulus Linux, you use ONIE (Open Network Install Environment), an extension to the traditional U-Boot software that allows for automatic discovery of a network installer image. This facilitates the ecosystem model of procuring switches, with a user’s own choice of operating system loaded, such as Cumulus Linux.

The easiest way to install Cumulus Linux with ONIE is with local HTTP discovery:

  1. If your host (like a laptop or server) is IPv6-enabled, make sure it is running a web server. If the host is IPv4-enabled, make sure it is running DHCP as well as a web server.
  2. Download the Cumulus Linux installation file to the root directory of the web server. Rename this file onie-installer.
  3. Connect your host using an Ethernet cable to the management Ethernet port of the switch.
  4. Power on the switch. The switch downloads the ONIE image installer and boots. You can watch the progress of the install in your terminal. After the installation completes, the Cumulus Linux login prompt appears in the terminal window.

{{% notice note %}} These steps describe a flexible unattended installation method. You do not need a console cable. A fresh install with ONIE using a local web server typically completes in less than ten minutes.

You have more options for installing Cumulus Linux with ONIE. Read Installing a New Cumulus Linux Image to install Cumulus Linux using ONIE in the following ways:

  • DHCP/web server with and without DHCP options
  • Web server without DHCP
  • FTP or TFTP without a web server
  • Local file
  • USB {{% /notice %}}

ONIE supports many other discovery mechanisms using USB (copy the installer to the root of the drive), DHCPv6 and DHCPv4, and image copy methods including HTTP, FTP, and TFTP. For more information on these discovery methods, refer to the ONIE documentation.

After installing Cumulus Linux, you are ready to:

  • Log in to Cumulus Linux on the switch.
  • Install the Cumulus Linux license.
  • Configure Cumulus Linux. This quick start guide provides instructions on configuring switch ports and a loopback interface.

Upgrade to the Latest Version

If you are running a Cumulus Linux version earlier than 3.0.0, you must perform a complete install, as described above. If you already have Cumulus Linux 3.0.0 or later installed on your switch, read Upgrading Cumulus Linux for considerations before starting the process.

Getting Started

When starting Cumulus Linux for the first time, the management port makes a DHCPv4 request. To determine the IP address of the switch, you can cross reference the MAC address of the switch with your DHCP server. The MAC address is typically located on the side of the switch or on the box in which the unit ships.

Login Credentials

The default installation includes one system account, root, with full system privileges, and one user account, cumulus, with sudo privileges. The root account password is set to null by default (which prohibits login), while the cumulus account is configured with this default password:

CumulusLinux!

In this quick start guide, you use the cumulus account to configure Cumulus Linux.

{{% notice warning %}} For optimum security, change the default password (using the passwd command) before you configure Cumulus Linux on the switch. {{% /notice %}}

All accounts except root are permitted remote SSH login; you can use sudo to grant a non-root account root-level access. Commands that change the system configuration require this elevated level of access.

For more information about sudo, read Using sudo to Delegate Privileges.

Serial Console Management

You are encouraged to perform management and configuration over the network, either in band or out of band. Use of the serial console is fully supported; however, many customers prefer the convenience of network-based management.

Typically, switches ship from the manufacturer with a mating DB9 serial cable. Switches with ONIE are always set to a 115200 baud rate.

Wired Ethernet Management

Switches supported in Cumulus Linux always contain at least one dedicated Ethernet management port, which is named eth0. This interface is geared specifically for out-of-band management use. The management interface uses DHCPv4 for addressing by default. You can set a static IP address with the Network Command Line Utility (NCLU).

Example IP Configuration

{{%expand “NCLU Commands” %}}

Set the static IP address with the interface address and interface gateway NCLU commands:

cumulus@switch:~$ net add interface eth0 ip address 192.0.2.42/24
cumulus@switch:~$ net add interface eth0 ip gateway 192.0.2.1
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

These commands produce the following snippet in the /etc/network/interfaces file:

auto eth0
iface eth0
    address 192.0.2.42/24
    gateway 192.0.2.1

{{% /expand %}}

{{%expand “Linux Commands” %}} Edit the /etc/network/interfaces file and add the following configuration:

cumulus@switch:~$ sudo nano /etc/network/interfaces

auto eth0
iface eth0
    address 192.0.2.42/24
    gateway 192.0.2.1

{{% /expand %}}

Configuring the Hostname and Timezone

To change the hostname, run net add hostname, which modifies both the/etc/hostnameand /etc/hosts files with the desired hostname.

cumulus@switch:~$ net add hostname <hostname>
cumulus@switch:~$ net pending
cumulus@switch:~$ net commit

{{%notice note %}} The command prompt in the terminal does not reflect the new hostname until you either log out of the switch or start a new shell.

When you use this NCLU command to set the hostname, DHCP does not override the hostname when you reboot the switch. However, if you disable the hostname setting with NCLU, DHCP does override the hostname the next time you reboot the switch. {{% /notice %}}

To update the timezone, use the NTP interactive mode:

  1. Run the following command in a terminal: sudo dpkg-reconfigure tzdata
  2. Follow the on screen menu options to select the geographic area and region.

{{%notice note %}} Programs that are already running (including log files) and users currently logged in, do not see timezone changes made with interactive mode. To have the timezone set for all services and daemons, a reboot is required. {{% /notice %}}

Verifying the System Time

Before you install the license, verify that the date and time on the switch are correct. You must correct the date and time if they are incorrect. The wrong date and time can have impacts on the switch, such as the inability to synchronize with Puppet or return errors like this one after you restart switchd:

Warning: Unit file of switchd.service changed on disk, ‘systemctl daemon-reload’ recommended.

Installing the License

Cumulus Linux is licensed on a per-instance basis. Each network system is fully operational, enabling any capability to be utilized on the switch with the exception of forwarding on switch panel ports. Only eth0 and console ports are activated on an unlicensed instance of Cumulus Linux. Enabling front panel ports requires a license.

You receive a license key from Cumulus Networks or an authorized reseller. Here is a sample license key:

user@company.com|thequickbrownfoxjumpsoverthelazydog312

There are three ways to install the license onto the switch:

  • Copy the license from a local server. Create a text file with the license and copy it to a server accessible from the switch. On the switch, use the following command to transfer the file directly on the switch, then install the license file:

    cumulus@switch:~$ scp user@my_server:/home/user/my_license_file.txt .
cumulus@switch:~$ sudo cl-license -i my_license_file.txt

- Copy the file to an HTTP server (not HTTPS), then reference the URL when you run `cl-license`:

cumulus@switch:~$ sudo cl-license -i 

- Copy and paste the license key into the  `cl-license` command:

cumulus@switch:~$ sudo cl-license -i

^+d


<table style="width: 1197px;" border="0">
<tbody>
<tr>
<td style="width: 192px;"><strong>Release Note ID</strong></td>
<td style="width: 167.8125px;"><strong>Summary</strong></td>
<td style="width: 622.1875px;"><strong>Description</strong></td>
</tr>
<tr>
<td style="white-space: nowrap; width: 192px;"><a name="RN1130"></a><a href="#RN1130"> <img src="https://theme.zdassets.com/theme_assets/257925/59be21c5912389f67290d58a9cf0a6e745b50226.svg" alt="" width="32" /> </a> <br />RN-1130 (CM-22618)</td>
<td style="font-size: small; width: 167.8125px;">On Cumulus RMP, multiple failed ACL installations result in an `out of table resource` message even after reverting to a good rule set</td>
<td style="font-size: small; width: 622.1875px;">
<p>After attempting to install unsupported ICMPv6-type rules, the hardware sync fails with an <code>Out of table resource</code> message even after you correct the rules.</p>
<p>This is a known issue that is currently being investigated.</p>
</td>
</tr>
<tr>
<td style="white-space: nowrap; width: 192px;"><a name="RN1133"></a><a href="#RN1133"> <img src="https://theme.zdassets.com/theme_assets/257925/59be21c5912389f67290d58a9cf0a6e745b50226.svg" alt="" width="32" /> </a> <br />RN-1133 (CM-22590)</td>
<td style="font-size: small; width: 167.8125px;">NCLU `net show configuration commands` does not show output for an IPv6 rsyslog host</td>
<td style="font-size: small; width: 622.1875px;">
<p>NCLU <code>net show configuration commands</code> does not display any output for IPv6 rsyslog hosts.</p>
<p>This is a known issue that is currently being investigated.</p>
</td>
</tr>
<tr>
<td style="white-space: nowrap; width: 192px;"><a name="RN1134"></a><a href="#RN1134"> <img src="https://theme.zdassets.com/theme_assets/257925/59be21c5912389f67290d58a9cf0a6e745b50226.svg" alt="" width="32" /> </a> <br />RN-1134 (CM-22589)</td>
<td style="font-size: small; width: 167.8125px;">NCLU `net show configuration commands` displays a syslog command with invalid syntax</td>
<td style="font-size: small; width: 622.1875px;">
<p>NCLU <code>net show configuration commands</code> displays a <code>net add syslog</code> command with invalid syntax. For example, if you run the following commands:</p>
<pre>cumulus@switch:~$ net add syslog host ipv4 10.0.0.1 port udp 514
cumulus@switch:~$ net commit
</pre>
<p>then run <code>net show configuration commands</code>, the output of the command syntax is invalid.</p>
<p>This is a known issue that is currently being investigated.</p>
</td>
</tr>
<tr>
<td style="white-space: nowrap; width: 192px;"><a name="RN1136"></a><a href="#RN1136"> <img src="https://theme.zdassets.com/theme_assets/257925/59be21c5912389f67290d58a9cf0a6e745b50226.svg" alt="" width="32" /> </a> <br />RN-1136 (CM-22554)</td>
<td style="font-size: small; width: 167.8125px;">The link state of a bond is not updated when several members are brought down remotely at once</td>
<td style="font-size: small; width: 622.1875px;">
<p>If you try to bring down several members of a bond remotely at the same time, the link state of one of the interfaces does not correctly transition to the down state; however, all links show down in hardware.</p>
<p>This is a known issue that is currently being investigated.</p>
</td>
</tr>
</tbody>
</table>

{{%notice note %}}
It is not necessary to reboot the switch to activate the switch ports. After you install the license, restart the `switchd` service. All front panel ports become active and show up as swp1, swp2, and so on.

If a license is not installed on a Cumulus Linux switch, the `switchd` service does not start. After you install the license, start `switchd` as described above.
{{% /notice %}}

{{%excerpt-include filename="cumulus-linux/system-configuration/switchd.md" panel="Restart switchd" /%}}

## Configuring Breakout Ports with Splitter Cables

If you are using 4x10G DAC or AOC cables, or want to break out 100G or 40G switch ports, configure the breakout ports. For more details, see [Layer 1 and Switch Port Attributes](https://docs.cumulusnetworks.com/display/DOCS/Switch+Port+Attributes#SwitchPortAttributes-breakout).

## Testing Cable Connectivity

By default, all data plane ports (every Ethernet port except the management interface, eth0) are disabled.

To test cable connectivity, administratively enable a port:

cumulus@switch:~$ net add interface swp1 cumulus@switch:~$ net pending cumulus@switch:~$ net commit


To administratively enable all physical ports, run the following command, where swp1-52 represents a switch with switch ports numbered from swp1 to swp52:

cumulus@switch:~$ net add interface swp1-52 cumulus@switch:~$ net pending cumulus@switch:~$ net commit


To view link status, use the `net show interface all` command. The following examples show the output of ports in `admin down`, `down`, and `up` modes:

cumulus@switch:~$ net show interface all Name Speed MTU Mode Summary

UP lo N/A 65536 Loopback IP: 10.0.0.11/32, 127.0.0.1/8, ::1/128 UP eth0 1G 1500 Mgmt IP: 192.168.0.11/24(DHCP) UP swp1 (hypervisor_port_1) 1G 1500 Access/L2 Untagged: br0 UP swp2 1G 1500 NotConfigured ADMDN swp45 0M 1500 NotConfigured ADMDN swp46 0M 1500 NotConfigured ADMDN swp47 0M 1500 NotConfigured ADMDN swp48 0M 1500 NotConfigured ADMDN swp49 0M 1500 NotConfigured ADMDN swp50 0M 1500 NotConfigured UP swp51 1G 1500 BondMember Master: bond0(DN) UP blue N/A 65536 NotConfigured DN bond0 N/A 1500 Bond Bond Members: swp51(UN) UP br0 N/A 1500 Bridge/L3 IP: 172.16.1.1/24 Untagged Members: swp1 802.1q Tag: Untagged STP: RootSwitch(32768) UP red N/A 65536 NotConfigured ADMDN rename13 0M 1500 NotConfigured ADMDN vagrant 0M 1500 NotConfigured


## Configuring Switch Ports

### Layer 2 Port Configuration

Cumulus Linux does not put all ports into a bridge by default. To create a bridge and configure one or more front panel ports as members of the bridge, use the following examples as guides.

#### Examples

{{% panel theme="info" header="Example One" %}}
In the following configuration example, the front panel port swp1 is placed into a bridge called *bridge*. The NCLU commands are:

cumulus@switch:~$ net add bridge bridge ports swp1 cumulus@switch:~$ net pending cumulus@switch:~$ net commit


The commands above produce the following `/etc/network/interfaces` snippet:

auto bridge iface bridge bridge-ports swp1 bridge-vlan-aware yes

{{% /panel %}}

{{% panel theme="info" header="Example Two" %}}
You can add a range of ports in one command. For example, add swp1 through swp10, swp12, and swp14 through swp20 to bridge:

cumulus@switch:~$ net add bridge bridge ports swp1-10,12,14-20 cumulus@switch:~$ net pending cumulus@switch:~$ net commit


The commands above produce the following snippet in the `/etc/network/interfaces` file:

auto bridge iface bridge bridge-ports swp1 swp2 swp3 swp4 swp5 swp6 swp7 swp8 swp9 swp10 swp12 swp14 swp15 swp16 swp17 swp18 swp19 swp20 bridge-vlan-aware yes

{{% /panel %}}

To view the changes in the kernel, use the  `brctl` command:

cumulus@switch:~$ brctl show bridge name bridge id STP enabled interfaces bridge 8000.443839000004 yes swp1 swp2


### Layer 3 Port Configuration

You can also use NCLU to configure a front panel port or bridge interface as a layer 3 port.

In the following configuration example, the front panel port swp1 is configured as a layer 3 access port:

cumulus@switch:~$ net add interface swp1 ip address 10.1.1.1/30 cumulus@switch:~$ net pending cumulus@switch:~$ net commit


The commands above produce the following snippet in the `/etc/network/interfaces` file:

auto swp1 iface swp1 address 10.1.1.1/30


To add an IP address to a bridge interface, you must put it into a VLAN interface:

cumulus@switch:~$ net add vlan 100 ip address 10.2.2.1/24 cumulus@switch:~$ net pending cumulus@switch:~$ net commit


The commands above produce the following snippet in the `/etc/network/interfaces` file:

auto bridge iface bridge bridge-vids 100 bridge-vlan-aware yes

auto vlan100 iface vlan100 address 192.168.10.1/24 vlan-id 100 vlan-raw-device bridge


To view the changes in the kernel, use the `ip addr show` command:

cumulus@switch:~$ ip addr show …

  1. swp1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bridge state UP group default qlen 1000 link/ether 44:38:39:00:6e:fe brd ff:ff:ff:ff:ff:ff

…

14: bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 44:38:39:00:00:04 brd ff:ff:ff:ff:ff:ff inet6 fe80::4638:39ff:fe00:4/64 scope link valid_lft forever preferred_lft forever…


## Configuring a Loopback Interface

Cumulus Linux has a loopback preconfigured in the `/etc/network/interfaces` file. When the switch boots up, it has a loopback interface, called *lo*, which is up and assigned an IP address of 127.0.0.1.

{{%notice tip %}}
The loopback interface *lo* must always be specified in the `/etc/network/interfaces` file and must always be up.
{{% /notice %}}

To see the status of the loopback interface (lo), use the `net show interface lo` command:

cumulus@switch:~$ net show interface lo Name MAC Speed MTU Mode

UP lo 00:00:00:00:00:00 N/A 65536 Loopback

IP Details

IP: 127.0.0.1/8, ::1/128 IP Neighbor(ARP) Entries: 0


Note that the loopback is up and is assigned an IP address of 127.0.0.1.

To add an IP address to a loopback interface, configure the *lo* interface with NCLU:

cumulus@switch:~$ net add loopback lo ip address 10.1.1.1/32 cumulus@switch:~$ net pending cumulus@switch:~$ net commit


You can configure multiple loopback addresses by adding additional `address` lines:

cumulus@switch:~$ net add loopback lo ip address 172.16.2.1/24 cumulus@switch:~$ net pending cumulus@switch:~$ net commit


The commands above produce the following snippet in the `/etc/network/interfaces` file:

auto lo iface lo inet loopback address 10.1.1.1/32 address 172.16.2.1/24