Automating Cisco Home and Remote Access Router Updates

Published: April 2019

Why: stronger security, fewer cases, and a better user experience

Cisco IT manages 30,000 Cisco Virtual Office routers in employees' home offices around the world. We make dozens of configuration changes every month--for security updates, operating system updates, new DNS or DHCP servers, new QoS policies, and so on.

Since 2006 we've automated configuration updates using custom scripts that push out the latest configuration file to 50-100 routers at a time. This worked well when we had 10,000 routers--but less well as the deployment grew.

"Pushing out a completely new file for every configuration change is like replacing every nail in a board instead of only the bent ones," says Joseph Bradley, Cisco IT senior engineer.

If the ninety-ninth of 100 routers failed to update, the configuration for all routers in the batch would sometimes revert to the previous state, and the script would try again the next day. In the case of a security update, the delay left the routers vulnerable.

Errors occurred often enough that eight engineers were needed to support Cisco Virtual Office. They spent 40-50 hours monthly testing configuration templates, another 5-6 hours updating router configurations, and still more time remediating errors.

We wanted a more efficient way to automate Cisco Virtual Office router configuration. Our goals included:

• Automated configuration based on router serial number.
• The flexibility to update just the parts of the configuration that had changed, not the whole file. Bradley calls this "surgical updates."
• Automated configuration checks every time a home router powers up. "Some employees switch off their home-office routers at the end of the workday, over the weekend, or when they're on vacation," says Madalina Ana, Cisco IT service manager. "A router powered off during the last security update is vulnerable when it's switched back on."
• A simpler user experience. Previously, employees had to visit a webpage and follow a series of instructions to provision a new router or reboot. We wanted to automate everything other than connecting the router to the network and power supply.

How: Network Services Orchestrator with Python scripts

We automated Cisco Virtual Office router configuration and compliance checks using Cisco Network Services Orchestrator (NSO) and Python scripts. We did not have to use APIs to connect the scripts because Python support is built right into NSO. We deployed NSO in three theaters, starting with router configuration management in August 2018 and using it for Plug and Play provisioning in February 2019. Here's how it works.

Day 1: initial provisioning

When we ship out a router, someone at the depot enters its serial number, which is recorded in the NSO database and mapped to a router configuration in the Plug and Play Connect cloud (available from software.cisco.com).

"Our user just connects the router to the network and powers it on--and they're off to the races," Bradley says. The user can walk away at that point. The router sends its serial number to the Plug and Play cloud, which directs the request to the correct NSO server and pulls the current configuration. About 20 minutes later, the router is ready to use--down from 45 minutes before.

"Employees like it because now they don't have to sit in front of a web interface during the setup," says Ana.

Day 2: regular updates

Several times a week, NSO checks each router's configuration against its database to see if updates are needed. It does the same whenever a router is powered back on. If the configuration is out of date, a Python script pushes out just the needed changes--not the entire configuration file. If one router fails to update, the others in the group are not affected.

Outcomes to date

• 15 percent decrease in support cases. "Since automating router updates, we've seen a big drop in certificate issues and offline routers," Bradley says. "And we expect 95 percent of provisioning cases to go away."
• Better compliance with policy. Checking the configuration whenever the router is powered on helps us more quickly detect and correct security vulnerabilities. For example, suppose a support engineer disabled 802.1X authentication during troubleshooting and then forgot to turn it back on. Before, we might not have become aware of the vulnerability for an hour. Now NSO detects and corrects misconfigurations as soon as the router is powered back up.
• Faster global configuration changes. We implemented 802.1X authentication on our 30,000 home-office routers before we started using Cisco NSO. With NSO, we completed a project of similar scope--migrating from IPv4 to IPv6--75 percent faster.
• Single source of truth about router configurations. Previously, we had multiple configuration databases--at least one for each theater. Now all configurations are stored centrally in the Cisco NSO database. Keeping all configurations in one place helps us more quickly inventory our hardware and firmware to see what upgrades are needed for security and performance.
• "We can also quickly see which routers haven't been powered on since the last update so that we can ask those employees to cycle the power," says Bradley.
• Better information for our InfoSec team. The consolidated report helps us answer questions from our security team more quickly, such as which routers received a particular update or why an update wasn't pushed out.
• Better user experience. When employees need to set up or reset a home router, they just cycle the power off and on. They can then walk away or do other work while the router is configured.
• Higher value from engineers and better job satisfaction. Freed from having to spend 50-60 hours a month testing and updating router configurations, engineers have more time to innovate. "They've shifted their focus from playing catch up and fixing issues to proactive improvements," Bradley says.

Conclusion

The entire router configuration lifecycle is now fully automated. We're continuing to fine-tune NSO and add new functions. One is automating changes that users have requested and Cisco IT has approved. Ana concludes, "In a matter of months, automating Cisco Virtual Office router configuration has improved security, reduced IT caseload, and improved our user experience. It's a win-win--for our users and for IT."

For more information

Automating Cisco Home and Remote Access Router Updates - Spanish (PDF)

Cisco Network Services Orchestrator

Cisco Virtual Office